aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/actions/cache
Commit message (Collapse)AuthorAge
* Change FileContentsProxy to use ctime instead of mtimeGravatar ulfjack2017-05-09
| | | | | | | | | This gives us better reliability for detecting file changes; especially in cases where tools intentionally do not update mtime. Fixes #1525. PiperOrigin-RevId: 155490849
* Move the chmod call from SkyframeActionExecutor to ActionMetadataHandlerGravatar ulfjack2017-04-07
| | | | | | | | | | | | | | | | | | | This relies on the explicit state transition in MetadataHandler to decide whether the chmod call is necessary (we must never call chmod if the action was not executed). This is a prerequisite for #1525. If we want to use ctime for detecting file content changes (which is more reliable than just mtime), then we must call chmod before stat, since chmod affects ctime. Before this change, we were caching the stat in ActionMetadataHandler, but calling chmod after action execution in SkyframeActionExecutor, which is the wrong order of calls. However, we must be able to stat in ActionMetadataHandler for cases where a single action runs multiple Spawns where one spawn's output is a subsequent spawn's input. Remove MetadataHandler.isInjected, which is no longer used anywhere. PiperOrigin-RevId: 152387133
* Explicitly document the state transition on MetadataHandlerGravatar ulfjack2017-04-06
| | | | | | | | | | | | | | | | The ActionMetadataHandler does an explicit state transition on discardOutputMetadata. Before the call, it may be used for action cache checking, and after the call it may be updated with execution results. Several of the methods now throw if they're used incorrectly, so I had to refactor the control flow in ActionExecutionFunction to correctly call discardOutputMetadata on the MetadataHandler in all cases. I discovered a resource leak (of FileOutErr) in IncludeParseFunction while I was at it, so I plugged that as well. One step towards #1525. PiperOrigin-RevId: 152363982
* Add caching of computed file digests based on file metadata.Gravatar Julio Merino2017-03-17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This change modifies DigestUtils to add a cache of (path, inode, mtime, size) to the digest of the file for those digests that are computed by reading the file contents. The cache itself is optional because relying on file metadata to cache the file digests could lead to correctness issues. Enabling the cache is exposed via a new (undocumented) --cache_computed_file_digests flag that we can use post-release to tune the built-in values if they prove to be incorrect or problematic. For Bazel, enable this cache unconditionally because the rest of Bazel already relies on mtimes and other file metadata to determine changes to files. The rationale for this change is performance: once we have lost the in-memory file metadata (e.g. because of a flag flip), Bazel has to redigest all of the output files to recompute action cache keys. For a pathological case of rebuilding a large app with 5GB of outputs and then flipping the --[no]check_visibility flag on the command line, we get the following numbers before this change: ____Elapsed time: 11.170s, Critical Path: 8.34s ____Elapsed time: 11.027s, Critical Path: 8.20s ____Elapsed time: 11.084s, Critical Path: 7.46s ____Elapsed time: 11.051s, Critical Path: 6.61s ____Elapsed time: 11.211s, Critical Path: 7.81s ____Elapsed time: 10.884s, Critical Path: 8.20s ____Elapsed time: 11.385s, Critical Path: 8.12s ____Elapsed time: 11.723s, Critical Path: 8.18s ____Elapsed time: 11.327s, Critical Path: 7.73s ____Elapsed time: 11.028s, Critical Path: 7.89s And after this change: ____Elapsed time: 4.294s, Critical Path: 0.27s ____Elapsed time: 4.376s, Critical Path: 0.83s ____Elapsed time: 8.083s, Critical Path: 0.52s ____Elapsed time: 4.302s, Critical Path: 0.64s ____Elapsed time: 4.282s, Critical Path: 0.37s ____Elapsed time: 4.219s, Critical Path: 0.61s ____Elapsed time: 4.214s, Critical Path: 0.97s ____Elapsed time: 4.185s, Critical Path: 0.71s ____Elapsed time: 7.962s, Critical Path: 4.30s ____Elapsed time: 4.149s, Critical Path: 1.03s -- PiperOrigin-RevId: 150351444 MOS_MIGRATED_REVID=150351444
* Convert --use_action_cache to a regular optionGravatar Googler2017-03-01
| | | | | | | | RELNOTES: Convert --use_action_cache to a regular option -- PiperOrigin-RevId: 148804881 MOS_MIGRATED_REVID=148804881
* Skip computing digests when --use_action_cache=false.Gravatar Googler2017-02-16
| | | | | | | | | | | | | Does this by delegating responsibility for constructing ActionCache.Entry instances to the ActionCache, and having the StubActionCache return null. Tests show a 1-2% elapsed-time reduction for clean builds: [] Doesn't use interface default methods, because we still need JDK7 for Mac. -- PiperOrigin-RevId: 147722062 MOS_MIGRATED_REVID=147722062
* Adds --[no]use_action_cache startup option to disable the action cache.Gravatar Googler2017-02-15
| | | | | | | | | Disabling the action cache is helpful in contexts where incremental builds are not required, or where actions need to be repeatedly executed for debugging. -- PiperOrigin-RevId: 147485055 MOS_MIGRATED_REVID=147485055
* Streamline Fingerprint implementationGravatar Michajlo Matijkiw2016-12-15
| | | | | | | | | | | | Thread all updates through a CodedOutputStream. This has the benefit of potentially hashing less data, as many int values can be represented more compactly, and reducing the churn of hashing strings (generally), since CodedOutputStream is already heavily optimized for this. While the buffer size is a little generous, it winds up paying off. -- PiperOrigin-RevId: 142151062 MOS_MIGRATED_REVID=142151062
* Adding an option to set the digest function that everything uses. Minor ↵Gravatar Ola Rozenfeld2016-11-18
| | | | | | | | | | | | | refactoring: enabling potential fast digest computation of more than one digest function type. Usage: bazel --host_jvm_args="-Dbazel.DigestFunction=SHA1" build ... Ugliness: using a system property (a static non-final variable), because the better way to do it (a flag) would result in a much, much larger refactoring. More ugliness: I have updated the minimal amount of tests. A lot of tests are still relying on the default value of MD5. Ideally, they need to be updated as well. -- MOS_MIGRATED_REVID=139490836
* Proper action output checks for TreeArtifacts. Instead of crashing Bazel, we ↵Gravatar Rumou Duan2016-10-19
| | | | | | | now handle failed TreeArtifact output checks gracefully. -- MOS_MIGRATED_REVID=136627086
* Add a method in MetadataHandler to retrieve contained artifacts from ↵Gravatar Rumou Duan2016-10-07
| | | | | | | TreeArtifacts. -- MOS_MIGRATED_REVID=135485914
* Track client environment in SkyframeGravatar Klaus Aehlig2016-09-14
| | | | | | | | | | | | | ...to determine which actions have to be recomputed based on changes to the client environment. Note that this change does it the simple way and reconsideres all actions on a changed client environment, while still only reexecuting those, where the part that was inherited from the environment actually did change. -- Change-Id: Ie1116d094642165e5e959447a6fcf49d19b37d6e Reviewed-on: https://bazel-review.googlesource.com/#/c/5431 MOS_MIGRATED_REVID=133010705
* Make Digest (renamed Md5Digest) a little more multi-purpose.Gravatar Shreya Bhattarai2016-08-23
| | | | | -- MOS_MIGRATED_REVID=130986194
* Drive-by cleanup to remove unnecessary function from the ActionCache interface.Gravatar Shreya Bhattarai2016-08-10
| | | | | -- MOS_MIGRATED_REVID=129753109
* Don't treat empty files specially with respect to mtime/digest.Gravatar Janak Ramakrishnan2016-07-14
| | | | | | | RELNOTES: Bazel no longer regards an empty file as changed if its mtime has changed. -- MOS_MIGRATED_REVID=127328552
* Rephrase Fingerprint in terms of guava HashFunction.Gravatar Eric Fellheimer2016-07-12
| | | | | | | This minimizes API divergence and allows us to make use of performance enhancements in guava, such as https://github.com/google/guava/issues/1197. -- MOS_MIGRATED_REVID=127108107
* Micro-optimizations to Digest.fromMetadata():Gravatar Googler2016-07-07
| | | | | | | | | | | | Don't encode paths as UTF-8 to md5sum them (the map keys are Strings derived from Paths, for which Fingerprint.addPath() takes the same shortcut). Remove defensive copy from the constructor (which did not need to be public). Remove redundant call to Fingerprint.reset(). -- MOS_MIGRATED_REVID=126682477
* Add --experimental_multi_threaded_digest which lets DigestUtils use multiple ↵Gravatar Philipp Wollermann2016-06-06
| | | | | | | | | threads when calculating the MD5 hash even for large files. Might improve performance when using an SSD. Fixes #835 and #1210. -- MOS_MIGRATED_REVID=124128233
* Log whenever a periodic flush of the action cache is slow.Gravatar Nathan Harmata2016-04-14
| | | | | -- MOS_MIGRATED_REVID=119813568
* Introduce TreeFileArtifact, which represents files under TreeArtifacts.Gravatar Rumou Duan2016-04-14
| | | | | | | Remove ArtifactFile, which is rendered obsolete by TreeFileArtifact. -- MOS_MIGRATED_REVID=119789154
* Support for handling TreeArtifact metadata and returning TreeArtifacts from ↵Gravatar Michael Thvedt2016-02-09
| | | | | | | ArtifactFunction. -- MOS_MIGRATED_REVID=114174899
* Improve space efficiency of Blaze action cache: For actions that don't ↵Gravatar Eric Fellheimer2016-01-22
| | | | | | | perform input discovery, there is no need to store the full set of edges in the action cache. This data is only used to formulate the set of input files for an action prior to checking the validity of a cache entry. For non-input-discovering actions, the set of input files is known statically and the action cache data is not used. -- MOS_MIGRATED_REVID=112704382
* Use Bazel Preconditions variant which avoids varargs array creationGravatar Mark Schaller2015-12-10
| | | | | | | Reduces garbage. -- MOS_MIGRATED_REVID=109914243
* Introduce Path#isSpecialFile, FileSystem#isSpecialFile, and ↵Gravatar Nathan Harmata2015-10-21
| | | | | | | FileStatus#isSpecialFile to help disambiguate between a regular file and a special file, since the file size of a special file cannot be trusted. -- MOS_MIGRATED_REVID=105903622
* Rationalize copyright headersGravatar Damien Martin-Guillerez2015-09-25
| | | | | | | | | | | The headers were modified with `find . -type f -exec 'sed' '-Ei' 's|Copyright 201([45]) Google|Copyright 201\1 The Bazel Authors|' '{}' ';'` And manual edit for not Google owned copyright. Because of the nature of ijar, I did not modified the header of file owned by Alan Donovan. The list of authors were extracted from the git log. It is missing older Google contributors that can be added on-demand. -- MOS_MIGRATED_REVID=103938715
* Add runfiles to metadata caches and make them isFile awareGravatar Michajlo Matijkiw2015-05-26
| | | | | -- MOS_MIGRATED_REVID=94318260
* Fix doc for CompactPersistentActionCache.validateIntegrityGravatar Mark Schaller2015-05-18
| | | | | -- MOS_MIGRATED_REVID=93713456
* Belated cleanup of MetadataHandler#discardMetadata. Since we have per-action ↵Gravatar Janak Ramakrishnan2015-04-14
| | | | | | | metadata handlers, we can just unconditionally discard all output metadata with this call. -- MOS_MIGRATED_REVID=91008004
* Add functionality to discard orphaned binaries and middlemen.Gravatar Michajlo Matijkiw2015-03-25
| | | | | | | | | | Here we additionally identify artifacts as orphaned if they are only inputs to middleman actions whose outputs are unused. The middleman actions are still part of the action graph, but we tolerate their inputs being orphaned. -- MOS_MIGRATED_REVID=89512745
* Don't print out action cache if it has more than 1000 entries. This prevents ↵Gravatar Janak Ramakrishnan2015-03-23
| | | | | | | IDE OOMing when a large action cache is in scope during debugging. -- MOS_MIGRATED_REVID=89127817
* add baseline functionality for not saving unused artifactsGravatar Michajlo Matijkiw2015-03-13
| | | | | | | | | | | | | | | | | | | We define unused artifacts as those that aren't consumed by any action. This can be because an action produced more outputs than a dependent action needed, or because it's a top level artifact and we don't care about its contents, just that it was built without issue. Actions may prevent outputs from being discarded by declaring them as mandatory. This is particularly useful for test outputs. The motivation behind this change is to reduce storage overhead for things we can do without. It is worth noting this change doesn't cover all cases. In particular it has difficulty identifying *_binary artifacts as orphaned. This is due to the insertion of a virtual runfiles artifact which depends upon the rule's outputs. -- MOS_MIGRATED_REVID=88467504
* Some cleanup changes.Gravatar Ulf Adams2015-03-05
| | | | | -- MOS_MIGRATED_REVID=87821306
* Update from Google.Gravatar Han-Wen Nienhuys2015-02-25
-- MOE_MIGRATED_REVID=85702957
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-07 19:36:33 +0000