aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/skyframe/ArtifactFunction.java
Commit message (Collapse)AuthorAge
* remote: add support for directory inputs in runfilesGravatar buchgr2018-08-02
| | | | | | | | | | | | | | | | Add support for tree artifacts (ctx.action.declare_directory(...)) in runfiles. Before this change we would throw away the information about the files inside a tree artifact before executing an action. That's fine for local execution where the sandbox just copies/symlinks a directory and doesn't care much what's inside. However, in remote execution we actually need to upload each individual file and so we need to be aware of all individual files not just directories. This change makes it so that this information is made available to a SpawnRunner via the SpawnInputExpander. RELNOTES: None PiperOrigin-RevId: 207091668
* Automated rollback of commit f309ad3be36363070e87eef0ee04b12f4956d601.Gravatar janakr2018-07-24
| | | | | | | | | | *** Reason for rollback *** Fixed duplicate derived inputs bug. Test is in diffbase. RELNOTES[INC]: If the same artifact is generated by two distinct but identical actions, and a downstream action has both those actions' outputs in its inputs, the artifact will now appear twice in the downstream action's inputs. If this causes problems in Skylark actions, you can use the uniquify=True argument in Args.add_args. PiperOrigin-RevId: 205863806
* Automated rollback of commit d491bf10f42e213292382c98a1dc439537f00f43.Gravatar janakr2018-07-17
| | | | | | | | | | | | | | | | | | *** Reason for rollback *** Still bugs lurking. See linked bug. *** Original change description *** Automated rollback of commit eb587075b0d6ffab1cf9e69ede1b7e547905e547. *** Reason for rollback *** Depot has been fixed. RELNOTES[INC]: If the same artifact is generated by two distinct but identical actions, and a downstream action has both those actions' outputs in its inputs, the artifact will now appear twice in the downstream action's inputs. If this causes problems in Skylark actions, you can use the uniquify=True argument in Args.add_args. PiperOrigin-RevId: 204997569
* Automated rollback of commit eb587075b0d6ffab1cf9e69ede1b7e547905e547.Gravatar janakr2018-07-16
| | | | | | | | | | *** Reason for rollback *** Depot has been fixed. RELNOTES[INC]: If the same artifact is generated by two distinct but identical actions, and a downstream action has both those actions' outputs in its inputs, the artifact will now appear twice in the downstream action's inputs. If this causes problems in Skylark actions, you can use the uniquify=True argument in Args.add_args. PiperOrigin-RevId: 204827477
* Sort expansions of aggregating middlemen by exec file path. These commonlyGravatar Googler2018-07-12
| | | | | | | | | | | | | contain a large set of files required for a particular toolchain that are added to every single compile action. Some exeutors in turn sort all action inputs in order to properly cache, deduplicate and serialize them. Sorting the expansion of these middlemen needs to be done only once and a later sorting together with a bunch of other action input files is much faster (as Java's TimSort algorithm is effectively linear with mostly sorted inputs). RELNOTES: None. PiperOrigin-RevId: 204285426
* Handles null value in createAggregatingValue env.getValues call.Gravatar shahan2018-06-27
| | | | PiperOrigin-RevId: 202389874
* Automated rollback of commit 45b308a62f42c2c0bcfe79dcd4046c4025a31059.Gravatar janakr2018-06-20
| | | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Breaks Javascript compilation. There's currently no way to iterate over a depset of Artifacts and deduplicate by identical paths in Skylark. This means that actions that want to do something once per Artifact in a depset (add a flag to the command line with the path of the Artifact for instance) will have duplicate entries if there are multiple Artifacts with the same path. This is not a true automated rollback, since I tried to make this as minimal as possible, to avoid merge conflicts and keep some of the benefits of the rolled back CL. In particular, the tests that were changed in the original CL to give artifacts their correct owners did not need to be changed back, since the owners are still correct. Moreover, this effectively contains rollbacks of unknown commit and https://github.com/bazelbuild/bazel/commit/39d6f89644107a8f7c080c4f4aec8527c1a73954, but keeps test coverage from those CLs as well. Comments added to CL thread where not a clean rollback (there are plenty of files not changed at all: ActionArtifactCycleReporter is the main wart, since it can still handle SkyKeys with Artifact as the argument, but Artifact is no longer the argument to any SkyKey). RELNOTES: None *** Original change description *** Make Artifact#equals take the owner into account for derived artifacts. Derived artifacts' owners are important because they are used to determine the artifact's generating action. Source artifacts' owners are not used in this way, so I left them alone. This allows us to get rid of most uses of ArtifactSkyKey. We may be able to delete it entirely in a follow-up. PiperOrigin-RevId: 201464780
* Refactoring: uses OutputService for ActionFileSystem injectionGravatar shahan2018-06-18
| | | | | | Moves more Action-oriented from skyframe package, which has become very unwieldy, to action package. This is needed to avoid circular dependencies caused when build-base is needed for skyframe. PiperOrigin-RevId: 200996982
* Add functionality to make certain SkyValues unshareable, meaning they are ↵Gravatar janakr2018-06-13
| | | | | | not serialized. Tag TestCompletionValue and any ActionExecutionValue coming from a NotifyOnActionCacheHit (i.e., tests) like that. To make such values really not shared, request the ActionExecutionValue from TestCompletionFunction as opposed to the ArtifactValue (propagating the unshareable bit up seemed like too much fuss, and I have a dream of getting rid of ArtifactValue anyway). PiperOrigin-RevId: 200504358
* Make Artifact#equals take the owner into account for derived artifacts.Gravatar janakr2018-06-08
| | | | | | | | Derived artifacts' owners are important because they are used to determine the artifact's generating action. Source artifacts' owners are not used in this way, so I left them alone. This allows us to get rid of most uses of ArtifactSkyKey. We may be able to delete it entirely in a follow-up. PiperOrigin-RevId: 199836436
* Refactoring: makes the code simpler by deleting Metadata and InputArtifactData.Gravatar shahan2018-06-06
| | | | | | (minor) ActionFS now implements MetadataProvider.getInput PiperOrigin-RevId: 199575194
* Carry tree artifacts' self data through aggregating middlemen.Gravatar Benjamin Peterson2018-06-03
| | | | | | | | | | | | | | | | | | | | | | | | This CL is a followup to c868c47. It is intended to fix #5296. The immediate problem in that issue is that remote caching finds a tree artifact in an input runfiles tree without metadata in the cache. The metadata isn't there because the runfiles artifacts are now all behind a middleman action, and ArtifactFunction didn't propagate tree artifact values through middlemen. This change fixes the problem by adding a tree artifact's self data value to the values carried through to dependent actions. Remote caching with tree artifact inputs seems to have been working mostly by accident. (Evidence: it wasn't tested, and remote execution of actions with tree artifact inputs results a Java traceback before and after c868c47.) This CL is only really a bandaid to return to the old "working" state. If remote execution wants to start supporting tree artifacts properly in the future, we'll likely need to carry the tree artifact child data through middleman actions, too. Closes #5299. Change-Id: I2e07d4fca0f6d2d34d97b7edb27f9d0063776225 PiperOrigin-RevId: 199079382
* Treat runfiles middlemen as aggregating middlemen.Gravatar Benjamin Peterson2018-05-28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The goal is to, for an action execution, make only a single skyframe edge to every required runfiles tree rather than an edge to every runfiles artifact directly. We accomplish this by pulling the runfiles artifacts' metadata for a particular runfiles tree through the appropriate runfiles middlemen artifact in one batch. This CL fixes https://github.com/bazelbuild/bazel/issues/3217. This change makes runfiles middlemen more similar to aggregating middlemen. We however continue to treat runfiles middlemen slightly differently than true aggregating middlemen. Namely, we do not expand runfiles middlemen into the final action inputs. The reasons for this are: 1. It can make latent bugs like https://github.com/bazelbuild/bazel/issues/4033 more severe. 2. The runfiles tree builder action's output MANIFEST contains absolute paths, which interferes with remote execution if its metadata is added to a cache hash. 3. In the sandbox, it causes the runfiles artifacts to be mounted at their exec path locations in addition to within the runfiles trees. This makes the sandbox less strict. (Perhaps tolerably?) 4. It saves a modicum of (transient) memory and time to not expand. If the first two issues are fixed, it could be a nice simplification to completely replace runfiles middlemen with aggregating middlemen. Change-Id: I2d2148297f897af4c4c6dc055d87f8a6fad0061e PiperOrigin-RevId: 198307109
* Stop storing ActionTemplate in a SkyKey: it's too heavyweight. Use the same ↵Gravatar janakr2018-02-15
| | | | | | mechanism as for normal actions, have the ActionTemplateExpansionFunction look the template up when needed. PiperOrigin-RevId: 185861672
* Make ActionLookupData a SkyKey. Should save some memory.Gravatar janakr2018-02-14
| | | | PiperOrigin-RevId: 185694489
* Refactor ArtifactSkyKey to get rid of an unnecessary wrapper class: actually ↵Gravatar janakr2018-02-05
| | | | | | | | essentially promote OwnedArtifact to ArtifactSkyKey and rename it to ArtifactSkyKey. The king is dead... Also add some other execution-phase codecs. PiperOrigin-RevId: 184552706
* Prevent broken cache entries on concurrent file changesGravatar ulfjack2018-01-19
| | | | | | | | | | | | | | | | Local execution has an inherent race condition: if a user modifies a file while an action is executed, then it is impossible for Bazel to tell which version of the file was actually read during action execution. The file may have been modified before or after the tool has read it, or, in the worst case, the tool may have read both the original and the modified version. In addition, the file may be changed back to the original state before Bazel can check the file, so computing the digest before / after may not be sufficient. This is a concern for both local and remote caches, although the cost of poisoning a shared remote cache is significantly higher, and is what has triggered this work. Fixes #3360. We solve this by keeping a reference to the FileContentsProxy, and using that to check for modificaitons before storing the cache entry. We output a warning if this check fails. This change does not increase memory consumption; Java objects are always allocated in multiples of 8 bytes, we use compressed oops, and the FileArtifactValue currently has 12 bytes worth of fields (excl. object overhead), so adding another pointer is effectively free. As a possible performance optimization on purely local builds, we could also consider not computing digests at all, and only use the FileContentsProxy for caching. PiperOrigin-RevId: 182510358
* Introduce Root class.Gravatar tomlu2018-01-17
| | | | | | | | | | | This class represents a root (such as a package path or an output root) used for file lookups and artifacts. It is meant to be as opaque as possible in order to hide the user's environment from sky keys and sky functions. Roots are used by RootedPaths and ArtifactRoots. This CL attempts to make the minimum number of modifications necessary to change RootedPath and ArtifactRoot to use these fields. Deprecated methods and invasive accessors are permitted to minimise the risk of any observable changes. RELNOTES: None PiperOrigin-RevId: 182271759
* Make FileSymlinkException and InconsistentFSException IOExceptionsGravatar ulfjack2018-01-12
| | | | | | | Most places handle them the same way as IOException, which seems like a safe default. The places that do care can still throw or catch the more specific type. PiperOrigin-RevId: 181719688
* Convert ActionLookupKey implementations to directly implement SkyKey, ↵Gravatar janakr2018-01-11
| | | | | | removing the layer of indirection of getting SkyKey out of ActionLookupKey, which uses more memory for no reason. PiperOrigin-RevId: 181658615
* Remove the concept of a mandatory input which is allowed to be missing.Gravatar Googler2017-11-29
| | | | | RELNOTES: None. PiperOrigin-RevId: 177332323
* Replace all usages of Blaze's Preconditions class with guava.Gravatar tomlu2017-11-09
| | | | | | | | Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own. This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually. PiperOrigin-RevId: 175033526
* Remove least important argument to 5+ arg Preconditions#checkState/checkNotNull.Gravatar tomlu2017-11-08
| | | | | | This change prepares a move to Guava's preconditions. Guava only has vararg-avoiding overloads up to 4 args. PiperOrigin-RevId: 175015502
* Simplify ArtifactFunction: inline missingInputFileGravatar ulfjack2017-07-26
| | | | | | Also fix two warnings about missing generic types. PiperOrigin-RevId: 163062670
* If --batch, --keep_going, --discard_analysis_cache, and the new ↵Gravatar janakr2017-03-31
| | | | | | | | | | | | | | | --noexperimental_enable_critical_path_profiling flags are all specified, then Bazel will delete Actions from ActionLookupValues as they are executed in order to save memory. Because an already-run action may output an artifact that is only requested later in the build, we need to maintain a way for the artifact to look up the action. But in most cases we don't need to keep the action itself, just its output metadata. Some actions unfortunately are needed post-execution, and so we special-case them. Also includes dependency change with description: Move action out of key. This keeps action references from polluting the graph -- actions are just stored in one SkyValue, instead of being present in SkyKeys. This does mean additional memory used: we have a separate ActionLookupData object per Action executed. That may reach ~24M for million-action builds. PiperOrigin-RevId: 151756383
* Make explicit that no I/O is done when constructing a FileArtifactValue for ↵Gravatar Janak Ramakrishnan2017-03-21
| | | | | | | | | | | | an ordinary Artifact in ArtifactFunction. If the data available in the FileValue is not sufficient to recreate the FileArtifactValue without disk access, then ActionMetadataHandler already stores the FileArtifactValue separately. This is approximately step -3 in a grand plan to save a bunch more memory when building with --batch --discard_analysis_cache --keep_going. -- PiperOrigin-RevId: 150646007 MOS_MIGRATED_REVID=150646007
* Introduce CppCompileActionTemplate, which expands into a list of ↵Gravatar Rumou Duan2017-02-10
| | | | | | | | CppCompileActions that to be executed at execution time. -- PiperOrigin-RevId: 147163077 MOS_MIGRATED_REVID=147163077
* Allow Skyframe graph lookups and value retrievals to throw InterruptedException.Gravatar Janak Ramakrishnan2016-08-16
| | | | | | | The only place we now don't handle InterruptedException is in the action graph created after analysis, since I'm not sure that will be around for that much longer. -- MOS_MIGRATED_REVID=130327770
* Refactor FileArtifactValue and ArtifactValue now that presence of mtime and ↵Gravatar Janak Ramakrishnan2016-08-01
| | | | | | | digest are mutually exclusive. -- MOS_MIGRATED_REVID=128843642
* 1. Create the TreeArtifact directory structure before expanding ActionTemplates.Gravatar Rumou Duan2016-06-14
| | | | | | | | 2. In PopulateTreeArtifactAction, create the parent directories for TreeFileArtifacts before executing the spawn. 3. Allow empty tree artifacts in CustomCommandLine and PopulateTreeArtifact. -- MOS_MIGRATED_REVID=124759286
* Skyframe changes to support SpawnActionTemplate.Gravatar Rumou Duan2016-06-07
| | | | | | | | 1. Adds ActionTemplateExpansion{Function, Value} for ActionTemplate expansion. 2. Changes ArtifactFunction to support evaluating TreeFileArtifacts and TreeArtifacts generated by ActionTemplate. -- MOS_MIGRATED_REVID=124160939
* Split ActionMetadata into ActionAnalysisMetadata and ActionExecutionMetadata.Gravatar Rumou Duan2016-04-26
| | | | | | | Except in action execution logic (ActionExecutionFunction, SkyframeActionExecutor, etc.), switch Action interface references to either ActionAnalysisMetadata if possible or ActionExecutionMetadata. -- MOS_MIGRATED_REVID=120723431
* Support for handling TreeArtifact metadata and returning TreeArtifacts from ↵Gravatar Michael Thvedt2016-02-09
| | | | | | | ArtifactFunction. -- MOS_MIGRATED_REVID=114174899
* Support unreadable filesGravatar Michajlo Matijkiw2016-01-20
| | | | | -- MOS_MIGRATED_REVID=112507181
* Use Bazel Preconditions variant which avoids varargs array creationGravatar Mark Schaller2015-12-10
| | | | | | | Reduces garbage. -- MOS_MIGRATED_REVID=109914243
* Rationalize copyright headersGravatar Damien Martin-Guillerez2015-09-25
| | | | | | | | | | | The headers were modified with `find . -type f -exec 'sed' '-Ei' 's|Copyright 201([45]) Google|Copyright 201\1 The Bazel Authors|' '{}' ';'` And manual edit for not Google owned copyright. Because of the nature of ijar, I did not modified the header of file owned by Alan Donovan. The list of authors were extracted from the git log. It is missing older Google contributors that can be added on-demand. -- MOS_MIGRATED_REVID=103938715
* Move Label from the lib.syntax to the lib.cmdline package so that:Gravatar Lukacs Berki2015-09-21
| | | | | | | | | | - Label parsing can be simplified - lib.syntax is only contains the code for Skylark and is reasonably independent from the problem domain of building things This change is mostly only changes to imports declarations. The rest is reversing the dependency between :cmdline and :syntax and moving a tiny amount of code between Printer and FilesetEntry and the addition of SkylarkPrintableValue that I couldn't be bothered to separate out into its own change. -- MOS_MIGRATED_REVID=103527877
* Elegantly handle unbounded file symlink resolutions, e.g. 'a' -> 'b' -> ↵Gravatar Nathan Harmata2015-07-29
| | | | | | | 'a/nope'. -- MOS_MIGRATED_REVID=99337668
* Update from Google.Gravatar Han-Wen Nienhuys2015-02-25
-- MOE_MIGRATED_REVID=85702957
generated by cgit v1.2.3 (git 2.39.1) at 2024-06-23 02:24:50 +0000