aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/skyframe/SkyframeActionExecutor.java
Commit message (Collapse)AuthorAge
* Support tree artifacts in target complete.Gravatar tomlu2018-08-15
| | | | | RELNOTES: None PiperOrigin-RevId: 208861642
* Automated rollback of commit 37bd5f665aa614c6dc640c9d19852dd8d5efb0d8.Gravatar felly2018-08-14
| | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Rollforward with fix: Don't use Immutable map builder to agggregate Filesets. We may reach the same one via direct and indirect runfiles, so we need to allow for duplicate entries. *** Original change description *** Automated rollback of commit 3bace1b937934fb2cea6260067ecc1cdbe526847. *** Reason for rollback *** b/112583337 RELNOTES: None *** Original change description *** Track Fileset in artifact expansion. PiperOrigin-RevId: 208748163
* Automated rollback of commit 3bace1b937934fb2cea6260067ecc1cdbe526847.Gravatar felly2018-08-14
| | | | | | | | | | | | | *** Reason for rollback *** b/112583337 RELNOTES: None *** Original change description *** Track Fileset in artifact expansion. PiperOrigin-RevId: 208658921
* Track Fileset in artifact expansion.Gravatar felly2018-08-13
| | | | PiperOrigin-RevId: 208549418
* Automated rollback of commit 9374ecf94ce44e8bc56e68678cb512abf4cf9ce2.Gravatar felly2018-08-10
| | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Rollforward with deduplication of traversals and regression test. RELNOTES: None *** Original change description *** Automated rollback of commit 39974a43abdd32e3a1acbc7da945b08da9983e4e. *** Reason for rollback *** b/112458627 *** Original change description *** Allow skyframe-aware actions to pass partial results through ActionExecutionContext. Remove FilesetProvider. PiperOrigin-RevId: 208231719
* Combine prefix event and payload stdout/stderrGravatar Klaus Aehlig2018-08-10
| | | | | | | | | ...in a single event instead of relying on the synchronisation on the reporter. In this way, it will be possible to focus on output of errors, and also to deduplicate warnings. Change-Id: I9669e8497ca6e22a01f3e285ec9ee42c1ecf3fb5 PiperOrigin-RevId: 208223882
* Automated rollback of commit 39974a43abdd32e3a1acbc7da945b08da9983e4e.Gravatar felly2018-08-10
| | | | | | | | | | | | | *** Reason for rollback *** b/112458627 *** Original change description *** Allow skyframe-aware actions to pass partial results through ActionExecutionContext. Remove FilesetProvider. PiperOrigin-RevId: 208213955
* Allow skyframe-aware actions to pass partial results through ↵Gravatar felly2018-08-09
| | | | | | | | ActionExecutionContext. Remove FilesetProvider. PiperOrigin-RevId: 208111251
* Pass Fileset information to the Action filesystem.Gravatar felly2018-08-09
| | | | PiperOrigin-RevId: 208096548
* Disables ActionFS sandboxing of discovered derived inputs forGravatar shahan2018-08-02
| | | | | | performance reasons. PiperOrigin-RevId: 207171305
* remote: add support for directory inputs in runfilesGravatar buchgr2018-08-02
| | | | | | | | | | | | | | | | Add support for tree artifacts (ctx.action.declare_directory(...)) in runfiles. Before this change we would throw away the information about the files inside a tree artifact before executing an action. That's fine for local execution where the sandbox just copies/symlinks a directory and doesn't care much what's inside. However, in remote execution we actually need to upload each individual file and so we need to be aware of all individual files not just directories. This change makes it so that this information is made available to a SpawnRunner via the SpawnInputExpander. RELNOTES: None PiperOrigin-RevId: 207091668
* Automated rollback of commit a6255612e4892729d3758775c76085b26b9bc584.Gravatar twerth2018-08-02
| | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Breaks FDO builds, see http://b/112112125 *** Original change description *** Remove PerActionFileCache Instead, make ActionMetadataHandler implement the MetadataProvider interface. This fixes an issue where an action that runs two spawns where one depends on an output of the other was unable to get the metadata for the intermediate output. We don't currently have actions that do this, but we will have in a future change (which will also implicitly act as a regression test). PiperOrigin-RevId: 207084179
* Pass a path to ActionExecutedEvent that is resolved via ActionExecutionContext.Gravatar tomlu2018-08-01
| | | | | | | If the output artifact is backed by a different file system then we need to use that file system later. RELNOTES: None PiperOrigin-RevId: 206981369
* Use the path resolver when dealing with tree artifacts.Gravatar felly2018-07-26
| | | | | RELNOTES: None PiperOrigin-RevId: 206157591
* Automated rollback of commit f309ad3be36363070e87eef0ee04b12f4956d601.Gravatar janakr2018-07-24
| | | | | | | | | | *** Reason for rollback *** Fixed duplicate derived inputs bug. Test is in diffbase. RELNOTES[INC]: If the same artifact is generated by two distinct but identical actions, and a downstream action has both those actions' outputs in its inputs, the artifact will now appear twice in the downstream action's inputs. If this causes problems in Skylark actions, you can use the uniquify=True argument in Args.add_args. PiperOrigin-RevId: 205863806
* Remove PerActionFileCacheGravatar ulfjack2018-07-23
| | | | | | | | | | | | | Instead, make ActionMetadataHandler implement the MetadataProvider interface. This fixes an issue where an action that runs two spawns where one depends on an output of the other was unable to get the metadata for the intermediate output. We don't currently have actions that do this, but we will have in a future change (which will also implicitly act as a regression test). PiperOrigin-RevId: 205629237
* Automated rollback of commit d491bf10f42e213292382c98a1dc439537f00f43.Gravatar janakr2018-07-17
| | | | | | | | | | | | | | | | | | *** Reason for rollback *** Still bugs lurking. See linked bug. *** Original change description *** Automated rollback of commit eb587075b0d6ffab1cf9e69ede1b7e547905e547. *** Reason for rollback *** Depot has been fixed. RELNOTES[INC]: If the same artifact is generated by two distinct but identical actions, and a downstream action has both those actions' outputs in its inputs, the artifact will now appear twice in the downstream action's inputs. If this causes problems in Skylark actions, you can use the uniquify=True argument in Args.add_args. PiperOrigin-RevId: 204997569
* Automated rollback of commit eb587075b0d6ffab1cf9e69ede1b7e547905e547.Gravatar janakr2018-07-16
| | | | | | | | | | *** Reason for rollback *** Depot has been fixed. RELNOTES[INC]: If the same artifact is generated by two distinct but identical actions, and a downstream action has both those actions' outputs in its inputs, the artifact will now appear twice in the downstream action's inputs. If this causes problems in Skylark actions, you can use the uniquify=True argument in Args.add_args. PiperOrigin-RevId: 204827477
* Instead of depending on mutable inputs of upstream CppCompileAction instances,Gravatar shahan2018-06-26
| | | | | | | | | CppCompileAction.discoverInputsStage2 retrieves values of discovered modules from ActionExecutionValue. This addresses a possible a correctness issue. PiperOrigin-RevId: 202162180
* Automated rollback of commit 45b308a62f42c2c0bcfe79dcd4046c4025a31059.Gravatar janakr2018-06-20
| | | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Breaks Javascript compilation. There's currently no way to iterate over a depset of Artifacts and deduplicate by identical paths in Skylark. This means that actions that want to do something once per Artifact in a depset (add a flag to the command line with the path of the Artifact for instance) will have duplicate entries if there are multiple Artifacts with the same path. This is not a true automated rollback, since I tried to make this as minimal as possible, to avoid merge conflicts and keep some of the benefits of the rolled back CL. In particular, the tests that were changed in the original CL to give artifacts their correct owners did not need to be changed back, since the owners are still correct. Moreover, this effectively contains rollbacks of unknown commit and https://github.com/bazelbuild/bazel/commit/39d6f89644107a8f7c080c4f4aec8527c1a73954, but keeps test coverage from those CLs as well. Comments added to CL thread where not a clean rollback (there are plenty of files not changed at all: ActionArtifactCycleReporter is the main wart, since it can still handle SkyKeys with Artifact as the argument, but Artifact is no longer the argument to any SkyKey). RELNOTES: None *** Original change description *** Make Artifact#equals take the owner into account for derived artifacts. Derived artifacts' owners are important because they are used to determine the artifact's generating action. Source artifacts' owners are not used in this way, so I left them alone. This allows us to get rid of most uses of ArtifactSkyKey. We may be able to delete it entirely in a follow-up. PiperOrigin-RevId: 201464780
* Refactoring: uses OutputService for ActionFileSystem injectionGravatar shahan2018-06-18
| | | | | | Moves more Action-oriented from skyframe package, which has become very unwieldy, to action package. This is needed to avoid circular dependencies caused when build-base is needed for skyframe. PiperOrigin-RevId: 200996982
* Add functionality to make certain SkyValues unshareable, meaning they are ↵Gravatar janakr2018-06-13
| | | | | | not serialized. Tag TestCompletionValue and any ActionExecutionValue coming from a NotifyOnActionCacheHit (i.e., tests) like that. To make such values really not shared, request the ActionExecutionValue from TestCompletionFunction as opposed to the ArtifactValue (propagating the unshareable bit up seemed like too much fuss, and I have a dream of getting rid of ArtifactValue anyway). PiperOrigin-RevId: 200504358
* Pass ActionFS paths through to action-level FileOutErr for the Action's ↵Gravatar felly2018-06-13
| | | | | | | stdout/stderr. RELNOTES: None PiperOrigin-RevId: 200459354
* Add AutoProfiler-like API to ProfilerGravatar ulfjack2018-06-11
| | | | | | - migrate all startTask/completeTask pairs to the new API PiperOrigin-RevId: 200038703
* Make Artifact#equals take the owner into account for derived artifacts.Gravatar janakr2018-06-08
| | | | | | | | Derived artifacts' owners are important because they are used to determine the artifact's generating action. Source artifacts' owners are not used in this way, so I left them alone. This allows us to get rid of most uses of ArtifactSkyKey. We may be able to delete it entirely in a follow-up. PiperOrigin-RevId: 199836436
* Deletes ActionInputFileCache.Gravatar shahan2018-06-07
| | | | PiperOrigin-RevId: 199732415
* Refactoring: makes the code simpler by deleting Metadata and InputArtifactData.Gravatar shahan2018-06-06
| | | | | | (minor) ActionFS now implements MetadataProvider.getInput PiperOrigin-RevId: 199575194
* MetadataProvider now provides ActionInput lookup by exec path.Gravatar shahan2018-06-05
| | | | | | Adds a helper class, ActionInputMap to do this with minimal wrapping overhead. PiperOrigin-RevId: 199391251
* Remove an unused method variable.Gravatar lberki2018-06-04
| | | | | RELNOTES: None. PiperOrigin-RevId: 199115199
* Removes most ActionInputFileCache functionality.Gravatar shahan2018-06-01
| | | | | | Actual class to be removed in a later change. PiperOrigin-RevId: 198937695
* With ActionFS, there is no need to delete previous output files. These ↵Gravatar felly2018-05-24
| | | | | | should not exist. PiperOrigin-RevId: 197895718
* Don't construct directories when using ActionFS - these are implicit.Gravatar felly2018-05-18
| | | | PiperOrigin-RevId: 197211669
* Allows ActionFS to delegate to other file systems.Gravatar shahan2018-05-17
| | | | PiperOrigin-RevId: 197055263
* Optimizes performance of ActionFS staging and eliminates ActionFS updates.Gravatar shahan2018-05-15
| | | | | | | | | Extracts a class, InputArtifactData to hold the input data instead of using a raw map. This provides the flexibility needed to support both ActionFS and existing code so ActionFS does not need to rekey the input data. Uses the smaller, getDeclaredIncludeSrcs instead of getAllowedDerivedInputs when possible for staging optional inputs in ActionFS. PiperOrigin-RevId: 196736703
* Adds an action-scoped filesystem.Gravatar shahan2018-05-09
| | | | PiperOrigin-RevId: 195973862
* Use the in-memory metadata in blaze as the source of truth for Fileset mappingsGravatar kush2018-05-02
| | | | | | | instead of the manifest files. RELNOTES: None PiperOrigin-RevId: 195149880
* Properly report completion of shared actions with input discoveryGravatar hlopko2018-04-24
| | | | | | | | | | | | | | Currently we report "Analyzing" when include scanning runs. But since we can have shared C++ compile actions, only one of the group will be executed and only one will be reported completed. Remaining shared actions currently stay with "analyzing" forever. This cl makes sure that these actions are properly handled when finished. This is an encore of https://github.com/bazelbuild/bazel/commit/24f19ec2679dd93b1ac5b06e46f3b35807d6e217. In this incarnation I make sure that all actions that discover inputs are consistent in reporting their Analyzing status. Originally only CppCompileAction was doing that. Apparently we have more actions that discover inputs (e.g. LtoBackendAction) but these were not reporting Analyzing and therefore crashing on preconditions. This cl makes sure that all actions discovering inputs report their analyzing status. RELNOTES: None PiperOrigin-RevId: 194066513
* Automated rollback of commit 24f19ec2679dd93b1ac5b06e46f3b35807d6e217.Gravatar ruperts2018-04-23
| | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** See: http://b/78455900 *** Original change description *** Properly report completion of shared actions with input discovery Currently we report "Analyzing" when include scanning runs. But since we can have shared C++ compile actions, only one of the group will be executed and only one will be reported completed. Remaining shared actions currently stay with "analyzing" forever. This cl makes sure that these actions are properly handled when finished. RELNOTES: None PiperOrigin-RevId: 193955856
* Properly report completion of shared actions with input discoveryGravatar hlopko2018-04-05
| | | | | | | | | | | | Currently we report "Analyzing" when include scanning runs. But since we can have shared C++ compile actions, only one of the group will be executed and only one will be reported completed. Remaining shared actions currently stay with "analyzing" forever. This cl makes sure that these actions are properly handled when finished. RELNOTES: None PiperOrigin-RevId: 191849728
* Change profiling to only accept strings for its "description" argument. ↵Gravatar janakr2018-04-01
| | | | | | Profiling can hold onto objects for the duration of the build, and some of those objects may be temporary that should not be persisted. In particular, UnixGlob and its inner classes should not outlive loading and analysis. For the most part, care was taken in this CL to only use strings that required no additional construction, mainly to minimize garbage (retaining references to newly created strings is not as great a concern since only the strings corresponding to the slowest K tasks are retained, for some relatively small values of K). Action descriptions for actually executing actions are eagerly expanded because that work is minimal compared to the work of actually executing an action. PiperOrigin-RevId: 191251488
* Replace path implementation.Gravatar tomlu2018-02-08
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Path and PathFragment have been replaced with String-based implementations. They are pretty similar, but each method is dissimilar enough that I did not feel sharing code was appropriate. A summary of changes: PATH ==== * Subsumes LocalPath (deleted, its tests repurposed) * Use a simple string to back Path * Path instances are no longer interned; Reference equality will no longer work * Always normalized (same as before) * Some operations will now be slower, like instance compares (which were previously just a reference check) * Multiple identical paths will now consume more memory since they are not interned PATH FRAGMENT ============= * Use a simple string to back PathFragment * No more segment arrays with interned strings * Always normalized * Remove isNormalized * Replace some isNormalizied uses with containsUpLevelReferences() to check if path fragments try to escape their scope * To check if user input is normalized, supply static methods on PathFragment to validate the string before constructing a PathFragment * Because PathFragments are always normalized, we have to replace checks for literal "." from PathFragment#getPathString to PathFragment#getSafePathString. The latter returns "." for the empty string. * The previous implementation supported efficient segment semantics (segment count, iterating over segments). This is now expensive since we do longer have a segment array. ARTIFACT ======== * Remove Path instance. It is instead dynamically constructed on request. This is necessary to avoid this CL becoming a memory regression. RELNOTES: None PiperOrigin-RevId: 185062932
* Remove EventBus from SkyframeActionExecutorGravatar ulfjack2018-01-16
| | | | | | Post to the Skyframe Environment instead. PiperOrigin-RevId: 182096559
* Pass file system to Path operations.Gravatar tomlu2017-12-06
| | | | PiperOrigin-RevId: 178116776
* Add ActionKeyContext to Action#getKey.Gravatar tomlu2017-11-29
| | | | | | | This key context can be used by actions to share partial key computations, for instance when computing MD5s for nested sets. RELNOTES: None PiperOrigin-RevId: 177359607
* Replace all usages of Blaze's Preconditions class with guava.Gravatar tomlu2017-11-09
| | | | | | | | Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own. This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually. PiperOrigin-RevId: 175033526
* Thread FileSystem through to a single Path#delete call site.Gravatar tomlu2017-10-30
| | | | | | This requires a fairly large amount of changes to fundamental objects like BlazeRuntime, Executor, and so on, as well as changing a lot of test code to thread the file system through. I expect future CLs to be much smaller. PiperOrigin-RevId: 173678144
* Post non-empty ActionResults for all completed Actions.Gravatar ruperts2017-10-23
| | | | | RELNOTES: None. PiperOrigin-RevId: 172955521
* Actions now have the option of returning an ActionResult, containing a ↵Gravatar ruperts2017-10-18
| | | | | | | (possibly empty) set of SpawnResults created during execution of the Action. RELNOTES: None. PiperOrigin-RevId: 172529328
* Set a bit in ActionExecutedEvent indicating if a failure occurred before the ↵Gravatar janakr2017-08-11
| | | | | | action actually executed. PiperOrigin-RevId: 164917959
* Move OutputService to lib.skyframeGravatar ulfjack2017-08-11
| | | | | | | | | | | This isn't ideal, but is the fastest way to split analysis, execution, and rules into separate java libraries, with lib.skyframe still being in the analysis library. Ideally, we'd split up the lib.skyframe package, move the analysis stuff to the analysis library, the execution stuff to the execution library, and so on. That wouldn't require us moving OutputService out of lib.exec. PiperOrigin-RevId: 164856998
generated by cgit v1.2.3 (git 2.39.1) at 2024-06-13 10:34:37 +0000