aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/skyframe/SkyframeActionExecutor.java
Commit message (Collapse)AuthorAge
* Remove EventBus from SkyframeActionExecutorGravatar ulfjack2018-01-16
| | | | | | Post to the Skyframe Environment instead. PiperOrigin-RevId: 182096559
* Pass file system to Path operations.Gravatar tomlu2017-12-06
| | | | PiperOrigin-RevId: 178116776
* Add ActionKeyContext to Action#getKey.Gravatar tomlu2017-11-29
| | | | | | | This key context can be used by actions to share partial key computations, for instance when computing MD5s for nested sets. RELNOTES: None PiperOrigin-RevId: 177359607
* Replace all usages of Blaze's Preconditions class with guava.Gravatar tomlu2017-11-09
| | | | | | | | Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own. This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually. PiperOrigin-RevId: 175033526
* Thread FileSystem through to a single Path#delete call site.Gravatar tomlu2017-10-30
| | | | | | This requires a fairly large amount of changes to fundamental objects like BlazeRuntime, Executor, and so on, as well as changing a lot of test code to thread the file system through. I expect future CLs to be much smaller. PiperOrigin-RevId: 173678144
* Post non-empty ActionResults for all completed Actions.Gravatar ruperts2017-10-23
| | | | | RELNOTES: None. PiperOrigin-RevId: 172955521
* Actions now have the option of returning an ActionResult, containing a ↵Gravatar ruperts2017-10-18
| | | | | | | (possibly empty) set of SpawnResults created during execution of the Action. RELNOTES: None. PiperOrigin-RevId: 172529328
* Set a bit in ActionExecutedEvent indicating if a failure occurred before the ↵Gravatar janakr2017-08-11
| | | | | | action actually executed. PiperOrigin-RevId: 164917959
* Move OutputService to lib.skyframeGravatar ulfjack2017-08-11
| | | | | | | | | | | This isn't ideal, but is the fastest way to split analysis, execution, and rules into separate java libraries, with lib.skyframe still being in the analysis library. Ideally, we'd split up the lib.skyframe package, move the analysis stuff to the analysis library, the execution stuff to the execution library, and so on. That wouldn't require us moving OutputService out of lib.exec. PiperOrigin-RevId: 164856998
* SkyframeActionExecutor.checkOutputs: clarify commentGravatar ulfjack2017-07-26
| | | | | | | The getMetadata call implicitly adds files to the metadata handler if they weren't known before, which may include statting the file. PiperOrigin-RevId: 163106987
* Add ActionInputPrefetcher to ActionExecutionContextGravatar ulfjack2017-07-24
| | | | | | | This is more consistent with other values, and removes the need to inject it into the constructor of the various strategy implementations. PiperOrigin-RevId: 162729187
* Avoid races while deleting output directory ancestorsGravatar mschaller2017-07-13
| | | | | | | | | | | | | When an output file becomes a directory from one build to another, the filesystem operations performed by several threads can race while they prepare output directories. Without synchronization, these races can cause failures. See the comments added in this CL for more detail. This CL adds synchronization around these filsystem operations and gets rid of the problematic races. RELNOTES: None. PiperOrigin-RevId: 161714702
* Simplify the ActionInputFileCacheGravatar ulfjack2017-07-06
| | | | | | | | | Add a single getMetadata method (matching MetadataHandler), and rewrite everything in those terms. This is in preparation for merging ActionInputFileCache and MetadataHandler. PiperOrigin-RevId: 161053535
* Extract ActionContext to a top-level classGravatar ulfjack2017-06-19
| | | | PiperOrigin-RevId: 159423459
* Rewrite the Executor/ActionExecutionContext splitGravatar ulfjack2017-06-19
| | | | | | | Move everything to ActionExecutionContext, and drop Executor whereever possible. This clarifies the API, makes it simpler to test, and simplifies the code. PiperOrigin-RevId: 159414816
* Add a custom interface for cache hit processing in actionsGravatar ulfjack2017-05-10
| | | | | | | | | | | | The new interface mirrors ActionExecutionContext, but is restricted to exactly the parts used right now. I did consider using ActionExecutionContext, but it contains some parts that we don't want to make available for cache hits. The end goal is to allow the build event stream access to artifact metadata, in particular for TestResult and TestSummary events, which in turn requires making artifact metadata available when the TestRunnerAction is a cache hit. PiperOrigin-RevId: 155612573
* Automated g4 rollback of commit 56e8d90674a35ebe727f74cd2d47108bfcaa4515.Gravatar ulfjack2017-05-09
| | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Looks like it broke stuff - the presubmit bypass was added by accident. *** Original change description *** Add a custom interface for cache hit processing in actions The new interface mirrors ActionExecutionContext, but is restricted to exactly the parts used right now. I did consider using ActionExecutionContext, but it contains some parts that we don't want to make available for cache hits. The end goal is to allow the build event stream access to artifact metadata, in particular for TestResult and TestSummary events, which in turn requires making artifact metadata available when the TestRunnerAc... *** PiperOrigin-RevId: 155493797
* Add a custom interface for cache hit processing in actionsGravatar ulfjack2017-05-09
| | | | | | | | | | | | The new interface mirrors ActionExecutionContext, but is restricted to exactly the parts used right now. I did consider using ActionExecutionContext, but it contains some parts that we don't want to make available for cache hits. The end goal is to allow the build event stream access to artifact metadata, in particular for TestResult and TestSummary events, which in turn requires making artifact metadata available when the TestRunnerAction is a cache hit. PiperOrigin-RevId: 155492447
* Rewrite repository lookup to return a failed value rather than throwGravatar ulfjack2017-04-20
| | | | | | | | | | | | | | | | | | | | | | | We need to lookup repositories as part of converting exec paths to artifacts, which in turn is needed for action cache lookups. These lookups should not cause a Skyframe exit, so we must not throw an exception here, unless the error makes it impossible to continue. Instead, we need to leave the decision whether to error out or not to the caller. Note that we may unnecessarily fetch a remote repository in order to do the action cache lookup, even if the action no longer depends on the input file, although this should only be possible for C++ compile actions. It's possible that there's another bug in the C++ compile action key computation that also contributes. This change also makes it so that the post-resolution action cache code ignores any errors wrt. repository lookup rather than throwing. If any of the paths could not be found, then the action cache lookup fails and we re-execute the corresponding action, which is exactly what should happen. Fixes #2759. PiperOrigin-RevId: 153696243
* Move the chmod call from SkyframeActionExecutor to ActionMetadataHandlerGravatar ulfjack2017-04-07
| | | | | | | | | | | | | | | | | | | This relies on the explicit state transition in MetadataHandler to decide whether the chmod call is necessary (we must never call chmod if the action was not executed). This is a prerequisite for #1525. If we want to use ctime for detecting file content changes (which is more reliable than just mtime), then we must call chmod before stat, since chmod affects ctime. Before this change, we were caching the stat in ActionMetadataHandler, but calling chmod after action execution in SkyframeActionExecutor, which is the wrong order of calls. However, we must be able to stat in ActionMetadataHandler for cases where a single action runs multiple Spawns where one spawn's output is a subsequent spawn's input. Remove MetadataHandler.isInjected, which is no longer used anywhere. PiperOrigin-RevId: 152387133
* Explicitly document the state transition on MetadataHandlerGravatar ulfjack2017-04-06
| | | | | | | | | | | | | | | | The ActionMetadataHandler does an explicit state transition on discardOutputMetadata. Before the call, it may be used for action cache checking, and after the call it may be updated with execution results. Several of the methods now throw if they're used incorrectly, so I had to refactor the control flow in ActionExecutionFunction to correctly call discardOutputMetadata on the MetadataHandler in all cases. I discovered a resource leak (of FileOutErr) in IncludeParseFunction while I was at it, so I plugged that as well. One step towards #1525. PiperOrigin-RevId: 152363982
* If --batch, --keep_going, --discard_analysis_cache, and the new ↵Gravatar janakr2017-03-31
| | | | | | | | | | | | | | | --noexperimental_enable_critical_path_profiling flags are all specified, then Bazel will delete Actions from ActionLookupValues as they are executed in order to save memory. Because an already-run action may output an artifact that is only requested later in the build, we need to maintain a way for the artifact to look up the action. But in most cases we don't need to keep the action itself, just its output metadata. Some actions unfortunately are needed post-execution, and so we special-case them. Also includes dependency change with description: Move action out of key. This keeps action references from polluting the graph -- actions are just stored in one SkyValue, instead of being present in SkyKeys. This does mean additional memory used: we have a separate ActionLookupData object per Action executed. That may reach ~24M for million-action builds. PiperOrigin-RevId: 151756383
* Remove only-in-tests null checks in SkyframeActionExecutor.Gravatar Janak Ramakrishnan2017-03-22
| | | | | | | | Still on step -0.5, let's be honest. -- PiperOrigin-RevId: 150783638 MOS_MIGRATED_REVID=150783638
* Rationalize local resource acquisitionGravatar Ulf Adams2017-03-03
| | | | | | | | | | | | | | | | | | | | | | Move all local resource acquisition to where local execution actually happens. Don't attempt to acquire resources per action, but only for individual spawns. This significantly simplifies the code. The downside is that we don't account for action-level work anymore. In general, actions should not perform any process execution themselves, but always delegate such work to a SpawnStrategy implementation. This change makes sure that every Spawn has local resources set in a way that is consistent with the previous state. However, there are two actions - Fileset and FileWrite -, which are not spawns, and so we now don't limit their concurrent execution anymore. For Fileset, all work is done in a custom Fileset-specific thread pool, so this shouldn't be a problem. I'm not sure about FileWriteAction. -- PiperOrigin-RevId: 149012600 MOS_MIGRATED_REVID=149012600
* Clean up the semantics of input discovering actions a bit by making ↵Gravatar Lukacs Berki2017-02-28
| | | | | | | | | | | | updateInputs() and inputsKnown() non-overridable and removing setInputs(). This comes at the cost of adding a flag to every action instance that's not used for non-input-discovering actions, but I think that's a deal. Simpler APIs are good, mmmmkay? Also fixed a few pre-existing issues in TestAction and ObjcCompileAction. -- PiperOrigin-RevId: 148749734 MOS_MIGRATED_REVID=148749734
* Tighten a precondition check during action execution.Gravatar Lukacs Berki2017-02-10
| | | | | | | | Input discovery is handled somewhere else, so the same Action instance should never be put twice into buildActionMap . -- PiperOrigin-RevId: 147132869 MOS_MIGRATED_REVID=147132869
* Support mapping of Paths to URIsGravatar Klaus Aehlig2017-01-18
| | | | | | | | | | | Bazel-created files (like log files of test runs) are internally reported as Paths. However, this is not always the most useful representation of the location of that artifact for a consumer of build events. Therefore, support a mapping of paths to more useful URIs. -- PiperOrigin-RevId: 144843525 MOS_MIGRATED_REVID=144843525
* Log the IOException when an output was not successfully created. It's ↵Gravatar Janak Ramakrishnan2017-01-13
| | | | | | | | possible the exception has information we've been suppressing. -- PiperOrigin-RevId: 144405370 MOS_MIGRATED_REVID=144405370
* Log if dangling symlink is found. This is an odd choice of thing to log, but ↵Gravatar Janak Ramakrishnan2017-01-13
| | | | | | | | shouldn't come up very often, except in the linked bug. -- PiperOrigin-RevId: 144367142 MOS_MIGRATED_REVID=144367142
* Description redacted.Gravatar Googler2016-12-12
| | | | | | -- PiperOrigin-RevId: 141561079 MOS_MIGRATED_REVID=141561079
* Proper action output checks for TreeArtifacts. Instead of crashing Bazel, we ↵Gravatar Rumou Duan2016-10-19
| | | | | | | now handle failed TreeArtifact output checks gracefully. -- MOS_MIGRATED_REVID=136627086
* Track client environment in SkyframeGravatar Klaus Aehlig2016-09-14
| | | | | | | | | | | | | ...to determine which actions have to be recomputed based on changes to the client environment. Note that this change does it the simple way and reconsideres all actions on a changed client environment, while still only reexecuting those, where the part that was inherited from the environment actually did change. -- Change-Id: Ie1116d094642165e5e959447a6fcf49d19b37d6e Reviewed-on: https://bazel-review.googlesource.com/#/c/5431 MOS_MIGRATED_REVID=133010705
* Add client environment to ActionExecutionContextGravatar Klaus Aehlig2016-08-26
| | | | | | | | | | | As the execution of an action now also depends on the client environment, make the latter part of the ActionExecutionContext, so that enough context is provided to actually execute an action. -- Change-Id: Ida7bf407ef0c0375728faba92494bfd47dcbaeb8 Reviewed-on: https://bazel-review.googlesource.com/#/c/5391 MOS_MIGRATED_REVID=131377490
* Action#discoverInputs returns an Iterable<Artifact> instead of aGravatar Cal Peyser2016-08-25
| | | | | | | Collection<Artifact> -- MOS_MIGRATED_REVID=131285541
* Allow Skyframe graph lookups and value retrievals to throw InterruptedException.Gravatar Janak Ramakrishnan2016-08-16
| | | | | | | The only place we now don't handle InterruptedException is in the action graph created after analysis, since I'm not sure that will be around for that much longer. -- MOS_MIGRATED_REVID=130327770
* Use byte[] rather than ByteString for file digests.Gravatar Googler2016-07-20
| | | | | | | | | | | | | | | | | ActionInputFileCache: Change getDigest() to return the underlying byte[16] owned by each FileArtifactValue. Remove throws clause from getInputFromDigest(); this should be an in-memory operation, and no implementation actually throws. PerActionFileCache: Invert mapping from artifact to digest only if needed. Remove interner, as it was used only for the reverse map keys, not the returned values. This should be a significant cpu savings as eagerly constructing the reverse maps was a noticeable hotspot. -- MOS_MIGRATED_REVID=127972359
* Rename FileOutErr.get{Output,Error}File to get{Output,Error}Path, because ↵Gravatar Philipp Wollermann2016-06-29
| | | | | | | they actually return a Path and not a File. -- MOS_MIGRATED_REVID=126102820
* 1. Create the TreeArtifact directory structure before expanding ActionTemplates.Gravatar Rumou Duan2016-06-14
| | | | | | | | 2. In PopulateTreeArtifactAction, create the parent directories for TreeFileArtifacts before executing the spawn. 3. Allow empty tree artifacts in CustomCommandLine and PopulateTreeArtifact. -- MOS_MIGRATED_REVID=124759286
* remove duplicate commentGravatar Benjamin Peterson2016-04-26
| | | | | | | -- Change-Id: I4c7cc2083cf0f886875b662440ed406e52dffae5 Reviewed-on: https://bazel-review.googlesource.com/#/c/3491/ MOS_MIGRATED_REVID=120804156
* Split ActionMetadata into ActionAnalysisMetadata and ActionExecutionMetadata.Gravatar Rumou Duan2016-04-26
| | | | | | | Except in action execution logic (ActionExecutionFunction, SkyframeActionExecutor, etc.), switch Action interface references to either ActionAnalysisMetadata if possible or ActionExecutionMetadata. -- MOS_MIGRATED_REVID=120723431
* Extract common logic for detecting action and artifact prefix conflicts.Gravatar Rumou Duan2016-04-19
| | | | | -- MOS_MIGRATED_REVID=120145833
* Introduce TreeFileArtifact, which represents files under TreeArtifacts.Gravatar Rumou Duan2016-04-14
| | | | | | | Remove ArtifactFile, which is rendered obsolete by TreeFileArtifact. -- MOS_MIGRATED_REVID=119789154
* Ensure that errors from a mismatch between include scanning and local ↵Gravatar Janak Ramakrishnan2016-03-09
| | | | | | | execution are reported. Previously, we would not report the error unless the build was already failing and this error ended up being the root cause of the first failed top-level target. -- MOS_MIGRATED_REVID=116767915
* Little refactorings on resource management stuff:Gravatar Philipp Wollermann2016-02-28
| | | | | | | | | | - Remove unneeded abstract override AbstractAction#getResourceConsumption() - Remove comments justifying returning ResourceSet.ZERO from getResourceConsumption() - instead document it once on the base method. - Replace ResourceSet.create(0, 0, 0) with ResourceSet.ZERO. - Make sure no one ever returns null from getResourceConsumption() and then simplify the code in SkyframeActionExecutor to not check for it. -- MOS_MIGRATED_REVID=115739250
* Rollback of commit a0eefb52f529b73c6cb92f0a762853646ea2eae6.Gravatar Googler2016-02-11
| | | | | | | | | | | | | *** Reason for rollback *** Rolling forward with the restored logic to avoid stat calls on injected Metadata. *** Original change description *** Automated [] rollback of commit df03e10f6552566982399b8779fe7bc7a17d75dc. -- MOS_MIGRATED_REVID=114447944
* Rollback of commit df03e10f6552566982399b8779fe7bc7a17d75dc.Gravatar Michajlo Matijkiw2016-02-11
| | | | | -- MOS_MIGRATED_REVID=114329043
* Make unpredictable action inputs and outputs available to Actions.Gravatar Michael Thvedt2016-02-10
| | | | | -- MOS_MIGRATED_REVID=114249806
* Make The Build Faster: Use AutoCloseable when dealing with resources.Gravatar Philipp Wollermann2016-02-10
| | | | | -- MOS_MIGRATED_REVID=114204404
* Support for handling TreeArtifact metadata and returning TreeArtifacts from ↵Gravatar Michael Thvedt2016-02-09
| | | | | | | ArtifactFunction. -- MOS_MIGRATED_REVID=114174899
* Rename MiddlemanExpander to ArtifactExpander, and refactor it to yield ↵Gravatar Michael Thvedt2016-02-09
| | | | | | | ArtifactFiles. -- MOS_MIGRATED_REVID=114166208
generated by cgit v1.2.3 (git 2.39.1) at 2024-08-18 10:08:01 +0000