aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/skyframe/SkyframeActionExecutor.java
Commit message (Collapse)AuthorAge
* Avoid races while deleting output directory ancestorsGravatar mschaller2017-07-13
| | | | | | | | | | | | | When an output file becomes a directory from one build to another, the filesystem operations performed by several threads can race while they prepare output directories. Without synchronization, these races can cause failures. See the comments added in this CL for more detail. This CL adds synchronization around these filsystem operations and gets rid of the problematic races. RELNOTES: None. PiperOrigin-RevId: 161714702
* Simplify the ActionInputFileCacheGravatar ulfjack2017-07-06
| | | | | | | | | Add a single getMetadata method (matching MetadataHandler), and rewrite everything in those terms. This is in preparation for merging ActionInputFileCache and MetadataHandler. PiperOrigin-RevId: 161053535
* Extract ActionContext to a top-level classGravatar ulfjack2017-06-19
| | | | PiperOrigin-RevId: 159423459
* Rewrite the Executor/ActionExecutionContext splitGravatar ulfjack2017-06-19
| | | | | | | Move everything to ActionExecutionContext, and drop Executor whereever possible. This clarifies the API, makes it simpler to test, and simplifies the code. PiperOrigin-RevId: 159414816
* Add a custom interface for cache hit processing in actionsGravatar ulfjack2017-05-10
| | | | | | | | | | | | The new interface mirrors ActionExecutionContext, but is restricted to exactly the parts used right now. I did consider using ActionExecutionContext, but it contains some parts that we don't want to make available for cache hits. The end goal is to allow the build event stream access to artifact metadata, in particular for TestResult and TestSummary events, which in turn requires making artifact metadata available when the TestRunnerAction is a cache hit. PiperOrigin-RevId: 155612573
* Automated g4 rollback of commit 56e8d90674a35ebe727f74cd2d47108bfcaa4515.Gravatar ulfjack2017-05-09
| | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Looks like it broke stuff - the presubmit bypass was added by accident. *** Original change description *** Add a custom interface for cache hit processing in actions The new interface mirrors ActionExecutionContext, but is restricted to exactly the parts used right now. I did consider using ActionExecutionContext, but it contains some parts that we don't want to make available for cache hits. The end goal is to allow the build event stream access to artifact metadata, in particular for TestResult and TestSummary events, which in turn requires making artifact metadata available when the TestRunnerAc... *** PiperOrigin-RevId: 155493797
* Add a custom interface for cache hit processing in actionsGravatar ulfjack2017-05-09
| | | | | | | | | | | | The new interface mirrors ActionExecutionContext, but is restricted to exactly the parts used right now. I did consider using ActionExecutionContext, but it contains some parts that we don't want to make available for cache hits. The end goal is to allow the build event stream access to artifact metadata, in particular for TestResult and TestSummary events, which in turn requires making artifact metadata available when the TestRunnerAction is a cache hit. PiperOrigin-RevId: 155492447
* Rewrite repository lookup to return a failed value rather than throwGravatar ulfjack2017-04-20
| | | | | | | | | | | | | | | | | | | | | | | We need to lookup repositories as part of converting exec paths to artifacts, which in turn is needed for action cache lookups. These lookups should not cause a Skyframe exit, so we must not throw an exception here, unless the error makes it impossible to continue. Instead, we need to leave the decision whether to error out or not to the caller. Note that we may unnecessarily fetch a remote repository in order to do the action cache lookup, even if the action no longer depends on the input file, although this should only be possible for C++ compile actions. It's possible that there's another bug in the C++ compile action key computation that also contributes. This change also makes it so that the post-resolution action cache code ignores any errors wrt. repository lookup rather than throwing. If any of the paths could not be found, then the action cache lookup fails and we re-execute the corresponding action, which is exactly what should happen. Fixes #2759. PiperOrigin-RevId: 153696243
* Move the chmod call from SkyframeActionExecutor to ActionMetadataHandlerGravatar ulfjack2017-04-07
| | | | | | | | | | | | | | | | | | | This relies on the explicit state transition in MetadataHandler to decide whether the chmod call is necessary (we must never call chmod if the action was not executed). This is a prerequisite for #1525. If we want to use ctime for detecting file content changes (which is more reliable than just mtime), then we must call chmod before stat, since chmod affects ctime. Before this change, we were caching the stat in ActionMetadataHandler, but calling chmod after action execution in SkyframeActionExecutor, which is the wrong order of calls. However, we must be able to stat in ActionMetadataHandler for cases where a single action runs multiple Spawns where one spawn's output is a subsequent spawn's input. Remove MetadataHandler.isInjected, which is no longer used anywhere. PiperOrigin-RevId: 152387133
* Explicitly document the state transition on MetadataHandlerGravatar ulfjack2017-04-06
| | | | | | | | | | | | | | | | The ActionMetadataHandler does an explicit state transition on discardOutputMetadata. Before the call, it may be used for action cache checking, and after the call it may be updated with execution results. Several of the methods now throw if they're used incorrectly, so I had to refactor the control flow in ActionExecutionFunction to correctly call discardOutputMetadata on the MetadataHandler in all cases. I discovered a resource leak (of FileOutErr) in IncludeParseFunction while I was at it, so I plugged that as well. One step towards #1525. PiperOrigin-RevId: 152363982
* If --batch, --keep_going, --discard_analysis_cache, and the new ↵Gravatar janakr2017-03-31
| | | | | | | | | | | | | | | --noexperimental_enable_critical_path_profiling flags are all specified, then Bazel will delete Actions from ActionLookupValues as they are executed in order to save memory. Because an already-run action may output an artifact that is only requested later in the build, we need to maintain a way for the artifact to look up the action. But in most cases we don't need to keep the action itself, just its output metadata. Some actions unfortunately are needed post-execution, and so we special-case them. Also includes dependency change with description: Move action out of key. This keeps action references from polluting the graph -- actions are just stored in one SkyValue, instead of being present in SkyKeys. This does mean additional memory used: we have a separate ActionLookupData object per Action executed. That may reach ~24M for million-action builds. PiperOrigin-RevId: 151756383
* Remove only-in-tests null checks in SkyframeActionExecutor.Gravatar Janak Ramakrishnan2017-03-22
| | | | | | | | Still on step -0.5, let's be honest. -- PiperOrigin-RevId: 150783638 MOS_MIGRATED_REVID=150783638
* Rationalize local resource acquisitionGravatar Ulf Adams2017-03-03
| | | | | | | | | | | | | | | | | | | | | | Move all local resource acquisition to where local execution actually happens. Don't attempt to acquire resources per action, but only for individual spawns. This significantly simplifies the code. The downside is that we don't account for action-level work anymore. In general, actions should not perform any process execution themselves, but always delegate such work to a SpawnStrategy implementation. This change makes sure that every Spawn has local resources set in a way that is consistent with the previous state. However, there are two actions - Fileset and FileWrite -, which are not spawns, and so we now don't limit their concurrent execution anymore. For Fileset, all work is done in a custom Fileset-specific thread pool, so this shouldn't be a problem. I'm not sure about FileWriteAction. -- PiperOrigin-RevId: 149012600 MOS_MIGRATED_REVID=149012600
* Clean up the semantics of input discovering actions a bit by making ↵Gravatar Lukacs Berki2017-02-28
| | | | | | | | | | | | updateInputs() and inputsKnown() non-overridable and removing setInputs(). This comes at the cost of adding a flag to every action instance that's not used for non-input-discovering actions, but I think that's a deal. Simpler APIs are good, mmmmkay? Also fixed a few pre-existing issues in TestAction and ObjcCompileAction. -- PiperOrigin-RevId: 148749734 MOS_MIGRATED_REVID=148749734
* Tighten a precondition check during action execution.Gravatar Lukacs Berki2017-02-10
| | | | | | | | Input discovery is handled somewhere else, so the same Action instance should never be put twice into buildActionMap . -- PiperOrigin-RevId: 147132869 MOS_MIGRATED_REVID=147132869
* Support mapping of Paths to URIsGravatar Klaus Aehlig2017-01-18
| | | | | | | | | | | Bazel-created files (like log files of test runs) are internally reported as Paths. However, this is not always the most useful representation of the location of that artifact for a consumer of build events. Therefore, support a mapping of paths to more useful URIs. -- PiperOrigin-RevId: 144843525 MOS_MIGRATED_REVID=144843525
* Log the IOException when an output was not successfully created. It's ↵Gravatar Janak Ramakrishnan2017-01-13
| | | | | | | | possible the exception has information we've been suppressing. -- PiperOrigin-RevId: 144405370 MOS_MIGRATED_REVID=144405370
* Log if dangling symlink is found. This is an odd choice of thing to log, but ↵Gravatar Janak Ramakrishnan2017-01-13
| | | | | | | | shouldn't come up very often, except in the linked bug. -- PiperOrigin-RevId: 144367142 MOS_MIGRATED_REVID=144367142
* Description redacted.Gravatar Googler2016-12-12
| | | | | | -- PiperOrigin-RevId: 141561079 MOS_MIGRATED_REVID=141561079
* Proper action output checks for TreeArtifacts. Instead of crashing Bazel, we ↵Gravatar Rumou Duan2016-10-19
| | | | | | | now handle failed TreeArtifact output checks gracefully. -- MOS_MIGRATED_REVID=136627086
* Track client environment in SkyframeGravatar Klaus Aehlig2016-09-14
| | | | | | | | | | | | | ...to determine which actions have to be recomputed based on changes to the client environment. Note that this change does it the simple way and reconsideres all actions on a changed client environment, while still only reexecuting those, where the part that was inherited from the environment actually did change. -- Change-Id: Ie1116d094642165e5e959447a6fcf49d19b37d6e Reviewed-on: https://bazel-review.googlesource.com/#/c/5431 MOS_MIGRATED_REVID=133010705
* Add client environment to ActionExecutionContextGravatar Klaus Aehlig2016-08-26
| | | | | | | | | | | As the execution of an action now also depends on the client environment, make the latter part of the ActionExecutionContext, so that enough context is provided to actually execute an action. -- Change-Id: Ida7bf407ef0c0375728faba92494bfd47dcbaeb8 Reviewed-on: https://bazel-review.googlesource.com/#/c/5391 MOS_MIGRATED_REVID=131377490
* Action#discoverInputs returns an Iterable<Artifact> instead of aGravatar Cal Peyser2016-08-25
| | | | | | | Collection<Artifact> -- MOS_MIGRATED_REVID=131285541
* Allow Skyframe graph lookups and value retrievals to throw InterruptedException.Gravatar Janak Ramakrishnan2016-08-16
| | | | | | | The only place we now don't handle InterruptedException is in the action graph created after analysis, since I'm not sure that will be around for that much longer. -- MOS_MIGRATED_REVID=130327770
* Use byte[] rather than ByteString for file digests.Gravatar Googler2016-07-20
| | | | | | | | | | | | | | | | | ActionInputFileCache: Change getDigest() to return the underlying byte[16] owned by each FileArtifactValue. Remove throws clause from getInputFromDigest(); this should be an in-memory operation, and no implementation actually throws. PerActionFileCache: Invert mapping from artifact to digest only if needed. Remove interner, as it was used only for the reverse map keys, not the returned values. This should be a significant cpu savings as eagerly constructing the reverse maps was a noticeable hotspot. -- MOS_MIGRATED_REVID=127972359
* Rename FileOutErr.get{Output,Error}File to get{Output,Error}Path, because ↵Gravatar Philipp Wollermann2016-06-29
| | | | | | | they actually return a Path and not a File. -- MOS_MIGRATED_REVID=126102820
* 1. Create the TreeArtifact directory structure before expanding ActionTemplates.Gravatar Rumou Duan2016-06-14
| | | | | | | | 2. In PopulateTreeArtifactAction, create the parent directories for TreeFileArtifacts before executing the spawn. 3. Allow empty tree artifacts in CustomCommandLine and PopulateTreeArtifact. -- MOS_MIGRATED_REVID=124759286
* remove duplicate commentGravatar Benjamin Peterson2016-04-26
| | | | | | | -- Change-Id: I4c7cc2083cf0f886875b662440ed406e52dffae5 Reviewed-on: https://bazel-review.googlesource.com/#/c/3491/ MOS_MIGRATED_REVID=120804156
* Split ActionMetadata into ActionAnalysisMetadata and ActionExecutionMetadata.Gravatar Rumou Duan2016-04-26
| | | | | | | Except in action execution logic (ActionExecutionFunction, SkyframeActionExecutor, etc.), switch Action interface references to either ActionAnalysisMetadata if possible or ActionExecutionMetadata. -- MOS_MIGRATED_REVID=120723431
* Extract common logic for detecting action and artifact prefix conflicts.Gravatar Rumou Duan2016-04-19
| | | | | -- MOS_MIGRATED_REVID=120145833
* Introduce TreeFileArtifact, which represents files under TreeArtifacts.Gravatar Rumou Duan2016-04-14
| | | | | | | Remove ArtifactFile, which is rendered obsolete by TreeFileArtifact. -- MOS_MIGRATED_REVID=119789154
* Ensure that errors from a mismatch between include scanning and local ↵Gravatar Janak Ramakrishnan2016-03-09
| | | | | | | execution are reported. Previously, we would not report the error unless the build was already failing and this error ended up being the root cause of the first failed top-level target. -- MOS_MIGRATED_REVID=116767915
* Little refactorings on resource management stuff:Gravatar Philipp Wollermann2016-02-28
| | | | | | | | | | - Remove unneeded abstract override AbstractAction#getResourceConsumption() - Remove comments justifying returning ResourceSet.ZERO from getResourceConsumption() - instead document it once on the base method. - Replace ResourceSet.create(0, 0, 0) with ResourceSet.ZERO. - Make sure no one ever returns null from getResourceConsumption() and then simplify the code in SkyframeActionExecutor to not check for it. -- MOS_MIGRATED_REVID=115739250
* Rollback of commit a0eefb52f529b73c6cb92f0a762853646ea2eae6.Gravatar Googler2016-02-11
| | | | | | | | | | | | | *** Reason for rollback *** Rolling forward with the restored logic to avoid stat calls on injected Metadata. *** Original change description *** Automated [] rollback of commit df03e10f6552566982399b8779fe7bc7a17d75dc. -- MOS_MIGRATED_REVID=114447944
* Rollback of commit df03e10f6552566982399b8779fe7bc7a17d75dc.Gravatar Michajlo Matijkiw2016-02-11
| | | | | -- MOS_MIGRATED_REVID=114329043
* Make unpredictable action inputs and outputs available to Actions.Gravatar Michael Thvedt2016-02-10
| | | | | -- MOS_MIGRATED_REVID=114249806
* Make The Build Faster: Use AutoCloseable when dealing with resources.Gravatar Philipp Wollermann2016-02-10
| | | | | -- MOS_MIGRATED_REVID=114204404
* Support for handling TreeArtifact metadata and returning TreeArtifacts from ↵Gravatar Michael Thvedt2016-02-09
| | | | | | | ArtifactFunction. -- MOS_MIGRATED_REVID=114174899
* Rename MiddlemanExpander to ArtifactExpander, and refactor it to yield ↵Gravatar Michael Thvedt2016-02-09
| | | | | | | ArtifactFiles. -- MOS_MIGRATED_REVID=114166208
* Refactor action output dumping to go through a single method.Gravatar Ulf Adams2016-01-15
| | | | | | | | | | | | | This is in preparation for moving the output dumping to the Reporter for two reasons: - we're currently relying on there being a Reporter which locks on itself; i.e., this code is tightly coupled with the Reporter code - the current Blaze output is confusing; we first print an info event, then the actual output, and then an error event - putting the Reporter in control allows us to fix that -- MOS_MIGRATED_REVID=112240684
* Use Bazel Preconditions variant which avoids varargs array creationGravatar Mark Schaller2015-12-10
| | | | | | | Reduces garbage. -- MOS_MIGRATED_REVID=109914243
* Track the entire OutputService instead of just the BatchStatter.Gravatar Eric Fellheimer2015-11-16
| | | | | -- MOS_MIGRATED_REVID=107800790
* Optimize how we mark files read-only upon action completion.Gravatar Eric Fellheimer2015-09-30
| | | | | -- MOS_MIGRATED_REVID=104219083
* Set the SkyframeActionExecutor.reporter in prepareForExecution.Gravatar Ulf Adams2015-09-30
| | | | | | | Remove a lot of now-unused Reporter parameters. -- MOS_MIGRATED_REVID=104100061
* Rationalize copyright headersGravatar Damien Martin-Guillerez2015-09-25
| | | | | | | | | | | The headers were modified with `find . -type f -exec 'sed' '-Ei' 's|Copyright 201([45]) Google|Copyright 201\1 The Bazel Authors|' '{}' ';'` And manual edit for not Google owned copyright. Because of the nature of ijar, I did not modified the header of file owned by Alan Donovan. The list of authors were extracted from the git log. It is missing older Google contributors that can be added on-demand. -- MOS_MIGRATED_REVID=103938715
* Move Label from the lib.syntax to the lib.cmdline package so that:Gravatar Lukacs Berki2015-09-21
| | | | | | | | | | - Label parsing can be simplified - lib.syntax is only contains the code for Skylark and is reasonably independent from the problem domain of building things This change is mostly only changes to imports declarations. The rest is reversing the dependency between :cmdline and :syntax and moving a tiny amount of code between Printer and FilesetEntry and the addition of SkylarkPrintableValue that I couldn't be bothered to separate out into its own change. -- MOS_MIGRATED_REVID=103527877
* Renaming ExecutorShutdownUtil to ExecutorUtil, since it doesn't just do ↵Gravatar Eric Fellheimer2015-06-19
| | | | | | | shutdown operations. -- MOS_MIGRATED_REVID=96351438
* Add some utilities and relax some visibility restrictions to make ↵Gravatar Janak Ramakrishnan2015-06-19
| | | | | | | alternative include scanning implementations possible. -- MOS_MIGRATED_REVID=96337469
* Handle exceptions encountered resolving packages during the execution phaseGravatar Michajlo Matijkiw2015-06-10
| | | | | | | | | | | | | | Currently we may do lookups of not-already-cached packages during the execution phase for actions that discover inputs. Exceptions encountered during this would go unhandled and result in a crash. Here we introduce PackageRootResolutionException which wraps these exceptions and triggers an ActionExecutionException which is cleanly handled in the exec phase. As part of this change SkyframeActionExecutor#getArtifactRoots(...) will fail properly on errors getting package roots. -- MOS_MIGRATED_REVID=95578891
* Add runfiles to metadata caches and make them isFile awareGravatar Michajlo Matijkiw2015-05-26
| | | | | -- MOS_MIGRATED_REVID=94318260
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-04 00:01:42 +0000