aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/skyframe/ActionExecutionFunction.java
Commit message (Collapse)AuthorAge
* Make SkyKey an interface, and start the migration of not creating SkyKey ↵Gravatar janakr2017-05-04
| | | | | | wrapper objects: for OwnedArtifacts, which are the most numerous during builds, and for Labels for TransitiveTraversalValues, which are the most numerous during queries. PiperOrigin-RevId: 154989520
* Rewrite repository lookup to return a failed value rather than throwGravatar ulfjack2017-04-20
| | | | | | | | | | | | | | | | | | | | | | | We need to lookup repositories as part of converting exec paths to artifacts, which in turn is needed for action cache lookups. These lookups should not cause a Skyframe exit, so we must not throw an exception here, unless the error makes it impossible to continue. Instead, we need to leave the decision whether to error out or not to the caller. Note that we may unnecessarily fetch a remote repository in order to do the action cache lookup, even if the action no longer depends on the input file, although this should only be possible for C++ compile actions. It's possible that there's another bug in the C++ compile action key computation that also contributes. This change also makes it so that the post-resolution action cache code ignores any errors wrt. repository lookup rather than throwing. If any of the paths could not be found, then the action cache lookup fails and we re-execute the corresponding action, which is exactly what should happen. Fixes #2759. PiperOrigin-RevId: 153696243
* Explicitly document the state transition on MetadataHandlerGravatar ulfjack2017-04-06
| | | | | | | | | | | | | | | | The ActionMetadataHandler does an explicit state transition on discardOutputMetadata. Before the call, it may be used for action cache checking, and after the call it may be updated with execution results. Several of the methods now throw if they're used incorrectly, so I had to refactor the control flow in ActionExecutionFunction to correctly call discardOutputMetadata on the MetadataHandler in all cases. I discovered a resource leak (of FileOutErr) in IncludeParseFunction while I was at it, so I plugged that as well. One step towards #1525. PiperOrigin-RevId: 152363982
* If --batch, --keep_going, --discard_analysis_cache, and the new ↵Gravatar janakr2017-03-31
| | | | | | | | | | | | | | | --noexperimental_enable_critical_path_profiling flags are all specified, then Bazel will delete Actions from ActionLookupValues as they are executed in order to save memory. Because an already-run action may output an artifact that is only requested later in the build, we need to maintain a way for the artifact to look up the action. But in most cases we don't need to keep the action itself, just its output metadata. Some actions unfortunately are needed post-execution, and so we special-case them. Also includes dependency change with description: Move action out of key. This keeps action references from polluting the graph -- actions are just stored in one SkyValue, instead of being present in SkyKeys. This does mean additional memory used: we have a separate ActionLookupData object per Action executed. That may reach ~24M for million-action builds. PiperOrigin-RevId: 151756383
* Clean up the semantics of input discovering actions a bit by making ↵Gravatar Lukacs Berki2017-02-28
| | | | | | | | | | | | updateInputs() and inputsKnown() non-overridable and removing setInputs(). This comes at the cost of adding a flag to every action instance that's not used for non-input-discovering actions, but I think that's a deal. Simpler APIs are good, mmmmkay? Also fixed a few pre-existing issues in TestAction and ObjcCompileAction. -- PiperOrigin-RevId: 148749734 MOS_MIGRATED_REVID=148749734
* Fix Cpp action cachingGravatar Ulf Adams2017-02-24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This combines both previous changes and extends them to work both with and without kchodorow@'s rollout of the exec root rearrangement. Unfortunately, each of these changes individually breaks something somewhere, so they must all go into a single commit. Change 1: CppCompileAction must return false from inputsKnown for .d pruning This is necessary (but not sufficient) for the action cache to work correctly. Consider the following sequence of events: 1. action is executed 2. .d pruning is performed 3. action cache writes entry with post-pruning inputs list 4. action gets regenerated (e.g., due to server restart) 5. the action cache calls inputsKnown(), which returns true 6. action cache checks entry from step 3 against pre-pruning inputs list, which results in a cache miss The action cache needs to know that the current list is not the final list, so inputsKnown() in step 5 must return false if .d pruning is enabled. Change 2: Fix artifact root discovery for external artifacts The SkyframeExecutor was assuming that all exec paths were coming from the main repository. Instead, rely on external exec paths to start with "../". Additional change 3: In addition, update the PackageRootResolverWithEnvironment and the HeaderDiscovery to use the single unified repository name guessing implementation. Previously, the PackageRootResolverWithEnvironment was poisoning the source artifact cache, which then caused subsequent lookups to return a bad artifact. Add a precondition to double-check that artifacts looked up by exec path have the correct source root. For compatibility with kchodorow@'s upcoming exec root refactor, if the exec path starts with "external", then assume it's coming from an external repository. This must be removed when that change is successfully rolled out, or it will break if anyone creates a package called 'external'. Additional change 4: On top of all of that, PackageRootResolverWithEnvironment and SkyframeExecutor must use the same source root computation as the Package class itself. I extracted the corresponding code to Root, and added comments both there and in Package to clearly indicate that these methods have to always be modified in sync. Fixes #2490. -- PiperOrigin-RevId: 148439309 MOS_MIGRATED_REVID=148439309
* Adds abstractions for include processing to CppSemanticsGravatar Googler2017-02-07
| | | | | | -- PiperOrigin-RevId: 146694092 MOS_MIGRATED_REVID=146694092
* Improve the error message added in commit ↵Gravatar Lukacs Berki2017-02-02
| | | | | | | | 41ab6db797bac30c8e39fc705254fef329b6fdbc . -- PiperOrigin-RevId: 146341606 MOS_MIGRATED_REVID=146341606
* Simplify action input discovery by removing ↵Gravatar Lukacs Berki2017-01-31
| | | | | | | | | | Action#getInputsWhenSkippingInputDiscovery(). This has the side effect of always allowing new inputs to be discovered when Action#discoversInputs() return true, but since those actions are very few and have mostly to do with C++, I think it's an acceptable tradeoff for th ecode being that much simpler. -- PiperOrigin-RevId: 146098150 MOS_MIGRATED_REVID=146098150
* Make "profile --html --html_details" work again.Gravatar Lukacs Berki2017-01-03
| | | | | | | | The start time of critical path component wasn't recorded correctly since #getStartWallTimeMillis() didn't work right. However, since profiler tasks store nanotime anyway, it's pointless to try to convert back and forth. -- PiperOrigin-RevId: 143438289 MOS_MIGRATED_REVID=143438289
* Prune modules when building modules themselves to reduce build times shortenGravatar Googler2016-12-02
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | critical paths. When the inputs of a module M are reduced to a set S, then that same set S also needs to be supplied to compile something that uses M. To do this, input discovery is divided into two stages. For each CppCompileAction, the first stage discovers the necessary modules (M above). These are then added as inputs to ensure that they are built. The second stage then finds all the modules (S above) that are required to use those and also adds them as inputs. For now, the new behavior is guarded by a new flag --experimental_prune_more_modules. This is currently implemented by reading the .d files of used modules add adding all their module dependencies. There are two noteworthy alternatives: 1. Hack up input discovery to understand modules, e.g. if a modular header is hit, continue scanning from all it's headers. However, this seems very brittle and a lot of additional information would have to be passed to the input discovery. 2. Directly pass the results from input discovery of one CppCompileAction to another one. However, this seems to tightly couple the execution of different CppCompileActions and might lead to a mess of different states, more memory consumption, etc. With the current implementation, there is a bit of runtime overhead of reading the .d files (many times). This could potentially be improved by caching the results. However, even without this caching, the runtime overhead is limited (<10%) for all builds I have tried (I have tried with builds where all the compile results are already in the executor's cache. -- MOS_MIGRATED_REVID=140793217
* Add experimental flag to stop requiring all transitive modules as inputs.Gravatar Googler2016-10-31
| | | | | | | | | Requiring all transitive modules to always be available can lead to long critical paths and even unnecessary compiles in combination with the prune_header_modules feature. -- MOS_MIGRATED_REVID=137696794
* Add a new concept of failure causesGravatar Klaus Aehlig2016-10-25
| | | | | | | | | | | | | | | | | | | | Not all possible reasons for failure are uniquely identified by a label. Therefore, add a new data type describing possible root causes of failures and use it. The new type is added in causes/*.java and coresponds to Haskell's one-line definition data Cause = LabelCause Label | ActionCause Path Label deriving Show With future clean up of other failure causes inadequately described by a label, we expect that type to be extended by new constructors (i.e., new classes implementing Cause). -- Change-Id: I6fec74c78cec6abb9c10e32743b05a792888fead Reviewed-on: https://bazel-review.googlesource.com/#/c/6617 MOS_MIGRATED_REVID=137156390
* Add type arguments for JDK7 build.Gravatar Janak Ramakrishnan2016-10-19
| | | | | -- MOS_MIGRATED_REVID=136575806
* Per action file caching for input TreeArtifact.Gravatar Rumou Duan2016-10-19
| | | | | -- MOS_MIGRATED_REVID=136475556
* Make environment dependency of actions factor through individual valuesGravatar Klaus Aehlig2016-09-15
| | | | | | | | | | | | | | With actions depending on the (white-listed part) of the environment as a whole, even though they are only re-executed if the used parts of the environment change, each action has to be reconsidered on any change of the environment. For large dependency graphs, this can be a considerable amount of effort; therefore add intermediate values for the individual variables and make actions only depend on those actually used. -- Change-Id: I283d289da3e0782dc4f9ac084a41425166cfede0 Reviewed-on: https://bazel-review.googlesource.com/#/c/5494 MOS_MIGRATED_REVID=133255911
* Track client environment in SkyframeGravatar Klaus Aehlig2016-09-14
| | | | | | | | | | | | | ...to determine which actions have to be recomputed based on changes to the client environment. Note that this change does it the simple way and reconsideres all actions on a changed client environment, while still only reexecuting those, where the part that was inherited from the environment actually did change. -- Change-Id: Ie1116d094642165e5e959447a6fcf49d19b37d6e Reviewed-on: https://bazel-review.googlesource.com/#/c/5431 MOS_MIGRATED_REVID=133010705
* Add an "is main repo" field to RootsGravatar Kristina Chodorow2016-09-09
| | | | | | | This is part of prepping for #1262. -- MOS_MIGRATED_REVID=132553178
* Refactor ActionExecutionFunction to avoid unnecessary unwrapping of SkyKeys ↵Gravatar Janak Ramakrishnan2016-08-22
| | | | | | | to Artifacts. Also avoid overhead of creating a set when we only need an iterable, and avoid an unnecessary iteration over the iterable as a contains check. In future, we may not want to pay the cost of constructing a true map in SkyFunction.Environment implementations. Part of this is a preliminary refactor to allow that. -- MOS_MIGRATED_REVID=130765995
* Allow Skyframe graph lookups and value retrievals to throw InterruptedException.Gravatar Janak Ramakrishnan2016-08-16
| | | | | | | The only place we now don't handle InterruptedException is in the action graph created after analysis, since I'm not sure that will be around for that much longer. -- MOS_MIGRATED_REVID=130327770
* Refactor FileArtifactValue and ArtifactValue now that presence of mtime and ↵Gravatar Janak Ramakrishnan2016-08-01
| | | | | | | digest are mutually exclusive. -- MOS_MIGRATED_REVID=128843642
* Avoid intermediate Map/Collections in ActionExecutionFunctionGravatar Michajlo Matijkiw2016-07-29
| | | | | -- MOS_MIGRATED_REVID=128731657
* Avoid copying map when adding discovered inputs to action metadata.Gravatar Janak Ramakrishnan2016-07-28
| | | | | -- MOS_MIGRATED_REVID=128641967
* Replace the occurrences of Constants.PRODUCT_NAME for a call toGravatar Luis Fernando Pino Duque2016-05-23
| | | | | | | | | BlazeRuntime#getProductName() or a reference to TestConstants.PRODUCT_NAME for tests. This CL prepares the codebase in order to delete the constant. -- MOS_MIGRATED_REVID=122993568
* Split ActionMetadata into ActionAnalysisMetadata and ActionExecutionMetadata.Gravatar Rumou Duan2016-04-26
| | | | | | | Except in action execution logic (ActionExecutionFunction, SkyframeActionExecutor, etc.), switch Action interface references to either ActionAnalysisMetadata if possible or ActionExecutionMetadata. -- MOS_MIGRATED_REVID=120723431
* Fix JDK 7 build of BazelGravatar Damien Martin-Guillerez2016-04-14
| | | | | -- MOS_MIGRATED_REVID=119832385
* Introduce TreeFileArtifact, which represents files under TreeArtifacts.Gravatar Rumou Duan2016-04-14
| | | | | | | Remove ArtifactFile, which is rendered obsolete by TreeFileArtifact. -- MOS_MIGRATED_REVID=119789154
* Move TimestampGranularityMonitor to CommandEnvironment.Gravatar Ulf Adams2016-03-23
| | | | | | | | This is one of the last pieces of state in BlazeRuntime that isn't safe to share across command invocations. -- MOS_MIGRATED_REVID=117910631
* Allow PackageRootResolver to take in directories for findPackageRoots().Gravatar Alex Humesky2016-03-18
| | | | | -- MOS_MIGRATED_REVID=117512506
* Don't keep packages in the default repository around after loading.Gravatar Brian Silverman2016-03-17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, this would get thrown when referring to the same package from both the main and default repositories: java.lang.IllegalArgumentException: Multiple entries with same key: tools/cpp=/home/brian/971-Robot-Code and tools/cpp=/home/brian/971-Robot-Code at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:136) at com.google.common.collect.RegularImmutableMap.checkNoConflictInKeyBucket(RegularImmutableMap.java:98) at com.google.common.collect.RegularImmutableMap.fromEntryArray(RegularImmutableMap.java:84) at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:295) at com.google.devtools.build.lib.buildtool.BuildTool.transformPackageRoots(BuildTool.java:301) at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:209) at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:334) at com.google.devtools.build.lib.runtime.commands.TestCommand.doTest(TestCommand.java:119) at com.google.devtools.build.lib.runtime.commands.TestCommand.exec(TestCommand.java:104) at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:371) at com.google.devtools.build.lib.runtime.BlazeRuntime$3.exec(BlazeRuntime.java:1016) at com.google.devtools.build.lib.server.RPCService.executeRequest(RPCService.java:65) at com.google.devtools.build.lib.server.RPCServer.executeRequest(RPCServer.java:434) at com.google.devtools.build.lib.server.RPCServer.serve(RPCServer.java:229) at com.google.devtools.build.lib.runtime.BlazeRuntime.serverMain(BlazeRuntime.java:975) at com.google.devtools.build.lib.runtime.BlazeRuntime.main(BlazeRuntime.java:772) at com.google.devtools.build.lib.bazel.BazelMain.main(BazelMain.java:55) And this would get thrown for any packages in the main repository loaded from other repositories: java.lang.RuntimeException: Unrecoverable error while evaluating node 'PACKAGE:@//tools/build_rules/go/toolchain' (requested by nodes ) at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:982) at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:499) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalArgumentException: Invalid BUILD file name for package '@//tools/build_rules/go/toolchain': /home/brian/bazel/tools/build_rules/go/toolchain/BUILD at com.google.devtools.build.lib.packages.Package.finishInit(Package.java:299) at com.google.devtools.build.lib.packages.Package$Builder.finishBuild(Package.java:1308) at com.google.devtools.build.lib.skyframe.PackageFunction.compute(PackageFunction.java:501) at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:933) ... 4 more Sponsor's comment: note the abundance of new Label.resolveRepositoryRelative() calls. They are ugly, but it's only making existing ugliness explicit. Yes, we should fix it, especially in the implementation of configurable attributes. Refs #940 -- Change-Id: I8bd7f7b00bec58a7157507595421bc50c81b404c Reviewed-on: https://bazel-review.googlesource.com/#/c/2591 MOS_MIGRATED_REVID=117429733
* Ensure that errors from a mismatch between include scanning and local ↵Gravatar Janak Ramakrishnan2016-03-09
| | | | | | | execution are reported. Previously, we would not report the error unless the build was already failing and this error ended up being the root cause of the first failed top-level target. -- MOS_MIGRATED_REVID=116767915
* Rollback of commit a0eefb52f529b73c6cb92f0a762853646ea2eae6.Gravatar Googler2016-02-11
| | | | | | | | | | | | | *** Reason for rollback *** Rolling forward with the restored logic to avoid stat calls on injected Metadata. *** Original change description *** Automated [] rollback of commit df03e10f6552566982399b8779fe7bc7a17d75dc. -- MOS_MIGRATED_REVID=114447944
* Rollback of commit df03e10f6552566982399b8779fe7bc7a17d75dc.Gravatar Michajlo Matijkiw2016-02-11
| | | | | -- MOS_MIGRATED_REVID=114329043
* Make unpredictable action inputs and outputs available to Actions.Gravatar Michael Thvedt2016-02-10
| | | | | -- MOS_MIGRATED_REVID=114249806
* Support for handling TreeArtifact metadata and returning TreeArtifacts from ↵Gravatar Michael Thvedt2016-02-09
| | | | | | | ArtifactFunction. -- MOS_MIGRATED_REVID=114174899
* Rename MiddlemanExpander to ArtifactExpander, and refactor it to yield ↵Gravatar Michael Thvedt2016-02-09
| | | | | | | ArtifactFiles. -- MOS_MIGRATED_REVID=114166208
* Allow an exit code to be specified in the ActionExecutionException when an ↵Gravatar Googler2016-01-25
| | | | | | | | | action fails, and propagate the exit code to BuildFailedException. And update all call sites that copy or wrap ActionExecutionExceptions. -- MOS_MIGRATED_REVID=112811857
* Use Bazel Preconditions variant which avoids varargs array creationGravatar Mark Schaller2015-12-10
| | | | | | | Reduces garbage. -- MOS_MIGRATED_REVID=109914243
* Fail build gracefully if an action discovers unexpected inputs.Gravatar Janak Ramakrishnan2015-12-08
| | | | | | | | | | | | | | | | | | | Blaze discovers inputs for some actions when running with some strategies. Those actions should not discover additional inputs when they run, regardless of the strategy they end up using. There are now no known legitimate cases of such additional input discovery, so we should reinstate this check and find new ones :) We also change the failure mode to be a normal error rather than a crash. This error does indicate a tooling issue, and a small chance of incorrect builds, but it doesn't create such an inconsistent state that a crash is warranted. -- Change-Id: I5d498d2fc1c5e23bfb5d77971f866c2027cbf03a Reviewed-on: https://bazel-review.googlesource.com/#/c/2500/3 MOS_MIGRATED_REVID=109703508
* When getting package roots for exec paths, don't start with the file, which ↵Gravatar Janak Ramakrishnan2015-11-13
| | | | | | | is guaranteed not to be a package. Instead, start with its parent. This will be faster and take less memory. -- MOS_MIGRATED_REVID=107725767
* Add debugging info in case of unexpected crash.Gravatar Janak Ramakrishnan2015-10-06
| | | | | -- MOS_MIGRATED_REVID=104658639
* Rationalize copyright headersGravatar Damien Martin-Guillerez2015-09-25
| | | | | | | | | | | The headers were modified with `find . -type f -exec 'sed' '-Ei' 's|Copyright 201([45]) Google|Copyright 201\1 The Bazel Authors|' '{}' ';'` And manual edit for not Google owned copyright. Because of the nature of ijar, I did not modified the header of file owned by Alan Donovan. The list of authors were extracted from the git log. It is missing older Google contributors that can be added on-demand. -- MOS_MIGRATED_REVID=103938715
* Move Label from the lib.syntax to the lib.cmdline package so that:Gravatar Lukacs Berki2015-09-21
| | | | | | | | | | - Label parsing can be simplified - lib.syntax is only contains the code for Skylark and is reasonably independent from the problem domain of building things This change is mostly only changes to imports declarations. The rest is reversing the dependency between :cmdline and :syntax and moving a tiny amount of code between Printer and FilesetEntry and the addition of SkylarkPrintableValue that I couldn't be bothered to separate out into its own change. -- MOS_MIGRATED_REVID=103527877
* Make exception in case a test tries to remote-log more informative.Gravatar Janak Ramakrishnan2015-09-17
| | | | | -- MOS_MIGRATED_REVID=103290841
* Move PackageIdentifier to cmdlineGravatar Kristina Chodorow2015-08-31
| | | | | | | | This is necessary to have TargetResolver depend on it without making it depend on the packages target. First step of #389. -- MOS_MIGRATED_REVID=101790345
* Actions can now request Skyframe dependencies and bypass action cache checking.Gravatar Laszlo Csomor2015-07-10
| | | | | | | | | | By implementing the SkyframeAwareAction interface an Action gains the power to request Skyframe dependencies after its declared inputs have been built and discovered but before execution. Such actions bypass the action cache check and rely on Skyframe's dependency tracking to decide when to be reexecuted. -- MOS_MIGRATED_REVID=97947856
* Relax invariant that an action's inputs discovered during execution must be ↵Gravatar Janak Ramakrishnan2015-07-01
| | | | | | | | | included in the action's inputs as found during the input discovery phase. We still require that no new metadata be discovered -- in other words, the "new" inputs are likely just symlinks to old inputs, with different nominal paths. -- MOS_MIGRATED_REVID=97257026
* Move dependency declaration out of finally block.Gravatar Janak Ramakrishnan2015-07-01
| | | | | | | This means that we will not declare additional dependencies if the action fails to execute, but that's correct -- if the action fails to execute, its inputs aren't updated, so there's nothing we'd discover. -- MOS_MIGRATED_REVID=97255120
* Add some utilities and relax some visibility restrictions to make ↵Gravatar Janak Ramakrishnan2015-06-19
| | | | | | | alternative include scanning implementations possible. -- MOS_MIGRATED_REVID=96337469
* Handle exceptions encountered resolving packages during the execution phaseGravatar Michajlo Matijkiw2015-06-10
| | | | | | | | | | | | | | Currently we may do lookups of not-already-cached packages during the execution phase for actions that discover inputs. Exceptions encountered during this would go unhandled and result in a crash. Here we introduce PackageRootResolutionException which wraps these exceptions and triggers an ActionExecutionException which is cleanly handled in the exec phase. As part of this change SkyframeActionExecutor#getArtifactRoots(...) will fail properly on errors getting package roots. -- MOS_MIGRATED_REVID=95578891
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-07 07:28:19 +0000