| Commit message (Collapse) | Author | Age |
|
|
|
|
|
|
|
| |
doesn't save memory in the 32-bit case, but makes it easier for people to see how many SkyKeys we have.
There's some unnecessary interning in tests, but it was easier to copypasta and doesn't harm anything, I think.
PiperOrigin-RevId: 187694309
|
|
|
|
| |
PiperOrigin-RevId: 185694489
|
|
|
|
|
|
| |
Post to the Skyframe Environment instead.
PiperOrigin-RevId: 182096559
|
|
|
|
|
|
| |
This is slightly more descriptive, and we will potentially want to use the name Root for a broader concept shared between ArtifactRoot and RootedPath.
PiperOrigin-RevId: 182082367
|
|
|
|
|
|
|
|
| |
Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own.
This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually.
PiperOrigin-RevId: 175033526
|
|
|
|
|
|
|
|
| |
This is both probably what the user most cares about, and also restores a Skyframe invariant, that as a catastrophic exception bubbles up, it stays catastrophic.
There is actually no hard Skyframe requirement for that, so we could consider relaxing it, but I'm willing to keep it for now.
PiperOrigin-RevId: 174759976
|
|
|
|
|
|
| |
cache if we're doing input discovery: input discovery may find new artifacts we don't yet know about. Similarly, in SingleBuildFileCache, use the Artifact's path if it's available, rather than the poor man's route of the execRoot.
PiperOrigin-RevId: 171635892
|
|
|
|
| |
PiperOrigin-RevId: 170503143
|
|
|
|
|
|
|
|
|
| |
Split collect, concurrent, vfs, windows into package-level BUILD files.
Move clock classes out of "util", into their own Java package.
Move CompactHashSet into its own Java package to break a dependency cycle.
Give nestedset and inmemoryfs their own package-level BUILD files.
PiperOrigin-RevId: 167702127
|
|
|
|
|
|
| |
wrapper objects: for OwnedArtifacts, which are the most numerous during builds, and for Labels for TransitiveTraversalValues, which are the most numerous during queries.
PiperOrigin-RevId: 154989520
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We need to lookup repositories as part of converting exec paths to artifacts,
which in turn is needed for action cache lookups. These lookups should not
cause a Skyframe exit, so we must not throw an exception here, unless the
error makes it impossible to continue. Instead, we need to leave the decision
whether to error out or not to the caller.
Note that we may unnecessarily fetch a remote repository in order to do the
action cache lookup, even if the action no longer depends on the input file,
although this should only be possible for C++ compile actions. It's possible
that there's another bug in the C++ compile action key computation that also
contributes.
This change also makes it so that the post-resolution action cache code
ignores any errors wrt. repository lookup rather than throwing. If any of the
paths could not be found, then the action cache lookup fails and we re-execute
the corresponding action, which is exactly what should happen.
Fixes #2759.
PiperOrigin-RevId: 153696243
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The ActionMetadataHandler does an explicit state transition on
discardOutputMetadata. Before the call, it may be used for action cache
checking, and after the call it may be updated with execution results.
Several of the methods now throw if they're used incorrectly, so I had to
refactor the control flow in ActionExecutionFunction to correctly call
discardOutputMetadata on the MetadataHandler in all cases. I discovered a
resource leak (of FileOutErr) in IncludeParseFunction while I was at it, so
I plugged that as well.
One step towards #1525.
PiperOrigin-RevId: 152363982
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
--noexperimental_enable_critical_path_profiling flags are all specified, then Bazel will delete Actions from ActionLookupValues as they are executed in order to save memory.
Because an already-run action may output an artifact that is only requested later in the build, we need to maintain a way for the artifact to look up the action. But in most cases we don't need to keep the action itself, just its output metadata.
Some actions unfortunately are needed post-execution, and so we special-case them.
Also includes dependency change with description:
Move action out of key. This keeps action references from polluting the graph -- actions are just stored in one SkyValue, instead of being present in SkyKeys.
This does mean additional memory used: we have a separate ActionLookupData object per Action executed. That may reach ~24M for million-action builds.
PiperOrigin-RevId: 151756383
|
|
|
|
|
|
|
|
|
|
|
|
| |
updateInputs() and inputsKnown() non-overridable and removing setInputs().
This comes at the cost of adding a flag to every action instance that's not used for non-input-discovering actions, but I think that's a deal. Simpler APIs are good, mmmmkay?
Also fixed a few pre-existing issues in TestAction and ObjcCompileAction.
--
PiperOrigin-RevId: 148749734
MOS_MIGRATED_REVID=148749734
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This combines both previous changes and extends them to work both with and
without kchodorow@'s rollout of the exec root rearrangement. Unfortunately,
each of these changes individually breaks something somewhere, so they must
all go into a single commit.
Change 1:
CppCompileAction must return false from inputsKnown for .d pruning
This is necessary (but not sufficient) for the action cache to work
correctly. Consider the following sequence of events:
1. action is executed
2. .d pruning is performed
3. action cache writes entry with post-pruning inputs list
4. action gets regenerated (e.g., due to server restart)
5. the action cache calls inputsKnown(), which returns true
6. action cache checks entry from step 3 against pre-pruning inputs list,
which results in a cache miss
The action cache needs to know that the current list is not the final list,
so inputsKnown() in step 5 must return false if .d pruning is enabled.
Change 2:
Fix artifact root discovery for external artifacts
The SkyframeExecutor was assuming that all exec paths were coming from the
main repository. Instead, rely on external exec paths to start with "../".
Additional change 3:
In addition, update the PackageRootResolverWithEnvironment and the
HeaderDiscovery to use the single unified repository name guessing
implementation. Previously, the PackageRootResolverWithEnvironment was
poisoning the source artifact cache, which then caused subsequent lookups
to return a bad artifact.
Add a precondition to double-check that artifacts looked up by exec path
have the correct source root.
For compatibility with kchodorow@'s upcoming exec root refactor, if the exec
path starts with "external", then assume it's coming from an external
repository. This must be removed when that change is successfully rolled out,
or it will break if anyone creates a package called 'external'.
Additional change 4:
On top of all of that, PackageRootResolverWithEnvironment and SkyframeExecutor
must use the same source root computation as the Package class itself. I
extracted the corresponding code to Root, and added comments both there and
in Package to clearly indicate that these methods have to always be modified
in sync.
Fixes #2490.
--
PiperOrigin-RevId: 148439309
MOS_MIGRATED_REVID=148439309
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 146694092
MOS_MIGRATED_REVID=146694092
|
|
|
|
|
|
|
|
| |
41ab6db797bac30c8e39fc705254fef329b6fdbc .
--
PiperOrigin-RevId: 146341606
MOS_MIGRATED_REVID=146341606
|
|
|
|
|
|
|
|
|
|
| |
Action#getInputsWhenSkippingInputDiscovery().
This has the side effect of always allowing new inputs to be discovered when Action#discoversInputs() return true, but since those actions are very few and have mostly to do with C++, I think it's an acceptable tradeoff for th ecode being that much simpler.
--
PiperOrigin-RevId: 146098150
MOS_MIGRATED_REVID=146098150
|
|
|
|
|
|
|
|
| |
The start time of critical path component wasn't recorded correctly since #getStartWallTimeMillis() didn't work right. However, since profiler tasks store nanotime anyway, it's pointless to try to convert back and forth.
--
PiperOrigin-RevId: 143438289
MOS_MIGRATED_REVID=143438289
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
critical paths.
When the inputs of a module M are reduced to a set S, then that same set S also
needs to be supplied to compile something that uses M. To do this, input
discovery is divided into two stages. For each CppCompileAction, the first
stage discovers the necessary modules (M above). These are then added as inputs
to ensure that they are built. The second stage then finds all the modules (S
above) that are required to use those and also adds them as inputs.
For now, the new behavior is guarded by a new flag
--experimental_prune_more_modules.
This is currently implemented by reading the .d files of used modules add
adding all their module dependencies. There are two noteworthy alternatives:
1. Hack up input discovery to understand modules, e.g. if a modular header is
hit, continue scanning from all it's headers. However, this seems very
brittle and a lot of additional information would have to be passed to the
input discovery.
2. Directly pass the results from input discovery of one CppCompileAction to
another one. However, this seems to tightly couple the execution of
different CppCompileActions and might lead to a mess of different states,
more memory consumption, etc.
With the current implementation, there is a bit of runtime overhead of
reading the .d files (many times). This could potentially be improved by
caching the results. However, even without this caching, the runtime overhead
is limited (<10%) for all builds I have tried (I have tried with builds where
all the compile results are already in the executor's cache.
--
MOS_MIGRATED_REVID=140793217
|
|
|
|
|
|
|
|
|
| |
Requiring all transitive modules to always be available can lead to long
critical paths and even unnecessary compiles in combination with the
prune_header_modules feature.
--
MOS_MIGRATED_REVID=137696794
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Not all possible reasons for failure are uniquely identified
by a label. Therefore, add a new data type describing possible
root causes of failures and use it.
The new type is added in causes/*.java and coresponds to Haskell's
one-line definition
data Cause = LabelCause Label | ActionCause Path Label deriving Show
With future clean up of other failure causes inadequately described
by a label, we expect that type to be extended by new constructors
(i.e., new classes implementing Cause).
--
Change-Id: I6fec74c78cec6abb9c10e32743b05a792888fead
Reviewed-on: https://bazel-review.googlesource.com/#/c/6617
MOS_MIGRATED_REVID=137156390
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=136575806
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=136475556
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
With actions depending on the (white-listed part) of the environment
as a whole, even though they are only re-executed if the used parts of
the environment change, each action has to be reconsidered on any change
of the environment. For large dependency graphs, this can be a considerable
amount of effort; therefore add intermediate values for the individual
variables and make actions only depend on those actually used.
--
Change-Id: I283d289da3e0782dc4f9ac084a41425166cfede0
Reviewed-on: https://bazel-review.googlesource.com/#/c/5494
MOS_MIGRATED_REVID=133255911
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
...to determine which actions have to be recomputed based on changes
to the client environment. Note that this change does it the simple way
and reconsideres all actions on a changed client environment, while still
only reexecuting those, where the part that was inherited from the environment
actually did change.
--
Change-Id: Ie1116d094642165e5e959447a6fcf49d19b37d6e
Reviewed-on: https://bazel-review.googlesource.com/#/c/5431
MOS_MIGRATED_REVID=133010705
|
|
|
|
|
|
|
| |
This is part of prepping for #1262.
--
MOS_MIGRATED_REVID=132553178
|
|
|
|
|
|
|
| |
to Artifacts. Also avoid overhead of creating a set when we only need an iterable, and avoid an unnecessary iteration over the iterable as a contains check. In future, we may not want to pay the cost of constructing a true map in SkyFunction.Environment implementations. Part of this is a preliminary refactor to allow that.
--
MOS_MIGRATED_REVID=130765995
|
|
|
|
|
|
|
| |
The only place we now don't handle InterruptedException is in the action graph created after analysis, since I'm not sure that will be around for that much longer.
--
MOS_MIGRATED_REVID=130327770
|
|
|
|
|
|
|
| |
digest are mutually exclusive.
--
MOS_MIGRATED_REVID=128843642
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=128731657
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=128641967
|
|
|
|
|
|
|
|
|
| |
BlazeRuntime#getProductName() or a reference to TestConstants.PRODUCT_NAME for tests.
This CL prepares the codebase in order to delete the constant.
--
MOS_MIGRATED_REVID=122993568
|
|
|
|
|
|
|
| |
Except in action execution logic (ActionExecutionFunction, SkyframeActionExecutor, etc.), switch Action interface references to either ActionAnalysisMetadata if possible or ActionExecutionMetadata.
--
MOS_MIGRATED_REVID=120723431
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=119832385
|
|
|
|
|
|
|
| |
Remove ArtifactFile, which is rendered obsolete by TreeFileArtifact.
--
MOS_MIGRATED_REVID=119789154
|
|
|
|
|
|
|
|
| |
This is one of the last pieces of state in BlazeRuntime that isn't safe to
share across command invocations.
--
MOS_MIGRATED_REVID=117910631
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=117512506
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously, this would get thrown when referring to the same package
from both the main and default repositories:
java.lang.IllegalArgumentException: Multiple entries with same key: tools/cpp=/home/brian/971-Robot-Code and tools/cpp=/home/brian/971-Robot-Code
at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:136)
at com.google.common.collect.RegularImmutableMap.checkNoConflictInKeyBucket(RegularImmutableMap.java:98)
at com.google.common.collect.RegularImmutableMap.fromEntryArray(RegularImmutableMap.java:84)
at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:295)
at com.google.devtools.build.lib.buildtool.BuildTool.transformPackageRoots(BuildTool.java:301)
at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:209)
at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:334)
at com.google.devtools.build.lib.runtime.commands.TestCommand.doTest(TestCommand.java:119)
at com.google.devtools.build.lib.runtime.commands.TestCommand.exec(TestCommand.java:104)
at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:371)
at com.google.devtools.build.lib.runtime.BlazeRuntime$3.exec(BlazeRuntime.java:1016)
at com.google.devtools.build.lib.server.RPCService.executeRequest(RPCService.java:65)
at com.google.devtools.build.lib.server.RPCServer.executeRequest(RPCServer.java:434)
at com.google.devtools.build.lib.server.RPCServer.serve(RPCServer.java:229)
at com.google.devtools.build.lib.runtime.BlazeRuntime.serverMain(BlazeRuntime.java:975)
at com.google.devtools.build.lib.runtime.BlazeRuntime.main(BlazeRuntime.java:772)
at com.google.devtools.build.lib.bazel.BazelMain.main(BazelMain.java:55)
And this would get thrown for any packages in the main repository loaded
from other repositories:
java.lang.RuntimeException: Unrecoverable error while evaluating node 'PACKAGE:@//tools/build_rules/go/toolchain' (requested by nodes )
at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:982)
at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:499)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Invalid BUILD file name for package '@//tools/build_rules/go/toolchain': /home/brian/bazel/tools/build_rules/go/toolchain/BUILD
at com.google.devtools.build.lib.packages.Package.finishInit(Package.java:299)
at com.google.devtools.build.lib.packages.Package$Builder.finishBuild(Package.java:1308)
at com.google.devtools.build.lib.skyframe.PackageFunction.compute(PackageFunction.java:501)
at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:933)
... 4 more
Sponsor's comment: note the abundance of new Label.resolveRepositoryRelative() calls. They are ugly, but it's only making existing ugliness explicit. Yes, we should fix it, especially in the implementation of configurable attributes.
Refs #940
--
Change-Id: I8bd7f7b00bec58a7157507595421bc50c81b404c
Reviewed-on: https://bazel-review.googlesource.com/#/c/2591
MOS_MIGRATED_REVID=117429733
|
|
|
|
|
|
|
| |
execution are reported. Previously, we would not report the error unless the build was already failing and this error ended up being the root cause of the first failed top-level target.
--
MOS_MIGRATED_REVID=116767915
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Rolling forward with the restored logic to avoid stat calls on injected Metadata.
*** Original change description ***
Automated [] rollback of commit df03e10f6552566982399b8779fe7bc7a17d75dc.
--
MOS_MIGRATED_REVID=114447944
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=114329043
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=114249806
|
|
|
|
|
|
|
| |
ArtifactFunction.
--
MOS_MIGRATED_REVID=114174899
|
|
|
|
|
|
|
| |
ArtifactFiles.
--
MOS_MIGRATED_REVID=114166208
|
|
|
|
|
|
|
|
|
| |
action fails, and propagate the exit code to BuildFailedException.
And update all call sites that copy or wrap ActionExecutionExceptions.
--
MOS_MIGRATED_REVID=112811857
|
|
|
|
|
|
|
| |
Reduces garbage.
--
MOS_MIGRATED_REVID=109914243
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Blaze discovers inputs for some actions when running with some
strategies. Those actions should not discover additional inputs
when they run, regardless of the strategy they end up using.
There are now no known legitimate cases of such additional input
discovery, so we should reinstate this check and find new ones :)
We also change the failure mode to be a normal error rather than
a crash. This error does indicate a tooling issue, and a small
chance of incorrect builds, but it doesn't create such an
inconsistent state that a crash is warranted.
--
Change-Id: I5d498d2fc1c5e23bfb5d77971f866c2027cbf03a
Reviewed-on: https://bazel-review.googlesource.com/#/c/2500/3
MOS_MIGRATED_REVID=109703508
|
|
|
|
|
|
|
| |
is guaranteed not to be a package. Instead, start with its parent. This will be faster and take less memory.
--
MOS_MIGRATED_REVID=107725767
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=104658639
|