| Commit message (Collapse) | Author | Age |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Rollforward with fix: Don't use Immutable map builder to agggregate Filesets. We may reach the same one via direct and indirect runfiles, so we need to allow for duplicate entries.
*** Original change description ***
Automated rollback of commit 3bace1b937934fb2cea6260067ecc1cdbe526847.
*** Reason for rollback ***
b/112583337
RELNOTES: None
*** Original change description ***
Track Fileset in artifact expansion.
PiperOrigin-RevId: 208748163
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
b/112583337
RELNOTES: None
*** Original change description ***
Track Fileset in artifact expansion.
PiperOrigin-RevId: 208658921
|
|
|
|
| |
PiperOrigin-RevId: 208549418
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Rollforward with deduplication of traversals and regression test.
RELNOTES: None
*** Original change description ***
Automated rollback of commit 39974a43abdd32e3a1acbc7da945b08da9983e4e.
*** Reason for rollback ***
b/112458627
*** Original change description ***
Allow skyframe-aware actions to pass partial results through ActionExecutionContext.
Remove FilesetProvider.
PiperOrigin-RevId: 208231719
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
b/112458627
*** Original change description ***
Allow skyframe-aware actions to pass partial results through ActionExecutionContext.
Remove FilesetProvider.
PiperOrigin-RevId: 208213955
|
|
|
|
|
|
|
|
| |
ActionExecutionContext.
Remove FilesetProvider.
PiperOrigin-RevId: 208111251
|
|
|
|
| |
PiperOrigin-RevId: 208096548
|
|
|
|
|
|
|
| |
looked-up artifact values.
RELNOTES: None
PiperOrigin-RevId: 207295716
|
|
|
|
|
|
| |
performance reasons.
PiperOrigin-RevId: 207171305
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add support for tree artifacts (ctx.action.declare_directory(...)) in
runfiles. Before this change we would throw away the information
about the files inside a tree artifact before executing an action.
That's fine for local execution where the sandbox just copies/symlinks
a directory and doesn't care much what's inside. However, in remote
execution we actually need to upload each individual file and so
we need to be aware of all individual files not just directories.
This change makes it so that this information is made available to a
SpawnRunner via the SpawnInputExpander.
RELNOTES: None
PiperOrigin-RevId: 207091668
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Breaks FDO builds, see http://b/112112125
*** Original change description ***
Remove PerActionFileCache
Instead, make ActionMetadataHandler implement the MetadataProvider interface.
This fixes an issue where an action that runs two spawns where one depends on
an output of the other was unable to get the metadata for the intermediate
output.
We don't currently have actions that do this, but we will have in a future
change (which will also implicitly act as a regression test).
PiperOrigin-RevId: 207084179
|
|
|
|
|
|
|
|
| |
function, discoverInputs() can be implemented to return null upon encountering
missing ActionExecutionValues.
RELNOTES: None.
PiperOrigin-RevId: 206913969
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Instead, make ActionMetadataHandler implement the MetadataProvider interface.
This fixes an issue where an action that runs two spawns where one depends on
an output of the other was unable to get the metadata for the intermediate
output.
We don't currently have actions that do this, but we will have in a future
change (which will also implicitly act as a regression test).
PiperOrigin-RevId: 205629237
|
|
|
|
|
|
|
|
|
| |
CppCompileAction.discoverInputsStage2 retrieves values of discovered modules
from ActionExecutionValue.
This addresses a possible a correctness issue.
PiperOrigin-RevId: 202162180
|
|
|
|
|
|
|
| |
The memory savings from this flag are not worth the complexity, and it interferes with action restarting.
RELNOTES: Remove support for --discard_actions_after_execution.
PiperOrigin-RevId: 201077905
|
|
|
|
|
|
| |
Moves more Action-oriented from skyframe package, which has become very unwieldy, to action package. This is needed to avoid circular dependencies caused when build-base is needed for skyframe.
PiperOrigin-RevId: 200996982
|
|
|
|
|
|
| |
not serialized. Tag TestCompletionValue and any ActionExecutionValue coming from a NotifyOnActionCacheHit (i.e., tests) like that. To make such values really not shared, request the ActionExecutionValue from TestCompletionFunction as opposed to the ArtifactValue (propagating the unshareable bit up seemed like too much fuss, and I have a dream of getting rid of ArtifactValue anyway).
PiperOrigin-RevId: 200504358
|
|
|
|
|
|
|
|
|
| |
unit test coverage for this.
Note that ActionFS is not generic enough to make use of FileSystemTest.
RELNOTES: None
PiperOrigin-RevId: 200304871
|
|
|
|
|
|
|
| |
The on-disk execRoot/blaze-out is now off limits to ActionFS.
RELNOTES: None
PiperOrigin-RevId: 200080287
|
|
|
|
|
| |
RELNOTES: None
PiperOrigin-RevId: 199880252
|
|
|
|
|
|
| |
(minor) ActionFS now implements MetadataProvider.getInput
PiperOrigin-RevId: 199575194
|
|
|
|
|
| |
RELNOTES: None
PiperOrigin-RevId: 199572597
|
|
|
|
|
|
| |
We now track Causes instead of plain Labels, which will allow us to do better reporting in the future. Add basic tests.
PiperOrigin-RevId: 198380468
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The goal is to, for an action execution, make only a single skyframe
edge to every required runfiles tree rather than an edge to every
runfiles artifact directly. We accomplish this by pulling the runfiles
artifacts' metadata for a particular runfiles tree through the
appropriate runfiles middlemen artifact in one batch. This CL fixes
https://github.com/bazelbuild/bazel/issues/3217.
This change makes runfiles middlemen more similar to aggregating
middlemen. We however continue to treat runfiles middlemen slightly
differently than true aggregating middlemen. Namely, we do not expand
runfiles middlemen into the final action inputs. The reasons for this
are:
1. It can make latent bugs like
https://github.com/bazelbuild/bazel/issues/4033 more severe.
2. The runfiles tree builder action's output MANIFEST contains
absolute paths, which interferes with remote execution if its metadata
is added to a cache hash.
3. In the sandbox, it causes the runfiles artifacts to be mounted at
their exec path locations in addition to within the runfiles
trees. This makes the sandbox less strict. (Perhaps tolerably?)
4. It saves a modicum of (transient) memory and time to not expand.
If the first two issues are fixed, it could be a nice simplification
to completely replace runfiles middlemen with aggregating middlemen.
Change-Id: I2d2148297f897af4c4c6dc055d87f8a6fad0061e
PiperOrigin-RevId: 198307109
|
|
|
|
| |
PiperOrigin-RevId: 197055263
|
|
|
|
|
|
|
|
|
| |
Extracts a class, InputArtifactData to hold the input data instead of using a raw map. This provides the flexibility needed to support both ActionFS and existing code so ActionFS does not need to rekey the input data.
Uses the smaller, getDeclaredIncludeSrcs instead of getAllowedDerivedInputs
when possible for staging optional inputs in ActionFS.
PiperOrigin-RevId: 196736703
|
|
|
|
| |
PiperOrigin-RevId: 195973862
|
|
|
|
|
|
|
| |
instead of the manifest files.
RELNOTES: None
PiperOrigin-RevId: 195149880
|
|
|
|
| |
PiperOrigin-RevId: 194998981
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Currently we report "Analyzing" when include scanning runs. But since we can
have shared C++ compile actions, only one of the group will be executed and only
one will be reported completed. Remaining shared actions currently stay with
"analyzing" forever.
This cl makes sure that these actions are properly handled when finished.
This is an encore of https://github.com/bazelbuild/bazel/commit/24f19ec2679dd93b1ac5b06e46f3b35807d6e217. In this incarnation I make sure that all actions that discover inputs are consistent in reporting their Analyzing status. Originally only CppCompileAction was doing that. Apparently we have more actions that discover inputs (e.g. LtoBackendAction) but these were not reporting Analyzing and therefore crashing on preconditions. This cl makes sure that all actions discovering inputs report their analyzing status.
RELNOTES: None
PiperOrigin-RevId: 194066513
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
See: http://b/78455900
*** Original change description ***
Properly report completion of shared actions with input discovery
Currently we report "Analyzing" when include scanning runs. But since we can
have shared C++ compile actions, only one of the group will be executed and only
one will be reported completed. Remaining shared actions currently stay with
"analyzing" forever.
This cl makes sure that these actions are properly handled when finished.
RELNOTES: None
PiperOrigin-RevId: 193955856
|
|
|
|
|
|
|
|
| |
the ones in SkyframeExecutor, called once for each Root in the package path list. This ensures there is a single canonical ArtifactRoot for each source root.
It is still the case that every Package loaded has its own Root (https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/packages/Package.java;l=287?q=Package.java), which is a waste of memory, but because of the map inside ArtifactFactory introduced in this change, all Artifacts with the same logical source root share a single ArtifactRoot instance.
PiperOrigin-RevId: 192513819
|
|
|
|
|
|
|
|
|
|
|
|
| |
Currently we report "Analyzing" when include scanning runs. But since we can
have shared C++ compile actions, only one of the group will be executed and only
one will be reported completed. Remaining shared actions currently stay with
"analyzing" forever.
This cl makes sure that these actions are properly handled when finished.
RELNOTES: None
PiperOrigin-RevId: 191849728
|
|
|
|
|
|
|
| |
Unconditionally inject the new values, and let skyframe change pruning avoid over-invalidation.
Change-Id: I3f478ea756121bde1078e9e79ddcfbeb54951cbb
PiperOrigin-RevId: 189156698
|
|
|
|
|
|
|
|
| |
doesn't save memory in the 32-bit case, but makes it easier for people to see how many SkyKeys we have.
There's some unnecessary interning in tests, but it was easier to copypasta and doesn't harm anything, I think.
PiperOrigin-RevId: 187694309
|
|
|
|
| |
PiperOrigin-RevId: 185694489
|
|
|
|
|
|
| |
Post to the Skyframe Environment instead.
PiperOrigin-RevId: 182096559
|
|
|
|
|
|
| |
This is slightly more descriptive, and we will potentially want to use the name Root for a broader concept shared between ArtifactRoot and RootedPath.
PiperOrigin-RevId: 182082367
|
|
|
|
|
|
|
|
| |
Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own.
This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually.
PiperOrigin-RevId: 175033526
|
|
|
|
|
|
|
|
| |
This is both probably what the user most cares about, and also restores a Skyframe invariant, that as a catastrophic exception bubbles up, it stays catastrophic.
There is actually no hard Skyframe requirement for that, so we could consider relaxing it, but I'm willing to keep it for now.
PiperOrigin-RevId: 174759976
|
|
|
|
|
|
| |
cache if we're doing input discovery: input discovery may find new artifacts we don't yet know about. Similarly, in SingleBuildFileCache, use the Artifact's path if it's available, rather than the poor man's route of the execRoot.
PiperOrigin-RevId: 171635892
|
|
|
|
| |
PiperOrigin-RevId: 170503143
|
|
|
|
|
|
|
|
|
| |
Split collect, concurrent, vfs, windows into package-level BUILD files.
Move clock classes out of "util", into their own Java package.
Move CompactHashSet into its own Java package to break a dependency cycle.
Give nestedset and inmemoryfs their own package-level BUILD files.
PiperOrigin-RevId: 167702127
|
|
|
|
|
|
| |
wrapper objects: for OwnedArtifacts, which are the most numerous during builds, and for Labels for TransitiveTraversalValues, which are the most numerous during queries.
PiperOrigin-RevId: 154989520
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We need to lookup repositories as part of converting exec paths to artifacts,
which in turn is needed for action cache lookups. These lookups should not
cause a Skyframe exit, so we must not throw an exception here, unless the
error makes it impossible to continue. Instead, we need to leave the decision
whether to error out or not to the caller.
Note that we may unnecessarily fetch a remote repository in order to do the
action cache lookup, even if the action no longer depends on the input file,
although this should only be possible for C++ compile actions. It's possible
that there's another bug in the C++ compile action key computation that also
contributes.
This change also makes it so that the post-resolution action cache code
ignores any errors wrt. repository lookup rather than throwing. If any of the
paths could not be found, then the action cache lookup fails and we re-execute
the corresponding action, which is exactly what should happen.
Fixes #2759.
PiperOrigin-RevId: 153696243
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The ActionMetadataHandler does an explicit state transition on
discardOutputMetadata. Before the call, it may be used for action cache
checking, and after the call it may be updated with execution results.
Several of the methods now throw if they're used incorrectly, so I had to
refactor the control flow in ActionExecutionFunction to correctly call
discardOutputMetadata on the MetadataHandler in all cases. I discovered a
resource leak (of FileOutErr) in IncludeParseFunction while I was at it, so
I plugged that as well.
One step towards #1525.
PiperOrigin-RevId: 152363982
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
--noexperimental_enable_critical_path_profiling flags are all specified, then Bazel will delete Actions from ActionLookupValues as they are executed in order to save memory.
Because an already-run action may output an artifact that is only requested later in the build, we need to maintain a way for the artifact to look up the action. But in most cases we don't need to keep the action itself, just its output metadata.
Some actions unfortunately are needed post-execution, and so we special-case them.
Also includes dependency change with description:
Move action out of key. This keeps action references from polluting the graph -- actions are just stored in one SkyValue, instead of being present in SkyKeys.
This does mean additional memory used: we have a separate ActionLookupData object per Action executed. That may reach ~24M for million-action builds.
PiperOrigin-RevId: 151756383
|
|
|
|
|
|
|
|
|
|
|
|
| |
updateInputs() and inputsKnown() non-overridable and removing setInputs().
This comes at the cost of adding a flag to every action instance that's not used for non-input-discovering actions, but I think that's a deal. Simpler APIs are good, mmmmkay?
Also fixed a few pre-existing issues in TestAction and ObjcCompileAction.
--
PiperOrigin-RevId: 148749734
MOS_MIGRATED_REVID=148749734
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This combines both previous changes and extends them to work both with and
without kchodorow@'s rollout of the exec root rearrangement. Unfortunately,
each of these changes individually breaks something somewhere, so they must
all go into a single commit.
Change 1:
CppCompileAction must return false from inputsKnown for .d pruning
This is necessary (but not sufficient) for the action cache to work
correctly. Consider the following sequence of events:
1. action is executed
2. .d pruning is performed
3. action cache writes entry with post-pruning inputs list
4. action gets regenerated (e.g., due to server restart)
5. the action cache calls inputsKnown(), which returns true
6. action cache checks entry from step 3 against pre-pruning inputs list,
which results in a cache miss
The action cache needs to know that the current list is not the final list,
so inputsKnown() in step 5 must return false if .d pruning is enabled.
Change 2:
Fix artifact root discovery for external artifacts
The SkyframeExecutor was assuming that all exec paths were coming from the
main repository. Instead, rely on external exec paths to start with "../".
Additional change 3:
In addition, update the PackageRootResolverWithEnvironment and the
HeaderDiscovery to use the single unified repository name guessing
implementation. Previously, the PackageRootResolverWithEnvironment was
poisoning the source artifact cache, which then caused subsequent lookups
to return a bad artifact.
Add a precondition to double-check that artifacts looked up by exec path
have the correct source root.
For compatibility with kchodorow@'s upcoming exec root refactor, if the exec
path starts with "external", then assume it's coming from an external
repository. This must be removed when that change is successfully rolled out,
or it will break if anyone creates a package called 'external'.
Additional change 4:
On top of all of that, PackageRootResolverWithEnvironment and SkyframeExecutor
must use the same source root computation as the Package class itself. I
extracted the corresponding code to Root, and added comments both there and
in Package to clearly indicate that these methods have to always be modified
in sync.
Fixes #2490.
--
PiperOrigin-RevId: 148439309
MOS_MIGRATED_REVID=148439309
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 146694092
MOS_MIGRATED_REVID=146694092
|