| Commit message (Collapse) | Author | Age |
|---|
| |
|
|
|
|
|
|
| |
node having the same priority, later enqueueings having higher priority, re-enqueued nodes having highest priority, and new root nodes having lowest priority. Experimentally, this can save significant RAM (1.4G in some builds!) while not affecting speed.
Also do a semi-drive-by deleting ExecutorFactory parameter to AbstractQueueVisitor, since it was always AbstractQueueVisitor.EXECUTOR_FACTORY.
PiperOrigin-RevId: 208560889
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
than the graph version when that is feasible.
* It's not feasible when the computation accesses outside state, i.e. is non-hermetic, so see below.
* It's also more complicated (and not worth the trouble) when the computation is taking place just for the error status.
Have SkyFunctionName declare whether the function it corresponds to is hermetic or non-hermetic. Only non-hermetically-generated SkyValues can be directly marked changed, and non-hermetic SkyFunctions have their values saved at the graph version, not the max of the child versions. All SkyFunctions are hermetic except for the ones that can be explicitly dirtied.
A marked-hermetic SkyFunction that has a transient error due to filesystem access can be re-evaluated and get the correct version: if it throws an IOException at version 1 and then, when re-evaluated at version 2 with unchanged dependencies, has a value, the version will be version 1.
All Skyframe unit tests that were doing non-hermetic things to nodes need to declare that those nodes are non-hermetic. I tried to make the minimal set of changes there, so that we had good incidental coverage of hermetic+non-hermetic nodes. Also did some drive-by clean-ups around that code.
Artifacts are a weird case, since they're doing untracked filesystem access (for source directories). Using max(child versions) for them gives rise to the following correctness bug: 1. do a build at v1 that creates a FileStateValue for dir/ at v1. Then at v2, add a file to dir/ and do a build that consumes dir/ as a source artifact. Now the artifact for dir/ will (incorrectly) have v1. Then at v1, do that build again. We'll consume the "artifact from the future". However, this can only have an effect when using the local action cache, since the incorrect value of the artifact (the mtime) is only consumed by the action cache. Bazel is already broken in this way (incremental builds don't invalidate directories), so this change doesn't make things worse.
PiperOrigin-RevId: 204210719
|
| |
|
|
|
|
|
|
|
|
| |
node's in-progress data should be forgotten, and its evaluation should be restarted from scratch, as if it were freshly created/dirtied. To guard against this happening unexpectedly, any such events are passed to a GraphInconsistencyReceiver, which can verify that the SkyFunction is behaving properly.
This is the first change in a series to permit action rewinding when it is discovered that a previously generated input file is no longer available. When an action detects that one of its inputs is unusable, it can return this sentinel value, causing it to be re-evaluated from scratch. Follow-up changes will make the node corresponding to the input, and the node corresponding to the action that generated the input, dirty when this happens, causing the upstream action to be re-run, regenerating the desired input.
Currently works for builds that do not keep edges, although follow-ups may make this possible for all builds.
PiperOrigin-RevId: 194863097
|
| |
|
|
|
|
|
|
| |
Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own.
This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually.
PiperOrigin-RevId: 175033526
|
| |
|
|
|
|
|
| |
With a few manual fixes for readability.
RELNOTES: None.
PiperOrigin-RevId: 160582556
|
| |
|
|
|
|
|
|
|
|
| |
-Have SkylarkImportLookupFunction include causes in the SkyFunctionExceptions it throws.
-Better transitive skyframe error declarations in ASTFileLookupFunction.
-Have ErrorInfo differentiate between direct and transitive transience.
-Introduce ErrorInfoManager and have ParallelEvaluator/ParallelEvaluatorContext use it.
RELNOTES: None
PiperOrigin-RevId: 159163186
|
| |
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 157446717
|
| |
|
|
|
|
|
|
| |
This is useful for dealing with all the existing implementations in the face of
interface changes that are irrelevant.
RELNOTES: None
PiperOrigin-RevId: 155525021
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
reverseDepsToConsolidate field in InMemoryNodeEntry. As part of that, revamp our logic of how we store pending operations: store adds bare on initial evaluations, and checks bare on incremental evaluations and operations on done nodes.
This should improve performance in two ways: BuildingState loses two fields, saving working memory intra-build. Storing pending reverse dep operations bare also saves memory intra-build. Note that neither of these changes helps resting memory state, only while a node is still evaluating.
Because of this, we can simplify ReverseDepsUtil a bit, making ReverseDepsUtilImpl a static class, which it always wanted to be (what it really wants to be is a superclass of InMemoryNodeEntry, but I don't want to spend the object alignment bits).
Finally, this makes it pretty tempting to get rid of BuildingState altogether on initial evaluations. We'd still keep DirtyBuildingState, but we could save another ~24 bytes by storing BuildingState's one remaining field, signaledDeps, directly inside InMemoryNodeEntry.
--
PiperOrigin-RevId: 151048879
MOS_MIGRATED_REVID=151048879
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
With more specific information to be reported by Skyfunctions, e.g.,
to inform the build-event protocol on missing files, the EventHandler
interface is no longer enough. Therefore, provide an enriched context
for reporting events.
--
Change-Id: I2d06166fe4d5b9054e24ad8c752fafc039e3f9f8
Reviewed-on: https://cr.bazel.build/8794
PiperOrigin-RevId: 148463437
MOS_MIGRATED_REVID=148463437
|
| |
|
|
|
|
|
|
|
|
| |
Remove DirtyKeyTracker and inflight node tracking as independent
progress tracking bits of code, and instead use an internal-facing
DirtyTrackingProgressReceiver wrapper of the user's
EvaluationProgressReceiver.
--
MOS_MIGRATED_REVID=135291924
|
| |
|
|
|
|
|
|
|
|
| |
EvaluationProgressReceiver objects have two common naming schemes
currently, and calling them invalidationReceiver is misleading, so to
make the naming convention standard, all object names are based on
"progressReceiver."
--
MOS_MIGRATED_REVID=134411011
|
| |
|
|
|
|
|
| |
parameters conveying the requesting node (if any), the requested node(s), as well as a reason for the skyframe graph lookup. Alternate graph implementations may be interested in this information.
--
MOS_MIGRATED_REVID=128496089
|
| |
|
|
|
|
|
| |
interfaces, in preparation for further changes.
--
MOS_MIGRATED_REVID=126924789
|
| |
|
|
|
|
|
| |
helpers to enforce concurrency synchronization points and determinism even if they are not using an InMemoryGraph-backed evaluator.
--
MOS_MIGRATED_REVID=121977783
|
| |
|
|
|
| |
--
MOS_MIGRATED_REVID=113197641
|
| |
|
|
|
|
|
| |
Reduces garbage.
--
MOS_MIGRATED_REVID=109914243
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Raises the level of abstraction of ValueVisitor's dependence on
AbstractQueueVisitor. Except for the "ForTestingOnly" methods now
available on the QuiescingExecutor interface, ValueVisitor is
agnostic to the implementation of its executor.
This also cleans up the full spectrum of visibility modifiers on
ValueVisitor methods, all of which ought to be private.
--
MOS_MIGRATED_REVID=106847453
|
| |
|
|
|
|
|
|
| |
Refactors SkyFunction map creation into the tester from tests.
Encapsulates modifiedValues better.
--
MOS_MIGRATED_REVID=105982046
|
| |
|
|
|
|
|
| |
Previously, only ThreadPoolExecutor implementations were allowed.
--
MOS_MIGRATED_REVID=105340237
|
| |
|
|
|
|
|
|
|
|
|
| |
The headers were modified with
`find . -type f -exec 'sed' '-Ei' 's|Copyright 201([45]) Google|Copyright 201\1 The Bazel Authors|' '{}' ';'`
And manual edit for not Google owned copyright. Because of the nature of ijar, I did not modified the header of file owned by Alan Donovan.
The list of authors were extracted from the git log. It is missing older Google contributors that can be added on-demand.
--
MOS_MIGRATED_REVID=103938715
|
| |
|
|
|
|
|
| |
avoid mutating the deps of nodes that are still going to be deps after evaluation is finished.
--
MOS_MIGRATED_REVID=103659429
|
| |
|
|
|
| |
--
MOS_MIGRATED_REVID=102835622
|
| |
|
|
|
|
|
|
|
| |
to interrupt evaluation in constant time.
Some ParallelEvaluator tests that implicitly relied on cleaning happening before the next evaluation were moved into MemoizingEvaluatorTest as a result.
--
MOS_MIGRATED_REVID=102696653
|
| |
|
|
|
|
|
| |
SkyFunction#compute calls.
--
MOS_MIGRATED_REVID=102268773
|
| |
|
|
|
|
|
| |
method wasn't guaranteed to catch all bugs. Also convert to a singleton since there's no reason to have multiple instances.
--
MOS_MIGRATED_REVID=102158719
|
| |
|
|
|
|
|
|
|
|
|
| |
This CL introduces a ThinNodeEntry, which is a NodeEntry without the
means of accessing its value. The InvalidatingNodeVisitor does not
need to access nodes' values while doing its work, so it is provided
with a ThinNodeQueryableGraph, capable of producing only
ThinNodeEntries.
--
MOS_MIGRATED_REVID=102088111
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The EvaluationProgressReceiver#evaluated method took a SkyValue,
but that parameter was only used by some of its implementations,
and only under some conditions.
Outside of tests, the main users are SkyframeBuildView's
ConfiguredTargetValueInvalidationReceiver and SkyframeBuilder's
ExecutionProgressReceiver.
The former builds up a set of built ConfiguredTarget keys when the
SkyValue is non-null and the EvaluationState is BUILT, and so its
nullity check can live behind those two conditions.
The latter cares about builting up a set of ConfiguredTargets, and
raising events on the eventBus when a TARGET_COMPLETION or
ASPECT_COMPLETION value is evaluated and is non-null. The extraction
of these values can live behind the conditions that check the type of
the SkyKey.
By making the SkyValue parameter lazy, this change enforces that it's
only accessed under these conditions.
This CL introduces a semantic change that should be small in effect.
The SkyframeBuildView keeps track of a set, dirtiedConfiguredTargetKeys,
and ConfiguredTarget keys evaluated as CLEAN were removed from it if
they had a non-null value. With this change, ConfiguredTarget keys
evaluated as CLEAN get removed regardless of whether their values are
null or non-null. The set is used to determine whether artifact conflict
search has to be rerun, and the extra removals that result from this CL
could cause fewer artifact conflict searches to run, but the only
affected searches would be those that were caused by dirtied configured
target values in error all of which were subsequently marked as clean,
which is probably rare.
--
MOS_MIGRATED_REVID=101144655
|
| |
|
|
|
|
|
| |
Ref equality for same-name SkyFunctionNames is not guaranteed.
--
MOS_MIGRATED_REVID=100322275
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The primary user of invalidation tracking is the SkyframeBuildView,
which monitored which ConfiguredTargetValues were invalidated. It
did that so the SkyframeExecutor could limit its search for artifact
conflicts to when the set of configured targets in the build changed.
For the newly introduced set of dirtied keys,
"dirtiedConfiguredTargetKeys" in SkyframeBuildView, to be as useful as
the "dirtyConfiguredTargets" set it replaced, the
ConfiguredTargetValueInvalidationReceiver must only remove a key from
the set if it was found to be clean when it was re-evaluated. If it
was rebuilt, then the key must stay in the set, to represent that the
set of configured target values has truly changed.
This CL introduces a semantic change that hopefully has a small effect,
if any. Previously, the informInvalidationReceiver method in
InvalidatingNodeVisitor only informed the invalidationReceiver when a
non-null value was invalidated. (This is usually, but not always,
equivalent to a non-error value. The uncommon exception is that in
keep-going builds, some nodes with errors may also have values, and
the invalidator would inform the receiver when such a node was
invalidated.) Now, the receiver is informed that a node is invalidated
regardless of whether its value is null. Because the receiver uses this
information to determine whether artifact conflict search has to be
rerun, and that search is expensive, it's possible this change will
negatively impact performance. However, the only way an extra search
could be invoked is if the invalidated configured target nodes are
all in error. That seems like it would happen rarely, if at all.
Further cleanup of unused SkyValues returned by markDirty to come in
a subsequent CL.
--
MOS_MIGRATED_REVID=100304744
|
| |
|
|
|
| |
--
MOS_MIGRATED_REVID=99961435
|
| |
|
|
|
|
|
| |
ParallelEvaluator, to allow users to customize which events they want to store.
--
MOS_MIGRATED_REVID=98299604
|
|
|
--
MOE_MIGRATED_REVID=85702957
|
janakr
tomlu
laurentlb
nharmata
lberki
Klaus Aehlig
Chloe Calvarin
Googler
Mark Schaller
Damien Martin-Guillerez
Han-Wen Nienhuys