aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/skyframe
Commit message (Collapse)AuthorAge
* Order Skyframe evaluations in a priority queue, with all children of a given ↵Gravatar janakr2018-08-13
| | | | | | | | node having the same priority, later enqueueings having higher priority, re-enqueued nodes having highest priority, and new root nodes having lowest priority. Experimentally, this can save significant RAM (1.4G in some builds!) while not affecting speed. Also do a semi-drive-by deleting ExecutorFactory parameter to AbstractQueueVisitor, since it was always AbstractQueueVisitor.EXECUTOR_FACTORY. PiperOrigin-RevId: 208560889
* ActionAnalysisMetadata exposes whether loose headers are present. ↵Gravatar shahan2018-08-12
| | | | | | TransitiveVersionTable uses futures. PiperOrigin-RevId: 208402465
* Refactoring of SkyFunctionEnvironment to iterate over events/postables only ↵Gravatar janakr2018-08-10
| | | | | | | | | | | | when they're actually being put into a committed value. The previous behavior submitted deps' events twice, when the dep was added and when the node finished building. The intention is to build on this refactoring to cut off events/postables across the analysis-execution boundary, so that actions are not carrying around nested sets of warnings coming from their configured targets. This will be safe because to execute an action, we must already have analyzed its configured target, so the warning would have been emitted there. As can be seen from the changed test, this is not a pure behavior no-op. We will now emit cached events slightly later, on value committal, rather than on first dep declaration. This should not be an issue: since the events are cached, the user must have already seen them on a prior build, so the delay should not be important. Inversely, we now report events slightly more quickly during bubbling up, since we report them at each stage, as opposed to just at ParallelEvaluator evaluation completion. PiperOrigin-RevId: 208316502
* In AbstractSkyFunctionEnvironment, don't check for filtered exceptions in ↵Gravatar janakr2018-08-08
| | | | | | the common case of no exceptions. We were already mostly tracking missing dependencies in the subclasses, so there's no need to check for missing dependencies here. PiperOrigin-RevId: 207934220
* When no new configured targets have been analyzed, only check for artifact ↵Gravatar janakr2018-08-03
| | | | | | | | | | conflicts if the current set of configured targets is not a subset of the largest set of configured targets that have been checked for conflicts. Also rework the flow between SkyframeBuildView and SkyframeActionExecutor to remove the SkyframeExecutor middleman. Also reword the error message in case of an ArtifactPrefixConflictException, since a clean should no longer be necessary. PiperOrigin-RevId: 207322139
* Add some additional debugging if a dep is unexpectedly absent.Gravatar janakr2018-08-02
| | | | PiperOrigin-RevId: 207178336
* Add the ParallelEvaluator.eval call to the ProfilerGravatar ulfjack2018-07-30
| | | | | | | We want to make sure that the profile captures all Skyframe invocations, so that we can catch cases where we accidentally introduce new ones. PiperOrigin-RevId: 206543307
* Skip invalidation entirely if the diff is emptyGravatar ulfjack2018-07-25
| | | | PiperOrigin-RevId: 205980620
* Configured targets register created source artifacts with Skyframe.Gravatar shahan2018-07-24
| | | | PiperOrigin-RevId: 205876673
* Fix crash bug in AbstractExceptionalParallelEvaluator#doMutatingEvaluation ↵Gravatar nharmata2018-07-23
| | | | | | | in a very specific window of time inbetween enqueueing one top-level node for evaluation and checking if another top-level node is done. See the added unit test for details. RELNOTES: None PiperOrigin-RevId: 205718683
* Set the version of a computed node to the max of its child versions rather ↵Gravatar janakr2018-07-11
| | | | | | | | | | | | | | | | | than the graph version when that is feasible. * It's not feasible when the computation accesses outside state, i.e. is non-hermetic, so see below. * It's also more complicated (and not worth the trouble) when the computation is taking place just for the error status. Have SkyFunctionName declare whether the function it corresponds to is hermetic or non-hermetic. Only non-hermetically-generated SkyValues can be directly marked changed, and non-hermetic SkyFunctions have their values saved at the graph version, not the max of the child versions. All SkyFunctions are hermetic except for the ones that can be explicitly dirtied. A marked-hermetic SkyFunction that has a transient error due to filesystem access can be re-evaluated and get the correct version: if it throws an IOException at version 1 and then, when re-evaluated at version 2 with unchanged dependencies, has a value, the version will be version 1. All Skyframe unit tests that were doing non-hermetic things to nodes need to declare that those nodes are non-hermetic. I tried to make the minimal set of changes there, so that we had good incidental coverage of hermetic+non-hermetic nodes. Also did some drive-by clean-ups around that code. Artifacts are a weird case, since they're doing untracked filesystem access (for source directories). Using max(child versions) for them gives rise to the following correctness bug: 1. do a build at v1 that creates a FileStateValue for dir/ at v1. Then at v2, add a file to dir/ and do a build that consumes dir/ as a source artifact. Now the artifact for dir/ will (incorrectly) have v1. Then at v1, do that build again. We'll consume the "artifact from the future". However, this can only have an effect when using the local action cache, since the incorrect value of the artifact (the mtime) is only consumed by the action cache. Bazel is already broken in this way (incremental builds don't invalidate directories), so this change doesn't make things worse. PiperOrigin-RevId: 204210719
* Native Skyframe support for node restartingGravatar mschaller2018-07-01
| | | | | | | | | | | | Useful for attempting to recover relationships between Skyframe graph state and external systems, when the evaluation of a Skyframe node has the side effect of creating that relationship. Currently, only supported in graph evaluations when reverse dependency edges are not tracked. RELNOTES: None. PiperOrigin-RevId: 202892953
* Restart node building if previous dep is dirty, fix check-then-act racesGravatar mschaller2018-07-01
| | | | | | | | | | | | | | | | | | | | | | While initializing the SkyFunctionEnvironment for a node being built, if a previously requested dep is found to be not done, reset and re-enqueue the building node. This lets the node handle the not-done dep like any other not-done dep (e.g. by enqueuing it or by waiting to be signalled by it). Similarly, while registering newly requested deps when building a node yields a value or an error, if a newly requested dep is found to be not done, return without completing the node, so that it may be signalled by the dep (without crashing; done nodes cannot be signalled). Also fixes a handful of remaining check-then-act races during Skyframe evaluation that were vulnerable to done->dirty node transitions. (Note that done->dirty node transitions during evaluation are planned, but not yet possible.) RELNOTES: None. PiperOrigin-RevId: 202886360
* Fix comment re: keep-going eval of SkyFunctions in error w undone depsGravatar mschaller2018-06-29
| | | | | | | | The ParallelEvaluator doesn't let such a SkyFunction complete; it has no control over whether the SkyFunction throws. RELNOTES: None. PiperOrigin-RevId: 202743267
* Fix asymmetric progress receiver state trackingGravatar mschaller2018-06-29
| | | | | | | | | The "elapsedTimeNanos > 0" conditional is highly likely to always be true, but if not, the progress receiver won't receive an "ending" event for the "compute" state. RELNOTES: None PiperOrigin-RevId: 202687683
* Reuse previously stored SkyValues during event/post collectionGravatar mschaller2018-06-29
| | | | | | | | | | | | | | | | | | | | | | | If a SkyFunction read a value (or its absence) from the graph during its evaluation, that value will be used to compute the event and post metadata for that evaluation. This CL modifies the assertion strategy for this code. Previously, registered deps which were not done would have gone undetected, and their events/posts skipped. This CL also makes a few minor changes that make SkyFunctionEnvironment more consistent: - Deps not already in previouslyRequestedDepsValues are added to newlyRequestedDeps regardless of whether evaluation was in error bubbling or whether the dep was done. - Previously requested deps of an inflight node are prefetched (by passing them to SkyFunctionEnvironment's ctor) during error bubbling in the same way as they are during normal eval or cycle checking. - Minor signature and documentation adjustments. RELNOTES: None. PiperOrigin-RevId: 202672709
* Store SkyValues for new deps from the graph intra-environmentGravatar mschaller2018-06-28
| | | | | | | | | | | | | | | | | | | | | | | | | This ensures that if a SkyFunction read a value (or its absence) for a dep during its evaluation, subsequent requests for that dep's value (or its absence) provide the same result. This does not yet necessarily apply to the process of collecting events and posts from a node's deps, which will be considered in a future refactoring. This CL adds SkyFunctionEnvironment#removeUndoneNewlyRequestedDeps because the prior strategy for removing undone deps for done parents, by re-requesting the dep's node from the graph and checking its doneness, could lead to deps being dropped from parents if those deps transitioned from done to dirty as the parent completes. Minor cleanup to the bubbleErrorInfo field, which is nullable, and now documented. (Note that done->dirty node transitions during evaluation are planned, but not yet possible.) RELNOTES: None. PiperOrigin-RevId: 202577098
* Convert directDeps to a map of SkyValuesGravatar mschaller2018-06-28
| | | | | | | | | | | | | | | | | | | | SkyFunctionEnvironment only cares about directDeps' values, not other NodeEntry data. This reduces the space of code which could be sensitive to nodes which transition from done to dirty during evaluation. To prevent check-then-act races in the refactored code (and only there; other code will be fixed in future refactorings), instead of checking deps' isDone() methods before accessing their value, allow getValueMaybeWithMetadata to be called when not done, and have it return null when not done. (Note that done->dirty node transitions during evaluation are planned, but not yet possible.) RELNOTES: None. PiperOrigin-RevId: 202518781
* Short-circuit done node dep registration when no new depsGravatar mschaller2018-06-28
| | | | | | | | The short-circuiting code was pulled into the !keepGoing conditional block inadvertently. RELNOTES: None. PiperOrigin-RevId: 202505510
* Add new BuildMetrics event to BEP.Gravatar tomlu2018-06-19
| | | | | | | To start we add just a single metric, the number of actions constructed in the current build. RELNOTES: None PiperOrigin-RevId: 201248490
* Clarify that ErrorInfoValue is a NotComparableSkyValue because we don't ↵Gravatar janakr2018-06-19
| | | | | | expect equality, not because there's no implementation. PiperOrigin-RevId: 201191262
* Permit marking dirty/changed a node more than onceGravatar mschaller2018-06-18
| | | | | | | | | | | | This functionality will be useful for node restarting. More than one parent node may request the restart of a shared child node, and this should not fail. Instead, the returned MarkedDirtyResult indicates whether the dirtying was redundant, and the calling code can assert on that. RELNOTES: None. PiperOrigin-RevId: 201005663
* Add functionality to make certain SkyValues unshareable, meaning they are ↵Gravatar janakr2018-06-13
| | | | | | not serialized. Tag TestCompletionValue and any ActionExecutionValue coming from a NotifyOnActionCacheHit (i.e., tests) like that. To make such values really not shared, request the ActionExecutionValue from TestCompletionFunction as opposed to the ArtifactValue (propagating the unshareable bit up seemed like too much fuss, and I have a dream of getting rid of ArtifactValue anyway). PiperOrigin-RevId: 200504358
* Add support for the Json trace file formatGravatar ulfjack2018-06-12
| | | | | | | | | | | | | | | | | Add a --experimental_generate_json_trace_profile option that puts a file into the output base (or uses --profile if set). There are still a lot of problems with this. - unexplained holes - way too many threads - nonsensical event titles - too many detail events, too little overview - it may also cause unnecessary load - it silently overwrites the existing file on subsequent invocations The format is documented here: goo.gl/oMZPLh PiperOrigin-RevId: 200259431
* Add AutoProfiler-like API to ProfilerGravatar ulfjack2018-06-11
| | | | | | - migrate all startTask/completeTask pairs to the new API PiperOrigin-RevId: 200038703
* Make Artifact#equals take the owner into account for derived artifacts.Gravatar janakr2018-06-08
| | | | | | | | Derived artifacts' owners are important because they are used to determine the artifact's generating action. Source artifacts' owners are not used in this way, so I left them alone. This allows us to get rid of most uses of ArtifactSkyKey. We may be able to delete it entirely in a follow-up. PiperOrigin-RevId: 199836436
* Add hacky method to InMemoryNodeEntry for fast but unsafe iteration over ↵Gravatar janakr2018-06-04
| | | | | | in-progress reverse deps. PiperOrigin-RevId: 199209256
* Remove ConfiguredTarget from TargetCompletionValue, since it is no longer ↵Gravatar janakr2018-06-04
| | | | | | needed: we can get the ConfiguredTargetKey directly from the TargetCompletionKey. Since that was the only use of the actual value in EvaluationProgressReceiver#evaluated, remove it, instead just provide a boolean enum that gives whether or not evaluation succeeded. PiperOrigin-RevId: 199178047
* Add events and get rid of ErrorInfoEncoder. Clean up some signatures and ↵Gravatar janakr2018-05-22
| | | | | | visibility in Skyframe classes. PiperOrigin-RevId: 197665817
* Potentially allow children of a dirty node to be missing from the graph. ↵Gravatar janakr2018-05-15
| | | | | | Also pass the GraphInconsistencyReciever into SkyframeExecutor as a parameter. PiperOrigin-RevId: 196716642
* Use ConcurrentHashMap directly instead of MapMaker.Gravatar Googler2018-05-03
| | | | | RELNOTES: None. PiperOrigin-RevId: 195280015
* Clean up code that directly imports nested classes like Builder, Entry, etc.Gravatar jcater2018-05-02
| | | | PiperOrigin-RevId: 195100670
* Allow SkyFunctions to return a sentinel value indicating that all of a ↵Gravatar janakr2018-04-30
| | | | | | | | | | node's in-progress data should be forgotten, and its evaluation should be restarted from scratch, as if it were freshly created/dirtied. To guard against this happening unexpectedly, any such events are passed to a GraphInconsistencyReceiver, which can verify that the SkyFunction is behaving properly. This is the first change in a series to permit action rewinding when it is discovered that a previously generated input file is no longer available. When an action detects that one of its inputs is unusable, it can return this sentinel value, causing it to be re-evaluated from scratch. Follow-up changes will make the node corresponding to the input, and the node corresponding to the action that generated the input, dirty when this happens, causing the upstream action to be re-run, regenerating the desired input. Currently works for builds that do not keep edges, although follow-ups may make this possible for all builds. PiperOrigin-RevId: 194863097
* Add hook to inform SkyframeExecutor that build is finished.Gravatar janakr2018-04-24
| | | | PiperOrigin-RevId: 194099006
* Remove use of bare Immutable{List,Map,Set} Builder classes.Gravatar jcater2018-04-20
| | | | | | Always use the more-qualified class name for clarity at the site of use. There are too many classes named Builder. PiperOrigin-RevId: 193649193
* Add warning to registerDependencies javadocGravatar shreyax2018-04-19
| | | | PiperOrigin-RevId: 193559600
* Notify Skyframe listeners about additional node states: CHECK_DIRTY, ↵Gravatar felly2018-04-17
| | | | | | INIT_ENV, and COMMIT. PiperOrigin-RevId: 193261989
* Simplify AbstractSkyFunctionEnvironment to more directly call ↵Gravatar shreyax2018-04-10
| | | | | | getValueOrUntypedExceptions. PiperOrigin-RevId: 192329649
* In inlined skylark import lookup function caching, don't actually fetch deps ↵Gravatar shreyax2018-04-06
| | | | | | when all we want to do is register an edge and don't require the value. PiperOrigin-RevId: 191966203
* Change profiling to only accept strings for its "description" argument. ↵Gravatar janakr2018-04-01
| | | | | | Profiling can hold onto objects for the duration of the build, and some of those objects may be temporary that should not be persisted. In particular, UnixGlob and its inner classes should not outlive loading and analysis. For the most part, care was taken in this CL to only use strings that required no additional construction, mainly to minimize garbage (retaining references to newly created strings is not as great a concern since only the strings corresponding to the slowest K tasks are retained, for some relatively small values of K). Action descriptions for actually executing actions are eagerly expanded because that work is minimal compared to the work of actually executing an action. PiperOrigin-RevId: 191251488
* Fix accidental eager expansion of grouped list in in-memory nodes.Gravatar felly2018-03-27
| | | | PiperOrigin-RevId: 190656902
* Add DONE_CHECKING as a reason for requesting nodes.Gravatar shreyax2018-03-26
| | | | | | | | Split registering the unique new deps of a node between those where we're enqueueing a known dependency from a prior build and one where we're adding a new dependency. Replace prefetchBatch with getBatchAsync and add createIfAbsentBatchAsync. PiperOrigin-RevId: 190471980
* Don't check for deps being done in a keep-going build as we expect this to ↵Gravatar shreyax2018-03-20
| | | | | | always be the case. PiperOrigin-RevId: 189792299
* Automated rollback of commit 72d28f3efc2842510a34cacd930c0204143f7412.Gravatar shreyax2018-03-19
| | | | | | | | | | | | | | | | | | | | | | | | Fix skylark caching to properly include transitive dependencies when there is a diamond-like dependency in the loaded bzl files. Also add guards to make sure we're not attempting to cache skylark files that transitively request a dependency that is in error. *** Reason for rollback *** Looking for source of non-determinism *** Original change description *** Automated rollback of commit 7ba939dfd5df48903929e9c14ebd0449656403e4. *** Reason for rollback *** Likely cause for non-determinism in skyframe *** Original change description *** Cache SkylarkLookupImportValues in memory so that we don't recompute them multiple times. PiperOrigin-RevId: 189686604
* Automated rollback of commit 7ba939dfd5df48903929e9c14ebd0449656403e4.Gravatar shreyax2018-03-12
| | | | | | | | | | | | *** Reason for rollback *** Likely cause for non-determinism in skyframe *** Original change description *** Cache SkylarkLookupImportValues in memory so that we don't recompute them multiple times. PiperOrigin-RevId: 188729929
* Cache SkylarkLookupImportValues in memory so that we don't recompute them ↵Gravatar shreyax2018-03-05
| | | | | | multiple times. PiperOrigin-RevId: 187941859
* Replace LegacySkyKey by AbstractSkyKey or custom SkyKeys. AbstractSkyKey ↵Gravatar janakr2018-03-02
| | | | | | | | doesn't save memory in the 32-bit case, but makes it easier for people to see how many SkyKeys we have. There's some unnecessary interning in tests, but it was easier to copypasta and doesn't harm anything, I think. PiperOrigin-RevId: 187694309
* Optimize GC usage of iterating over all elements of GroupedLists when we ↵Gravatar shreyax2018-03-02
| | | | | | don't care about the group structure, and simplify the logic for prefetching old deps. PiperOrigin-RevId: 187681887
* Deletes AutoCodec.Strategy.SINGLETON now that we have @AutoCodec field tags.Gravatar shahan2018-02-28
| | | | | | This also gets rid of some static initialization cycles which we should try very hard to avoid in the future. PiperOrigin-RevId: 187334087
* Tags mapped singletons in SkyValueEncoderGravatar shahan2018-02-27
| | | | PiperOrigin-RevId: 187209783
generated by cgit v1.2.3 (git 2.39.1) at 2024-06-24 01:19:05 +0000