aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/skyframe/ParallelEvaluator.java
Commit message (Collapse)AuthorAge
* Refactors ParallelEvaluator to support eval-wide exceptions.Gravatar mschaller2018-02-01
| | | | | | | Also clarifies a comment on preventNewEvaluations. RELNOTES: None. PiperOrigin-RevId: 184198568
* ParallelEvaluator: report events early for cache hitsGravatar ulfjack2017-12-21
| | | | | | | | | | | | | | | | | | In the case that a node is already done when evaluation starts, we now report events and postables early, rather than waiting until the end of evaluation. This makes reporting more timely, and ensures reporting even if the evaluation is interrupted. This caused a problem with moving the TargetCompleteEvent into Skyframe (unknown commit). I added two unit tests at the Skyframe level to cover the guarantees that we need for that. Note that the replay call in constructResult can duplicate the events from a cache hit - this is not a problem since the replaying visitor automatically removes duplicates (and it wasn't obvious which keys correspond to cache hits). PiperOrigin-RevId: 179788157
* Replace all usages of Blaze's Preconditions class with guava.Gravatar tomlu2017-11-09
| | | | | | | | Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own. This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually. PiperOrigin-RevId: 175033526
* Remove least important argument to 5+ arg Preconditions#checkState/checkNotNull.Gravatar tomlu2017-11-08
| | | | | | This change prepares a move to Guava's preconditions. Guava only has vararg-avoiding overloads up to 4 args. PiperOrigin-RevId: 175015502
* Don't require --keep_going to discard graph edges. It's unnecessary.Gravatar janakr2017-11-02
| | | | PiperOrigin-RevId: 174202685
* Bubble errors up even in the case of keep_going builds that failed due to ↵Gravatar janakr2017-10-07
| | | | | | | | catastrophes. Our stricter behavior in the face of errors means that it is no longer possible for a done node to depend on a not-done node in this build. This opens up the possibility to discard graph edges on all --batch builds, or at least those with --discard_analysis_cache. PiperOrigin-RevId: 171375405
* Introduce AbstractParallelEvaluatorGravatar mschaller2017-08-07
| | | | | | | | | | This change enables alternate evaluation strategies. Drive-by fix to GraphTester, making TestFunction static because it has no dependencies on its outer class, and adding #unsetConstantValue. RELNOTES: None. PiperOrigin-RevId: 164327055
* Now that SkyKey is an interface, accept Iterable<? extends SkyKey>Gravatar ulfjack2017-07-19
| | | | | | | A subsequent CL makes TargetPatternKey implement SkyKey, and it's much nicer if I can pass lists of TargetPatternKey to the various Skyframe APIs. PiperOrigin-RevId: 162359843
* Fix Postable forwarding and replayGravatar ulfjack2017-07-18
| | | | | | | We were previously duplicate-posting Postable events posted to the Skyframe environment. PiperOrigin-RevId: 162323598
* A bunch of unrelated cleanups:Gravatar nharmata2017-06-16
| | | | | | | | | | -Have SkylarkImportLookupFunction include causes in the SkyFunctionExceptions it throws. -Better transitive skyframe error declarations in ASTFileLookupFunction. -Have ErrorInfo differentiate between direct and transitive transience. -Introduce ErrorInfoManager and have ParallelEvaluator/ParallelEvaluatorContext use it. RELNOTES: None PiperOrigin-RevId: 159163186
* Propagate postable events further upGravatar Klaus Aehlig2017-06-14
| | | | | | | | | | | | With the introduction of the ExtendedEventHandler, SkyFunctions were given the possibility to post additional Postable events in addition to the standard events. As SkyFunctions have to be restartable, events are collected first and only posted after the function is finished. Make sure that this also applies to postable events and they are not dropped. Change-Id: Ie1c3a0134935c75ea984fa2cc924e7327a9da81f PiperOrigin-RevId: 158964337
* Avoid waste in registerNewlyDiscovered... when there aren't anyGravatar mschaller2017-05-22
| | | | | RELNOTES: None. PiperOrigin-RevId: 156553687
* Add a EvaluationProgressReceiver#computing method.Gravatar nharmata2017-05-11
| | | | | RELNOTES: None PiperOrigin-RevId: 155665128
* Allow graph implementations to filter out deps that are known to be done ↵Gravatar janakr2017-04-10
| | | | | | when change pruning. This can speed up change pruning. PiperOrigin-RevId: 152538144
* Stop storing reverse deps to signal in BuildingState. Instead, re-use the ↵Gravatar Janak Ramakrishnan2017-03-24
| | | | | | | | | | | | | | reverseDepsToConsolidate field in InMemoryNodeEntry. As part of that, revamp our logic of how we store pending operations: store adds bare on initial evaluations, and checks bare on incremental evaluations and operations on done nodes. This should improve performance in two ways: BuildingState loses two fields, saving working memory intra-build. Storing pending reverse dep operations bare also saves memory intra-build. Note that neither of these changes helps resting memory state, only while a node is still evaluating. Because of this, we can simplify ReverseDepsUtil a bit, making ReverseDepsUtilImpl a static class, which it always wanted to be (what it really wants to be is a superclass of InMemoryNodeEntry, but I don't want to spend the object alignment bits). Finally, this makes it pretty tempting to get rid of BuildingState altogether on initial evaluations. We'd still keep DirtyBuildingState, but we could save another ~24 bytes by storing BuildingState's one remaining field, signaledDeps, directly inside InMemoryNodeEntry. -- PiperOrigin-RevId: 151048879 MOS_MIGRATED_REVID=151048879
* Tighten invariants around dirtiness checking. We should never need to create ↵Gravatar Janak Ramakrishnan2017-03-06
| | | | | | | | a child that's being checked by a parent (that was a legacy of when we delegated to enqueueChild), and such a child that is being checked should always be dirty or done, never fresh. -- PiperOrigin-RevId: 149136909 MOS_MIGRATED_REVID=149136909
* Provide more reporting options to SkyFunctions Gravatar Klaus Aehlig2017-02-27
| | | | | | | | | | | | | With more specific information to be reported by Skyfunctions, e.g., to inform the build-event protocol on missing files, the EventHandler interface is no longer enough. Therefore, provide an enriched context for reporting events. -- Change-Id: I2d06166fe4d5b9054e24ad8c752fafc039e3f9f8 Reviewed-on: https://cr.bazel.build/8794 PiperOrigin-RevId: 148463437 MOS_MIGRATED_REVID=148463437
* Inform progress receiver that node has been computed before any exception ↵Gravatar Janak Ramakrishnan2017-02-13
| | | | | | | | handling. -- PiperOrigin-RevId: 147185674 MOS_MIGRATED_REVID=147185674
* Add some logging in case we fail an evaluation. This should be generally ↵Gravatar Janak Ramakrishnan2017-01-13
| | | | | | | | useful, but may also help identify Bazel as the culprit in the linked bug if we're lucky and the timestamps are clear. -- PiperOrigin-RevId: 144359628 MOS_MIGRATED_REVID=144359628
* Stop storing a set in GroupedListHelper to deduplicate SkyKey dep requests. ↵Gravatar Janak Ramakrishnan2016-10-14
| | | | | | | Instead, deduplicate when the helper is actually added to a GroupedList. -- MOS_MIGRATED_REVID=136145321
* EvaluationProgressReceiver absorbs other progress trackers.Gravatar Chloe Calvarin2016-10-06
| | | | | | | | | | Remove DirtyKeyTracker and inflight node tracking as independent progress tracking bits of code, and instead use an internal-facing DirtyTrackingProgressReceiver wrapper of the user's EvaluationProgressReceiver. -- MOS_MIGRATED_REVID=135291924
* Allow reverse dependency lookups on Skyframe graph to throw InteruptedExceptionGravatar Googler2016-09-14
| | | | | -- MOS_MIGRATED_REVID=132999234
* Allow cycle detector to be injected into ParallelEvaluator, so that callers ↵Gravatar Janak Ramakrishnan2016-08-26
| | | | | | | can use different cycle detection algorithms if they wish. -- MOS_MIGRATED_REVID=131347160
* Factor out cycle detection.Gravatar Janak Ramakrishnan2016-08-26
| | | | | -- MOS_MIGRATED_REVID=131344134
* Refactor ParallelEvaluator in preparation for making it more modular with ↵Gravatar Janak Ramakrishnan2016-08-26
| | | | | | | | | | | respect to cycle checking. Reducing the size of ParallelEvaluator.java is also probably long overdue. I believe this change stands on its own, but if you don't think the third change is worth it, and this isn't worth it on its own, feel free to push back. -- MOS_MIGRATED_REVID=131340165
* Avoid copying SkyKeys into a set and then into a map when retrieving them. ↵Gravatar Janak Ramakrishnan2016-08-23
| | | | | | | | | | | | | Instead, just put them directly into a map. This avoids the memory churn and CPU cost of the set. As a result, we have to use HashMaps instead of ImmutableMap.Builders, which I hope is ok (especially since we're not keeping them around), and due to that, we have some nice nondeterminism in the returned order, which matters for some cycle-checking tests. Also, don't use a map at all when we don't need to (when building events). Note that, since we have to deduplicate at some point, this means that changing the return type of SkyFunction.Environment#getValues to not be a random-access map is probably not worth it. Changing the return type of ProcessableGraph#getBatch to not be a random access map might still be worthwhile, although it might require some funny operations. -- MOS_MIGRATED_REVID=131070418
* Allow Skyframe graph lookups and value retrievals to throw InterruptedException.Gravatar Janak Ramakrishnan2016-08-16
| | | | | | | The only place we now don't handle InterruptedException is in the action graph created after analysis, since I'm not sure that will be around for that much longer. -- MOS_MIGRATED_REVID=130327770
* Have AQV propagate the most severe error encountered by any of its tasks. ↵Gravatar Nathan Harmata2016-08-12
| | | | | | | | | Make sure that ParallelEvaluator treats SchedulerExceptions as less severe than other RuntimeExceptions (it already did, but add a comment emphasizing this). The combination of the above means that we don't ignore hypothetical crashes in ParallelEvaluator worker threads _after_ a SchedulerException (e.g. due to an error in nokeep_going evaluation). -- MOS_MIGRATED_REVID=130044230
* Fix bug in ParallelEvaluator when an error dep was not newly requested in a ↵Gravatar Janak Ramakrishnan2016-08-11
| | | | | | | | | nokeep_going build because it finished building in between the time it was first requested and when we checked it for done-ness after the SkyFunction evaluation. This results in an IllegalStateException that gets ignored (and, importantly, not propagated) by AbstractQueueVisitor because AbstractQueueVisitor only records and propagates the first Throwable encountered. For nokeep_going evaluations, this will be the SchedulerException that we use for control flow. The IllegalStateException in question is benign in this case because it's merely from a Preconditions failure and doesn't leave anything in a bad state. It's possible, though, that we have other bugs that are being masked in this way. -- MOS_MIGRATED_REVID=129919336
* Delete NodeEntryField since it's now superfluous in the presence of the new ↵Gravatar Nathan Harmata2016-08-03
| | | | | | | QueryableGraph.Reason which conveys more information. Add a few more Reason enum values to make this refactor benign. -- MOS_MIGRATED_REVID=129118462
* Augment the QueryableGraph#get[BatchWithFieldHints] method to take in ↵Gravatar Nathan Harmata2016-07-27
| | | | | | | parameters conveying the requesting node (if any), the requested node(s), as well as a reason for the skyframe graph lookup. Alternate graph implementations may be interested in this information. -- MOS_MIGRATED_REVID=128496089
* In keep-going mode, don't fail-fast on the evaluation of an individual ↵Gravatar Janak Ramakrishnan2016-07-12
| | | | | | | SkyFunction if it throws an exception but still has missing deps. Instead, pretend it didn't throw, and restart it when its known deps are all done, presumably to throw the same exception. This removes a basic source of non-determinism. -- MOS_MIGRATED_REVID=127129202
* Replace QueryableGraph#getBatch with #getBatchWithFieldHints. This allows ↵Gravatar Janak Ramakrishnan2016-07-11
| | | | | | | alternate graph implementations to optimize how they construct node entries. -- MOS_MIGRATED_REVID=126932020
* Filter out already-removed deps when removing a node from the rdeps of its ↵Gravatar Janak Ramakrishnan2016-07-04
| | | | | | | last build's deps. This is only an issue with cycles, since there we try to maintain the invariant that a parent is not done before its children by removing its deps on its children before we construct its cycle value. -- MOS_MIGRATED_REVID=126494009
* Batch prefetch now also retrieves the deps from the node's previous run.Gravatar Eric Fellheimer2016-07-01
| | | | | -- MOS_MIGRATED_REVID=126344189
* Batchify child node lookups in Skyframe cycle detection.Gravatar Eric Fellheimer2016-06-21
| | | | | -- MOS_MIGRATED_REVID=125376554
* Minor change to add QueryableGraphBackedSkyFunctionEnvironment.Gravatar Janak Ramakrishnan2016-06-21
| | | | | -- MOS_MIGRATED_REVID=125362963
* Minor improvement to cycle detection algorithm: Do not recurse into done ↵Gravatar Eric Fellheimer2016-06-21
| | | | | | | | | | | nodes, where there is no work to do anyway. This triggered some non-determinism that we explicitly workaround in the unit tests. Also add a comment about a potential but unrelated optimization. -- MOS_MIGRATED_REVID=125355303
* Log how long Skyframe cycle checking takes.Gravatar Eric Fellheimer2016-06-15
| | | | | -- MOS_MIGRATED_REVID=124874002
* Remove ability of AbstractQueueVisitor to continue after an interrupt. That ↵Gravatar Janak Ramakrishnan2016-06-15
| | | | | | | functionality was only used in tests. -- MOS_MIGRATED_REVID=124841573
* Allow AQV users to inject arbitrary handling of classified errors.Gravatar Nathan Harmata2016-05-27
| | | | | -- MOS_MIGRATED_REVID=123347295
* Fix bug in lazy removal of reverse deps -- there was one remaining case ↵Gravatar Janak Ramakrishnan2016-05-18
| | | | | | | where we weren't checking to see if a reverse dep already existed when we declared a reverse dep. -- MOS_MIGRATED_REVID=122581019
* Remove reverse deps lazily, only when the node has finished building and we ↵Gravatar Janak Ramakrishnan2016-05-17
| | | | | | | | | | | discover that it no longer has certain deps. In the common case, where a node's deps do not change in the end, this reduces lock contention and CPU. The downside of this is that we now create a set of the previous reverse deps during each evaluation of a node. We don't store this set in order to conserve memory, so we pay for it in CPU. We will probably only construct it two or three times (most SkyFunctions don't have so many groups), so the cost shouldn't be so high, but we can try to mitigate if it shows up in profiling. -- MOS_MIGRATED_REVID=122566267
* Stop converting temporary direct deps to a set. In almost all cases, this ↵Gravatar Janak Ramakrishnan2016-05-16
| | | | | | | conversion is unnecessary and wasteful. In the remaining cases, the set conversion can be explicit. -- MOS_MIGRATED_REVID=122294939
* Allow Skyframe tests to use the NotifyingGraph and DeterministicGraph ↵Gravatar Janak Ramakrishnan2016-05-11
| | | | | | | helpers to enforce concurrency synchronization points and determinism even if they are not using an InMemoryGraph-backed evaluator. -- MOS_MIGRATED_REVID=121977783
* Fix keep-going build with an existing cycle by only setting errorDepKey in a ↵Gravatar Janak Ramakrishnan2016-03-29
| | | | | | | non-keep-going build. Setting errorDepKey was only mostly harmless, to mix memes. (Actually, it was quite harmful.) -- MOS_MIGRATED_REVID=118410594
* Fix debugging info when crashing with missing SkyFunction.Gravatar Janak Ramakrishnan2016-03-23
| | | | | -- MOS_MIGRATED_REVID=117894693
* Transform the getBatch result in SkyFunctionEnvironment instead of copying ↵Gravatar Janak Ramakrishnan2016-03-22
| | | | | | | it. The copying showed up as a source of memory spikiness. -- MOS_MIGRATED_REVID=117741939
* Use static creation method for SkyKey. This allows interning SkyKeys as they ↵Gravatar Janak Ramakrishnan2016-03-03
| | | | | | | are created, as opposed to when they are requested from the ParallelEvaluator. That delay can lead to large memory spikes and churn. -- MOS_MIGRATED_REVID=116224565
* Improve the error message when the ParallelEvaluator is unable to create a ↵Gravatar Luis Fernando Pino Duque2016-02-28
| | | | | | | SkyFunction -- MOS_MIGRATED_REVID=115671161
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-02 06:39:09 +0000