aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/skyframe
Commit message (Collapse)AuthorAge
* Refactor WalkableGraph and BuildDriver interfacesGravatar Googler2017-12-21
| | | | | | | | Remove WalkableGraph#isUpToDate and BuildDriver#alreadyEvaluated and delegate the work to implementation. RELNOTES: None PiperOrigin-RevId: 179815374
* ParallelEvaluator: report events early for cache hitsGravatar ulfjack2017-12-21
| | | | | | | | | | | | | | | | | | In the case that a node is already done when evaluation starts, we now report events and postables early, rather than waiting until the end of evaluation. This makes reporting more timely, and ensures reporting even if the evaluation is interrupted. This caused a problem with moving the TargetCompleteEvent into Skyframe (unknown commit). I added two unit tests at the Skyframe level to cover the guarantees that we need for that. Note that the replay call in constructResult can duplicate the events from a cache hit - this is not a problem since the replaying visitor automatically removes duplicates (and it wasn't obvious which keys correspond to cache hits). PiperOrigin-RevId: 179788157
* Simplify tagged event handling.Gravatar ulfjack2017-11-30
| | | | | | | | | | | | Don't make copies of Events on replay. The same events may be replayed a lot, so it's better to copy before storing the events. Also avoid a copy if the tag doesn't actually change. The intent of this change is to reduce gc churn on incremental builds. When I wrote this change (~a year ago), this was a noticable source of gc churn in some benchmarks I ran at the time. PiperOrigin-RevId: 177450696
* Add a static method GroupedList#numElements to allow to count the number of ↵Gravatar shreyax2017-11-29
| | | | | | deps from a compressed GroupedList without uncompressing it. Also some minor GC improvements. PiperOrigin-RevId: 177338852
* Allow InMemoryNodeEntry subclasses to change which reverse dep operation to ↵Gravatar janakr2017-11-21
| | | | | | store bare. PiperOrigin-RevId: 176505963
* Clear interrupted bit in thread when throwing an interrupted exception that ↵Gravatar janakr2017-11-21
| | | | | | | | came from an AbstractParallelEvaluator evaluation. It's against the standard Java contract to throw but still have the thread's interrupted bit set. Also get rid of some unnecessary initializeTester() calls in MemoizingEvaluatorTest: we already call it via a @Before annotation. PiperOrigin-RevId: 176496034
* RELNOTES: --keep_incrementality_data flag allows Bazel servers to be run in ↵Gravatar janakr2017-11-12
| | | | | | | | | | | | | | memory-saving non-incremental mode independent of --batch and --discard_analysis_cache. A command run with --nokeep_incrementality_data will discard data that would be needed for incremental builds. Subsequent commands can be sent to the same server, but they will not get the benefit of incrementality from this command. However, if --keep_incrementality_data is specified on a subsequent command, the commands after that will get the benefits of incrementality. There are two benefits to not being dependent on --batch. First, this allows Bazel servers to be run in extreme memory-saving mode without the startup penalties (JVM startup, JITting) that --batch execution imposes. Second, this allows Bazel developers to inspect the state of a Bazel server after an extreme memory-saving build. In order to avoid discarding data unnecessarily (for instance, on a "bazel info used-heap-size-after-gc" or "bazel dump --skyframe=summary") the actual resetting of the graph is done lazily, right before its use in SequencedSkyframeExecutor#sync. This is morally a partial rollback of https://github.com/bazelbuild/bazel/commit/98cd82cbdcac7c48164a611c5a9aa8fc2f1720ef. For now, our tests specify all of the flags. After this change sticks, I plan to get rid of the --batch flag from these tests, which should allow for some clean-ups. Eventually --batch and --discard_analysis_cache may not imply that we don't keep incremental state: we can require that it be specified explicitly. PiperOrigin-RevId: 175335075
* Replace all usages of Blaze's Preconditions class with guava.Gravatar tomlu2017-11-09
| | | | | | | | Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own. This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually. PiperOrigin-RevId: 175033526
* Remove least important argument to 5+ arg Preconditions#checkState/checkNotNull.Gravatar tomlu2017-11-08
| | | | | | This change prepares a move to Guava's preconditions. Guava only has vararg-avoiding overloads up to 4 args. PiperOrigin-RevId: 175015502
* Make ErrorInfo#toString more informative.Gravatar janakr2017-11-06
| | | | PiperOrigin-RevId: 174508154
* Don't require --keep_going to discard graph edges. It's unnecessary.Gravatar janakr2017-11-02
| | | | PiperOrigin-RevId: 174202685
* Fixes EvaluationResult BuilderGravatar Googler2017-10-31
| | | | | RELNOTES: None PiperOrigin-RevId: 173950304
* Push NodeEntry#keepEdges down to InMemoryNodeEntry. It's not needed on the ↵Gravatar janakr2017-10-18
| | | | | | general interface. PiperOrigin-RevId: 172606623
* Allow NodeEntry implementations to keep just deps, as opposed to all edges ↵Gravatar janakr2017-10-10
| | | | | | or no edges. Also add option to disable checks in MemoizingEvaluatorTest that don't make sense for implementations that don't keep track of dirty nodes. Also extract RecordingDifferencer to an interface. And add a test for the situation that a node changes during a build that it's not requested, and which fails, necessitating cleanup. PiperOrigin-RevId: 171616817
* Bubble errors up even in the case of keep_going builds that failed due to ↵Gravatar janakr2017-10-07
| | | | | | | | catastrophes. Our stricter behavior in the face of errors means that it is no longer possible for a done node to depend on a not-done node in this build. This opens up the possibility to discard graph edges on all --batch builds, or at least those with --discard_analysis_cache. PiperOrigin-RevId: 171375405
* Stop injecting WorkspaceStatusAction into the Skyframe graph as a ↵Gravatar janakr2017-09-26
| | | | | | | | | | | | precomputed value. Instead, manually check if the value has changed, and if it has, invalidate its consuming WorkspaceStatusValue node, forcing its re-evaluation, where it will pick up the new value. This seems more awkward than the original code, but it is more correct in spirit: injecting a precomputed value which can change even while the source state does not is a smell. Long-term, the key for the WorkspaceStatusValue should incorporate a hash of the action, and that hash should be in the configuration, just as other configuration flags are. That isn't possible right now just because we don't have configuration trimming, and we drop all nodes on configuration changes, so putting workspace status options into the configuration would lose change pruning whenever we changed workspace status options. If/when those problems are fixed, we can extend this change to have WorkspaceStatusFunction continue to request the action out-of-band, but keyed by the hash. Then we can stop invalidating stale nodes. See also https://github.com/bazelbuild/bazel/issues/3785. PiperOrigin-RevId: 169947071
* Switch from using Iterable to Collection in the return type to be more explicit.Gravatar shreyax2017-09-20
| | | | PiperOrigin-RevId: 169278760
* Automatic code cleanup.Gravatar cushon2017-09-15
| | | | PiperOrigin-RevId: 168802886
* More BUILD file refactorings.Gravatar philwo2017-09-06
| | | | | | | | | Split collect, concurrent, vfs, windows into package-level BUILD files. Move clock classes out of "util", into their own Java package. Move CompactHashSet into its own Java package to break a dependency cycle. Give nestedset and inmemoryfs their own package-level BUILD files. PiperOrigin-RevId: 167702127
* Split the cycle between vfs and profiler.Gravatar philwo2017-08-31
| | | | | | | | | | - Move ProfilerInfo into a subpackage (it's not necessary for profiling, just for analyzing a profile). - Make some fields in Profiler public for ProfileInfo. - Mark Profiler as ThreadSafe; there's no cyclic dependency here. This is based on ulfjack's microbazel patch series: https://github.com/ulfjack/bazel/commit/44553fcac0fc876784d8f48c2e577d8c999712de PiperOrigin-RevId: 167121952
* Change WalkableGraphFactory#prepareAndGet to take multiple SkyKeys as graph ↵Gravatar Googler2017-08-17
| | | | | | | | | | roots It also changes a few accessors of utility methods in Skyframe library. It refactors the QueryExpressionMapper to use a general QueryExpressionVisitor. RELNOTES: None PiperOrigin-RevId: 165534908
* Introduce AbstractParallelEvaluatorGravatar mschaller2017-08-07
| | | | | | | | | | This change enables alternate evaluation strategies. Drive-by fix to GraphTester, making TestFunction static because it has no dependencies on its outer class, and adding #unsetConstantValue. RELNOTES: None. PiperOrigin-RevId: 164327055
* Adding a ConstantVersion to Skyframe, to be used when only a single ↵Gravatar kush2017-08-03
| | | | | | | evaluation of the graph with a single constant version is expected RELNOTES: None PiperOrigin-RevId: 164059941
* Add --toolchain_resolution_debug option to give more information aboutGravatar John Cater2017-07-26
| | | | | | | | | toolchain selection. Fixes #3431. Change-Id: Ia38415575b6a121cbb6a028bfc0276691cd11b6d PiperOrigin-RevId: 163196646
* Make SkyKey.argument a default methodGravatar ulfjack2017-07-25
| | | | | | | Almost all implementations simply return this, all of which can be removed now. PiperOrigin-RevId: 163046912
* Make InMemoryGraph public.Gravatar janakr2017-07-24
| | | | PiperOrigin-RevId: 162788157
* Tolerate injected nodes having deps.Gravatar janakr2017-07-21
| | | | | | | | | | We don't check explicitly that these are the only two ways, but this can happen if the error transience node is a dep of a node that's being injected, or if an injected node is an "external" file that needs to depend on an external package. The first possibility can happen if there was an IOException reading the node on the previous build. We handle the situation by just dirtying the node, not injecting it. Actual evaluation can handle the re-stat. PiperOrigin-RevId: 162622092
* Automated rollback of commit 6e72f78e1c2e573787ec862e671f3b3b4c33dc96.Gravatar kush2017-07-20
| | | | | | | | | | | | | | *** Reason for rollback *** Causing TGP issues with tool failures: b/63839245 Was finally able to repro the issue at HEAD, and didn't occur without this change. *** Original change description *** Small changes to skyframe package. RELNOTES: None PiperOrigin-RevId: 162565994
* Now that SkyKey is an interface, accept Iterable<? extends SkyKey>Gravatar ulfjack2017-07-19
| | | | | | | A subsequent CL makes TargetPatternKey implement SkyKey, and it's much nicer if I can pass lists of TargetPatternKey to the various Skyframe APIs. PiperOrigin-RevId: 162359843
* Fix Postable forwarding and replayGravatar ulfjack2017-07-18
| | | | | | | We were previously duplicate-posting Postable events posted to the Skyframe environment. PiperOrigin-RevId: 162323598
* Small changes to skyframe package.Gravatar kush2017-07-18
| | | | PiperOrigin-RevId: 162288376
* A bunch of unrelated cleanups:Gravatar nharmata2017-06-16
| | | | | | | | | | -Have SkylarkImportLookupFunction include causes in the SkyFunctionExceptions it throws. -Better transitive skyframe error declarations in ASTFileLookupFunction. -Have ErrorInfo differentiate between direct and transitive transience. -Introduce ErrorInfoManager and have ParallelEvaluator/ParallelEvaluatorContext use it. RELNOTES: None PiperOrigin-RevId: 159163186
* Propagate postable events further upGravatar Klaus Aehlig2017-06-14
| | | | | | | | | | | | With the introduction of the ExtendedEventHandler, SkyFunctions were given the possibility to post additional Postable events in addition to the standard events. As SkyFunctions have to be restartable, events are collected first and only posted after the function is finished. Make sure that this also applies to postable events and they are not dropped. Change-Id: Ie1c3a0134935c75ea984fa2cc924e7327a9da81f PiperOrigin-RevId: 158964337
* Also report stored posts further up in SkyframeGravatar Klaus Aehlig2017-05-31
| | | | | | | | | | As SkyFunctions are supposed to be restartable, e.g., if prerequisites are missing, events generated by them are temporarily stored. So are BuildEvents posted by them. However, once the evaluation of a SkyFunction is finalized, the stored posts need to be reposted. Do this. Change-Id: I4ce20266fbfcbb298e93eb53086fa9916874f5d8 PiperOrigin-RevId: 157575119
* Avoid waste in registerNewlyDiscovered... when there aren't anyGravatar mschaller2017-05-22
| | | | | RELNOTES: None. PiperOrigin-RevId: 156553687
* Add a EvaluationProgressReceiver#computing method.Gravatar nharmata2017-05-11
| | | | | RELNOTES: None PiperOrigin-RevId: 155665128
* Introduce CompoundEvaluationProgressReceiver for combining multiple ↵Gravatar nharmata2017-05-10
| | | | | | | EvaluationProgressReceivers. RELNOTES: None PiperOrigin-RevId: 155542146
* Introduce an EvaluationProgressReceiver implementation that does nothing.Gravatar nharmata2017-05-10
| | | | | | | | This is useful for dealing with all the existing implementations in the face of interface changes that are irrelevant. RELNOTES: None PiperOrigin-RevId: 155525021
* Clean up AbstractQueueVisitor's constructors.Gravatar janakr2017-05-09
| | | | | | | | The "concurrent" bit was supposedly around for testing purposes, but who knows if it even works anymore. Making other callsites explicitly state their ErrorClassifier gets us down to two constructors, one of which can delegate to the other. I think having both these constructors is useful because there's a linkage between creating a new executor service and specifying that the AQV should shut down the service at the end of the visitation. And using a static create() method doesn't work because of AQV's inheritance model. PiperOrigin-RevId: 155406771
* Make SkyKey an interface, and start the migration of not creating SkyKey ↵Gravatar janakr2017-05-04
| | | | | | wrapper objects: for OwnedArtifacts, which are the most numerous during builds, and for Labels for TransitiveTraversalValues, which are the most numerous during queries. PiperOrigin-RevId: 154989520
* Allow graph implementations to filter out deps that are known to be done ↵Gravatar janakr2017-04-10
| | | | | | when change pruning. This can speed up change pruning. PiperOrigin-RevId: 152538144
* Drop loading-phase values if --discard_analysis_cache is true and we're not ↵Gravatar janakr2017-03-31
| | | | | | keeping incremental state. PiperOrigin-RevId: 151639711
* Remove BuildingState, since it only has one field. Instead, keep the ↵Gravatar Janak Ramakrishnan2017-03-27
| | | | | | | | | | signaledDeps field directly in InMemoryNodeEntry. Should save ~24 bytes per freshly evaluating node entry (I haven't calculated object alignment for InMemoryNodeEntry now, so could be more or less). Also might save some memory for re-evaluating node entries, since the BuildingState class had to be padded out to a multiple of 8 bytes before the DirtyBuildingState fields could start. Don't actually know if that was happening. -- PiperOrigin-RevId: 151138224 MOS_MIGRATED_REVID=151138224
* Stop storing reverse deps to signal in BuildingState. Instead, re-use the ↵Gravatar Janak Ramakrishnan2017-03-24
| | | | | | | | | | | | | | reverseDepsToConsolidate field in InMemoryNodeEntry. As part of that, revamp our logic of how we store pending operations: store adds bare on initial evaluations, and checks bare on incremental evaluations and operations on done nodes. This should improve performance in two ways: BuildingState loses two fields, saving working memory intra-build. Storing pending reverse dep operations bare also saves memory intra-build. Note that neither of these changes helps resting memory state, only while a node is still evaluating. Because of this, we can simplify ReverseDepsUtil a bit, making ReverseDepsUtilImpl a static class, which it always wanted to be (what it really wants to be is a superclass of InMemoryNodeEntry, but I don't want to spend the object alignment bits). Finally, this makes it pretty tempting to get rid of BuildingState altogether on initial evaluations. We'd still keep DirtyBuildingState, but we could save another ~24 bytes by storing BuildingState's one remaining field, signaledDeps, directly inside InMemoryNodeEntry. -- PiperOrigin-RevId: 151048879 MOS_MIGRATED_REVID=151048879
* Tighten invariants around dirtiness checking. We should never need to create ↵Gravatar Janak Ramakrishnan2017-03-06
| | | | | | | | a child that's being checked by a parent (that was a legacy of when we delegated to enqueueChild), and such a child that is being checked should always be dirty or done, never fresh. -- PiperOrigin-RevId: 149136909 MOS_MIGRATED_REVID=149136909
* Add some more debugging info in case of rdep inconsistencies in Skyframe -- ↵Gravatar Janak Ramakrishnan2017-03-03
| | | | | | | | the full list of rdep mods that are being performed on this entry, not just the current one that failed. -- PiperOrigin-RevId: 149055655 MOS_MIGRATED_REVID=149055655
* In SkyFunctionEnvironment post progress-like events immediately Gravatar Klaus Aehlig2017-02-28
| | | | | | | | | | | | | In Skyframe Evaluation, events reporting about progress are meant to be seen immediately by the environment. More over, they are not to be replayed when taking this SkyFunction out of cache. So change the custom implementation of the StoredEventHandler in the SkyFunctionEnvironment to forward appropriately. -- Change-Id: I30baf0088595ef684f16270f11668e980e65aa41 Reviewed-on: https://cr.bazel.build/9112 PiperOrigin-RevId: 148766400 MOS_MIGRATED_REVID=148766400
* Provide more reporting options to SkyFunctions Gravatar Klaus Aehlig2017-02-27
| | | | | | | | | | | | | With more specific information to be reported by Skyfunctions, e.g., to inform the build-event protocol on missing files, the EventHandler interface is no longer enough. Therefore, provide an enriched context for reporting events. -- Change-Id: I2d06166fe4d5b9054e24ad8c752fafc039e3f9f8 Reviewed-on: https://cr.bazel.build/8794 PiperOrigin-RevId: 148463437 MOS_MIGRATED_REVID=148463437
* Inform progress receiver that node has been computed before any exception ↵Gravatar Janak Ramakrishnan2017-02-13
| | | | | | | | handling. -- PiperOrigin-RevId: 147185674 MOS_MIGRATED_REVID=147185674
* Add "CLEAN_UP" as a reason for retrieving a node.Gravatar Shreya Bhattarai2017-02-09
| | | | | | -- PiperOrigin-RevId: 146982656 MOS_MIGRATED_REVID=146982656
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-04 18:28:52 +0000