aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/skyframe/InMemoryNodeEntry.java
Commit message (Collapse)AuthorAge
* Set the version of a computed node to the max of its child versions rather ↵Gravatar janakr2018-07-11
| | | | | | | | | | | | | | | | | than the graph version when that is feasible. * It's not feasible when the computation accesses outside state, i.e. is non-hermetic, so see below. * It's also more complicated (and not worth the trouble) when the computation is taking place just for the error status. Have SkyFunctionName declare whether the function it corresponds to is hermetic or non-hermetic. Only non-hermetically-generated SkyValues can be directly marked changed, and non-hermetic SkyFunctions have their values saved at the graph version, not the max of the child versions. All SkyFunctions are hermetic except for the ones that can be explicitly dirtied. A marked-hermetic SkyFunction that has a transient error due to filesystem access can be re-evaluated and get the correct version: if it throws an IOException at version 1 and then, when re-evaluated at version 2 with unchanged dependencies, has a value, the version will be version 1. All Skyframe unit tests that were doing non-hermetic things to nodes need to declare that those nodes are non-hermetic. I tried to make the minimal set of changes there, so that we had good incidental coverage of hermetic+non-hermetic nodes. Also did some drive-by clean-ups around that code. Artifacts are a weird case, since they're doing untracked filesystem access (for source directories). Using max(child versions) for them gives rise to the following correctness bug: 1. do a build at v1 that creates a FileStateValue for dir/ at v1. Then at v2, add a file to dir/ and do a build that consumes dir/ as a source artifact. Now the artifact for dir/ will (incorrectly) have v1. Then at v1, do that build again. We'll consume the "artifact from the future". However, this can only have an effect when using the local action cache, since the incorrect value of the artifact (the mtime) is only consumed by the action cache. Bazel is already broken in this way (incremental builds don't invalidate directories), so this change doesn't make things worse. PiperOrigin-RevId: 204210719
* Convert directDeps to a map of SkyValuesGravatar mschaller2018-06-28
| | | | | | | | | | | | | | | | | | | | SkyFunctionEnvironment only cares about directDeps' values, not other NodeEntry data. This reduces the space of code which could be sensitive to nodes which transition from done to dirty during evaluation. To prevent check-then-act races in the refactored code (and only there; other code will be fixed in future refactorings), instead of checking deps' isDone() methods before accessing their value, allow getValueMaybeWithMetadata to be called when not done, and have it return null when not done. (Note that done->dirty node transitions during evaluation are planned, but not yet possible.) RELNOTES: None. PiperOrigin-RevId: 202518781
* Permit marking dirty/changed a node more than onceGravatar mschaller2018-06-18
| | | | | | | | | | | | This functionality will be useful for node restarting. More than one parent node may request the restart of a shared child node, and this should not fail. Instead, the returned MarkedDirtyResult indicates whether the dirtying was redundant, and the calling code can assert on that. RELNOTES: None. PiperOrigin-RevId: 201005663
* Add hacky method to InMemoryNodeEntry for fast but unsafe iteration over ↵Gravatar janakr2018-06-04
| | | | | | in-progress reverse deps. PiperOrigin-RevId: 199209256
* Allow SkyFunctions to return a sentinel value indicating that all of a ↵Gravatar janakr2018-04-30
| | | | | | | | | | node's in-progress data should be forgotten, and its evaluation should be restarted from scratch, as if it were freshly created/dirtied. To guard against this happening unexpectedly, any such events are passed to a GraphInconsistencyReceiver, which can verify that the SkyFunction is behaving properly. This is the first change in a series to permit action rewinding when it is discovered that a previously generated input file is no longer available. When an action detects that one of its inputs is unusable, it can return this sentinel value, causing it to be re-evaluated from scratch. Follow-up changes will make the node corresponding to the input, and the node corresponding to the action that generated the input, dirty when this happens, causing the upstream action to be re-run, regenerating the desired input. Currently works for builds that do not keep edges, although follow-ups may make this possible for all builds. PiperOrigin-RevId: 194863097
* Fix accidental eager expansion of grouped list in in-memory nodes.Gravatar felly2018-03-27
| | | | PiperOrigin-RevId: 190656902
* Optimize GC usage of iterating over all elements of GroupedLists when we ↵Gravatar shreyax2018-03-02
| | | | | | don't care about the group structure, and simplify the logic for prefetching old deps. PiperOrigin-RevId: 187681887
* Remove synchronization from InMemoryNodeEntry#getValue{,MaybeWithMetadata} ↵Gravatar janakr2018-01-10
| | | | | | and #toValue. I saw significant contention from this method in some experiments, and the synchronization isn't needed, since we only call these methods on done nodes (as determined by #isDone), and a node that is observably done cannot racily change its value. PiperOrigin-RevId: 181507133
* Add a static method GroupedList#numElements to allow to count the number of ↵Gravatar shreyax2017-11-29
| | | | | | deps from a compressed GroupedList without uncompressing it. Also some minor GC improvements. PiperOrigin-RevId: 177338852
* Allow InMemoryNodeEntry subclasses to change which reverse dep operation to ↵Gravatar janakr2017-11-21
| | | | | | store bare. PiperOrigin-RevId: 176505963
* Replace all usages of Blaze's Preconditions class with guava.Gravatar tomlu2017-11-09
| | | | | | | | Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own. This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually. PiperOrigin-RevId: 175033526
* Push NodeEntry#keepEdges down to InMemoryNodeEntry. It's not needed on the ↵Gravatar janakr2017-10-18
| | | | | | general interface. PiperOrigin-RevId: 172606623
* Allow NodeEntry implementations to keep just deps, as opposed to all edges ↵Gravatar janakr2017-10-10
| | | | | | or no edges. Also add option to disable checks in MemoizingEvaluatorTest that don't make sense for implementations that don't keep track of dirty nodes. Also extract RecordingDifferencer to an interface. And add a test for the situation that a node changes during a build that it's not requested, and which fails, necessitating cleanup. PiperOrigin-RevId: 171616817
* Switch from using Iterable to Collection in the return type to be more explicit.Gravatar shreyax2017-09-20
| | | | PiperOrigin-RevId: 169278760
* Remove BuildingState, since it only has one field. Instead, keep the ↵Gravatar Janak Ramakrishnan2017-03-27
| | | | | | | | | | signaledDeps field directly in InMemoryNodeEntry. Should save ~24 bytes per freshly evaluating node entry (I haven't calculated object alignment for InMemoryNodeEntry now, so could be more or less). Also might save some memory for re-evaluating node entries, since the BuildingState class had to be padded out to a multiple of 8 bytes before the DirtyBuildingState fields could start. Don't actually know if that was happening. -- PiperOrigin-RevId: 151138224 MOS_MIGRATED_REVID=151138224
* Stop storing reverse deps to signal in BuildingState. Instead, re-use the ↵Gravatar Janak Ramakrishnan2017-03-24
| | | | | | | | | | | | | | reverseDepsToConsolidate field in InMemoryNodeEntry. As part of that, revamp our logic of how we store pending operations: store adds bare on initial evaluations, and checks bare on incremental evaluations and operations on done nodes. This should improve performance in two ways: BuildingState loses two fields, saving working memory intra-build. Storing pending reverse dep operations bare also saves memory intra-build. Note that neither of these changes helps resting memory state, only while a node is still evaluating. Because of this, we can simplify ReverseDepsUtil a bit, making ReverseDepsUtilImpl a static class, which it always wanted to be (what it really wants to be is a superclass of InMemoryNodeEntry, but I don't want to spend the object alignment bits). Finally, this makes it pretty tempting to get rid of BuildingState altogether on initial evaluations. We'd still keep DirtyBuildingState, but we could save another ~24 bytes by storing BuildingState's one remaining field, signaledDeps, directly inside InMemoryNodeEntry. -- PiperOrigin-RevId: 151048879 MOS_MIGRATED_REVID=151048879
* Refactor DirtyBuildingState to allow custom implementation on storing theGravatar Googler2017-02-01
| | | | | | | | dependencies used in last build -- PiperOrigin-RevId: 146169454 MOS_MIGRATED_REVID=146169454
* Unsynchronize InMemoryNodeEntry#isDone. The buildingState variable's state ↵Gravatar Shreya Bhattarai2017-01-09
| | | | | | | | changes are visibile to all other threads by volatility and we don't read the variable at any intermediate or inconsistent state (simply check against null or a constant). reverseDepsToSignal is also made volatile for subclasses that need volatile reads to it. -- PiperOrigin-RevId: 143787032 MOS_MIGRATED_REVID=143787032
* Stop storing a set in GroupedListHelper to deduplicate SkyKey dep requests. ↵Gravatar Janak Ramakrishnan2016-10-14
| | | | | | | Instead, deduplicate when the helper is actually added to a GroupedList. -- MOS_MIGRATED_REVID=136145321
* Allow Skyframe graph lookups and value retrievals to throw InterruptedException.Gravatar Janak Ramakrishnan2016-08-16
| | | | | | | The only place we now don't handle InterruptedException is in the action graph created after analysis, since I'm not sure that will be around for that much longer. -- MOS_MIGRATED_REVID=130327770
* Refactor BuildingState to save memory.Gravatar Janak Ramakrishnan2016-07-07
| | | | | | | | | Collapse the "evaluating" boolean into the "signaledDeps" int field, since signaledDeps is always 0 if evaluating is false, so we can use the sentinel value -1 to indicate that evaluation has not yet started. This leads to a slightly less tolerant node entry: it must "start evaluating" before you can do things like set its value. Places that wasn't being done have been fixed, at least as far as we have test coverage for. Also, factor the "dirty" parts of BuildingState out into a subclass. It would probably be cleaner to use composition here, but I don't want to pay the price of another object. -- MOS_MIGRATED_REVID=126729331
* Re-use the InMemoryNodeEntry#directDeps field for temporary direct deps.Gravatar Janak Ramakrishnan2016-07-07
| | | | | | | The biggest savings here is that we were not eagerly discarding the InMemoryNodeEntry#directDeps field after an entry was marked dirty, even though we would never read its value again. But rather than just fix that, by getting rid of the field in BuildingState, we can potentially save memory with smaller BuildingState objects as well. -- MOS_MIGRATED_REVID=126709632
* Minor documentation and visibility tweaks to InMemoryNodeEntry and ↵Gravatar Janak Ramakrishnan2016-06-29
| | | | | | | InvalidatedNodeEntry. -- MOS_MIGRATED_REVID=126139179
* Get rid of boolean field in InMemoryNodeEntry. After adding the ↵Gravatar Janak Ramakrishnan2016-06-28
| | | | | | | lastEvaluated/lastChanged version fields, we lost memory alignment, so this boolean was costing us 8 bytes per instance. -- MOS_MIGRATED_REVID=125998857
* Remove reverse deps lazily, only when the node has finished building and we ↵Gravatar Janak Ramakrishnan2016-05-17
| | | | | | | | | | | discover that it no longer has certain deps. In the common case, where a node's deps do not change in the end, this reduces lock contention and CPU. The downside of this is that we now create a set of the previous reverse deps during each evaluation of a node. We don't store this set in order to conserve memory, so we pay for it in CPU. We will probably only construct it two or three times (most SkyFunctions don't have so many groups), so the cost shouldn't be so high, but we can try to mitigate if it shows up in profiling. -- MOS_MIGRATED_REVID=122566267
* Stop converting temporary direct deps to a set. In almost all cases, this ↵Gravatar Janak Ramakrishnan2016-05-16
| | | | | | | conversion is unnecessary and wasteful. In the remaining cases, the set conversion can be explicit. -- MOS_MIGRATED_REVID=122294939
* Don't check direct deps when doing change pruning. Since dependents of a ↵Gravatar Janak Ramakrishnan2016-02-09
| | | | | | | node don't have access to its dependencies ("grandparents don't know grandchildren", or vice versa, depending on your point of view), changes to a node's dependencies can't affect downstream nodes. -- MOS_MIGRATED_REVID=114143894
* Intern IntVersions.Gravatar Janak Ramakrishnan2016-01-28
| | | | | -- MOS_MIGRATED_REVID=113197641
* Use Bazel Preconditions variant which avoids varargs array creationGravatar Mark Schaller2015-12-10
| | | | | | | Reduces garbage. -- MOS_MIGRATED_REVID=109914243
* Avoid list copy in BuildingState.getNextDirtyDirectDepsGravatar Mark Schaller2015-12-09
| | | | | | | | | Also, in GroupedList, short-circuit expensive group equality check with a reference check, saving time and garbage when groups are the same object. -- MOS_MIGRATED_REVID=109795332
* Avoid extra rdeps copy when finishing nodeGravatar Mark Schaller2015-11-24
| | | | | | | Reduces garbage and wasted work -- MOS_MIGRATED_REVID=108566164
* Extract ReverseDepsUtil interface so that InMemoryNodeEntry can be partially ↵Gravatar Janak Ramakrishnan2015-11-24
| | | | | | | isolated from implementation details. -- MOS_MIGRATED_REVID=108523104
* Avoid re-evaluating a parent node when a child is found to be unchanged from ↵Gravatar Janak Ramakrishnan2015-11-19
| | | | | | | | | | | an earlier version at which the child changed but the parent did not. Concrete scenario: Parent depends on Child. We first evaluate at version v1, Child has value A1, Parent has value B1. We then evaluate at version v2, which changes a dependency of Child. Child has value A2, and Child.getVersion() returns v2. Parent re-evaluates to B1, so is unchanged. Parent.getVersion() returns v1. Now evaluate at version v3, which also changes a dependency of Child. Child re-evaluates to A2, so Child.getVersion() returns v2. If we signal Parent with v2 and Parent only knows that it is at version v1, then Parent must unnecessarily re-evaluate. To fix this, we store an additional version in the entry -- the version at which the node was last evaluated, even if the evaluation did not result in a new value. Parent can then compare that version to its children's versions. If that version is at least as recent as their versions, it knows that the result of evaluating will be the same as it was at that last evaluated version, which is its current value. An alternative solution might be to just signal the parent with a boolean, saying whether or not the child was changed on this evaluation. However, this would be incorrect in the scenario above, with the modification that in the second evaluation, the user just requests the value of Child -- Parent is not updated. In that case, during the third evaluation, Child would report that it was not changed during this evaluation, but we must still re-evaluate Parent since it has not yet picked up the value of Child from the earlier build. -- MOS_MIGRATED_REVID=108163443
* Return rdeps when marking a node dirtyGravatar Mark Schaller2015-11-18
| | | | | | | | | | The thread that succeeds at marking a node dirty during invalidation must then schedule that node's reverse deps for invalidation. Providing the set of reverse deps as a return value from marking a node dirty makes some future optimizations possible. -- MOS_MIGRATED_REVID=108045473
* Delay additions as well as removals of reverse deps. Now that removals are ↵Gravatar Janak Ramakrishnan2015-10-12
| | | | | | | | | | | not all done during invalidation, repeated adding/removing means that we are consolidating more often, negating the benefit of delayed removals. To work around this, delay adds as well until we consolidate and verify the integrity of our data. Since there is no well-defined point that a consolidation should trigger for a done node, we delay until our pending list is as large as the done list. We can tweak this if necessary for a memory/performance tradeoff. The alternative to this that I could think of is giving up our strong integrity checks, which I'm not a fan of. -- MOS_MIGRATED_REVID=105095886
* Be more specific about InMemoryNodeEntry.getGroupedDirectDepsGravatar Mark Schaller2015-10-09
| | | | | -- MOS_MIGRATED_REVID=104983790
* Increase visibility of an InMemoryNodeEntry methodGravatar Mark Schaller2015-10-08
| | | | | | | | | | For use in alternate graph implementations. Also adds some user-friendly methods to StringValue, a SkyValue implementation used in tests. -- MOS_MIGRATED_REVID=104899226
* Simplify buildingState check in markCleanGravatar Mark Schaller2015-09-30
| | | | | | | The value getting checked was the same as the value it was compared to. -- MOS_MIGRATED_REVID=104144544
* Rationalize copyright headersGravatar Damien Martin-Guillerez2015-09-25
| | | | | | | | | | | The headers were modified with `find . -type f -exec 'sed' '-Ei' 's|Copyright 201([45]) Google|Copyright 201\1 The Bazel Authors|' '{}' ';'` And manual edit for not Google owned copyright. Because of the nature of ijar, I did not modified the header of file owned by Alan Donovan. The list of authors were extracted from the git log. It is missing older Google contributors that can be added on-demand. -- MOS_MIGRATED_REVID=103938715
* Don't remove reverse deps until node is known to be changed. This helps ↵Gravatar Janak Ramakrishnan2015-09-23
| | | | | | | avoid mutating the deps of nodes that are still going to be deps after evaluation is finished. -- MOS_MIGRATED_REVID=103659429
* Strictly enforce reverse deps removals.Gravatar Janak Ramakrishnan2015-09-11
| | | | | | | Now that we don't clean the graph during catastrophes, we should be crashing hard in these cases. -- MOS_MIGRATED_REVID=102785226
* Delay cleaning of in-flight nodes until the following build. This allows us ↵Gravatar Janak Ramakrishnan2015-09-11
| | | | | | | | | to interrupt evaluation in constant time. Some ParallelEvaluator tests that implicitly relied on cleaning happening before the next evaluation were moved into MemoizingEvaluatorTest as a result. -- MOS_MIGRATED_REVID=102696653
* Return raw SkyValue instead of wrapped ValueWithMetadata.Gravatar Janak Ramakrishnan2015-08-25
| | | | | -- MOS_MIGRATED_REVID=101377320
* Fix some nits and allow alternate evaluators to use MemoizingEvaluatorTest.Gravatar Janak Ramakrishnan2015-08-21
| | | | | -- MOS_MIGRATED_REVID=101150250
* Don't return SkyValue when dirtying nodeGravatar Mark Schaller2015-08-11
| | | | | | | | The invalidator is no longer using the SkyValue, so there's no need to return it when dirtying a node. -- MOS_MIGRATED_REVID=100307211
* Change return type of InMemoryNodeEntry#getReverseDeps to Collection<SkyKey>.Gravatar Janak Ramakrishnan2015-08-05
| | | | | -- MOS_MIGRATED_REVID=99943491
* Modify BuildingState to allow for alternative implementations.Gravatar Janak Ramakrishnan2015-08-04
| | | | | -- MOS_MIGRATED_REVID=99747336
* Replaced other occurrences of Objects#toStringHelper with ↵Gravatar Googler2015-07-17
| | | | | | | MoreObjects#toStringHelper -- MOS_MIGRATED_REVID=98383075
* Revert the recent changes that made some skyframe-internal data structures ↵Gravatar Nathan Harmata2015-03-10
| | | | | | | serializable; alternative graph implementations no longer need these. -- MOS_MIGRATED_REVID=88003503
* Add the method InMemoryNodeEntry#getGroupedDirectDeps so callers can access ↵Gravatar Nathan Harmata2015-03-10
| | | | | | | the ordered list of grouped deps. -- MOS_MIGRATED_REVID=87969259
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-05 15:29:12 +0000