aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/test/java/com/google/devtools/build/skyframe
Commit message (Collapse)AuthorAge
* Order Skyframe evaluations in a priority queue, with all children of a given ↵Gravatar janakr2018-08-13
| | | | | | | | node having the same priority, later enqueueings having higher priority, re-enqueued nodes having highest priority, and new root nodes having lowest priority. Experimentally, this can save significant RAM (1.4G in some builds!) while not affecting speed. Also do a semi-drive-by deleting ExecutorFactory parameter to AbstractQueueVisitor, since it was always AbstractQueueVisitor.EXECUTOR_FACTORY. PiperOrigin-RevId: 208560889
* Refactoring of SkyFunctionEnvironment to iterate over events/postables only ↵Gravatar janakr2018-08-10
| | | | | | | | | | | | when they're actually being put into a committed value. The previous behavior submitted deps' events twice, when the dep was added and when the node finished building. The intention is to build on this refactoring to cut off events/postables across the analysis-execution boundary, so that actions are not carrying around nested sets of warnings coming from their configured targets. This will be safe because to execute an action, we must already have analyzed its configured target, so the warning would have been emitted there. As can be seen from the changed test, this is not a pure behavior no-op. We will now emit cached events slightly later, on value committal, rather than on first dep declaration. This should not be an issue: since the events are cached, the user must have already seen them on a prior build, so the delay should not be important. Inversely, we now report events slightly more quickly during bubbling up, since we report them at each stage, as opposed to just at ParallelEvaluator evaluation completion. PiperOrigin-RevId: 208316502
* Fix crash bug in AbstractExceptionalParallelEvaluator#doMutatingEvaluation ↵Gravatar nharmata2018-07-23
| | | | | | | in a very specific window of time inbetween enqueueing one top-level node for evaluation and checking if another top-level node is done. See the added unit test for details. RELNOTES: None PiperOrigin-RevId: 205718683
* Remove gender specific prononuns from Bazel codebaseGravatar hlopko2018-07-23
| | | | | RELNOTES: None. PiperOrigin-RevId: 205635805
* PiperOrigin-RevId: 205308422Gravatar shahan2018-07-19
|
* Set the version of a computed node to the max of its child versions rather ↵Gravatar janakr2018-07-11
| | | | | | | | | | | | | | | | | than the graph version when that is feasible. * It's not feasible when the computation accesses outside state, i.e. is non-hermetic, so see below. * It's also more complicated (and not worth the trouble) when the computation is taking place just for the error status. Have SkyFunctionName declare whether the function it corresponds to is hermetic or non-hermetic. Only non-hermetically-generated SkyValues can be directly marked changed, and non-hermetic SkyFunctions have their values saved at the graph version, not the max of the child versions. All SkyFunctions are hermetic except for the ones that can be explicitly dirtied. A marked-hermetic SkyFunction that has a transient error due to filesystem access can be re-evaluated and get the correct version: if it throws an IOException at version 1 and then, when re-evaluated at version 2 with unchanged dependencies, has a value, the version will be version 1. All Skyframe unit tests that were doing non-hermetic things to nodes need to declare that those nodes are non-hermetic. I tried to make the minimal set of changes there, so that we had good incidental coverage of hermetic+non-hermetic nodes. Also did some drive-by clean-ups around that code. Artifacts are a weird case, since they're doing untracked filesystem access (for source directories). Using max(child versions) for them gives rise to the following correctness bug: 1. do a build at v1 that creates a FileStateValue for dir/ at v1. Then at v2, add a file to dir/ and do a build that consumes dir/ as a source artifact. Now the artifact for dir/ will (incorrectly) have v1. Then at v1, do that build again. We'll consume the "artifact from the future". However, this can only have an effect when using the local action cache, since the incorrect value of the artifact (the mtime) is only consumed by the action cache. Bazel is already broken in this way (incremental builds don't invalidate directories), so this change doesn't make things worse. PiperOrigin-RevId: 204210719
* Native Skyframe support for node restartingGravatar mschaller2018-07-01
| | | | | | | | | | | | Useful for attempting to recover relationships between Skyframe graph state and external systems, when the evaluation of a Skyframe node has the side effect of creating that relationship. Currently, only supported in graph evaluations when reverse dependency edges are not tracked. RELNOTES: None. PiperOrigin-RevId: 202892953
* Add new BuildMetrics event to BEP.Gravatar tomlu2018-06-19
| | | | | | | To start we add just a single metric, the number of actions constructed in the current build. RELNOTES: None PiperOrigin-RevId: 201248490
* Permit marking dirty/changed a node more than onceGravatar mschaller2018-06-18
| | | | | | | | | | | | This functionality will be useful for node restarting. More than one parent node may request the restart of a shared child node, and this should not fail. Instead, the returned MarkedDirtyResult indicates whether the dirtying was redundant, and the calling code can assert on that. RELNOTES: None. PiperOrigin-RevId: 201005663
* Remove ConfiguredTarget from TargetCompletionValue, since it is no longer ↵Gravatar janakr2018-06-04
| | | | | | needed: we can get the ConfiguredTargetKey directly from the TargetCompletionKey. Since that was the only use of the actual value in EvaluationProgressReceiver#evaluated, remove it, instead just provide a boolean enum that gives whether or not evaluation succeeded. PiperOrigin-RevId: 199178047
* Allow injection of custom SkyFunctions into GraphTester.Gravatar janakr2018-06-04
| | | | PiperOrigin-RevId: 199126212
* Add events and get rid of ErrorInfoEncoder. Clean up some signatures and ↵Gravatar janakr2018-05-22
| | | | | | visibility in Skyframe classes. PiperOrigin-RevId: 197665817
* Potentially allow children of a dirty node to be missing from the graph. ↵Gravatar janakr2018-05-15
| | | | | | Also pass the GraphInconsistencyReciever into SkyframeExecutor as a parameter. PiperOrigin-RevId: 196716642
* Allow SkyFunctions to return a sentinel value indicating that all of a ↵Gravatar janakr2018-04-30
| | | | | | | | | | node's in-progress data should be forgotten, and its evaluation should be restarted from scratch, as if it were freshly created/dirtied. To guard against this happening unexpectedly, any such events are passed to a GraphInconsistencyReceiver, which can verify that the SkyFunction is behaving properly. This is the first change in a series to permit action rewinding when it is discovered that a previously generated input file is no longer available. When an action detects that one of its inputs is unusable, it can return this sentinel value, causing it to be re-evaluated from scratch. Follow-up changes will make the node corresponding to the input, and the node corresponding to the action that generated the input, dirty when this happens, causing the upstream action to be re-run, regenerating the desired input. Currently works for builds that do not keep edges, although follow-ups may make this possible for all builds. PiperOrigin-RevId: 194863097
* Add tester method that allows us to unconditionally create a TestFunction ↵Gravatar shreyax2018-03-19
| | | | | | builder. PiperOrigin-RevId: 189592782
* Replace LegacySkyKey by AbstractSkyKey or custom SkyKeys. AbstractSkyKey ↵Gravatar janakr2018-03-02
| | | | | | | | doesn't save memory in the 32-bit case, but makes it easier for people to see how many SkyKeys we have. There's some unnecessary interning in tests, but it was easier to copypasta and doesn't harm anything, I think. PiperOrigin-RevId: 187694309
* Expose SimpleCycleDetector. Also, allow MemoizingEvaluatorTest to support ↵Gravatar cpeyser2018-02-26
| | | | | | evaluators that do not store errors alongside values, but which still support error transience. PiperOrigin-RevId: 187058808
* Automated rollback of commit 7fe59b98eefc96a6310f0b0221d4e0f18e2a9000.Gravatar shreyax2018-02-23
| | | | | | | | | | | | *** Reason for rollback *** Fixed bug due to TransitiveTargetFunction requesting multiple Package dependencies when computing its aspect deps by only applying the optimization to TransitiveTraversalFunction. *** Original change description *** Automated rollback of commit cce164aed44aba1de244f0d764cd33a5cc6980b2. PiperOrigin-RevId: 186766812
* Deflake ParallelEvaluatorTest.errorBubblesToParentsOfTopLevelValue: error ↵Gravatar janakr2018-02-21
| | | | | | cannot finish building until top registers a dep on it, and we can only know that by listening to the addReverseDep call. PiperOrigin-RevId: 186499232
* Automated rollback of commit cce164aed44aba1de244f0d764cd33a5cc6980b2.Gravatar fwe2018-02-19
| | | | PiperOrigin-RevId: 186211672
* Re-use previously computed deps for TransitiveBaseTraversalFunction#compute ↵Gravatar shreyax2018-02-16
| | | | | | if we have them instead of re-computing them each time on a skyframe restart. PiperOrigin-RevId: 186017079
* Automatic code cleanup.Gravatar cpovirk2018-02-13
| | | | PiperOrigin-RevId: 185624059
* Fix bug where an was-inflight-and-is-about-to-be-done NodeEntry has ↵Gravatar nharmata2018-01-18
| | | | | | | | | | | | | | | | | | | | incomplete deps that need to be removed, and these deps are currently duplicated in the NodeEntry's newly requested deps GroupedList. Also add a fast-path to GroupedListHelper#remove(List<Object> elements, Set<E> toRemove) for the incredibly common case where toRemove is empty. This saves a wasteful O(elements.size()) scan over elements. This method is unconditionally called each time a SkyFunction restart causes us to add new direct deps (with elements=<the new direct deps> and toRemove=<unfinished direct deps>); in the case where there are a ton of new direct deps, this scan entails wasted cpu and gc churn. The bug only occurs in uncommon case that there are deps to remove. The bug has existed since GroupedList was first introduced into the codebase. In Skyframe-land, this is only observable in nokeep_going mode because in keep_going mode "we do not let SkyFunctions throw errors with missing deps" (quote from comment in AbstractParallelEvaluator). A Bazel-on-Skyframe-land example how this bug could occur in practice is PackageFunction's Skyframe hybrid globbing. If an io error is encountered during legacy globbing, the PackageFunction eagerly throws a SkyFunctionException but it has already requested the Skyframe GlobValue deps. RELNOTES: None PiperOrigin-RevId: 182403943
* ParallelEvaluator: report events early for cache hitsGravatar ulfjack2017-12-21
| | | | | | | | | | | | | | | | | | In the case that a node is already done when evaluation starts, we now report events and postables early, rather than waiting until the end of evaluation. This makes reporting more timely, and ensures reporting even if the evaluation is interrupted. This caused a problem with moving the TargetCompleteEvent into Skyframe (unknown commit). I added two unit tests at the Skyframe level to cover the guarantees that we need for that. Note that the replay call in constructResult can duplicate the events from a cache hit - this is not a problem since the replaying visitor automatically removes duplicates (and it wasn't obvious which keys correspond to cache hits). PiperOrigin-RevId: 179788157
* Have GraphTest test rdep consistency conditionally since some graph ↵Gravatar nharmata2017-12-18
| | | | | | | implementations don't store rdeps for done nodes. RELNOTES: None PiperOrigin-RevId: 179455698
* Put more information in failure message of occasionally flaky test, and ↵Gravatar janakr2017-11-22
| | | | | | deduplicate code. PiperOrigin-RevId: 176686795
* Clear interrupted bit in thread when throwing an interrupted exception that ↵Gravatar janakr2017-11-21
| | | | | | | | came from an AbstractParallelEvaluator evaluation. It's against the standard Java contract to throw but still have the thread's interrupted bit set. Also get rid of some unnecessary initializeTester() calls in MemoizingEvaluatorTest: we already call it via a @Before annotation. PiperOrigin-RevId: 176496034
* Replace truth.FailureStrategy with truth.FailureMetadata in bazel's Subjects.Gravatar Googler2017-11-10
| | | | | | | | | | Also changed truth.SubjectFactory to truth.Subject.Factory (plain renaming) and use method reference instead of anonymous class to create the factory when applicable. FailureMetadata, an opaque object to its users, is introduced to replace FailureStrategy in in custom Subject in order to resolve some existing flaws of FailureStrategy as well as enable new features to be added to Truth. New API is available in Truth-0.36, if there is a build/pom.xml, it's also updated to use this version. PiperOrigin-RevId: 175308179
* Replace all usages of Blaze's Preconditions class with guava.Gravatar tomlu2017-11-09
| | | | | | | | Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own. This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually. PiperOrigin-RevId: 175033526
* Make ErrorInfo#toString more informative.Gravatar janakr2017-11-06
| | | | PiperOrigin-RevId: 174508154
* Fix local repository detection when the repository path is absolute.Gravatar John Cater2017-10-13
| | | | | | | Fixes #3874. Change-Id: Ibbe3ea27b77426f551e2f70f082478edb2234749 PiperOrigin-RevId: 171957230
* Allow NodeEntry implementations to keep just deps, as opposed to all edges ↵Gravatar janakr2017-10-10
| | | | | | or no edges. Also add option to disable checks in MemoizingEvaluatorTest that don't make sense for implementations that don't keep track of dirty nodes. Also extract RecordingDifferencer to an interface. And add a test for the situation that a node changes during a build that it's not requested, and which fails, necessitating cleanup. PiperOrigin-RevId: 171616817
* Bubble errors up even in the case of keep_going builds that failed due to ↵Gravatar janakr2017-10-07
| | | | | | | | catastrophes. Our stricter behavior in the face of errors means that it is no longer possible for a done node to depend on a not-done node in this build. This opens up the possibility to discard graph edges on all --batch builds, or at least those with --discard_analysis_cache. PiperOrigin-RevId: 171375405
* CI,windows: create test_suites for Windows testsGravatar L?szl? Csomor2017-09-19
| | | | | | | | | | | | | | | Add recursive test_suite rules for all tests that ci.bazel.io runs for Windows, and set the top-level test_suite as the CI test target. Doing so shortens the command line and works around https://github.com/bazelbuild/bazel/issues/3742 I verified that the old set of tests are the same as the new set. Change-Id: Id8d5da3f0c03c9b8969a9f8e1e9a3096888365aa PiperOrigin-RevId: 169242858
* Automatic code cleanup.Gravatar cushon2017-09-15
| | | | PiperOrigin-RevId: 168802886
* More BUILD file refactorings.Gravatar philwo2017-09-06
| | | | | | | | | Split collect, concurrent, vfs, windows into package-level BUILD files. Move clock classes out of "util", into their own Java package. Move CompactHashSet into its own Java package to break a dependency cycle. Give nestedset and inmemoryfs their own package-level BUILD files. PiperOrigin-RevId: 167702127
* Introduce AbstractParallelEvaluatorGravatar mschaller2017-08-07
| | | | | | | | | | This change enables alternate evaluation strategies. Drive-by fix to GraphTester, making TestFunction static because it has no dependencies on its outer class, and adding #unsetConstantValue. RELNOTES: None. PiperOrigin-RevId: 164327055
* Add an Iterable<Event> Truth subject so that we can easily assert things on ↵Gravatar janakr2017-07-24
| | | | | | | | | | an iterable of events without calling into the heavyweight MoreAsserts. Use that in Skyframe instead of MoreAsserts. Also delete an unused method in MoreAsserts. PiperOrigin-RevId: 162754283
* Tolerate injected nodes having deps.Gravatar janakr2017-07-21
| | | | | | | | | | We don't check explicitly that these are the only two ways, but this can happen if the error transience node is a dep of a node that's being injected, or if an injected node is an "external" file that needs to depend on an external package. The first possibility can happen if there was an IOException reading the node on the previous build. We handle the situation by just dirtying the node, not injecting it. Actual evaluation can handle the re-stat. PiperOrigin-RevId: 162622092
* Now that SkyKey is an interface, accept Iterable<? extends SkyKey>Gravatar ulfjack2017-07-19
| | | | | | | A subsequent CL makes TargetPatternKey implement SkyKey, and it's much nicer if I can pass lists of TargetPatternKey to the various Skyframe APIs. PiperOrigin-RevId: 162359843
* Fix Postable forwarding and replayGravatar ulfjack2017-07-18
| | | | | | | We were previously duplicate-posting Postable events posted to the Skyframe environment. PiperOrigin-RevId: 162323598
* Automated conversion to Java 8Gravatar laurentlb2017-06-30
| | | | | | | With a few manual fixes for readability. RELNOTES: None. PiperOrigin-RevId: 160582556
* Give MemoizingEvaluatorTest subclasses a bit more control over how transient ↵Gravatar nharmata2017-06-19
| | | | | | | errors are used in some of the test cases. RELNOTES: None PiperOrigin-RevId: 159275483
* A bunch of unrelated cleanups:Gravatar nharmata2017-06-16
| | | | | | | | | | -Have SkylarkImportLookupFunction include causes in the SkyFunctionExceptions it throws. -Better transitive skyframe error declarations in ASTFileLookupFunction. -Have ErrorInfo differentiate between direct and transitive transience. -Introduce ErrorInfoManager and have ParallelEvaluator/ParallelEvaluatorContext use it. RELNOTES: None PiperOrigin-RevId: 159163186
* Propagate postable events further upGravatar Klaus Aehlig2017-06-14
| | | | | | | | | | | | With the introduction of the ExtendedEventHandler, SkyFunctions were given the possibility to post additional Postable events in addition to the standard events. As SkyFunctions have to be restartable, events are collected first and only posted after the function is finished. Make sure that this also applies to postable events and they are not dropped. Change-Id: Ie1c3a0134935c75ea984fa2cc924e7327a9da81f PiperOrigin-RevId: 158964337
* Remove all imports of org.junit.Assert except for static imports of #fail().Gravatar lberki2017-06-01
| | | | | RELNOTES: None. PiperOrigin-RevId: 157685150
* Migrate all tests to Truth (except the ones in the examples).Gravatar lberki2017-05-31
| | | | | RELNOTES: None. PiperOrigin-RevId: 157576492
* Migrate most of the assertions to Truth that the auto-migration tool did not ↵Gravatar lberki2017-05-31
| | | | | | | | | catch. IntelliJ's "replace structurally" command was surprisingly useful. RELNOTES: None. PiperOrigin-RevId: 157463734
* Migrate Java tests to Truth.Gravatar lberki2017-05-30
| | | | | | RELNOTES: None. PiperOrigin-RevId: 157446717
* Fixes incorrectly-ordered arguments to calls to assertEqualsGravatar Googler2017-05-19
| | | | | | | | | | | | | | | [] This change has been automatically generated by an Error Prone check that detects incorrect argument ordering on calls to assertEquals-style methods. See [] Cleanup change automatically generated by javacflume/refactory Refactoring: third_party/java_src/error_prone/project/core/src/main/java/com/google/errorprone/bugpatterns/argumentselectiondefects:AssertEqualsArgumentOrderChecker_refactoring Tested: TAP --sample for global presubmit queue [] PiperOrigin-RevId: 156539781
generated by cgit v1.2.3 (git 2.39.1) at 2024-06-26 09:57:29 +0000