aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/query2/engine
Commit message (Collapse)AuthorAge
* Some code cleanups and improvements around the various DTC visitor ↵Gravatar nharmata2018-07-30
| | | | | | | | | | | implementations in SkyQuery: (1) Get rid of the 'errorReporter' param used in the two unbounded 'deps' implementations. In practice, this callback is always used right alongside the normal query callback. So we can re-implement it as a single, composite callback. (2) Revert the recent change to processResultsAndReturnTargets (and restore the old method name). It general, it's a bad idea to return Targets outside that method, because it means we aren't respecting the MultisetSemaphore<PackageIdentifier>. Along with (1), this fixes a real issue where DepsUnboundedVisitor#processPartialResults was using Targets (and thus keeping their Packages live in memory) outside of the package semaphore. RELNOTES: None PiperOrigin-RevId: 206606747
* Allow uniquifiers to throw QueryExceptions.Gravatar shreyax2018-07-02
| | | | | RELNOTES: None. PiperOrigin-RevId: 202961077
* Rename VariableContext to the more general purpose QueryExpressionContext ↵Gravatar shreyax2018-06-12
| | | | | | and thread it through to graph traversal functions. Some other light refactorings as well. PiperOrigin-RevId: 200292556
* Fix bug in skyframe-based test-suite expansionGravatar ulfjack2018-06-07
| | | | | | | | | | It was tracking filtered tests and then applying the filter at the next higher level. I also added a bunch of comments - we actually have four implementations of test suite expansion, and they are not consistent. Sorry about that. PiperOrigin-RevId: 199629485
* Implement unbounded deps() using ParallelVisitor for SkyQuery.Gravatar shreyax2018-05-25
| | | | PiperOrigin-RevId: 198074986
* Use ConcurrentHashMap directly instead of MapMaker.Gravatar Googler2018-05-03
| | | | | RELNOTES: None. PiperOrigin-RevId: 195280015
* Provide parallel implementations of bounded allrdeps and rdeps.Gravatar nharmata2018-04-12
| | | | | RELNOTES: None PiperOrigin-RevId: 192681579
* Fix bug in SkyQuery's parallel allrdeps with --nohost_deps.Gravatar nharmata2018-04-10
| | | | | | | | | The was that the presence of invalid dependency edges would incorrectly cause 'allrdeps' to not visit targets. Concrete example: if --nohost_deps was set and T1 depended on T via a host edge and T2 depended on T via a non-host edge then 'allrdeps(T)' might incorrectly not contain T2. Along with fixing the bug, refactor ParallelVisitor's deduping logic: instead of deduping visitations inside #getVisitResult, dedupe them before adding them to #processingQueue. This should be a strict, small, performance win, especially for trivial visitations (e.g. RBuildFilesVisitor). RELNOTES: None PiperOrigin-RevId: 192327607
* Fix check-then-act race for some function potentially returning multiple ↵Gravatar shreyax2018-04-05
| | | | | | targets. PiperOrigin-RevId: 191774693
* As promised in an earlier commit, remove subinclude machinery from ↵Gravatar nharmata2018-03-26
| | | | | | | | | PackageFactory, Package, PackageFunction, and also all things that make use of Package#getSubincludeLabels. This code is completely dead, and has been for a while. RELNOTES: None PiperOrigin-RevId: 190486792
* Create a new output formatter for cquery which output transition information ↵Gravatar juliexxia2018-03-23
| | | | | | in either a FULL or LITE version. Trigger new output with the new --transitions cquery flag in the new CqueryOptions class. PiperOrigin-RevId: 190278664
* Generalize QueryExpressionVisitor's visitation by permitting an arbitrary ↵Gravatar nharmata2018-01-31
| | | | | | | "context" object to be passed along. Also make some query internals public, for use in fancy QueryExpressionVisitor implementations. RELNOTES: None PiperOrigin-RevId: 184014063
* Automated rollback of commit 9e63684a58025d98bfe2f180d56ba7ece69545ae.Gravatar juliexxia2018-01-05
| | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Roll forward of commit 86b4532769c22cca2ed7068a60f3326beaad34af after fixing bad import. +small misc fixes suggested by critique *** Original change description *** Automated rollback of commit 86b4532769c22cca2ed7068a60f3326beaad34af. *** Reason for rollback *** Probably breaking //javatests/com/google/devtools/build/lib:Query2Tests *** Original change description *** Restructure how universeScope is used when testing configured query to mimick impending changes to the configured query interface (CL/179872445) which will pull build targets out of the query expression. Fill in testTopLevelTransitions on the way! PiperOrigin-RevId: 180930388
* Automated rollback of commit 86b4532769c22cca2ed7068a60f3326beaad34af.Gravatar juliexxia2018-01-04
| | | | | | | | | | | | | | *** Reason for rollback *** Probably breaking //javatests/com/google/devtools/build/lib:Query2Tests *** Original change description *** Restructure how universeScope is used when testing configured query to mimick impending changes to the configured query interface (CL/179872445) which will pull build targets out of the query expression. Fill in testTopLevelTransitions on the way! PiperOrigin-RevId: 180880350
* Restructure how universeScope is used when testing configured query to ↵Gravatar juliexxia2018-01-04
| | | | | | | | mimick impending changes to the configured query interface (CL/179872445) which will pull build targets out of the query expression. Fill in testTopLevelTransitions on the way! PiperOrigin-RevId: 180854150
* Replace all usages of Blaze's Preconditions class with guava.Gravatar tomlu2017-11-09
| | | | | | | | Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own. This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually. PiperOrigin-RevId: 175033526
* More BUILD file refactorings.Gravatar philwo2017-09-06
| | | | | | | | | Split collect, concurrent, vfs, windows into package-level BUILD files. Move clock classes out of "util", into their own Java package. Move CompactHashSet into its own Java package to break a dependency cycle. Give nestedset and inmemoryfs their own package-level BUILD files. PiperOrigin-RevId: 167702127
* Add method getRdepsUnboundedInUniverseParallel to StremableQueryEnvironmentGravatar Googler2017-09-04
| | | | | RELNOTES: None PiperOrigin-RevId: 167335614
* Change WalkableGraphFactory#prepareAndGet to take multiple SkyKeys as graph ↵Gravatar Googler2017-08-17
| | | | | | | | | | roots It also changes a few accessors of utility methods in Skyframe library. It refactors the QueryExpressionMapper to use a general QueryExpressionVisitor. RELNOTES: None PiperOrigin-RevId: 165534908
* Introduce the 'siblings' query function.Gravatar nharmata2017-08-14
| | | | | | RELNOTES[NEW]: There is now a 'siblings' query function. See the query documentation for more details. PiperOrigin-RevId: 165010653
* Generalize some of methods in TargetPattern, PrepareDepsOfPatternValue, and ↵Gravatar nharmata2017-07-26
| | | | | | | RecursivePackageProvider dealing with the concept of "excluded directories". RELNOTES: None PiperOrigin-RevId: 163074794
* Automated conversion to Java 8Gravatar laurentlb2017-06-30
| | | | | | | With a few manual fixes for readability. RELNOTES: None. PiperOrigin-RevId: 160582556
* Refactor SkyQueryEnvironment to allow preprocessing or postprocessing targetsGravatar Googler2017-06-27
| | | | | | | | Refactor SkyQueryEnvironment and a few other query helpers to make it easier to work with targets. RELNOTES: None PiperOrigin-RevId: 160165398
* In the Blaze Query implementation, use Set and Map implementations backed by ↵Gravatar nharmata2017-06-22
| | | | | | | the same KeyExtractor used that the Uniquifier implementation uses. This fixes a hypothetical issue where we were previously relying on Target#equals/hashCode. RELNOTES: None PiperOrigin-RevId: 159741545
* Ban the combination of buildfiles/loadfiles and --output=location.Gravatar nharmata2017-06-19
| | | | | | RELNOTES[INC]: The --output=location flag to 'bazel query' cannot be used with query expressions that involve the 'buildfiles' or 'loadfiles' operators. This also applies to 'genquery' rules. PiperOrigin-RevId: 159259061
* Don't call function names keyword in the query reference because they are not.Gravatar lberki2017-06-13
| | | | | RELNOTES: None. PiperOrigin-RevId: 158721043
* Don't throw runtime exception on interruption - since we know the future is ↵Gravatar Googler2017-05-26
| | | | | | already done, we should leave the thread in an interrupted state and proceed. This fixes a blaze crash when a 'genquery' execution is interrupted at the right time. PiperOrigin-RevId: 157000269
* Have QueryExpressionMapper#compose take arbitrary number of mappersGravatar Googler2017-05-04
| | | | | | | AttrFunction and LabelsFunction are also made public. RELNOTES: None PiperOrigin-RevId: 155108260
* Global cleanup change.Gravatar Googler2017-03-23
| | | | | | -- PiperOrigin-RevId: 151019690 MOS_MIGRATED_REVID=151019690
* Fix inadvertent performance regression introduced by the recent rewrite of ↵Gravatar Nathan Harmata2017-03-14
| | | | | | | | | | | | | | | | | | 'blaze query'. The "streaming" callbacks used by some query functions, e.g. 'deps', make calls to QueryEnvironment#buildTransitiveClosure. For a cold blaze server, these calls do package loading via LabelVisitor (which calls into Skyframe via a top-level #evaluate call). So we'd prefer a single massive call which can make full use of blaze's loading-phase parallelism via Skyframe over a bunch of sequential small calls. For a hot blaze server, there are two problems: (1) LabelVisitor's meager up-to-date check isn't useful (as in we cannot reuse old visitations) when we do a whole bunch of small visitations instead of one massive one. (2) The actual work of the LabelVisitor (building up a portion of a temporary graph) isn't being effectively parallelized when we do it sequentially in small chunks. This issue is yet another subtle reason why the old BlazeQueryEnvironment#eval made sense (and why it was unfortunately not compatible with the streaming query evaluation model from the beginning). -- PiperOrigin-RevId: 150081619 MOS_MIGRATED_REVID=150081619
* Description redacted.Gravatar Nathan Harmata2017-03-09
| | | | | | -- PiperOrigin-RevId: 149585165 MOS_MIGRATED_REVID=149585165
* Fix bug with streaming bounded deps/allrdeps/rdeps.Gravatar Nathan Harmata2017-03-08
| | | | | | -- PiperOrigin-RevId: 149431500 MOS_MIGRATED_REVID=149431500
* Global cleanup change.Gravatar Googler2017-03-03
| | | | | | -- PiperOrigin-RevId: 149089903 MOS_MIGRATED_REVID=149089903
* Rollback of commit 822c37816ac669e51bec3853b41849a19ec5e230.Gravatar Nathan Harmata2017-03-01
| | | | | | -- PiperOrigin-RevId: 148844518 MOS_MIGRATED_REVID=148844518
* Reimplement blaze query using an async evaluation model. Use a concurrent ↵Gravatar Nathan Harmata2017-02-28
| | | | | | | | | | | | | | | | | | | backend for SkyQueryEnvironment's implementation in order to achieve parallelism. Advantages: -New design has no flaws that the old design had. -Code is structured so that deadlocks due to thread starvation are impossible (yup!). Disadvantages: -The meat of this change needs to all be in a single CL because every single QueryFunction and QueryExpression needs to be rewritten in the async style. Still TODO: -Fully embrace the async model in all QueryFunctions (e.g. 'rdeps', 'allpaths'). -Use concurrency in BlazeQueryEnvironment to achieve parallel evaluation for (non SkyQuery) 'blaze query' and genquery. -- PiperOrigin-RevId: 148690279 MOS_MIGRATED_REVID=148690279
* Reword comment about the current implementation of ^.Gravatar Nathan Harmata2017-02-27
| | | | | | -- PiperOrigin-RevId: 148488719 MOS_MIGRATED_REVID=148488719
* Fix a hypothetical issue with 'buildfiles' and 'loadfiles' with duplicate ↵Gravatar Nathan Harmata2017-02-16
| | | | | | | | | | | | | | | | | | targets. These two functions don't use a Uniquifier internally so if the same bzl file foo.bzl is loaded multiple ways in 'e' in 'loadfiles(e)' then 'f' in 'f(loadfiles(e))' will observe foo.bzl multiple times. This isn't observable in practice (i.e. I cannot write a black-box query test that would fail without this CL) because: (1) BlazeQueryEnvironment#eval(QueryExpression, VariableContext<Target>, Callback<Target>) uses an intermediate aggregating callback and thus doesn't use the streaming query evaluation model. This means that the internal deduping in BlazeQueryEnvironment#getBuildFiles is sufficient. (2) SkyQueryEnvironment uses an outer BatchStreamedCallback which internally uses a Uniquifier. Still, this CL is useful for SkyQuery because without we're doing wasted work (e.g. in my example above, 'f' will wastefully do duplicate work with the .bzl files from the inner loadfiles(e)) because the deduping happens on the final query result, not on the intermediate result from 'buildfiles'/'loadfiles'. For 'buildfiles', there's an additional wart with (1) above because FakeSubincludeTarget doesn't override Object#equals/hashCode. -- PiperOrigin-RevId: 147656183 MOS_MIGRATED_REVID=147656183
* Give a more helpful error message on malformed regexes.Gravatar Nathan Harmata2017-02-16
| | | | | | -- PiperOrigin-RevId: 147621296 MOS_MIGRATED_REVID=147621296
* Global cleanup change.Gravatar Googler2017-01-25
| | | | | | -- PiperOrigin-RevId: 145473478 MOS_MIGRATED_REVID=145473478
* Remove dead code referencing deleted "suites" attribute.Gravatar Greg Estren2016-12-21
| | | | | | | | | This used to be how test_suites depended on other test_suites. Now they just go in "tests". -- PiperOrigin-RevId: 142607603 MOS_MIGRATED_REVID=142607603
* Rollback of commit e0a330577d9fe98169645cb68d9fc22cc787eeb6.Gravatar Janak Ramakrishnan2016-12-21
| | | | | | | | | | *** Reason for rollback *** Investigating if causes deadlock/thread starvation. -- PiperOrigin-RevId: 142575769 MOS_MIGRATED_REVID=142575769
* Some improvements to ParallelQueryUtils.Gravatar Nathan Harmata2016-12-16
| | | | | | | | | | | | | | | | | | | | | | | | (i) Use a CountDownLatch in ParallelQueryUtils#executeQueryTasksAndWaitInterruptibly to avoid busy-looping while waiting for query subtask completion (this busy-looping unnecessarily ties up a thread). But we still retain the fail-fast semantics we want (I renamed the method to emphasize this). (ii) Also have a special-case in ParallelQueryUtils#executeQueryTasksAndWaitInterruptibly for evaluating one query subtask so we don't wastefully use another thread. (iii) Also add ThreadSafety annotations to ParallelQueryUtils. ---- (i) and (ii) combine to address the following theoretical issue. Suppose we're evaluating a query expression of the form "(e1 - e2) + (e3 - e4)". The old code would (with the worst-case FJP thread scheduling) have the following threads at the _same_ time: Main QueryCommand thread - executeQueryTasksAndWaitInterruptibly(queryTasks = [(e1 - e2), (e3 - e4)] FJP thread - executeQueryTasksAndWaitInterruptibly(queryTasks = [e2]) FJP thread - eval(e2) FJP thread - executeQueryTasksAndWaitInterruptibly(queryTasks = [e4]) FJP thread - eval(e4) So of those 5 concurrent threads, 3 would be doing busy-loop waiting. For more pathological query expressions, we could end up tying up lots of threads doing wasteful busy-loops. -- PiperOrigin-RevId: 142215680 MOS_MIGRATED_REVID=142215680
* Make Argument factory methods public.Gravatar Nathan Harmata2016-12-13
| | | | | | -- PiperOrigin-RevId: 141920696 MOS_MIGRATED_REVID=141920696
* Rephrase query transformation in terms of composition of QueryExpressionMappers.Gravatar Nathan Harmata2016-12-13
| | | | | | -- PiperOrigin-RevId: 141904124 MOS_MIGRATED_REVID=141904124
* Fix bad bug with the parallel implementation of BinaryOperatorExpression. ↵Gravatar Nathan Harmata2016-12-12
| | | | | | | | | | | | | | Turns out that ForkJoinTask#adapt(Callable) returns a ForkJoinTask whose Future#get on error throws a ExecutionException wrapping a RuntimeException wrapping the thrown checked exception from the callable. This is documented behavior [1] that I incorrectly didn't know about. The additional level of wrapping meant that the catch-block of the parallel implementation of BinaryOperatorExpression wasn't rethrowing the InterruptedException/QueryException that the parallel task threw. The subtly in this bug is that the query expression being evaluated needs to be of the form "e1 + e2", where evaluation of "e1" throws a QueryException even in keepGoing mode (note that most of the query errors actually go through AbstractBlazeQueryEnvironment#reportBuildFileError). The test I wrote picks on LetExpression's evaluation-time (rather than e.g. parsing time) validation of the variable name. [1] https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ForkJoinTask.html#adapt(java.util.concurrent.Callable) -- PiperOrigin-RevId: 141772584 MOS_MIGRATED_REVID=141772584
* Fix spelling mistake.Gravatar Nathan Harmata2016-12-06
| | | | | | -- PiperOrigin-RevId: 141178325 MOS_MIGRATED_REVID=141178325
* For all function expressions of the form f(..., e1, ..., e2, ..., eK, ...), ↵Gravatar Nathan Harmata2016-12-02
| | | | | | | | ensure that all of e1, e2, ..., and eK are elligble for parallel evaluation. This is _not_ the same as providing a parallel implementation of f, which we can do separately in followup CLs. -- PiperOrigin-RevId: 140861694 MOS_MIGRATED_REVID=140861694
* Provide a parallel implementation of "e1 - e2 - e3" by noting its ↵Gravatar Nathan Harmata2016-11-21
| | | | | | | equivalence to "e1 - (e2 + e3)" and the fact that we already have a parallel implementation of "e2 + e3". -- MOS_MIGRATED_REVID=139792288
* Enable parallel evaluation of 'e' in query expressions of the form ↵Gravatar Nathan Harmata2016-11-21
| | | | | | | 'buildfiles(e)'. -- MOS_MIGRATED_REVID=139787078
* Provide a parallel implementation of "e1 + e2".Gravatar Nathan Harmata2016-11-21
| | | | | -- MOS_MIGRATED_REVID=139613681
generated by cgit v1.2.3 (git 2.39.1) at 2024-06-28 19:45:07 +0000