| Commit message (Collapse) | Author | Age |
|
|
|
|
|
|
| |
"context" object to be passed along. Also make some query internals public, for use in fancy QueryExpressionVisitor implementations.
RELNOTES: None
PiperOrigin-RevId: 184014063
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Roll forward of commit 86b4532769c22cca2ed7068a60f3326beaad34af after fixing bad import.
+small misc fixes suggested by critique
*** Original change description ***
Automated rollback of commit 86b4532769c22cca2ed7068a60f3326beaad34af.
*** Reason for rollback ***
Probably breaking //javatests/com/google/devtools/build/lib:Query2Tests
*** Original change description ***
Restructure how universeScope is used when testing configured query to mimick impending changes to the configured query interface (CL/179872445) which will pull build targets out of the query expression.
Fill in testTopLevelTransitions on the way!
PiperOrigin-RevId: 180930388
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Probably breaking //javatests/com/google/devtools/build/lib:Query2Tests
*** Original change description ***
Restructure how universeScope is used when testing configured query to mimick impending changes to the configured query interface (CL/179872445) which will pull build targets out of the query expression.
Fill in testTopLevelTransitions on the way!
PiperOrigin-RevId: 180880350
|
|
|
|
|
|
|
|
| |
mimick impending changes to the configured query interface (CL/179872445) which will pull build targets out of the query expression.
Fill in testTopLevelTransitions on the way!
PiperOrigin-RevId: 180854150
|
|
|
|
|
|
|
|
| |
Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own.
This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually.
PiperOrigin-RevId: 175033526
|
|
|
|
|
|
|
|
|
| |
Split collect, concurrent, vfs, windows into package-level BUILD files.
Move clock classes out of "util", into their own Java package.
Move CompactHashSet into its own Java package to break a dependency cycle.
Give nestedset and inmemoryfs their own package-level BUILD files.
PiperOrigin-RevId: 167702127
|
|
|
|
|
| |
RELNOTES: None
PiperOrigin-RevId: 167335614
|
|
|
|
|
|
|
|
|
|
| |
roots
It also changes a few accessors of utility methods in Skyframe library. It
refactors the QueryExpressionMapper to use a general QueryExpressionVisitor.
RELNOTES: None
PiperOrigin-RevId: 165534908
|
|
|
|
|
|
| |
RELNOTES[NEW]: There is now a 'siblings' query function. See the query documentation for more details.
PiperOrigin-RevId: 165010653
|
|
|
|
|
|
|
| |
RecursivePackageProvider dealing with the concept of "excluded directories".
RELNOTES: None
PiperOrigin-RevId: 163074794
|
|
|
|
|
|
|
| |
With a few manual fixes for readability.
RELNOTES: None.
PiperOrigin-RevId: 160582556
|
|
|
|
|
|
|
|
| |
Refactor SkyQueryEnvironment and a few other query helpers to make it easier to
work with targets.
RELNOTES: None
PiperOrigin-RevId: 160165398
|
|
|
|
|
|
|
| |
the same KeyExtractor used that the Uniquifier implementation uses. This fixes a hypothetical issue where we were previously relying on Target#equals/hashCode.
RELNOTES: None
PiperOrigin-RevId: 159741545
|
|
|
|
|
|
| |
RELNOTES[INC]: The --output=location flag to 'bazel query' cannot be used with query expressions that involve the 'buildfiles' or 'loadfiles' operators. This also applies to 'genquery' rules.
PiperOrigin-RevId: 159259061
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 158721043
|
|
|
|
|
|
| |
already done, we should leave the thread in an interrupted state and proceed. This fixes a blaze crash when a 'genquery' execution is interrupted at the right time.
PiperOrigin-RevId: 157000269
|
|
|
|
|
|
|
| |
AttrFunction and LabelsFunction are also made public.
RELNOTES: None
PiperOrigin-RevId: 155108260
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 151019690
MOS_MIGRATED_REVID=151019690
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
'blaze query'.
The "streaming" callbacks used by some query functions, e.g. 'deps', make calls to QueryEnvironment#buildTransitiveClosure.
For a cold blaze server, these calls do package loading via LabelVisitor (which calls into Skyframe via a top-level #evaluate call). So we'd prefer a single massive call which can make full use of blaze's loading-phase parallelism via Skyframe over a bunch of sequential small calls.
For a hot blaze server, there are two problems:
(1) LabelVisitor's meager up-to-date check isn't useful (as in we cannot reuse old visitations) when we do a whole bunch of small visitations instead of one massive one.
(2) The actual work of the LabelVisitor (building up a portion of a temporary graph) isn't being effectively parallelized when we do it sequentially in small chunks.
This issue is yet another subtle reason why the old BlazeQueryEnvironment#eval made sense (and why it was unfortunately not compatible with the streaming query evaluation model from the beginning).
--
PiperOrigin-RevId: 150081619
MOS_MIGRATED_REVID=150081619
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 149585165
MOS_MIGRATED_REVID=149585165
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 149431500
MOS_MIGRATED_REVID=149431500
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 149089903
MOS_MIGRATED_REVID=149089903
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 148844518
MOS_MIGRATED_REVID=148844518
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
backend for SkyQueryEnvironment's implementation in order to achieve parallelism.
Advantages:
-New design has no flaws that the old design had.
-Code is structured so that deadlocks due to thread starvation are impossible (yup!).
Disadvantages:
-The meat of this change needs to all be in a single CL because every single QueryFunction and QueryExpression needs to be rewritten in the async style.
Still TODO:
-Fully embrace the async model in all QueryFunctions (e.g. 'rdeps', 'allpaths').
-Use concurrency in BlazeQueryEnvironment to achieve parallel evaluation for (non SkyQuery) 'blaze query' and genquery.
--
PiperOrigin-RevId: 148690279
MOS_MIGRATED_REVID=148690279
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 148488719
MOS_MIGRATED_REVID=148488719
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
targets.
These two functions don't use a Uniquifier internally so if the same bzl file foo.bzl is loaded multiple ways in 'e' in 'loadfiles(e)' then 'f' in 'f(loadfiles(e))' will observe foo.bzl multiple times.
This isn't observable in practice (i.e. I cannot write a black-box query test that would fail without this CL) because:
(1) BlazeQueryEnvironment#eval(QueryExpression, VariableContext<Target>, Callback<Target>) uses an intermediate aggregating callback and thus doesn't use the streaming query evaluation model. This means that the internal deduping in BlazeQueryEnvironment#getBuildFiles is sufficient.
(2) SkyQueryEnvironment uses an outer BatchStreamedCallback which internally uses a Uniquifier.
Still, this CL is useful for SkyQuery because without we're doing wasted work (e.g. in my example above, 'f' will wastefully do duplicate work with the .bzl files from the inner loadfiles(e)) because the deduping happens on the final query result, not on the intermediate result from 'buildfiles'/'loadfiles'.
For 'buildfiles', there's an additional wart with (1) above because FakeSubincludeTarget doesn't override Object#equals/hashCode.
--
PiperOrigin-RevId: 147656183
MOS_MIGRATED_REVID=147656183
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 147621296
MOS_MIGRATED_REVID=147621296
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 145473478
MOS_MIGRATED_REVID=145473478
|
|
|
|
|
|
|
|
|
| |
This used to be how test_suites depended on other test_suites.
Now they just go in "tests".
--
PiperOrigin-RevId: 142607603
MOS_MIGRATED_REVID=142607603
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Investigating if causes deadlock/thread starvation.
--
PiperOrigin-RevId: 142575769
MOS_MIGRATED_REVID=142575769
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(i) Use a CountDownLatch in ParallelQueryUtils#executeQueryTasksAndWaitInterruptibly to avoid busy-looping while waiting for query subtask completion (this busy-looping unnecessarily ties up a thread). But we still retain the fail-fast semantics we want (I renamed the method to emphasize this).
(ii) Also have a special-case in ParallelQueryUtils#executeQueryTasksAndWaitInterruptibly for evaluating one query subtask so we don't wastefully use another thread.
(iii) Also add ThreadSafety annotations to ParallelQueryUtils.
----
(i) and (ii) combine to address the following theoretical issue. Suppose we're evaluating a query expression of the form "(e1 - e2) + (e3 - e4)". The old code would (with the worst-case FJP thread scheduling) have the following threads at the _same_ time:
Main QueryCommand thread - executeQueryTasksAndWaitInterruptibly(queryTasks = [(e1 - e2), (e3 - e4)]
FJP thread - executeQueryTasksAndWaitInterruptibly(queryTasks = [e2])
FJP thread - eval(e2)
FJP thread - executeQueryTasksAndWaitInterruptibly(queryTasks = [e4])
FJP thread - eval(e4)
So of those 5 concurrent threads, 3 would be doing busy-loop waiting. For more pathological query expressions, we could end up tying up lots of threads doing wasteful busy-loops.
--
PiperOrigin-RevId: 142215680
MOS_MIGRATED_REVID=142215680
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 141920696
MOS_MIGRATED_REVID=141920696
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 141904124
MOS_MIGRATED_REVID=141904124
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Turns out that ForkJoinTask#adapt(Callable) returns a ForkJoinTask whose Future#get on error throws a ExecutionException wrapping a RuntimeException wrapping the thrown checked exception from the callable. This is documented behavior [1] that I incorrectly didn't know about.
The additional level of wrapping meant that the catch-block of the parallel implementation of BinaryOperatorExpression wasn't rethrowing the InterruptedException/QueryException that the parallel task threw.
The subtly in this bug is that the query expression being evaluated needs to be of the form "e1 + e2", where evaluation of "e1" throws a QueryException even in keepGoing mode (note that most of the query errors actually go through AbstractBlazeQueryEnvironment#reportBuildFileError). The test I wrote picks on LetExpression's evaluation-time (rather than e.g. parsing time) validation of the variable name.
[1] https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ForkJoinTask.html#adapt(java.util.concurrent.Callable)
--
PiperOrigin-RevId: 141772584
MOS_MIGRATED_REVID=141772584
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 141178325
MOS_MIGRATED_REVID=141178325
|
|
|
|
|
|
|
|
| |
ensure that all of e1, e2, ..., and eK are elligble for parallel evaluation. This is _not_ the same as providing a parallel implementation of f, which we can do separately in followup CLs.
--
PiperOrigin-RevId: 140861694
MOS_MIGRATED_REVID=140861694
|
|
|
|
|
|
|
| |
equivalence to "e1 - (e2 + e3)" and the fact that we already have a parallel implementation of "e2 + e3".
--
MOS_MIGRATED_REVID=139792288
|
|
|
|
|
|
|
| |
'buildfiles(e)'.
--
MOS_MIGRATED_REVID=139787078
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=139613681
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=139508838
|
|
|
|
|
|
|
| |
by introducing TargetPattern#parEval, which allows TargetPatterns' evaluations to explicitly have parallel implementations (no need to secretly use a FJP).
--
MOS_MIGRATED_REVID=139101922
|
|
|
|
|
|
|
|
| |
No need for the char[] in the middle, prevents us from accidentally modifying
input, or sucking up ram on huge queries.
--
MOS_MIGRATED_REVID=137872573
|
|
|
|
|
|
|
| |
overrides. Also, have AbstractBlazeQueryEnvironment#evaluateQuery take an OutputFormatterCallback instance rather than a Callback instance. This is more sensible since the latter is only intended to be used intra-query, while the former is intended for usage in end-to-end query evaluation. This lets us slightly simplify QueryCommand, by shifting the responsibility for managing the OutputFormatterCallback to AbstractBlazeQueryEnvironment#evaluateQuery.
--
MOS_MIGRATED_REVID=134827588
|
|
|
|
|
|
|
| |
shared naive BFS implementation. Also implement RegexFilterExpression#parEval.
--
MOS_MIGRATED_REVID=134598046
|
|
|
|
|
|
|
| |
evaluation in SkyQueryEnvironment. FJP is nicer to program against, imo.
--
MOS_MIGRATED_REVID=133844508
|
|
|
|
|
|
|
| |
once. This is helpful when there are multiple skylark reverse 'load' paths from the same popular .bzl file to a single BUILD file. As an implementation detail, introduce a few ThreadSafeUniquifier things (these will also be used in followup CLs).
--
MOS_MIGRATED_REVID=132680777
|
|
|
|
|
|
|
| |
nodes.
--
MOS_MIGRATED_REVID=132558031
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
evaluation:
-Rename QueryExpression#evalConcurrently to QueryExpression#parEval. (parallelism is not concurrency! See https://existentialtype.wordpress.com/2011/03/17/parallelism-is-not-concurrency/)
-Have SkyQueryEnvironment#eval (used recursively in #evaluateQuery) dynamically call QueryExpression#parEval when appropriate.
-Delete QueryExpression#canEvalConcurrently.
-Add ThreadSafety annotations in a bunch of relevant places in the query codebase.
-A bunch of testing infrastructure to test parallel query evaluation.
-TODOs for implementing parallel evaluation of all QueryExpression nodes.
--
MOS_MIGRATED_REVID=132436340
|
|
|
|
|
|
|
| |
The only place we now don't handle InterruptedException is in the action graph created after analysis, since I'm not sure that will be around for that much longer.
--
MOS_MIGRATED_REVID=130327770
|
|
|
|
|
|
|
|
|
| |
the left-hand side needs to be pinned fully into memory.
Intersection is not associative, so we can't do the same thing there. For example, "abc" ^ "abc" = "abc", but "abc" ^ "a" ^ "b" ^ "c" = <null>.
--
MOS_MIGRATED_REVID=130015098
|