aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/query2
Commit message (Collapse)AuthorAge
* Remove dead code referencing deleted "suites" attribute.Gravatar Greg Estren2016-12-21
| | | | | | | | | This used to be how test_suites depended on other test_suites. Now they just go in "tests". -- PiperOrigin-RevId: 142607603 MOS_MIGRATED_REVID=142607603
* Rollback of commit e0a330577d9fe98169645cb68d9fc22cc787eeb6.Gravatar Janak Ramakrishnan2016-12-21
| | | | | | | | | | *** Reason for rollback *** Investigating if causes deadlock/thread starvation. -- PiperOrigin-RevId: 142575769 MOS_MIGRATED_REVID=142575769
* Add some logging when query evaluation halts abruptly.Gravatar Nathan Harmata2016-12-16
| | | | | | -- PiperOrigin-RevId: 142295816 MOS_MIGRATED_REVID=142295816
* Some improvements to ParallelQueryUtils.Gravatar Nathan Harmata2016-12-16
| | | | | | | | | | | | | | | | | | | | | | | | (i) Use a CountDownLatch in ParallelQueryUtils#executeQueryTasksAndWaitInterruptibly to avoid busy-looping while waiting for query subtask completion (this busy-looping unnecessarily ties up a thread). But we still retain the fail-fast semantics we want (I renamed the method to emphasize this). (ii) Also have a special-case in ParallelQueryUtils#executeQueryTasksAndWaitInterruptibly for evaluating one query subtask so we don't wastefully use another thread. (iii) Also add ThreadSafety annotations to ParallelQueryUtils. ---- (i) and (ii) combine to address the following theoretical issue. Suppose we're evaluating a query expression of the form "(e1 - e2) + (e3 - e4)". The old code would (with the worst-case FJP thread scheduling) have the following threads at the _same_ time: Main QueryCommand thread - executeQueryTasksAndWaitInterruptibly(queryTasks = [(e1 - e2), (e3 - e4)] FJP thread - executeQueryTasksAndWaitInterruptibly(queryTasks = [e2]) FJP thread - eval(e2) FJP thread - executeQueryTasksAndWaitInterruptibly(queryTasks = [e4]) FJP thread - eval(e4) So of those 5 concurrent threads, 3 would be doing busy-loop waiting. For more pathological query expressions, we could end up tying up lots of threads doing wasteful busy-loops. -- PiperOrigin-RevId: 142215680 MOS_MIGRATED_REVID=142215680
* Make Argument factory methods public.Gravatar Nathan Harmata2016-12-13
| | | | | | -- PiperOrigin-RevId: 141920696 MOS_MIGRATED_REVID=141920696
* Rephrase query transformation in terms of composition of QueryExpressionMappers.Gravatar Nathan Harmata2016-12-13
| | | | | | -- PiperOrigin-RevId: 141904124 MOS_MIGRATED_REVID=141904124
* Fix bad bug with the parallel implementation of BinaryOperatorExpression. ↵Gravatar Nathan Harmata2016-12-12
| | | | | | | | | | | | | | Turns out that ForkJoinTask#adapt(Callable) returns a ForkJoinTask whose Future#get on error throws a ExecutionException wrapping a RuntimeException wrapping the thrown checked exception from the callable. This is documented behavior [1] that I incorrectly didn't know about. The additional level of wrapping meant that the catch-block of the parallel implementation of BinaryOperatorExpression wasn't rethrowing the InterruptedException/QueryException that the parallel task threw. The subtly in this bug is that the query expression being evaluated needs to be of the form "e1 + e2", where evaluation of "e1" throws a QueryException even in keepGoing mode (note that most of the query errors actually go through AbstractBlazeQueryEnvironment#reportBuildFileError). The test I wrote picks on LetExpression's evaluation-time (rather than e.g. parsing time) validation of the variable name. [1] https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ForkJoinTask.html#adapt(java.util.concurrent.Callable) -- PiperOrigin-RevId: 141772584 MOS_MIGRATED_REVID=141772584
* Global cleanup change.Gravatar Googler2016-12-07
| | | | | | -- PiperOrigin-RevId: 141307853 MOS_MIGRATED_REVID=141307853
* Fix spelling mistake.Gravatar Nathan Harmata2016-12-06
| | | | | | -- PiperOrigin-RevId: 141178325 MOS_MIGRATED_REVID=141178325
* For all function expressions of the form f(..., e1, ..., e2, ..., eK, ...), ↵Gravatar Nathan Harmata2016-12-02
| | | | | | | | ensure that all of e1, e2, ..., and eK are elligble for parallel evaluation. This is _not_ the same as providing a parallel implementation of f, which we can do separately in followup CLs. -- PiperOrigin-RevId: 140861694 MOS_MIGRATED_REVID=140861694
* Update ParallelSkyQueryUtils to use QuiescingExecutor instead of ForkJoinPoolGravatar Googler2016-11-29
| | | | | | | | | | | for concurrent visitations. During BFS visitation of rdeps and rbuildfiles, it uses a centralized pool (backed by a LinkedBlockingQueue) to store all pending visits, and a periodically running scheduler to schedule tasks for each pending visit. -- MOS_MIGRATED_REVID=140398162
* Provide a parallel implementation of "e1 - e2 - e3" by noting its ↵Gravatar Nathan Harmata2016-11-21
| | | | | | | equivalence to "e1 - (e2 + e3)" and the fact that we already have a parallel implementation of "e2 + e3". -- MOS_MIGRATED_REVID=139792288
* Enable parallel evaluation of 'e' in query expressions of the form ↵Gravatar Nathan Harmata2016-11-21
| | | | | | | 'buildfiles(e)'. -- MOS_MIGRATED_REVID=139787078
* Provide a parallel implementation of "e1 + e2".Gravatar Nathan Harmata2016-11-21
| | | | | -- MOS_MIGRATED_REVID=139613681
* Introduce a failFast mode to OutputFormatterCallback#close.Gravatar Nathan Harmata2016-11-18
| | | | | -- MOS_MIGRATED_REVID=139508838
* Cache all previously computed values in ↵Gravatar Janak Ramakrishnan2016-11-17
| | | | | | | | | SkyQueryEnvironment#beforeEvaluateQuery if possible to save on latency for small queries. This assumes that if the graph is up to date, then the data in SkyQueryEnvironment is also up to date. It also assumes that a ForkJoinPool remains usable until #shutdownNow is called. -- MOS_MIGRATED_REVID=139386363
* Re-use the precomputed universe key in WalkableGraphFactory#prepareAndGet.Gravatar Janak Ramakrishnan2016-11-17
| | | | | -- MOS_MIGRATED_REVID=139357569
* Avoid AutoProfiler overhead in AbstractBlazeQueryEnvironment#evaluateQuery.Gravatar Janak Ramakrishnan2016-11-15
| | | | | -- MOS_MIGRATED_REVID=139113333
* Make TargetPattern evaluation during query evaluation more parallel-friendly ↵Gravatar Nathan Harmata2016-11-15
| | | | | | | by introducing TargetPattern#parEval, which allows TargetPatterns' evaluations to explicitly have parallel implementations (no need to secretly use a FJP). -- MOS_MIGRATED_REVID=139101922
* In SkyQueryEnvironment, cache some more objects that are just functions of ↵Gravatar Janak Ramakrishnan2016-11-14
| | | | | | | final fields. -- MOS_MIGRATED_REVID=138894804
* Add a mechanism for bounding the number of Packages SkyQueryEnvironment's ↵Gravatar Nathan Harmata2016-11-11
| | | | | | | expensive parallel operations can operate on at once. -- MOS_MIGRATED_REVID=138779172
* Precompute the universe key in SkyQueryEnvironment. Since it's a function of ↵Gravatar Janak Ramakrishnan2016-11-09
| | | | | | | the scope and the parser prefix, both of which are final, it never changes. -- MOS_MIGRATED_REVID=138536185
* Loosen visibility and finality of *QueryEnvironment fields.Gravatar Janak Ramakrishnan2016-11-09
| | | | | -- MOS_MIGRATED_REVID=138534021
* Have query Lexer operated directly on StringGravatar Michajlo Matijkiw2016-11-02
| | | | | | | | No need for the char[] in the middle, prevents us from accidentally modifying input, or sucking up ram on huge queries. -- MOS_MIGRATED_REVID=137872573
* bazel query: Gravatar Greg Estren2016-10-28
| | | | | | | | | - Extend https://github.com/bazelbuild/bazel/commit/83ffb5e7bc034ea2dd1e957249079b3884a9450d to --output=xml, --output=proto, too. This makes query much more resilient against freezing / OOMing from "select() + select() + select() + ..." patterns. - Make the above logic accessible to any output type. - Move AggregatingAttributeMapper.flattenAttributeValues to ProtoOutputFormatter, since this is really a utility method for ProtoOutputFormatter. -- MOS_MIGRATED_REVID=137429116
* Flip the order of recursive ForkJoinTasks generated during parallel allrdeps ↵Gravatar Nathan Harmata2016-10-27
| | | | | | | evaluation: put the outputting of targets before the recursive visitation. Since ForkJoinTasks are fork'd in order, the new order optimizes for throughput to the underlying Callback. -- MOS_MIGRATED_REVID=137281325
* bazel query --output=xml: avoid iterating over all possible values forGravatar Greg Estren2016-10-26
| | | | | | | | attr = select({"a": LABEL_LIST, "b": LABEL_LIST, ...}). This can be exponential (w.r.t. the # of select conditions) and produce OOMs. -- MOS_MIGRATED_REVID=137200979
* Add generic parameter to make ParallelSkyQueryUtil Java7 compliantGravatar Googler2016-10-25
| | | | | -- MOS_MIGRATED_REVID=137192333
* Defer targetification of SkyKeys during unbounded allrdeps evaluationGravatar Googler2016-10-24
| | | | | -- MOS_MIGRATED_REVID=137064426
* Rollback of commit e6191f7209ebf3f67480bc7aa9db5ac58beb2156.Gravatar Damien Martin-Guillerez2016-10-19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Roll-forward with fix. Tested with `bazel build src:srcs //src/test/...` using a bootstrapped bazel. Fixes #1923. RELNOTES[INC]: Non test-only targets can no longer depends on test-only targets. *** Original change description *** Automated [] rollback of commit a9f20b0d6459d395444c45cf5e94a899f3443633. *** Reason for rollback *** Broke Bazel CI: ERROR: /home/ci/workspace/Bazel/JAVA_VERSION/1.8/PLATFORM_NAME/linux-x86_64/src/BUILD:284:1: in filegroup rule //src:srcs: non-test target '//src:srcs' depends on testonly target '//src/java_tools/junitrunner/java/com/google/testing/junit/junit4:srcs' and doesn't have testonly attribute set. ERROR: Analysis of target '//scripts/packages:packages' failed; build aborted. http://ci.bazel.io/view/Bazel%20b... *** -- MOS_MIGRATED_REVID=136589937
* In AllRdepsUnboundedVisitor, group rdeps by package before partitioning them ↵Gravatar Nathan Harmata2016-10-11
| | | | | | | into batches. This way nodes corresponding to targets in the same package are likely to be in the same batch. -- MOS_MIGRATED_REVID=135684088
* Split SkyQueryEnvironment#getReverseDeps into a few helper methods and use ↵Gravatar Nathan Harmata2016-10-11
| | | | | | | | | them in ParallelSkyQueryUtils. In AllRdepsUnboundedVisitor#getVisitResult, only get rdeps of the nodes in question. As long as we're careful to only visit nodes have corresponding Targets (which we already are doing), there's no need to also get the Targets corresponding to the nodes in question. -- MOS_MIGRATED_REVID=135680956
* Make a method in ProtoOutputFormatter visible for tests.Gravatar Janak Ramakrishnan2016-10-10
| | | | | -- MOS_MIGRATED_REVID=135495077
* Unify AbstractBlazeQueryEnvironment#evaluateQuery with its subclass ↵Gravatar Nathan Harmata2016-10-04
| | | | | | | overrides. Also, have AbstractBlazeQueryEnvironment#evaluateQuery take an OutputFormatterCallback instance rather than a Callback instance. This is more sensible since the latter is only intended to be used intra-query, while the former is intended for usage in end-to-end query evaluation. This lets us slightly simplify QueryCommand, by shifting the responsibility for managing the OutputFormatterCallback to AbstractBlazeQueryEnvironment#evaluateQuery. -- MOS_MIGRATED_REVID=134827588
* Add type annotation for the benefit of java 7Gravatar Klaus Aehlig2016-09-30
| | | | | | | Fixes #1863 -- MOS_MIGRATED_REVID=134767527
* Don't force PrintStream on streaming output formattersGravatar Michajlo Matijkiw2016-09-30
| | | | | | | | Usually an OutputStream will do. Forgo the extra layer of indirection and stream directly to the output. -- MOS_MIGRATED_REVID=134682243
* Provide parallel implementations of 'allrdeps' and 'rbuildfiles', using a ↵Gravatar Nathan Harmata2016-09-29
| | | | | | | shared naive BFS implementation. Also implement RegexFilterExpression#parEval. -- MOS_MIGRATED_REVID=134598046
* Make AbstractBlazeQueryEnvironment no longer implement AutoCloseable. ↵Gravatar Nathan Harmata2016-09-29
| | | | | | | | | | | Instead have SkyQueryEnvironment#evaluateQuery be responsible for handling cleanup of its internal ForkJoinPool. In addition to being a more sensible way of organizing the code (imo, it makes sense for SkyQueryEnvironment to clean up after itself), this fixes several issues: (i) If query evaluation is interrupted, the AbstractBlazeQueryEnvironment#close call at the end of the try-with-resources statement in QueryCommand would be blocking and non-urgent. N.B. This was not an issue with the current ForkJoinPool usage, but was an issue with the old ThreadPoolExecutor usage. (ii) Because of how the code in QueryCommand was structured, OutputFormatterCallback#close would happen _before_ the AbstractBlazeQueryEnvironment#close call. If query evaluation is interrupted, threads executing query tasks may be invoking the callback after the callback had been shut down! -- MOS_MIGRATED_REVID=134573395
* Use ForkJoinPool, rather than ListeningExecutorService, for parallel query ↵Gravatar Nathan Harmata2016-09-22
| | | | | | | evaluation in SkyQueryEnvironment. FJP is nicer to program against, imo. -- MOS_MIGRATED_REVID=133844508
* Make a couple helper methods in SkyQueryEnvironment public.Gravatar Nathan Harmata2016-09-21
| | | | | | | FIXED=31439098 -- MOS_MIGRATED_REVID=133751771
* Introduce a new method on the StreamedFormatter interface for creating a ↵Gravatar Nathan Harmata2016-09-21
| | | | | | | callback for streaming a precomputed result. -- MOS_MIGRATED_REVID=133720742
* Have SkyQueryEnvironment#getRBuildFiles not visit skyframe nodes more than ↵Gravatar Nathan Harmata2016-09-12
| | | | | | | once. This is helpful when there are multiple skylark reverse 'load' paths from the same popular .bzl file to a single BUILD file. As an implementation detail, introduce a few ThreadSafeUniquifier things (these will also be used in followup CLs). -- MOS_MIGRATED_REVID=132680777
* Boilerplate for parallel evaluation of FunctionExpression query expression ↵Gravatar Nathan Harmata2016-09-09
| | | | | | | nodes. -- MOS_MIGRATED_REVID=132558031
* A bunch of small changes to prepare SkyQueryEnvironment for full-parallel ↵Gravatar Nathan Harmata2016-09-08
| | | | | | | | | | | | | | evaluation: -Rename QueryExpression#evalConcurrently to QueryExpression#parEval. (parallelism is not concurrency! See https://existentialtype.wordpress.com/2011/03/17/parallelism-is-not-concurrency/) -Have SkyQueryEnvironment#eval (used recursively in #evaluateQuery) dynamically call QueryExpression#parEval when appropriate. -Delete QueryExpression#canEvalConcurrently. -Add ThreadSafety annotations in a bunch of relevant places in the query codebase. -A bunch of testing infrastructure to test parallel query evaluation. -TODOs for implementing parallel evaluation of all QueryExpression nodes. -- MOS_MIGRATED_REVID=132436340
* Remove unnecessary wrapping of InterruptedException in SkyQueryEnvironment.Gravatar Janak Ramakrishnan2016-08-18
| | | | | -- MOS_MIGRATED_REVID=130535008
* In BlazeQueryEnvironment, stop wrapping InterruptedExceptions in ↵Gravatar Janak Ramakrishnan2016-08-17
| | | | | | | | | QueryExceptions now that all relevant methods declare that they throw InterruptedException. Small side benefit of commit 3c0adb26bac6d756fb97e4bcc6d4e5b2cefa5eeb. -- MOS_MIGRATED_REVID=130402917
* Allow Skyframe graph lookups and value retrievals to throw InterruptedException.Gravatar Janak Ramakrishnan2016-08-16
| | | | | | | The only place we now don't handle InterruptedException is in the action graph created after analysis, since I'm not sure that will be around for that much longer. -- MOS_MIGRATED_REVID=130327770
* Stream the right-hand side of subtractive query expressions. This means only ↵Gravatar Eric Fellheimer2016-08-12
| | | | | | | | | the left-hand side needs to be pinned fully into memory. Intersection is not associative, so we can't do the same thing there. For example, "abc" ^ "abc" = "abc", but "abc" ^ "a" ^ "b" ^ "c" = <null>. -- MOS_MIGRATED_REVID=130015098
* Properly size compact hash map in order to mitigate GC churn.Gravatar Eric Fellheimer2016-08-11
| | | | | -- MOS_MIGRATED_REVID=129912633
* Added support for null terminated queries.Gravatar Andreas Bergmeier2016-08-10
| | | | | | | | Closes #1496. -- Reviewed-on: https://github.com/bazelbuild/bazel/pull/1496 MOS_MIGRATED_REVID=129846158
generated by cgit v1.2.3 (git 2.39.1) at 2024-10-21 07:49:29 +0000