aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/query2/SkyQueryEnvironment.java
Commit message (Collapse)AuthorAge
...
* Precompute the universe key in SkyQueryEnvironment. Since it's a function of ↵Gravatar Janak Ramakrishnan2016-11-09
| | | | | | | the scope and the parser prefix, both of which are final, it never changes. -- MOS_MIGRATED_REVID=138536185
* Loosen visibility and finality of *QueryEnvironment fields.Gravatar Janak Ramakrishnan2016-11-09
| | | | | -- MOS_MIGRATED_REVID=138534021
* Defer targetification of SkyKeys during unbounded allrdeps evaluationGravatar Googler2016-10-24
| | | | | -- MOS_MIGRATED_REVID=137064426
* In AllRdepsUnboundedVisitor, group rdeps by package before partitioning them ↵Gravatar Nathan Harmata2016-10-11
| | | | | | | into batches. This way nodes corresponding to targets in the same package are likely to be in the same batch. -- MOS_MIGRATED_REVID=135684088
* Split SkyQueryEnvironment#getReverseDeps into a few helper methods and use ↵Gravatar Nathan Harmata2016-10-11
| | | | | | | | | them in ParallelSkyQueryUtils. In AllRdepsUnboundedVisitor#getVisitResult, only get rdeps of the nodes in question. As long as we're careful to only visit nodes have corresponding Targets (which we already are doing), there's no need to also get the Targets corresponding to the nodes in question. -- MOS_MIGRATED_REVID=135680956
* Unify AbstractBlazeQueryEnvironment#evaluateQuery with its subclass ↵Gravatar Nathan Harmata2016-10-04
| | | | | | | overrides. Also, have AbstractBlazeQueryEnvironment#evaluateQuery take an OutputFormatterCallback instance rather than a Callback instance. This is more sensible since the latter is only intended to be used intra-query, while the former is intended for usage in end-to-end query evaluation. This lets us slightly simplify QueryCommand, by shifting the responsibility for managing the OutputFormatterCallback to AbstractBlazeQueryEnvironment#evaluateQuery. -- MOS_MIGRATED_REVID=134827588
* Provide parallel implementations of 'allrdeps' and 'rbuildfiles', using a ↵Gravatar Nathan Harmata2016-09-29
| | | | | | | shared naive BFS implementation. Also implement RegexFilterExpression#parEval. -- MOS_MIGRATED_REVID=134598046
* Make AbstractBlazeQueryEnvironment no longer implement AutoCloseable. ↵Gravatar Nathan Harmata2016-09-29
| | | | | | | | | | | Instead have SkyQueryEnvironment#evaluateQuery be responsible for handling cleanup of its internal ForkJoinPool. In addition to being a more sensible way of organizing the code (imo, it makes sense for SkyQueryEnvironment to clean up after itself), this fixes several issues: (i) If query evaluation is interrupted, the AbstractBlazeQueryEnvironment#close call at the end of the try-with-resources statement in QueryCommand would be blocking and non-urgent. N.B. This was not an issue with the current ForkJoinPool usage, but was an issue with the old ThreadPoolExecutor usage. (ii) Because of how the code in QueryCommand was structured, OutputFormatterCallback#close would happen _before_ the AbstractBlazeQueryEnvironment#close call. If query evaluation is interrupted, threads executing query tasks may be invoking the callback after the callback had been shut down! -- MOS_MIGRATED_REVID=134573395
* Use ForkJoinPool, rather than ListeningExecutorService, for parallel query ↵Gravatar Nathan Harmata2016-09-22
| | | | | | | evaluation in SkyQueryEnvironment. FJP is nicer to program against, imo. -- MOS_MIGRATED_REVID=133844508
* Make a couple helper methods in SkyQueryEnvironment public.Gravatar Nathan Harmata2016-09-21
| | | | | | | FIXED=31439098 -- MOS_MIGRATED_REVID=133751771
* Have SkyQueryEnvironment#getRBuildFiles not visit skyframe nodes more than ↵Gravatar Nathan Harmata2016-09-12
| | | | | | | once. This is helpful when there are multiple skylark reverse 'load' paths from the same popular .bzl file to a single BUILD file. As an implementation detail, introduce a few ThreadSafeUniquifier things (these will also be used in followup CLs). -- MOS_MIGRATED_REVID=132680777
* A bunch of small changes to prepare SkyQueryEnvironment for full-parallel ↵Gravatar Nathan Harmata2016-09-08
| | | | | | | | | | | | | | evaluation: -Rename QueryExpression#evalConcurrently to QueryExpression#parEval. (parallelism is not concurrency! See https://existentialtype.wordpress.com/2011/03/17/parallelism-is-not-concurrency/) -Have SkyQueryEnvironment#eval (used recursively in #evaluateQuery) dynamically call QueryExpression#parEval when appropriate. -Delete QueryExpression#canEvalConcurrently. -Add ThreadSafety annotations in a bunch of relevant places in the query codebase. -A bunch of testing infrastructure to test parallel query evaluation. -TODOs for implementing parallel evaluation of all QueryExpression nodes. -- MOS_MIGRATED_REVID=132436340
* Remove unnecessary wrapping of InterruptedException in SkyQueryEnvironment.Gravatar Janak Ramakrishnan2016-08-18
| | | | | -- MOS_MIGRATED_REVID=130535008
* Allow Skyframe graph lookups and value retrievals to throw InterruptedException.Gravatar Janak Ramakrishnan2016-08-16
| | | | | | | The only place we now don't handle InterruptedException is in the action graph created after analysis, since I'm not sure that will be around for that much longer. -- MOS_MIGRATED_REVID=130327770
* Properly size compact hash map in order to mitigate GC churn.Gravatar Eric Fellheimer2016-08-11
| | | | | -- MOS_MIGRATED_REVID=129912633
* Re-implement variables in the blaze query language. Instead of using a ↵Gravatar Nathan Harmata2016-07-14
| | | | | | | | | mutable global context of variable bindings, pass around immutable local contexts. The motivation is so we can safely evaluate all blaze query expressions concurrently under the hood. A global context is hostile to this goal. -- MOS_MIGRATED_REVID=127324600
* No longer allow QueryExpressionMapper#map to throw a QueryException. This ↵Gravatar Nathan Harmata2016-07-04
| | | | | | | was unnecessary. -- MOS_MIGRATED_REVID=126457804
* (1) Allow QueryExpressionMapper#map to throw a QueryException.Gravatar Nathan Harmata2016-07-01
| | | | | | | | (2) Refactor the simple query concurrency support by introducing QueryExpression#canEvalConcurrently and removing the primitive BinaryOperatorExpression#evalConcurrently and the hardcoded SkyQueryEnvironment logic around it. This way each QueryExpression can safely manage its own concurrency. A followup CL will ensure that concurrent evaluation occurs for as much of the query expression as possible, rather than just for the top-level query expression node. (3) Make a few query internals public. -- MOS_MIGRATED_REVID=126324637
* Cap depserver log lines of blaze stderr lines at 1000 chars. Implement both ↵Gravatar Nathan Harmata2016-06-30
| | | | | | | this and commit 116c2f6724718fc408e0c73c9026d89281c75e87 using Guava's Ascii#truncate. -- MOS_MIGRATED_REVID=126198762
* Truncate the log line for transformed query expressions larger than 1000 chars.Gravatar Nathan Harmata2016-06-29
| | | | | -- MOS_MIGRATED_REVID=126079533
* Fix threadpool leak in SkyQueryEnvironmentGravatar Mark Schaller2016-06-27
| | | | | | | | Shutdown the SkyQueryEnvironment's threadpool after query evaluation is complete and the environment is ready for disposal. -- MOS_MIGRATED_REVID=125975317
* Add SkyQuery-specific allrdeps implementation to allow batch getReverseDepsGravatar Googler2016-06-27
| | | | | | | with maximum limit -- MOS_MIGRATED_REVID=125959807
* Parallelize (some of) BinaryOperatorExpressionGravatar Mark Schaller2016-06-22
| | | | | | | | | | | | | | | | | | | | | | | | Adds evalConcurrently to QueryExpression so that expression implementations that support concurrent evaluation can do so using the supplied executor service. Implements concurrent evaluation for the PLUS/UNION cases of BinaryOperatorExpression. Because evalConcurrently requires its callback to be threadsafe, but the callback passed to evaluateQuery may only be called by one thread at a time, this change makes the BatchStreamedCallback constructed by SkyQueryEnvironment threadsafe, including its uniquifier. However, there is a thread-safety problem when the operands of BinaryOperatorExpression are LetExpressions, because their evaluation involves mutating state in the query environment. A future change will fix that. For now, concurrent evaluation is only attempted when the query expression is a BinaryOperatorExpression and all its operands are target literals. -- MOS_MIGRATED_REVID=125505247
* Don't create batching callback in SkyQueryEnvironment.getTargetsMatchingPatternGravatar Mark Schaller2016-06-22
| | | | | | | | | | | | | | For SkyQueryEnvironment, the labelFilter parameter was a predicate that always returned true, so the value of strictScope (read only when the predicate returned false) didn't matter. In addition to batching, the responsibility of the callback constructed in getTargetsMatchingPattern was to apply the filter. Because the filter always returned true, and because there's already a batching callback constructed in evaluateQuery, the getTargetsMatchingPattern callback wasn't doing any useful work. -- MOS_MIGRATED_REVID=125502498
* Create one TargetPatternResolver per SkyQueryEnvironment initializationGravatar Mark Schaller2016-06-22
| | | | | | | | | | | | | | | | | | Instead of creating one GraphBackedRecursivePackageProvider, thread pool, and RecursivePackageProviderBackedTargetPatternResolver per call to getTargetsMatchingPattern, create them once during field initialization and in the init method. Previously, Recursive[..]TargetPatternResolver expected to be given ownership of the executor, so it could shut it down. With this change, the resolver now employs a different technique for blocking on the completion of its asynchronous calls. The change also fixes an issue where the use of the resolver along with the EnvironmentBackedRecursivePackageProvider worked only because the provided executor was a direct executor. -- MOS_MIGRATED_REVID=125499330
* Skip preloading in SkyQueryEnvironmentGravatar Mark Schaller2016-06-22
| | | | | | | | | SkyQueryEnvironment almost always does no useful work in preloading target patterns. With this CL, preloading no longer happens, and we can delete code paths that depended on preloading. -- MOS_MIGRATED_REVID=125492408
* Avoid nested BatchStreamedCallbacks in SkyQueryEnvironmentGravatar Mark Schaller2016-06-22
| | | | | | | | | | | | | | | | | | | | Previously each call to SkyQueryEnvironment.eval wrapped the provided callback in a new batched callback. Nesting batched callbacks results in unnecessary work. With this change, SkyQueryEnvironment will construct one batched callback in evaluateQuery (the top-level function called during query evaluation). Because batched callbacks need to be flushed using processLastPending(), it was convenient not to call through to the base class's implementation of evaluateQuery, but to inline it in SkyQueryEnvironment and add the processLastPending call at the appropriate place. This also clears the way for future customizations to SkyQueryEnvironment's implementation of evaluateQuery. -- MOS_MIGRATED_REVID=125477770
* Allow blacklisted package prefixes to be passed in lazily via a Supplier.Gravatar Eric Fellheimer2016-05-30
| | | | | -- MOS_MIGRATED_REVID=123424674
* Batch package retrieval for rbuildfiles so that even if a single file has a ↵Gravatar Janak Ramakrishnan2016-05-17
| | | | | | | large number of rbuildfile dependencies, we retrieve them in batches of at most BATCH_CALLBACK_SIZE. -- MOS_MIGRATED_REVID=122540401
* Stream output of rbuildfiles operator to a callback. This function was ↵Gravatar Janak Ramakrishnan2016-05-06
| | | | | | | missed when we transformed to callbacks. -- MOS_MIGRATED_REVID=121619124
* Delay retrieval of the blacklist patterns file from the graph until we ↵Gravatar Janak Ramakrishnan2016-05-02
| | | | | | | actually need it. -- MOS_MIGRATED_REVID=121160209
* Introduce Label.EXTERNAL_PACKAGE_FILE_NAME as a convenience. Note that we ↵Gravatar Nathan Harmata2016-04-27
| | | | | | | already have Label.EXTERNAL_PACKAGE_NAME. -- MOS_MIGRATED_REVID=120828276
* Fix SkyQuery bug where we weren't respecting the package blacklist. We do ↵Gravatar Nathan Harmata2016-04-18
| | | | | | | | | this by changing both the relevant Skyframe and the SkyQuery code to propagate (minimal!) blacklist information in the SkyKeys themselves. There are other approaches to solving this problem, but I like how this solution doesn't involve duplication of logic. Also, it has the following nice benefit: previously, RecursiveDirectoryTraversalFunction would declare a dep on the blacklist for every directory traversed which adds an edge for each directory traversed. -- MOS_MIGRATED_REVID=120049635
* Remove the WalkableGraphFactory#afterUse hook, which was only called in ↵Gravatar Nathan Harmata2016-03-21
| | | | | | | SkyQueryEnvironment. QueryEnvironmentFactory, recently introduced by unknown commit, is a much more general purpose mechanism. -- MOS_MIGRATED_REVID=117590252
* Don't keep packages in the default repository around after loading.Gravatar Brian Silverman2016-03-17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, this would get thrown when referring to the same package from both the main and default repositories: java.lang.IllegalArgumentException: Multiple entries with same key: tools/cpp=/home/brian/971-Robot-Code and tools/cpp=/home/brian/971-Robot-Code at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:136) at com.google.common.collect.RegularImmutableMap.checkNoConflictInKeyBucket(RegularImmutableMap.java:98) at com.google.common.collect.RegularImmutableMap.fromEntryArray(RegularImmutableMap.java:84) at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:295) at com.google.devtools.build.lib.buildtool.BuildTool.transformPackageRoots(BuildTool.java:301) at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:209) at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:334) at com.google.devtools.build.lib.runtime.commands.TestCommand.doTest(TestCommand.java:119) at com.google.devtools.build.lib.runtime.commands.TestCommand.exec(TestCommand.java:104) at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:371) at com.google.devtools.build.lib.runtime.BlazeRuntime$3.exec(BlazeRuntime.java:1016) at com.google.devtools.build.lib.server.RPCService.executeRequest(RPCService.java:65) at com.google.devtools.build.lib.server.RPCServer.executeRequest(RPCServer.java:434) at com.google.devtools.build.lib.server.RPCServer.serve(RPCServer.java:229) at com.google.devtools.build.lib.runtime.BlazeRuntime.serverMain(BlazeRuntime.java:975) at com.google.devtools.build.lib.runtime.BlazeRuntime.main(BlazeRuntime.java:772) at com.google.devtools.build.lib.bazel.BazelMain.main(BazelMain.java:55) And this would get thrown for any packages in the main repository loaded from other repositories: java.lang.RuntimeException: Unrecoverable error while evaluating node 'PACKAGE:@//tools/build_rules/go/toolchain' (requested by nodes ) at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:982) at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:499) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalArgumentException: Invalid BUILD file name for package '@//tools/build_rules/go/toolchain': /home/brian/bazel/tools/build_rules/go/toolchain/BUILD at com.google.devtools.build.lib.packages.Package.finishInit(Package.java:299) at com.google.devtools.build.lib.packages.Package$Builder.finishBuild(Package.java:1308) at com.google.devtools.build.lib.skyframe.PackageFunction.compute(PackageFunction.java:501) at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:933) ... 4 more Sponsor's comment: note the abundance of new Label.resolveRepositoryRelative() calls. They are ugly, but it's only making existing ugliness explicit. Yes, we should fix it, especially in the implementation of configurable attributes. Refs #940 -- Change-Id: I8bd7f7b00bec58a7157507595421bc50c81b404c Reviewed-on: https://bazel-review.googlesource.com/#/c/2591 MOS_MIGRATED_REVID=117429733
* In SkyQueryEnvironment, rewrite queries using the semantics-preserving ↵Gravatar Nathan Harmata2016-03-02
| | | | | | | | | transformation 'rdeps(<sky_query_environment_universe_scope>, T, depth)' -> 'allrdeps(T, depth)'. SkyQueryEnvironment can evaluate such allrdeps queries much more efficiently since it doesn't need to bother filtering out targets outside of universe, meaning it doesn't need to have all targets in the universe in memory at the same time. -- MOS_MIGRATED_REVID=116075008
* Parallelize Package retrieval during Sky-query operation. To maintain ↵Gravatar Eric Fellheimer2016-01-29
| | | | | | | type-safety, we now must pass in the exception type of the callback. -- MOS_MIGRATED_REVID=113313312
* Refactor dependency filtering out of Rule class.Gravatar Dmitry Lomov2016-01-22
| | | | | -- MOS_MIGRATED_REVID=112717648
* Stop filtering out targets not in the graph in SkyQueryEnvironment. Instead, ↵Gravatar Janak Ramakrishnan2016-01-15
| | | | | | | | | | | | | | just warn if a target turns out to not be present. This means that queries may return unexpected results. For instance, if "query" means "bazel query --order_output=no ", then here are the results of two queries: query --universe_scope=//foo/... //foo:output_file //foo:output_file query --universe_scope=//foo/... deps(//foo:output_file) WARNING: Targets not in graph [//foo:output_file generated_file] //foo:output_file -- MOS_MIGRATED_REVID=112163475
* Use AutoProfiler in SkyQueryEnvironment#init. Also augment the ↵Gravatar Nathan Harmata2016-01-14
| | | | | | | WalkableGraphFactory interface with a hook to inform the factory when the WalkableGraph is done being used. -- MOS_MIGRATED_REVID=112074284
* When transforming labels into targets in SkyQueryEnvironment, stream the ↵Gravatar Janak Ramakrishnan2016-01-11
| | | | | | | | | result to a callback instead of returning it directly. This means that the targets a precomputed pattern resolves to can be processed incrementally. This is the sixth and hopefully final step in a series to allow processing large sets of targets in query target patterns via streaming batches rather than all at once. This should improve performance for SkyQueryEnvironment for certain classes of large queries. -- MOS_MIGRATED_REVID=111697983
* Stream result of TargetPattern#eval to a callback instead of returning it ↵Gravatar Janak Ramakrishnan2016-01-11
| | | | | | | | | directly, and pass a Query callback in when resolving target patterns. This means that the targets a pattern resolves to can be processed incrementally. This is the fifth step in a series to allow processing large sets of targets in query target patterns via streaming batches rather than all at once. This should improve performance for SkyQueryEnvironment for certain classes of large queries. -- MOS_MIGRATED_REVID=111696713
* Resolve target patterns on the fly in SkyQueryEnvironment. Cache only the ↵Gravatar Janak Ramakrishnan2016-01-07
| | | | | | | | | label sets that are precomputed in the graph. This is the fourth step in a series to allow processing large sets of targets in query target patterns via streaming batches rather than all at once. This may make SkyQueryEnvironment slower when evaluating queries with repeated target patterns, or many target patterns that would benefit from graph lookups that were batched across all patterns. But that is not currently a bottleneck we're concerned about. -- MOS_MIGRATED_REVID=111626483
* Make QueryEnvironment#getTargetsMatchingPattern pass its result to a callback.Gravatar Janak Ramakrishnan2016-01-07
| | | | | | | This is the third step in a series to allow processing large sets of targets in query target patterns via streaming batches rather than all at once. This should also be a functional no-op. -- MOS_MIGRATED_REVID=111620668
* In SkyQueryEnvironment, don't silently give up when there's a cycle in the ↵Gravatar Janak Ramakrishnan2016-01-07
| | | | | | | graph. We can compute the universe target patterns outside of skyframe, which is the only reason we need the value we were requesting. Giving up was preventing us from evaluating "..." patterns even if the "..." pattern didn't contain any cycles itself. -- MOS_MIGRATED_REVID=111605976
* Don't throw a TargetParsingException/QueryException in keepGoing mode when ↵Gravatar Janak Ramakrishnan2016-01-07
| | | | | | | we encounter a bad target pattern. -- MOS_MIGRATED_REVID=111374219
* RELNOTES: add loadfiles() query operator, to find skylark files loaded by ↵Gravatar Han-Wen Nienhuys2015-12-16
| | | | | | | targets. -- MOS_MIGRATED_REVID=110275427
* Use Bazel Preconditions variant which avoids varargs array creationGravatar Mark Schaller2015-12-10
| | | | | | | Reduces garbage. -- MOS_MIGRATED_REVID=109914243
* Allow output formatters to work in stream mode.Gravatar Miguel Alcon Pinto2015-12-10
| | | | | -- MOS_MIGRATED_REVID=109908202
* Fast-path sky-query environment in the case where we have no dependency ↵Gravatar Eric Fellheimer2015-12-07
| | | | | | | | | filter on edge types (eg, implicit deps). In this case we can avoid doing an iteration over a potential reverse dependency's dependencies, since all edges are assumed to be valid. -- MOS_MIGRATED_REVID=109575545
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-07 06:55:37 +0000