| Commit message (Collapse) | Author | Age |
... | |
|
|
|
|
|
|
| |
final fields.
--
MOS_MIGRATED_REVID=138894804
|
|
|
|
|
|
|
| |
expensive parallel operations can operate on at once.
--
MOS_MIGRATED_REVID=138779172
|
|
|
|
|
|
|
| |
the scope and the parser prefix, both of which are final, it never changes.
--
MOS_MIGRATED_REVID=138536185
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=138534021
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=137064426
|
|
|
|
|
|
|
| |
into batches. This way nodes corresponding to targets in the same package are likely to be in the same batch.
--
MOS_MIGRATED_REVID=135684088
|
|
|
|
|
|
|
|
|
| |
them in ParallelSkyQueryUtils.
In AllRdepsUnboundedVisitor#getVisitResult, only get rdeps of the nodes in question. As long as we're careful to only visit nodes have corresponding Targets (which we already are doing), there's no need to also get the Targets corresponding to the nodes in question.
--
MOS_MIGRATED_REVID=135680956
|
|
|
|
|
|
|
| |
overrides. Also, have AbstractBlazeQueryEnvironment#evaluateQuery take an OutputFormatterCallback instance rather than a Callback instance. This is more sensible since the latter is only intended to be used intra-query, while the former is intended for usage in end-to-end query evaluation. This lets us slightly simplify QueryCommand, by shifting the responsibility for managing the OutputFormatterCallback to AbstractBlazeQueryEnvironment#evaluateQuery.
--
MOS_MIGRATED_REVID=134827588
|
|
|
|
|
|
|
| |
shared naive BFS implementation. Also implement RegexFilterExpression#parEval.
--
MOS_MIGRATED_REVID=134598046
|
|
|
|
|
|
|
|
|
|
|
| |
Instead have SkyQueryEnvironment#evaluateQuery be responsible for handling cleanup of its internal ForkJoinPool.
In addition to being a more sensible way of organizing the code (imo, it makes sense for SkyQueryEnvironment to clean up after itself), this fixes several issues:
(i) If query evaluation is interrupted, the AbstractBlazeQueryEnvironment#close call at the end of the try-with-resources statement in QueryCommand would be blocking and non-urgent. N.B. This was not an issue with the current ForkJoinPool usage, but was an issue with the old ThreadPoolExecutor usage.
(ii) Because of how the code in QueryCommand was structured, OutputFormatterCallback#close would happen _before_ the AbstractBlazeQueryEnvironment#close call. If query evaluation is interrupted, threads executing query tasks may be invoking the callback after the callback had been shut down!
--
MOS_MIGRATED_REVID=134573395
|
|
|
|
|
|
|
| |
evaluation in SkyQueryEnvironment. FJP is nicer to program against, imo.
--
MOS_MIGRATED_REVID=133844508
|
|
|
|
|
|
|
| |
FIXED=31439098
--
MOS_MIGRATED_REVID=133751771
|
|
|
|
|
|
|
| |
once. This is helpful when there are multiple skylark reverse 'load' paths from the same popular .bzl file to a single BUILD file. As an implementation detail, introduce a few ThreadSafeUniquifier things (these will also be used in followup CLs).
--
MOS_MIGRATED_REVID=132680777
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
evaluation:
-Rename QueryExpression#evalConcurrently to QueryExpression#parEval. (parallelism is not concurrency! See https://existentialtype.wordpress.com/2011/03/17/parallelism-is-not-concurrency/)
-Have SkyQueryEnvironment#eval (used recursively in #evaluateQuery) dynamically call QueryExpression#parEval when appropriate.
-Delete QueryExpression#canEvalConcurrently.
-Add ThreadSafety annotations in a bunch of relevant places in the query codebase.
-A bunch of testing infrastructure to test parallel query evaluation.
-TODOs for implementing parallel evaluation of all QueryExpression nodes.
--
MOS_MIGRATED_REVID=132436340
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=130535008
|
|
|
|
|
|
|
| |
The only place we now don't handle InterruptedException is in the action graph created after analysis, since I'm not sure that will be around for that much longer.
--
MOS_MIGRATED_REVID=130327770
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=129912633
|
|
|
|
|
|
|
|
|
| |
mutable global context of variable bindings, pass around immutable local contexts.
The motivation is so we can safely evaluate all blaze query expressions concurrently under the hood. A global context is hostile to this goal.
--
MOS_MIGRATED_REVID=127324600
|
|
|
|
|
|
|
| |
was unnecessary.
--
MOS_MIGRATED_REVID=126457804
|
|
|
|
|
|
|
|
| |
(2) Refactor the simple query concurrency support by introducing QueryExpression#canEvalConcurrently and removing the primitive BinaryOperatorExpression#evalConcurrently and the hardcoded SkyQueryEnvironment logic around it. This way each QueryExpression can safely manage its own concurrency. A followup CL will ensure that concurrent evaluation occurs for as much of the query expression as possible, rather than just for the top-level query expression node.
(3) Make a few query internals public.
--
MOS_MIGRATED_REVID=126324637
|
|
|
|
|
|
|
| |
this and commit 116c2f6724718fc408e0c73c9026d89281c75e87 using Guava's Ascii#truncate.
--
MOS_MIGRATED_REVID=126198762
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=126079533
|
|
|
|
|
|
|
|
| |
Shutdown the SkyQueryEnvironment's threadpool after query evaluation
is complete and the environment is ready for disposal.
--
MOS_MIGRATED_REVID=125975317
|
|
|
|
|
|
|
| |
with maximum limit
--
MOS_MIGRATED_REVID=125959807
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds evalConcurrently to QueryExpression so that expression
implementations that support concurrent evaluation can do so using the
supplied executor service.
Implements concurrent evaluation for the PLUS/UNION cases of
BinaryOperatorExpression.
Because evalConcurrently requires its callback to be threadsafe, but
the callback passed to evaluateQuery may only be called by one thread
at a time, this change makes the BatchStreamedCallback constructed by
SkyQueryEnvironment threadsafe, including its uniquifier.
However, there is a thread-safety problem when the operands of
BinaryOperatorExpression are LetExpressions, because their evaluation
involves mutating state in the query environment. A future change will
fix that. For now, concurrent evaluation is only attempted when the
query expression is a BinaryOperatorExpression and all its operands are
target literals.
--
MOS_MIGRATED_REVID=125505247
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For SkyQueryEnvironment, the labelFilter parameter was a predicate
that always returned true, so the value of strictScope (read only
when the predicate returned false) didn't matter. In addition to
batching, the responsibility of the callback constructed in
getTargetsMatchingPattern was to apply the filter. Because the filter
always returned true, and because there's already a batching callback
constructed in evaluateQuery, the getTargetsMatchingPattern callback
wasn't doing any useful work.
--
MOS_MIGRATED_REVID=125502498
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Instead of creating one GraphBackedRecursivePackageProvider, thread
pool, and RecursivePackageProviderBackedTargetPatternResolver per call
to getTargetsMatchingPattern, create them once during field
initialization and in the init method.
Previously, Recursive[..]TargetPatternResolver expected to be given
ownership of the executor, so it could shut it down. With this change,
the resolver now employs a different technique for blocking on the
completion of its asynchronous calls. The change also fixes an issue
where the use of the resolver along with the
EnvironmentBackedRecursivePackageProvider worked only because the
provided executor was a direct executor.
--
MOS_MIGRATED_REVID=125499330
|
|
|
|
|
|
|
|
|
| |
SkyQueryEnvironment almost always does no useful work in preloading
target patterns. With this CL, preloading no longer happens, and we
can delete code paths that depended on preloading.
--
MOS_MIGRATED_REVID=125492408
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously each call to SkyQueryEnvironment.eval wrapped the provided
callback in a new batched callback. Nesting batched callbacks results
in unnecessary work.
With this change, SkyQueryEnvironment will construct one batched
callback in evaluateQuery (the top-level function called during query
evaluation). Because batched callbacks need to be flushed using
processLastPending(), it was convenient not to call through to the
base class's implementation of evaluateQuery, but to inline it in
SkyQueryEnvironment and add the processLastPending call at the
appropriate place.
This also clears the way for future customizations to
SkyQueryEnvironment's implementation of evaluateQuery.
--
MOS_MIGRATED_REVID=125477770
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=123424674
|
|
|
|
|
|
|
| |
large number of rbuildfile dependencies, we retrieve them in batches of at most BATCH_CALLBACK_SIZE.
--
MOS_MIGRATED_REVID=122540401
|
|
|
|
|
|
|
| |
missed when we transformed to callbacks.
--
MOS_MIGRATED_REVID=121619124
|
|
|
|
|
|
|
| |
actually need it.
--
MOS_MIGRATED_REVID=121160209
|
|
|
|
|
|
|
| |
already have Label.EXTERNAL_PACKAGE_NAME.
--
MOS_MIGRATED_REVID=120828276
|
|
|
|
|
|
|
|
|
| |
this by changing both the relevant Skyframe and the SkyQuery code to propagate (minimal!) blacklist information in the SkyKeys themselves.
There are other approaches to solving this problem, but I like how this solution doesn't involve duplication of logic. Also, it has the following nice benefit: previously, RecursiveDirectoryTraversalFunction would declare a dep on the blacklist for every directory traversed which adds an edge for each directory traversed.
--
MOS_MIGRATED_REVID=120049635
|
|
|
|
|
|
|
| |
SkyQueryEnvironment. QueryEnvironmentFactory, recently introduced by unknown commit, is a much more general purpose mechanism.
--
MOS_MIGRATED_REVID=117590252
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously, this would get thrown when referring to the same package
from both the main and default repositories:
java.lang.IllegalArgumentException: Multiple entries with same key: tools/cpp=/home/brian/971-Robot-Code and tools/cpp=/home/brian/971-Robot-Code
at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:136)
at com.google.common.collect.RegularImmutableMap.checkNoConflictInKeyBucket(RegularImmutableMap.java:98)
at com.google.common.collect.RegularImmutableMap.fromEntryArray(RegularImmutableMap.java:84)
at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:295)
at com.google.devtools.build.lib.buildtool.BuildTool.transformPackageRoots(BuildTool.java:301)
at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:209)
at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:334)
at com.google.devtools.build.lib.runtime.commands.TestCommand.doTest(TestCommand.java:119)
at com.google.devtools.build.lib.runtime.commands.TestCommand.exec(TestCommand.java:104)
at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:371)
at com.google.devtools.build.lib.runtime.BlazeRuntime$3.exec(BlazeRuntime.java:1016)
at com.google.devtools.build.lib.server.RPCService.executeRequest(RPCService.java:65)
at com.google.devtools.build.lib.server.RPCServer.executeRequest(RPCServer.java:434)
at com.google.devtools.build.lib.server.RPCServer.serve(RPCServer.java:229)
at com.google.devtools.build.lib.runtime.BlazeRuntime.serverMain(BlazeRuntime.java:975)
at com.google.devtools.build.lib.runtime.BlazeRuntime.main(BlazeRuntime.java:772)
at com.google.devtools.build.lib.bazel.BazelMain.main(BazelMain.java:55)
And this would get thrown for any packages in the main repository loaded
from other repositories:
java.lang.RuntimeException: Unrecoverable error while evaluating node 'PACKAGE:@//tools/build_rules/go/toolchain' (requested by nodes )
at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:982)
at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:499)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Invalid BUILD file name for package '@//tools/build_rules/go/toolchain': /home/brian/bazel/tools/build_rules/go/toolchain/BUILD
at com.google.devtools.build.lib.packages.Package.finishInit(Package.java:299)
at com.google.devtools.build.lib.packages.Package$Builder.finishBuild(Package.java:1308)
at com.google.devtools.build.lib.skyframe.PackageFunction.compute(PackageFunction.java:501)
at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:933)
... 4 more
Sponsor's comment: note the abundance of new Label.resolveRepositoryRelative() calls. They are ugly, but it's only making existing ugliness explicit. Yes, we should fix it, especially in the implementation of configurable attributes.
Refs #940
--
Change-Id: I8bd7f7b00bec58a7157507595421bc50c81b404c
Reviewed-on: https://bazel-review.googlesource.com/#/c/2591
MOS_MIGRATED_REVID=117429733
|
|
|
|
|
|
|
|
|
| |
transformation 'rdeps(<sky_query_environment_universe_scope>, T, depth)' -> 'allrdeps(T, depth)'.
SkyQueryEnvironment can evaluate such allrdeps queries much more efficiently since it doesn't need to bother filtering out targets outside of universe, meaning it doesn't need to have all targets in the universe in memory at the same time.
--
MOS_MIGRATED_REVID=116075008
|
|
|
|
|
|
|
| |
type-safety, we now must pass in the exception type of the callback.
--
MOS_MIGRATED_REVID=113313312
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=112717648
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
just warn if a target turns out to not be present. This means that queries may return unexpected results. For instance, if "query" means "bazel query --order_output=no ", then here are the results of two queries:
query --universe_scope=//foo/... //foo:output_file
//foo:output_file
query --universe_scope=//foo/... deps(//foo:output_file)
WARNING: Targets not in graph [//foo:output_file generated_file]
//foo:output_file
--
MOS_MIGRATED_REVID=112163475
|
|
|
|
|
|
|
| |
WalkableGraphFactory interface with a hook to inform the factory when the WalkableGraph is done being used.
--
MOS_MIGRATED_REVID=112074284
|
|
|
|
|
|
|
|
|
| |
result to a callback instead of returning it directly. This means that the targets a precomputed pattern resolves to can be processed incrementally.
This is the sixth and hopefully final step in a series to allow processing large sets of targets in query target patterns via streaming batches rather than all at once. This should improve performance for SkyQueryEnvironment for certain classes of large queries.
--
MOS_MIGRATED_REVID=111697983
|
|
|
|
|
|
|
|
|
| |
directly, and pass a Query callback in when resolving target patterns. This means that the targets a pattern resolves to can be processed incrementally.
This is the fifth step in a series to allow processing large sets of targets in query target patterns via streaming batches rather than all at once. This should improve performance for SkyQueryEnvironment for certain classes of large queries.
--
MOS_MIGRATED_REVID=111696713
|
|
|
|
|
|
|
|
|
| |
label sets that are precomputed in the graph.
This is the fourth step in a series to allow processing large sets of targets in query target patterns via streaming batches rather than all at once. This may make SkyQueryEnvironment slower when evaluating queries with repeated target patterns, or many target patterns that would benefit from graph lookups that were batched across all patterns. But that is not currently a bottleneck we're concerned about.
--
MOS_MIGRATED_REVID=111626483
|
|
|
|
|
|
|
| |
This is the third step in a series to allow processing large sets of targets in query target patterns via streaming batches rather than all at once. This should also be a functional no-op.
--
MOS_MIGRATED_REVID=111620668
|
|
|
|
|
|
|
| |
graph. We can compute the universe target patterns outside of skyframe, which is the only reason we need the value we were requesting. Giving up was preventing us from evaluating "..." patterns even if the "..." pattern didn't contain any cycles itself.
--
MOS_MIGRATED_REVID=111605976
|
|
|
|
|
|
|
| |
we encounter a bad target pattern.
--
MOS_MIGRATED_REVID=111374219
|
|
|
|
|
|
|
| |
targets.
--
MOS_MIGRATED_REVID=110275427
|
|
|
|
|
|
|
| |
Reduces garbage.
--
MOS_MIGRATED_REVID=109914243
|