aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/query2/AbstractBlazeQueryEnvironment.java
Commit message (Collapse)AuthorAge
* Description redacted.Gravatar Nathan Harmata2017-03-09
| | | | | | -- PiperOrigin-RevId: 149585165 MOS_MIGRATED_REVID=149585165
* Fix bug with streaming bounded deps/allrdeps/rdeps.Gravatar Nathan Harmata2017-03-08
| | | | | | -- PiperOrigin-RevId: 149431500 MOS_MIGRATED_REVID=149431500
* Rollback of commit 822c37816ac669e51bec3853b41849a19ec5e230.Gravatar Nathan Harmata2017-03-01
| | | | | | -- PiperOrigin-RevId: 148844518 MOS_MIGRATED_REVID=148844518
* Reimplement blaze query using an async evaluation model. Use a concurrent ↵Gravatar Nathan Harmata2017-02-28
| | | | | | | | | | | | | | | | | | | backend for SkyQueryEnvironment's implementation in order to achieve parallelism. Advantages: -New design has no flaws that the old design had. -Code is structured so that deadlocks due to thread starvation are impossible (yup!). Disadvantages: -The meat of this change needs to all be in a single CL because every single QueryFunction and QueryExpression needs to be rewritten in the async style. Still TODO: -Fully embrace the async model in all QueryFunctions (e.g. 'rdeps', 'allpaths'). -Use concurrency in BlazeQueryEnvironment to achieve parallel evaluation for (non SkyQuery) 'blaze query' and genquery. -- PiperOrigin-RevId: 148690279 MOS_MIGRATED_REVID=148690279
* Provide more reporting options to SkyFunctions Gravatar Klaus Aehlig2017-02-27
| | | | | | | | | | | | | With more specific information to be reported by Skyfunctions, e.g., to inform the build-event protocol on missing files, the EventHandler interface is no longer enough. Therefore, provide an enriched context for reporting events. -- Change-Id: I2d06166fe4d5b9054e24ad8c752fafc039e3f9f8 Reviewed-on: https://cr.bazel.build/8794 PiperOrigin-RevId: 148463437 MOS_MIGRATED_REVID=148463437
* Rollback of commit e0a330577d9fe98169645cb68d9fc22cc787eeb6.Gravatar Janak Ramakrishnan2016-12-21
| | | | | | | | | | *** Reason for rollback *** Investigating if causes deadlock/thread starvation. -- PiperOrigin-RevId: 142575769 MOS_MIGRATED_REVID=142575769
* For all function expressions of the form f(..., e1, ..., e2, ..., eK, ...), ↵Gravatar Nathan Harmata2016-12-02
| | | | | | | | ensure that all of e1, e2, ..., and eK are elligble for parallel evaluation. This is _not_ the same as providing a parallel implementation of f, which we can do separately in followup CLs. -- PiperOrigin-RevId: 140861694 MOS_MIGRATED_REVID=140861694
* Introduce a failFast mode to OutputFormatterCallback#close.Gravatar Nathan Harmata2016-11-18
| | | | | -- MOS_MIGRATED_REVID=139508838
* Avoid AutoProfiler overhead in AbstractBlazeQueryEnvironment#evaluateQuery.Gravatar Janak Ramakrishnan2016-11-15
| | | | | -- MOS_MIGRATED_REVID=139113333
* Loosen visibility and finality of *QueryEnvironment fields.Gravatar Janak Ramakrishnan2016-11-09
| | | | | -- MOS_MIGRATED_REVID=138534021
* Unify AbstractBlazeQueryEnvironment#evaluateQuery with its subclass ↵Gravatar Nathan Harmata2016-10-04
| | | | | | | overrides. Also, have AbstractBlazeQueryEnvironment#evaluateQuery take an OutputFormatterCallback instance rather than a Callback instance. This is more sensible since the latter is only intended to be used intra-query, while the former is intended for usage in end-to-end query evaluation. This lets us slightly simplify QueryCommand, by shifting the responsibility for managing the OutputFormatterCallback to AbstractBlazeQueryEnvironment#evaluateQuery. -- MOS_MIGRATED_REVID=134827588
* Make AbstractBlazeQueryEnvironment no longer implement AutoCloseable. ↵Gravatar Nathan Harmata2016-09-29
| | | | | | | | | | | Instead have SkyQueryEnvironment#evaluateQuery be responsible for handling cleanup of its internal ForkJoinPool. In addition to being a more sensible way of organizing the code (imo, it makes sense for SkyQueryEnvironment to clean up after itself), this fixes several issues: (i) If query evaluation is interrupted, the AbstractBlazeQueryEnvironment#close call at the end of the try-with-resources statement in QueryCommand would be blocking and non-urgent. N.B. This was not an issue with the current ForkJoinPool usage, but was an issue with the old ThreadPoolExecutor usage. (ii) Because of how the code in QueryCommand was structured, OutputFormatterCallback#close would happen _before_ the AbstractBlazeQueryEnvironment#close call. If query evaluation is interrupted, threads executing query tasks may be invoking the callback after the callback had been shut down! -- MOS_MIGRATED_REVID=134573395
* A bunch of small changes to prepare SkyQueryEnvironment for full-parallel ↵Gravatar Nathan Harmata2016-09-08
| | | | | | | | | | | | | | evaluation: -Rename QueryExpression#evalConcurrently to QueryExpression#parEval. (parallelism is not concurrency! See https://existentialtype.wordpress.com/2011/03/17/parallelism-is-not-concurrency/) -Have SkyQueryEnvironment#eval (used recursively in #evaluateQuery) dynamically call QueryExpression#parEval when appropriate. -Delete QueryExpression#canEvalConcurrently. -Add ThreadSafety annotations in a bunch of relevant places in the query codebase. -A bunch of testing infrastructure to test parallel query evaluation. -TODOs for implementing parallel evaluation of all QueryExpression nodes. -- MOS_MIGRATED_REVID=132436340
* In BlazeQueryEnvironment, stop wrapping InterruptedExceptions in ↵Gravatar Janak Ramakrishnan2016-08-17
| | | | | | | | | QueryExceptions now that all relevant methods declare that they throw InterruptedException. Small side benefit of commit 3c0adb26bac6d756fb97e4bcc6d4e5b2cefa5eeb. -- MOS_MIGRATED_REVID=130402917
* Allow Skyframe graph lookups and value retrievals to throw InterruptedException.Gravatar Janak Ramakrishnan2016-08-16
| | | | | | | The only place we now don't handle InterruptedException is in the action graph created after analysis, since I'm not sure that will be around for that much longer. -- MOS_MIGRATED_REVID=130327770
* Re-implement variables in the blaze query language. Instead of using a ↵Gravatar Nathan Harmata2016-07-14
| | | | | | | | | mutable global context of variable bindings, pass around immutable local contexts. The motivation is so we can safely evaluate all blaze query expressions concurrently under the hood. A global context is hostile to this goal. -- MOS_MIGRATED_REVID=127324600
* No longer allow QueryExpressionMapper#map to throw a QueryException. This ↵Gravatar Nathan Harmata2016-07-04
| | | | | | | was unnecessary. -- MOS_MIGRATED_REVID=126457804
* (1) Allow QueryExpressionMapper#map to throw a QueryException.Gravatar Nathan Harmata2016-07-01
| | | | | | | | (2) Refactor the simple query concurrency support by introducing QueryExpression#canEvalConcurrently and removing the primitive BinaryOperatorExpression#evalConcurrently and the hardcoded SkyQueryEnvironment logic around it. This way each QueryExpression can safely manage its own concurrency. A followup CL will ensure that concurrent evaluation occurs for as much of the query expression as possible, rather than just for the top-level query expression node. (3) Make a few query internals public. -- MOS_MIGRATED_REVID=126324637
* Fix threadpool leak in SkyQueryEnvironmentGravatar Mark Schaller2016-06-27
| | | | | | | | Shutdown the SkyQueryEnvironment's threadpool after query evaluation is complete and the environment is ready for disposal. -- MOS_MIGRATED_REVID=125975317
* Add an afterCommand hook to AbstractBlazeQueryEnvironment.Gravatar Nathan Harmata2016-04-15
| | | | | -- MOS_MIGRATED_REVID=119868282
* Move AbstractBlazeQueryEnvironment to a factory class, and have BlazeModule ↵Gravatar Nathan Harmata2016-03-21
| | | | | | | optionally expose a custom factory implementation. -- MOS_MIGRATED_REVID=117546934
* In SkyQueryEnvironment, rewrite queries using the semantics-preserving ↵Gravatar Nathan Harmata2016-03-02
| | | | | | | | | transformation 'rdeps(<sky_query_environment_universe_scope>, T, depth)' -> 'allrdeps(T, depth)'. SkyQueryEnvironment can evaluate such allrdeps queries much more efficiently since it doesn't need to bother filtering out targets outside of universe, meaning it doesn't need to have all targets in the universe in memory at the same time. -- MOS_MIGRATED_REVID=116075008
* Extract AttributeInfoProvider interface for DependencyFilters.Gravatar Dmitry Lomov2016-01-22
| | | | | | | | This is in preparation of implementing dependency filtering correctly for aspects. -- MOS_MIGRATED_REVID=112721440
* Refactor dependency filtering out of Rule class.Gravatar Dmitry Lomov2016-01-22
| | | | | -- MOS_MIGRATED_REVID=112717648
* Resolve target patterns on the fly in SkyQueryEnvironment. Cache only the ↵Gravatar Janak Ramakrishnan2016-01-07
| | | | | | | | | label sets that are precomputed in the graph. This is the fourth step in a series to allow processing large sets of targets in query target patterns via streaming batches rather than all at once. This may make SkyQueryEnvironment slower when evaluating queries with repeated target patterns, or many target patterns that would benefit from graph lookups that were batched across all patterns. But that is not currently a bottleneck we're concerned about. -- MOS_MIGRATED_REVID=111626483
* Make QueryEnvironment#getTargetsMatchingPattern pass its result to a callback.Gravatar Janak Ramakrishnan2016-01-07
| | | | | | | This is the third step in a series to allow processing large sets of targets in query target patterns via streaming batches rather than all at once. This should also be a functional no-op. -- MOS_MIGRATED_REVID=111620668
* Remove empty put when preloadOrThrow throws a TargetParsingException. ↵Gravatar Janak Ramakrishnan2016-01-07
| | | | | | | preloadOrThrow now only throws when in nokeep-going mode, and so the put now does nothing. -- MOS_MIGRATED_REVID=111417679
* Use Bazel Preconditions variant which avoids varargs array creationGravatar Mark Schaller2015-12-10
| | | | | | | Reduces garbage. -- MOS_MIGRATED_REVID=109914243
* Allow output formatters to work in stream mode.Gravatar Miguel Alcon Pinto2015-12-10
| | | | | -- MOS_MIGRATED_REVID=109908202
* Rollback of commit 8e7da8704dba78dba47fbaa0eafce3a91c2deb13.Gravatar Peter Schmitt2015-12-01
| | | | | | | *** Reason for rollback *** -- MOS_MIGRATED_REVID=109005097
* Allow output formatters to work in stream mode.Gravatar Miguel Alcon Pinto2015-11-19
| | | | | -- MOS_MIGRATED_REVID=108142169
* Transform Blaze query to be able to work in streamed mode.Gravatar Miguel Alcon Pinto2015-11-06
| | | | | -- MOS_MIGRATED_REVID=107249788
* Rationalize copyright headersGravatar Damien Martin-Guillerez2015-09-25
| | | | | | | | | | | The headers were modified with `find . -type f -exec 'sed' '-Ei' 's|Copyright 201([45]) Google|Copyright 201\1 The Bazel Authors|' '{}' ';'` And manual edit for not Google owned copyright. Because of the nature of ijar, I did not modified the header of file owned by Alan Donovan. The list of authors were extracted from the git log. It is missing older Google contributors that can be added on-demand. -- MOS_MIGRATED_REVID=103938715
* Move Label from the lib.syntax to the lib.cmdline package so that:Gravatar Lukacs Berki2015-09-21
| | | | | | | | | | - Label parsing can be simplified - lib.syntax is only contains the code for Skylark and is reasonably independent from the problem domain of building things This change is mostly only changes to imports declarations. The rest is reversing the dependency between :cmdline and :syntax and moving a tiny amount of code between Printer and FilesetEntry and the addition of SkylarkPrintableValue that I couldn't be bothered to separate out into its own change. -- MOS_MIGRATED_REVID=103527877
* Use AutoProfiler in the Bazel codebase.Gravatar Nathan Harmata2015-09-08
| | | | | -- MOS_MIGRATED_REVID=102584924
* Use Set<Target> instead of ResolvedTargets<Target> when saving resolved ↵Gravatar Janak Ramakrishnan2015-08-18
| | | | | | | | | target patterns in query. We never use anything but the getTargets() field, so it's useless to store other things. -- MOS_MIGRATED_REVID=100846573
* Add some useful logging of "blaze query" timing.Gravatar Eric Fellheimer2015-07-29
| | | | | -- MOS_MIGRATED_REVID=99302436
* Throw InterruptedException directly if a query is interrupted, instead of ↵Gravatar Janak Ramakrishnan2015-07-10
| | | | | | | | | wrapping it in a QueryException. QueryException should usually indicate a persistent failure, while an InterruptedException is transient. Wrapping the InterruptedException in a QueryException just obfuscates state. -- MOS_MIGRATED_REVID=97815388
* Preload the universe for query testsGravatar Mark Schaller2015-03-24
| | | | | | | | | | | | | This change means that tests no longer need to specify the patterns they use themselves, unless the test involves excluding some pattern. Also fixes a bug in GraphBackedRecursivePackageProvider revealed by the test changes where it behaved differently from EnvironmentBackedRecursivePackageProvider when looking up packages with errors. -- MOS_MIGRATED_REVID=89297008
* On-the-fly target pattern resolution in SkyQueryEnvironmentGravatar Mark Schaller2015-03-18
| | | | | | | | | | | | | Moves pattern resolving logic from TargetPatternFunction.Resolver to a top level class. Adds a layer of abstraction to the Resolver implementation enabling it to be backed by either an Environment or a Graph, for use in SkyFunction evaluation or on-the-fly evaluation, respectively. Finally, SkyQueryEnvironment#preloadOrThrow now checks to see if each target pattern exists in the graph, and any that don't will be resolved on-the-fly. -- MOS_MIGRATED_REVID=88861201
* Add a SkyQueryEnvironment which runs queries by examining the Skyframe graph.Gravatar Janak Ramakrishnan2015-02-26
| | | | | | | | | This environment eagerly preloads the transitive closure of a specified query "universe", and so may not be as efficient as the standard query for limited-scope queries. It is activated when the universe is specified and ordered results are not requested (since it is currently unable to order results). Tests were modified/added to exercise this environment where deemed interesting. Some ugly hacks were done to add coverage in AbstractQueryTest and friends, because currently even if the full depot is loaded (using //...), individual target patterns most likely won't be present in the graph. A better way to deal with this situation, suggested by felly, is probably to extract target pattern resolution logic to an auxiliary function so that query is able to resolve target patterns without mutating the graph, and then call into the read-only graph with the resolved patterns. That may be done in a follow-up, in which case the "scope" of every query could be //... . -- MOS_MIGRATED_REVID=87257028
* Refactor Query code a bit. Get rid of separate SkyframeQueryEnvironment, ↵Gravatar Janak Ramakrishnan2015-02-11
| | | | | | | | | move some utility methods to utility class. The real goal of this CL is to make sure all users of BlazeQueryEnvironment are constructing one through EvaluableBlazeQueryEnvironment#newQueryEnvironment, and not directly. This ensures that alternate implementations of EvaluableBlazeQueryEnvironment will be automatically injected via #newQueryEnvironment. -- MOS_MIGRATED_REVID=86112439
* Extract an EvaluableBlazeQueryEnvironment abstract class from ↵Gravatar Janak Ramakrishnan2015-02-11
BlazeQueryEnvironment that exposes an evaluateQuery method, and also implements the non-LabelVisitor-specific parts of BlazeQueryEnvironment, for other implementations' uses. Most of the code is just a straight refactoring of BlazeQueryEnvironment into EvaluableBlazeQueryEnvironment (and BlazeTargetAccessor). Ignoring whitespace changes in [] may be your friend for seeing that it's a straight move. This also allows us to change tests to use QueryCommand.newQueryEnvironment, in preparation for newQueryEnvironment potentially returning a different EvaluableBlazeQueryEnvironment subclass depending on the circumstances. -- MOS_MIGRATED_REVID=86088953
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-07 20:18:28 +0000