| Commit message (Collapse) | Author | Age |
|
|
|
|
|
|
| |
AttrFunction and LabelsFunction are also made public.
RELNOTES: None
PiperOrigin-RevId: 155108260
|
|
|
|
|
|
| |
wrapper objects: for OwnedArtifacts, which are the most numerous during builds, and for Labels for TransitiveTraversalValues, which are the most numerous during queries.
PiperOrigin-RevId: 154989520
|
|
|
|
|
|
|
|
| |
Add static methods to return target kinds of InputFile and OutFile, as well as
suffix of target kind of Rule.
RELNOTES: None
PiperOrigin-RevId: 154866926
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 153449924
|
|
|
|
|
|
|
|
|
| |
These are two different concepts. Do not remove category overload compatibility in this CL, to keep this change limited to converting the current uses of category.
With some flyby formatting fixes on affected OptionsBases.
RELNOTES: None.
PiperOrigin-RevId: 153390002
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 153058374
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Before this change, attributes with select() were completely skipped.
This doesn't attempt to merge "trivial" selects, e.g. even though:
attr = [":foo"] + select({"//conditions:default": [":bar"]})
always resolves to:
attr = [":foo", ":bar"]
this change still produces:
attr = [":foo"] + [":bar"]
We could merge these in a future change, if desired. But it's not
even clear that's desired. There's conceptual value in keeping the
lists separate since that's how they were originally written. That
gives users a cue to maybe refactor their rules.
RELNOTES[NEW]: "query --output=build" now includes select()s
PiperOrigin-RevId: 152956939
|
|
|
|
|
|
|
|
|
|
|
|
| |
'create' method.
This paves the way for changing PathFragment to e.g. an abstract class with multiple subclasses. This way we can split out the windows-specific stuff into one of these concrete classes, making the code more readable and also saving memory (since the shallow heap size of the NonWindowsPathFragment subclass will hopefully be smaller than that of the current PathFragment).
This also lets us pursue gc churn optimizations. We can now do interning in PathFragment#create and can also get rid of unnecessary intermediate PathFragment allocations.
RELNOTES: None
PiperOrigin-RevId: 152145768
|
|
|
|
|
|
|
| |
This isn't used anymore, it's the same as STRING_DICT, deleting so no one tries
to use it.
PiperOrigin-RevId: 151738915
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 151019690
MOS_MIGRATED_REVID=151019690
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
'blaze query'.
The "streaming" callbacks used by some query functions, e.g. 'deps', make calls to QueryEnvironment#buildTransitiveClosure.
For a cold blaze server, these calls do package loading via LabelVisitor (which calls into Skyframe via a top-level #evaluate call). So we'd prefer a single massive call which can make full use of blaze's loading-phase parallelism via Skyframe over a bunch of sequential small calls.
For a hot blaze server, there are two problems:
(1) LabelVisitor's meager up-to-date check isn't useful (as in we cannot reuse old visitations) when we do a whole bunch of small visitations instead of one massive one.
(2) The actual work of the LabelVisitor (building up a portion of a temporary graph) isn't being effectively parallelized when we do it sequentially in small chunks.
This issue is yet another subtle reason why the old BlazeQueryEnvironment#eval made sense (and why it was unfortunately not compatible with the streaming query evaluation model from the beginning).
--
PiperOrigin-RevId: 150081619
MOS_MIGRATED_REVID=150081619
|
|
|
|
|
|
|
|
|
|
|
|
| |
Universe evaluation only occurs in the SkyQueryEnvironment
implementation. This setting is a no-op for other QueryEnvironment
implementations.
This support is needed to correctly test an upcoming bugfix.
--
PiperOrigin-RevId: 149786616
MOS_MIGRATED_REVID=149786616
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 149585165
MOS_MIGRATED_REVID=149585165
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 149431500
MOS_MIGRATED_REVID=149431500
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 149089903
MOS_MIGRATED_REVID=149089903
|
|
|
|
|
|
|
|
|
|
|
|
| |
prevent thread starvation
This change gets rid of the additional thread needed for task scheduling during
BFS visitation, which eliminates the possibility of thread starvation while a
single thread pool is used for multiple concurrent evaluations.
--
PiperOrigin-RevId: 148911346
MOS_MIGRATED_REVID=148911346
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 148844518
MOS_MIGRATED_REVID=148844518
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
backend for SkyQueryEnvironment's implementation in order to achieve parallelism.
Advantages:
-New design has no flaws that the old design had.
-Code is structured so that deadlocks due to thread starvation are impossible (yup!).
Disadvantages:
-The meat of this change needs to all be in a single CL because every single QueryFunction and QueryExpression needs to be rewritten in the async style.
Still TODO:
-Fully embrace the async model in all QueryFunctions (e.g. 'rdeps', 'allpaths').
-Use concurrency in BlazeQueryEnvironment to achieve parallel evaluation for (non SkyQuery) 'blaze query' and genquery.
--
PiperOrigin-RevId: 148690279
MOS_MIGRATED_REVID=148690279
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 148643993
MOS_MIGRATED_REVID=148643993
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 148488719
MOS_MIGRATED_REVID=148488719
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
With more specific information to be reported by Skyfunctions, e.g.,
to inform the build-event protocol on missing files, the EventHandler
interface is no longer enough. Therefore, provide an enriched context
for reporting events.
--
Change-Id: I2d06166fe4d5b9054e24ad8c752fafc039e3f9f8
Reviewed-on: https://cr.bazel.build/8794
PiperOrigin-RevId: 148463437
MOS_MIGRATED_REVID=148463437
|
|
|
|
|
|
|
|
|
| |
This enables both native and Skylark rules to declare attributes which
have labels/Targets as keys, and have string values.
--
PiperOrigin-RevId: 148365033
MOS_MIGRATED_REVID=148365033
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
targets.
These two functions don't use a Uniquifier internally so if the same bzl file foo.bzl is loaded multiple ways in 'e' in 'loadfiles(e)' then 'f' in 'f(loadfiles(e))' will observe foo.bzl multiple times.
This isn't observable in practice (i.e. I cannot write a black-box query test that would fail without this CL) because:
(1) BlazeQueryEnvironment#eval(QueryExpression, VariableContext<Target>, Callback<Target>) uses an intermediate aggregating callback and thus doesn't use the streaming query evaluation model. This means that the internal deduping in BlazeQueryEnvironment#getBuildFiles is sufficient.
(2) SkyQueryEnvironment uses an outer BatchStreamedCallback which internally uses a Uniquifier.
Still, this CL is useful for SkyQuery because without we're doing wasted work (e.g. in my example above, 'f' will wastefully do duplicate work with the .bzl files from the inner loadfiles(e)) because the deduping happens on the final query result, not on the intermediate result from 'buildfiles'/'loadfiles'.
For 'buildfiles', there's an additional wart with (1) above because FakeSubincludeTarget doesn't override Object#equals/hashCode.
--
PiperOrigin-RevId: 147656183
MOS_MIGRATED_REVID=147656183
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 147621296
MOS_MIGRATED_REVID=147621296
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This overrides the traditional has(String name, Type<>T> type)
with has(String name) and removes the type check outright from
isConfigurable.
Ideally we'd remove the old version in this same change. But there
are enough uses of it that that's not a risk-free change and
is safer as followup changes.
--
PiperOrigin-RevId: 147513593
MOS_MIGRATED_REVID=147513593
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The name attribute gets special treatment in the codebase, in that
it's not simply yet another attribute but stored in it's own field.
Thus, every callside dealing with attributes needs to be aware of
this special case and explicitly handle the name attribute. It's
easy to see that this can lead to bugs. For example, querying for
the name attribute is currently broken due the querying code not
being aware of the special case [1].
Discussions with experienced bazel developers came to the conclusion
that there is no need (anymore) to treat the name attribute specially
and thus we decided it's best to remove the special treatment and
handle the name attribute as any other attribute.
This change removes the handling of name attributes and also adds a test
case to verify that bug [1] is fixed.
[1] https://github.com/bazelbuild/bazel/issues/278
--
PiperOrigin-RevId: 147446345
MOS_MIGRATED_REVID=147446345
|
|
|
|
|
|
|
|
|
|
| |
I was trying to simplify GenQuery, and found this during an audit of the
code; it's not sufficient to make any larger changes, but it may help
eventuall.
--
PiperOrigin-RevId: 147023626
MOS_MIGRATED_REVID=147023626
|
|
|
|
|
|
|
|
|
|
| |
Two of the functions are never used, several of the parameter are not
used or can be simplified (require callers to convert from Target ->
Label).
--
PiperOrigin-RevId: 146787380
MOS_MIGRATED_REVID=146787380
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 145473478
MOS_MIGRATED_REVID=145473478
|
|
|
|
|
|
|
|
|
|
|
|
| |
Because the event handler's inner handlers are removed after each query
command, caching the handler caused a subset of subsequent commands'
errors (those reported through the resolver's handler) to go unlogged.
Also fix a bug with cycle detection in DelegatingWalkableGraph.
--
PiperOrigin-RevId: 145124271
MOS_MIGRATED_REVID=145124271
|
|
|
|
|
|
|
|
| |
WalkableGraph#getException to be given non-existent keys without crashing. Add WalkableGraph#isCycle to fill the gap in testing for the difference between non-existence and depending on a cycle.
--
PiperOrigin-RevId: 143205289
MOS_MIGRATED_REVID=143205289
|
|
|
|
|
|
|
|
|
| |
This used to be how test_suites depended on other test_suites.
Now they just go in "tests".
--
PiperOrigin-RevId: 142607603
MOS_MIGRATED_REVID=142607603
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Investigating if causes deadlock/thread starvation.
--
PiperOrigin-RevId: 142575769
MOS_MIGRATED_REVID=142575769
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 142295816
MOS_MIGRATED_REVID=142295816
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(i) Use a CountDownLatch in ParallelQueryUtils#executeQueryTasksAndWaitInterruptibly to avoid busy-looping while waiting for query subtask completion (this busy-looping unnecessarily ties up a thread). But we still retain the fail-fast semantics we want (I renamed the method to emphasize this).
(ii) Also have a special-case in ParallelQueryUtils#executeQueryTasksAndWaitInterruptibly for evaluating one query subtask so we don't wastefully use another thread.
(iii) Also add ThreadSafety annotations to ParallelQueryUtils.
----
(i) and (ii) combine to address the following theoretical issue. Suppose we're evaluating a query expression of the form "(e1 - e2) + (e3 - e4)". The old code would (with the worst-case FJP thread scheduling) have the following threads at the _same_ time:
Main QueryCommand thread - executeQueryTasksAndWaitInterruptibly(queryTasks = [(e1 - e2), (e3 - e4)]
FJP thread - executeQueryTasksAndWaitInterruptibly(queryTasks = [e2])
FJP thread - eval(e2)
FJP thread - executeQueryTasksAndWaitInterruptibly(queryTasks = [e4])
FJP thread - eval(e4)
So of those 5 concurrent threads, 3 would be doing busy-loop waiting. For more pathological query expressions, we could end up tying up lots of threads doing wasteful busy-loops.
--
PiperOrigin-RevId: 142215680
MOS_MIGRATED_REVID=142215680
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 141920696
MOS_MIGRATED_REVID=141920696
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 141904124
MOS_MIGRATED_REVID=141904124
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Turns out that ForkJoinTask#adapt(Callable) returns a ForkJoinTask whose Future#get on error throws a ExecutionException wrapping a RuntimeException wrapping the thrown checked exception from the callable. This is documented behavior [1] that I incorrectly didn't know about.
The additional level of wrapping meant that the catch-block of the parallel implementation of BinaryOperatorExpression wasn't rethrowing the InterruptedException/QueryException that the parallel task threw.
The subtly in this bug is that the query expression being evaluated needs to be of the form "e1 + e2", where evaluation of "e1" throws a QueryException even in keepGoing mode (note that most of the query errors actually go through AbstractBlazeQueryEnvironment#reportBuildFileError). The test I wrote picks on LetExpression's evaluation-time (rather than e.g. parsing time) validation of the variable name.
[1] https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ForkJoinTask.html#adapt(java.util.concurrent.Callable)
--
PiperOrigin-RevId: 141772584
MOS_MIGRATED_REVID=141772584
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 141307853
MOS_MIGRATED_REVID=141307853
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 141178325
MOS_MIGRATED_REVID=141178325
|
|
|
|
|
|
|
|
| |
ensure that all of e1, e2, ..., and eK are elligble for parallel evaluation. This is _not_ the same as providing a parallel implementation of f, which we can do separately in followup CLs.
--
PiperOrigin-RevId: 140861694
MOS_MIGRATED_REVID=140861694
|
|
|
|
|
|
|
|
|
|
|
| |
for concurrent visitations.
During BFS visitation of rdeps and rbuildfiles, it uses a centralized pool
(backed by a LinkedBlockingQueue) to store all pending visits, and a
periodically running scheduler to schedule tasks for each pending visit.
--
MOS_MIGRATED_REVID=140398162
|
|
|
|
|
|
|
| |
equivalence to "e1 - (e2 + e3)" and the fact that we already have a parallel implementation of "e2 + e3".
--
MOS_MIGRATED_REVID=139792288
|
|
|
|
|
|
|
| |
'buildfiles(e)'.
--
MOS_MIGRATED_REVID=139787078
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=139613681
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=139508838
|
|
|
|
|
|
|
|
|
| |
SkyQueryEnvironment#beforeEvaluateQuery if possible to save on latency for small queries.
This assumes that if the graph is up to date, then the data in SkyQueryEnvironment is also up to date. It also assumes that a ForkJoinPool remains usable until #shutdownNow is called.
--
MOS_MIGRATED_REVID=139386363
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=139357569
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=139113333
|
|
|
|
|
|
|
| |
by introducing TargetPattern#parEval, which allows TargetPatterns' evaluations to explicitly have parallel implementations (no need to secretly use a FJP).
--
MOS_MIGRATED_REVID=139101922
|