| Commit message (Collapse) | Author | Age |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It turns out that the SUCCESS status is often misunderstood to mean "zero exit",
even though this is clearly documented. I've decided to add another status for
non-zero exit, and use success only for zero exit to avoid this pitfall.
Also, many of the status codes are set, but never used. I decided to reduce the
number of status codes to only those that are actually relevant, which
simplifies further processing. Instead, we should add a string message for the
error case when we need one - we're not using it right now, so I decided not to
add that yet.
PiperOrigin-RevId: 177129441
|
|
|
|
|
|
|
|
|
| |
Currently we don't care about the list order of SpawnResults. However, we may care about the list order later. Also, if the equals() method for SpawnResults is ever changed then it may be problematic to return SpawnResults in a Set.
Aside: ActionResults use SpawnResults to calculate cumulative execution times for Actions, and may provide other metrics in future.
RELNOTES: None.
PiperOrigin-RevId: 176579460
|
|
|
|
|
|
|
|
|
|
| |
Fix for #4035
@laszlocsomor
Closes #4110.
PiperOrigin-RevId: 176346381
|
|
|
|
|
|
|
|
| |
Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own.
This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually.
PiperOrigin-RevId: 175033526
|
|
|
|
|
|
|
| |
TerminationStatus, and also add a TerminationStatus.Builder and tests.
RELNOTES: None.
PiperOrigin-RevId: 174557303
|
|
|
|
|
|
|
| |
and make cumulative execution times available in ActionResults.
RELNOTES: None
PiperOrigin-RevId: 174553272
|
|
|
|
|
|
| |
This requires a fairly large amount of changes to fundamental objects like BlazeRuntime, Executor, and so on, as well as changing a lot of test code to thread the file system through. I expect future CLs to be much smaller.
PiperOrigin-RevId: 173678144
|
|
|
|
|
|
|
|
| |
Even if the test action produced no output, which it really shouldn't, Bazel should create an empty test.log file.
TESTED=unit tests
RELNOTES: Fixes #3834
PiperOrigin-RevId: 172412615
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Every build and test action that creates a Spawn
will now have platform-specific environment
variables for temp directories:
- on Windows: TMP and TEMP
- on Linux/Darwin: TMPDIR
This is particularly important on Windows where
e.g. Java programs cannot create temp directories
unless there's a valid TMP or TEMP environment
variable set.
Fixes:
- https://github.com/bazelbuild/bazel/issues/1590
- https://github.com/bazelbuild/bazel/issues/2349
- https://github.com/bazelbuild/bazel/issues/2870
Change-Id: Ib758307daf6b3a51b0f71ae5e65e5bb564dad643
PiperOrigin-RevId: 172326371
|
|
|
|
|
|
| |
TESTED=added remote test
RELNOTES: Fixes #2574
PiperOrigin-RevId: 172294781
|
|
|
|
|
|
|
| |
Whether or not there was a catastrophic error is stored in the SpawnResult, so
we can just use that instead of passing in an additional boolean.
PiperOrigin-RevId: 172083752
|
|
|
|
|
|
| |
cache if we're doing input discovery: input discovery may find new artifacts we don't yet know about. Similarly, in SingleBuildFileCache, use the Artifact's path if it's available, rather than the poor man's route of the execRoot.
PiperOrigin-RevId: 171635892
|
|
|
|
|
|
|
|
|
| |
ActionContexts, etc., so that SpawnResult metadata is returned upwards.
Note that the TODOs mostly refer to changes that will appear in a subsequent CL (a CL to return SpawnResults, contained in ActionResults, from Actions/AbstractActions). I split off the remaining SpawnResult-related changes into this CL and kept the ActionResult-related changes separate.
RELNOTES: None.
PiperOrigin-RevId: 171355611
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 170955877
|
|
|
|
|
|
|
|
|
|
| |
- add a mode to control how to handle relative symlinks
- if set, attempt to resolve relative symlinks in manifests by looking for
another entry that defines a file with the intended symlink target (we don't
do recursive resolution for now)
- add more test coverage; fix a bug in handling empty target locations
PiperOrigin-RevId: 170691492
|
|
|
|
|
|
|
| |
them to be created.
RELNOTES: None.
PiperOrigin-RevId: 170058295
|
|
|
|
|
|
|
|
| |
Include the coverage data file into the list of files repeorted in the
test_result event in the BEP.
Change-Id: Ia7b653addd5589268ce919b3e0349371dbc18bf2
PiperOrigin-RevId: 169956864
|
|
|
|
|
|
|
| |
build.lib.actions.SpawnActionContext can import SpawnResult without creating a cyclic dependency.
RELNOTES: None.
PiperOrigin-RevId: 169642267
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
executions and tests with multiple attempts. Previously, non-cached test execution would incorrectly only include test.log and test.xml.
Implement this by unifying test output list building in a static TestResult method.
For additional output files in multi-attempt tests, we follow the existing convention:
Existing example: test.xml => test_attempts/attempt_N.xml
New example (all new outputs done similarly): test.outputs/outputs.zip => test_attempts/attempt_N.outputs/outputs.zip
RELNOTES: All test output files included for cached, uncached, and multiple attempt tests.
PiperOrigin-RevId: 169556608
|
|
|
|
|
|
|
|
|
|
|
|
| |
NetUtil.getShortHostName can take seconds on mac and on windows (like, 20!),
since it performs reverse dns lookup. We already cached hostname for the
BazelWorkspaceStatusModule, let's cache it for entire bazel server. Also make
sure that users of the method understand it's cached.
Fixes #3586.
RELNOTES: None.
PiperOrigin-RevId: 168691615
|
|
|
|
|
|
|
|
|
| |
Split collect, concurrent, vfs, windows into package-level BUILD files.
Move clock classes out of "util", into their own Java package.
Move CompactHashSet into its own Java package to break a dependency cycle.
Give nestedset and inmemoryfs their own package-level BUILD files.
PiperOrigin-RevId: 167702127
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 167505493
|
|
|
|
|
|
|
|
|
|
| |
For any errors that are due to failures in the remote caching /
execution layers Bazel now returns exit code 34 (ExitCode.REMOTE_ERROR).
This includes errors where the remote cache / executor is unreachable or
crashes. It does not include errors if the test / build failure is due
to user errors i.e. compilation or test failures.
PiperOrigin-RevId: 167259236
|
|
|
|
|
|
|
|
|
| |
If there is a build failure, don't clobber the terse test summary
by naming all the (usually many) tests that were skipped due to
this failure.
Change-Id: I6daae3efb1594c2b1018f87a50cf63949a34535b
PiperOrigin-RevId: 166983264
|
|
|
|
| |
PiperOrigin-RevId: 166849610
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This paves the way for Skylark-side compact command lines that can fail during expansion.
In general, expansion happens in these scenarios:
* Action execution expands the command line to execute it. This is fine since we are well equipped already to handle failing actions.
* In the analysis phase we expand command lines to investigate whether we need a params file. This could be moved to the execution phase, which would have the benefit of getting params files out of the action graph and saving memory.
* Key computation expands the command line. This could be fixed by allowing command lines to compute their own keys (which wouldn't try to expand the command line). This could have the benefit of being more efficient.
* Extra actions expand the command line to construct the extra action proto. This could maybe be deferred to the execution phase (writing the extra action), which would also be more memory efficient.
For failed key computations, we return a singleton "error" key. This means multiple actions that will fail will map to the same key. These actions will necessarily fail if executed, so we should not need to worry about these ending up in the action cache. If we do manage to make the command line compute its own keys then this will become moot in the future.
RELNOTES: None
PiperOrigin-RevId: 166266385
|
|
|
|
|
|
|
|
|
|
| |
AbstractSpawnRunner now uses a SpawnCache if one is registered, this allows
adding caching to any spawn runner without having to be aware of the
implementations.
I will delete the old CachedLocalSpawnRunner in a follow-up CL.
PiperOrigin-RevId: 165024382
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Change the persistent worker spawn strategy to extend
AbstractSpawnStrategy and put the actual logic into
WorkerSpawnRunner. WorkerTestStrategy is unaffected.
I had to extend SpawnPolicy with a speculating() method. Persistent
workers need to know if speculation is happening in order to require
sandboxing.
Additionally, I added java_test rules for the local runner tests and
worker tests. See https://github.com/bazelbuild/bazel/issues/3481.
NOTE: ulfjack@ made some changes to this change before merging:
- changed Reporter to EventHandler; added TODO about its usage
- reverted non-semantic indentation change in AbstractSpawnStrategy
- reverted a non-semantic indentation change in WorkerSpawnRunner
- updated some internal classes to match
- removed catch IOException in WorkerSpawnRunner in some cases,
removed verboseFailures flag from WorkerSpawnRunner, updated callers
- disable some tests on Windows; we were previously not running them,
now that we do, they fail :-(
Change-Id: I207b3938f0dc84d374ab052d5030020886451d47
PiperOrigin-RevId: 164965398
|
|
|
|
|
|
|
| |
...instead of injecting it through the constructor. Simplify all the callers
accordingly.
PiperOrigin-RevId: 164955391
|
|
|
|
|
|
|
|
|
|
|
| |
This isn't ideal, but is the fastest way to split analysis, execution, and
rules into separate java libraries, with lib.skyframe still being in the
analysis library. Ideally, we'd split up the lib.skyframe package, move the
analysis stuff to the analysis library, the execution stuff to the execution
library, and so on. That wouldn't require us moving OutputService out of
lib.exec.
PiperOrigin-RevId: 164856998
|
|
|
|
|
|
|
|
| |
These are depended upon by analysis code, so need to live in the same library
as lib.analysis. Moving them here makes it possible to split the build-base
library into separate libraries for analysis, execution, and rules.
PiperOrigin-RevId: 164847161
|
|
|
|
| |
PiperOrigin-RevId: 164844520
|
|
|
|
|
|
|
| |
All prefetching now goes through AbstractSpawnStrategy's implementation of
SpawnExecutionPolicy. Make sure the sandbox runners also do this consistently.
PiperOrigin-RevId: 164836877
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Important: the simplified API now defaults to forwarding interrupts to
subprocesses. I did audit all the call sites, and I think this is a safe change
to make.
- Properly support timeouts with all implementations
- Simplify the API
- only provide two flavours of blocking calls, which require no input and
forward interrupts; this is the most common usage
- provide a number of async calls, which optionally takes input, and a flag
whether to forward interrupts
- only support input streams, no byte arrays or other 'convenience features'
that are rarely needed and unnecessarily increase the surface area
- use java.time.Duration to specify timeout; for consistency, interpret a
timeout of <= 0 as no timeout (i.e., including rather than excluding 0)
- KillableObserver and subclasses are no longer part of the public API, but
still used to implement timeouts if the Subprocess.Factory does not support
them
- Update the documentation for Command
- Update all callers; most callers now use the simplified API
PiperOrigin-RevId: 164716782
|
|
|
|
| |
PiperOrigin-RevId: 164577062
|
|
|
|
|
| |
RELNOTES: None
PiperOrigin-RevId: 163471182
|
|
|
|
|
|
|
|
| |
Also added tests specifically for the output, to ensure we don't break it again.
TESTED=remote worker, unit tests
RELNOTES: fixes #3380
PiperOrigin-RevId: 163283558
|
|
|
|
|
|
| |
This removes a bunch of code duplication that I previously introduced.
PiperOrigin-RevId: 162909430
|
|
|
|
| |
PiperOrigin-RevId: 162906926
|
|
|
|
|
|
| |
The plan is to add it to ActionExecutionContext, which is also there.
PiperOrigin-RevId: 162656835
|
|
|
|
|
|
|
| |
The option filters proto dependency can be removed from the OptionsParser. This is in response to option parser users that want to avoid the bazel-internal proto file in their dependencies.
RELNOTES: None.
PiperOrigin-RevId: 162249778
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- add an id for logging; this allows us to correlate log entries for the same
spawn from multiple spawn runner implementations in the future
- add a prefetch method to the SpawnExecutionPolicy; better than relying on
the ActionInputPrefetcher being injected in the constructor
- add a name parameter to the report method; this is in preparation for a
single unified SpawnStrategy implementation - it's basically the last bit of
difference between SandboxStrategy and RemoteSpawnStrategy; they're otherwise
equivalent (if not identical)
PiperOrigin-RevId: 162194684
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The main change here is to only catch SpawnExecException in
StandaloneTestStrategy, so all other exceptions simplify propagate up. As a
result, Bazel no longer retries tests that fail with an exception, we only
retry tests that actually ran, had a spawn result, and resulted in a
UserExecException. That is probably what we want.
Also do some cleanup:
- Remove ExecException.timedOut; nobody was calling it (but there's still
SpawnExecException.timedOut)
- Remove SpawnActionContext.shouldPropagateExecException; all exceptions
(except SpawnExecException) are now propagated by default
- Remote the SandboxOptions from the SandboxStrategies; all sandboxing options
are now handled by the underlying SpawnRunner implementations
I'll send a followup CL to remove the UserExecException and
EnvironmentalExecException types; the types don't do anything special, and
there are no catch blocks in production code that catch one of these more
specific types.
This should fix #3322 by removing a bunch of special handling.
PiperOrigin-RevId: 161960919
|
|
|
|
| |
PiperOrigin-RevId: 161944209
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 161937673
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Make use of existing abstractions like SpawnRunner and SpawnExecutionPolicy.
- Instead of having the *Strategy create a *Runner, and then call back into
SandboxStrategy, create a single SandboxContainer which contains the full
command line, environment, and everything needed to create and delete the
sandbox directory.
- Do all the work in SandboxStrategy, including creation and deletion of the
sandbox directory.
- Use SpawnResult instead of throwing, catching, and rethrowing.
- Simplify the control flow a bit.
PiperOrigin-RevId: 161644979
|
|
|
|
|
|
|
|
|
|
| |
OptionMetadataTags.
These are similar, no need to have both fields. Removing the "DOCUMENTED" default, the absence of UNDOCUMENTED will be used instead.
Since requiring a documentation category for undocumented options doesn't make sense, list that as one of the OptionDocumentationCategories, but list HIDDEN and INTERNAL as part of OptionMetadata. These options should list UNDOCUMENTED as their category.
PiperOrigin-RevId: 161515674
|
|
|
|
| |
PiperOrigin-RevId: 161512574
|
|
|
|
|
|
|
|
| |
Also report test warnings, if any, for completed test actions in
the build event protocol.
RELNOTES: None
PiperOrigin-RevId: 161384061
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The cachable flag was only set to false for flaky tests.
Before this CL, we did not cache flaky tests on subsequent runs, instead
rerunning them, even though this is both very expensive and inconsistent with
our normal handling, which is to cache passes but not errors.
If cache_test_results is enabled, we now also cached timed out test results.
If cache_test_results=auto (the default), we now also cache flaky tests.
We still do not cache failed tests; the TestRunnerAction checks if the previous
test result was marked as passed (which also applies to flaky tests, but not
to failed or timed-out tests).
Also add some unit tests.
PiperOrigin-RevId: 161187950
|