| Commit message (Collapse) | Author | Age |
|
|
|
|
|
|
| |
Fixes https://github.com/bazelbuild/bazel/issues/4376
Change-Id: Id78bb0930044626304e54f07735db4d4b2c84720
PiperOrigin-RevId: 181959528
|
|
|
|
|
|
| |
They need this to parse input manifests. Previously we would grab the exec root from the Root, but wish to unsupport this.
PiperOrigin-RevId: 181669143
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Create a PosixLocalEnvProvider and
WindowsLocalEnvProvider class, with singleton
instances for now.
This refactoring should not change functionality,
it's just a requirement for an upcoming change.
That upcoming change is for these classes to
respect the client environment's TMPDIR or
TMP/TEMP envvars.
See https://github.com/bazelbuild/bazel/issues/4376
Change-Id: I032bb6f18adf8af9e43e6bc543c09c58adae3863
PiperOrigin-RevId: 180799936
|
|
|
|
|
|
|
|
| |
This is in preparation for merging FileArtifactValue and FileStateValue.
Progress on #3360.
PiperOrigin-RevId: 179832948
|
|
|
|
|
|
|
|
|
| |
everywhere instead of duplicating process-wrapper --shell_arguments in Blaze.
To avoid a cyclic dependency, I broke up exec/local:local into exec/local:local and exec/local:options.
RELNOTES: None.
PiperOrigin-RevId: 177419268
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It turns out that the SUCCESS status is often misunderstood to mean "zero exit",
even though this is clearly documented. I've decided to add another status for
non-zero exit, and use success only for zero exit to avoid this pitfall.
Also, many of the status codes are set, but never used. I decided to reduce the
number of status codes to only those that are actually relevant, which
simplifies further processing. Instead, we should add a string message for the
error case when we need one - we're not using it right now, so I decided not to
add that yet.
PiperOrigin-RevId: 177129441
|
|
|
|
|
|
|
|
|
| |
Currently we don't care about the list order of SpawnResults. However, we may care about the list order later. Also, if the equals() method for SpawnResults is ever changed then it may be problematic to return SpawnResults in a Set.
Aside: ActionResults use SpawnResults to calculate cumulative execution times for Actions, and may provide other metrics in future.
RELNOTES: None.
PiperOrigin-RevId: 176579460
|
|
|
|
|
|
|
|
| |
Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own.
This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually.
PiperOrigin-RevId: 175033526
|
|
|
|
|
|
|
| |
and make cumulative execution times available in ActionResults.
RELNOTES: None
PiperOrigin-RevId: 174553272
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
These two close operations were added to work around #1708, but caused #2675.
We found the root cause of the hanging problem in #1708 is a race
condition when creating Windows processes:
When Bazel trys to create two processes, one for a local command
execution, one for starting the worker process. The worker process
might accidentally inherits handles opened when creating the local
command process, and it holds those handles as long as it lives.
Therefore, ReadFile function hangs when handles for the write end of
stdout/stderr pipes are released by the worker.
The solution is to make Bazel native createProcess JNI function
explicitly inheirts handles as needed, and use this function to start
worker process.
Related: http://support.microsoft.com/kb/315939
Fixed https://github.com/bazelbuild/bazel/issues/2675
Change-Id: I1c9b1ac3c9383ed2fd28ea92f528f19649693275
PiperOrigin-RevId: 173244832
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Every build and test action that creates a Spawn
will now have platform-specific environment
variables for temp directories:
- on Windows: TMP and TEMP
- on Linux/Darwin: TMPDIR
This is particularly important on Windows where
e.g. Java programs cannot create temp directories
unless there's a valid TMP or TEMP environment
variable set.
Fixes:
- https://github.com/bazelbuild/bazel/issues/1590
- https://github.com/bazelbuild/bazel/issues/2349
- https://github.com/bazelbuild/bazel/issues/2870
Change-Id: Ib758307daf6b3a51b0f71ae5e65e5bb564dad643
PiperOrigin-RevId: 172326371
|
|
|
|
|
|
|
|
|
| |
ActionContexts, etc., so that SpawnResult metadata is returned upwards.
Note that the TODOs mostly refer to changes that will appear in a subsequent CL (a CL to return SpawnResults, contained in ActionResults, from Actions/AbstractActions). I split off the remaining SpawnResult-related changes into this CL and kept the ActionResult-related changes separate.
RELNOTES: None.
PiperOrigin-RevId: 171355611
|
|
|
|
|
|
|
| |
build.lib.actions.SpawnActionContext can import SpawnResult without creating a cyclic dependency.
RELNOTES: None.
PiperOrigin-RevId: 169642267
|
|
|
|
|
|
|
|
|
| |
Split collect, concurrent, vfs, windows into package-level BUILD files.
Move clock classes out of "util", into their own Java package.
Move CompactHashSet into its own Java package to break a dependency cycle.
Give nestedset and inmemoryfs their own package-level BUILD files.
PiperOrigin-RevId: 167702127
|
|
|
|
|
|
|
|
| |
We must check that a given artifact is a file, before calling getDigest, according to the interface contract specified in actions/Metadata.java.
Add a regression test to bazel_worker_test.sh, too. It's enough to simply add a directory entry to the worker's data to trigger the original bug.
PiperOrigin-RevId: 166856512
|
|
|
|
| |
PiperOrigin-RevId: 165571541
|
|
|
|
|
|
| |
Fixes #3329.
PiperOrigin-RevId: 165443367
|
|
|
|
|
|
|
| |
disk have changed since it was launched, print *which* files have changed.
RELNOTES: Improved logging when workers have to be restarted due to its files having changed.
PiperOrigin-RevId: 165419664
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Change the persistent worker spawn strategy to extend
AbstractSpawnStrategy and put the actual logic into
WorkerSpawnRunner. WorkerTestStrategy is unaffected.
I had to extend SpawnPolicy with a speculating() method. Persistent
workers need to know if speculation is happening in order to require
sandboxing.
Additionally, I added java_test rules for the local runner tests and
worker tests. See https://github.com/bazelbuild/bazel/issues/3481.
NOTE: ulfjack@ made some changes to this change before merging:
- changed Reporter to EventHandler; added TODO about its usage
- reverted non-semantic indentation change in AbstractSpawnStrategy
- reverted a non-semantic indentation change in WorkerSpawnRunner
- updated some internal classes to match
- removed catch IOException in WorkerSpawnRunner in some cases,
removed verboseFailures flag from WorkerSpawnRunner, updated callers
- disable some tests on Windows; we were previously not running them,
now that we do, they fail :-(
Change-Id: I207b3938f0dc84d374ab052d5030020886451d47
PiperOrigin-RevId: 164965398
|
|
|
|
|
|
|
|
| |
These are depended upon by analysis code, so need to live in the same library
as lib.analysis. Moving them here makes it possible to split the build-base
library into separate libraries for analysis, execution, and rules.
PiperOrigin-RevId: 164847161
|
|
|
|
|
|
|
| |
The option filters proto dependency can be removed from the OptionsParser. This is in response to option parser users that want to avoid the bazel-internal proto file in their dependencies.
RELNOTES: None.
PiperOrigin-RevId: 162249778
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The main change here is to only catch SpawnExecException in
StandaloneTestStrategy, so all other exceptions simplify propagate up. As a
result, Bazel no longer retries tests that fail with an exception, we only
retry tests that actually ran, had a spawn result, and resulted in a
UserExecException. That is probably what we want.
Also do some cleanup:
- Remove ExecException.timedOut; nobody was calling it (but there's still
SpawnExecException.timedOut)
- Remove SpawnActionContext.shouldPropagateExecException; all exceptions
(except SpawnExecException) are now propagated by default
- Remote the SandboxOptions from the SandboxStrategies; all sandboxing options
are now handled by the underlying SpawnRunner implementations
I'll send a followup CL to remove the UserExecException and
EnvironmentalExecException types; the types don't do anything special, and
there are no catch blocks in production code that catch one of these more
specific types.
This should fix #3322 by removing a bunch of special handling.
PiperOrigin-RevId: 161960919
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Make use of existing abstractions like SpawnRunner and SpawnExecutionPolicy.
- Instead of having the *Strategy create a *Runner, and then call back into
SandboxStrategy, create a single SandboxContainer which contains the full
command line, environment, and everything needed to create and delete the
sandbox directory.
- Do all the work in SandboxStrategy, including creation and deletion of the
sandbox directory.
- Use SpawnResult instead of throwing, catching, and rethrowing.
- Simplify the control flow a bit.
PiperOrigin-RevId: 161644979
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The cachable flag was only set to false for flaky tests.
Before this CL, we did not cache flaky tests on subsequent runs, instead
rerunning them, even though this is both very expensive and inconsistent with
our normal handling, which is to cache passes but not errors.
If cache_test_results is enabled, we now also cached timed out test results.
If cache_test_results=auto (the default), we now also cache flaky tests.
We still do not cache failed tests; the TestRunnerAction checks if the previous
test result was marked as passed (which also applies to flaky tests, but not
to failed or timed-out tests).
Also add some unit tests.
PiperOrigin-RevId: 161187950
|
|
|
|
|
|
|
|
|
| |
Add a single getMetadata method (matching MetadataHandler), and rewrite
everything in those terms.
This is in preparation for merging ActionInputFileCache and MetadataHandler.
PiperOrigin-RevId: 161053535
|
|
|
|
|
|
|
|
| |
Move the default from the annotation to every mention. This makes the incompleteness explicit. Will add the defaults to test targets in a separate change.
Once all dependencies are cleaned up, the Option annotation will no longer allow options without the documentationCategory or effectTag, to prevent new options being added without categories while we migrate to the new option categorization.
PiperOrigin-RevId: 160281252
|
|
|
|
| |
PiperOrigin-RevId: 159423459
|
|
|
|
|
|
|
| |
Move everything to ActionExecutionContext, and drop Executor whereever possible.
This clarifies the API, makes it simpler to test, and simplifies the code.
PiperOrigin-RevId: 159414816
|
|
|
|
| |
PiperOrigin-RevId: 159221067
|
|
|
|
|
|
|
|
|
|
|
| |
Don't pass the Command annotation explicitly, but add it to CommandEnvironment
instead; most modules don't need it in the first place, so it was a lot of
boilerplate for not much. Also change it so that the command is passed to the
constructor.
Add some documentation to the beforeCommand method.
PiperOrigin-RevId: 158847128
|
|
|
|
|
| |
RELNOTES: None
PiperOrigin-RevId: 158498863
|
|
|
|
|
|
|
|
| |
Instead of passing a client env into the test strategies, use the same
mechansim as --action_env, by depending on the right set of Skyframe nodes that
correspond to client env entries.
PiperOrigin-RevId: 158401670
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Roll forward of directory name change
*** Original change description ***
Automated g4 rollback of commit 1d9e1ac90197b1d3d7b137ba3c1ada67bb9ba31b.
*** Reason for rollback ***
Breaks //src/test/shell/integration:force_delete_output_test
*** Original change description ***
Symlink output directories to the correct directory name
If the workspace directory is /path/to/my/proj and the name in the WORKSPACE
file is "floop", this will symlink the output directories to
output_base/execroot/floop instead of output_base/execroot/proj.
More prep for #1262, fixes #1681.
PiperOrigin-RevId: 156892980
|
|
|
|
|
|
|
|
|
| |
This unifies our code to use just one standard implementation to get the
entire expanded input files for a Spawn, including from Filesets and
Runfiles.
Change-Id: I1e286508adf0a9aeddf70934b010e6fcc144c4a7
PiperOrigin-RevId: 155497273
|
|
|
|
|
|
| |
Part of #2855.
PiperOrigin-RevId: 154477854
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This might sound strange at first, but the reasoning is this: A worker should never simply exit. Bazel controls the lifetime of its subprocesses, so a worker quitting is considered a failure (also because it cannot be distinguished from a crash).
With that set, we can improve two things:
- Bazel will now only notice that a worker crashed / quit when it tries to use one during a build. This is also the only time when we can print error messages to the user. Earlier we might have noticed that a worker crashed during validation, but had no mechanism to alert the user to this, because this wasn't necessarily during a build.
- This also fixes a race condition where a worker is still alive during validation, then quits, then the WorkerSpawnStrategy tries to send a WorkRequest, which fails, triggering an IOException.
This fixes the flaky test test_worker_restarts_after_exit (which is now called test_build_fails_when_worker_exits).
Part of #2855.
RELNOTES: Bazel will no longer gracefully restart workers that crashed / quit, instead this triggers a build failure.
PiperOrigin-RevId: 154470257
|
|
|
|
|
|
| |
Part of #2855.
PiperOrigin-RevId: 154461639
|
|
|
|
|
|
|
|
| |
Prevents it from potentially throwing an UnsupportedEncodingException.
Part of #2855.
PiperOrigin-RevId: 154446897
|
|
|
|
|
|
| |
Part of #2855.
PiperOrigin-RevId: 154427566
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It wasn't a good idea to have in the first place:
- It only ever worked in certain random circumstances (e.g. worker returning an unparseable protobuf, but not when it crashed).
- No other strategy implements a behavior like this.
- Confusing to users and hard to describe what use case this is good for.
- Complex and error prone code.
Part of #2855.
RELNOTES: The flag --worker_max_retries was removed. The WorkerSpawnStrategy no longer retries execution of failed Spawns, the reason being that this just masks compiler bugs and isn't done for any other execution strategy either.
PiperOrigin-RevId: 154422561
|
|
|
|
|
|
|
|
| |
check for the response and shows a helpful error message in that case.
Part of #2855.
PiperOrigin-RevId: 154416882
|
|
|
|
|
|
|
|
| |
This should help users and developers a lot figuring out *why* the worker crashed as the log file was very undiscoverable for them.
Part of the "make error messages great" effort (#2855).
PiperOrigin-RevId: 154165665
|
|
|
|
|
|
|
| |
WorkerTestStrategy as well, to better log errors.
RELNOTES: None
PiperOrigin-RevId: 154080821
|
|
|
|
| |
PiperOrigin-RevId: 152799488
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Breaks //src/test/shell/integration:force_delete_output_test
*** Original change description ***
Symlink output directories to the correct directory name
If the workspace directory is /path/to/my/proj and the name in the WORKSPACE
file is "floop", this will symlink the output directories to
output_base/execroot/floop instead of output_base/execroot/proj.
More prep for #1262, fixes #1681.
PiperOrigin-RevId: 152126545
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Note 1) Explanation of the flags:
--explicit_java_test_deps: This is a flag independent of the others, and forces users to specify the JUnit deps. We should try to make this flag default to true in a future release, irrespective of the work around persistent java tests.
--experimental_testrunner: Bazel-only flag affecting switching from the BazelTestRunner to the ExperimentalTestRunner which will run the tests in a separate classloader. --explicit_java_test_deps is desired for this to ensure once we split the classpaths of the TestRunner and the TestTarget, the TestTarget does not hit a ClassNotFoundException, due to missing JUnit deps.
--test_strategy=experimental_worker: This is the existing flag, which in turn depends on --experimental_testrunner flag, since only the ExperimentalTestRunner is capable to running java tests persistently.
Note 2) There was no clean way to check for the flags defined in JavaOptions within TestActionBuilder (as I was checking the "tag=experimental_testrunner" before), without making TeasActionBuilder's build rules depend on java rules (yikes!). Hence, I created a new method compatibleWithStrategy() within each fragment, so I could check if WorkerTestStrategy could be compatible with the user specified flags. Thanks to Greg for suggesting this approach!
RELNOTES: None
PiperOrigin-RevId: 151729869
|
|
|
|
|
|
|
|
|
|
| |
If the workspace directory is /path/to/my/proj and the name in the WORKSPACE
file is "floop", this will symlink the output directories to
output_base/execroot/floop instead of output_base/execroot/proj.
More prep for #1262, fixes #1681.
PiperOrigin-RevId: 151712384
|
|
|
|
|
|
|
|
| |
the ExperimentalTestRunner.
--
PiperOrigin-RevId: 149636903
MOS_MIGRATED_REVID=149636903
|
|
|
|
|
|
|
|
| |
This is essentially a rollforward of commit 7d0561b6ca92d72bd8767d4dca50e5437976812c, and changes triggering the perisitent runner using an environment variable instead of argument as suggested in commit 7d0561b6ca92d72bd8767d4dca50e5437976812c
--
PiperOrigin-RevId: 149540564
MOS_MIGRATED_REVID=149540564
|
|
|
|
|
|
|
|
|
|
| |
@flagfile.txt style.
Let the worker strategy correctly handle multiple flagfiles, instead of just assuming that the last argument will be the one and only @flagfile.
--
PiperOrigin-RevId: 149291230
MOS_MIGRATED_REVID=149291230
|