| Commit message (Collapse) | Author | Age |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
as a list, rather than as a single line (i.e., newline delimited rather than
space delimited).
Before:
SUBCOMMAND: # //src/main/java/com/google/devtools/build/lib:string_util [action 'Building src/main/java/com/google/devtools/build/lib/libstring_util.jar (2 source files) [for host]']
(cd /tmp/devbazel_output_base/execroot/io_bazel && \
exec env - \
LC_CTYPE=en_US.UTF-8 \
external/embedded_jdk/bin/java -XX:+UseParallelOldGC -XX:-CompactStrings '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--patch-module=java.compiler=external/bazel_tools/third_party/java/jdk/langtools/java_compiler.jar' '--patch-module=jdk.compiler=external/bazel_tools/third_party/java/jdk/langtools/jdk_compiler.jar' '--add-opens=java.base/java.nio=ALL-UNNAMED' -jar external/bazel_tools/tools/jdk/JavaBuilder_deploy.jar @bazel-out/host/bin/src/main/java/com/google/devtools/build/lib/libstring_util.jar-2.params)
After:
SUBCOMMAND: # //src/main/java/com/google/devtools/build/lib:string_util [action 'Building src/main/java/com/google/devtools/build/lib/libstring_util.jar (2 source files) [for host]']
(cd /tmp/devbazel_output_base/execroot/io_bazel && \
exec env - \
LC_CTYPE=en_US.UTF-8 \
external/embedded_jdk/bin/java \
-XX:+UseParallelOldGC \
-XX:-CompactStrings \
'--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' \
'--add-exports=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED' \
'--add-exports=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED' \
'--add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' \
'--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' \
'--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED' \
'--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' \
'--add-opens=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' \
'--patch-module=java.compiler=external/bazel_tools/third_party/java/jdk/langtools/java_compiler.jar' \
'--patch-module=jdk.compiler=external/bazel_tools/third_party/java/jdk/langtools/jdk_compiler.jar' \
'--add-opens=java.base/java.nio=ALL-UNNAMED' \
-jar \
external/bazel_tools/tools/jdk/JavaBuilder_deploy.jar \
@bazel-out/host/bin/src/main/java/com/google/devtools/build/lib/libstring_util.jar-2.params)
RELNOTES: --subcommands can now take a "pretty_print" value ("--subcommands=pretty_print") to print the
arguments of subcommands as a list for easier reading.
PiperOrigin-RevId: 206213009
|
|
|
|
| |
PiperOrigin-RevId: 202092962
|
|
|
|
|
|
|
|
| |
of the --worker_verbose setting.
This message is important because it can catch cases where people are unknowingly running consecutive builds with different options, leading to much slower builds. In this regard it is a lot like "Build options have changed, discarding analysis cache".
PiperOrigin-RevId: 201648554
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This also gets rid of some boilerplate. The ExecutionTool.addActionContext
method has been around for a while, but is underused.
There are still a few ActionContextProvider implementations left, which are
implementing other functionality besides adding action contexts.
As a side effect, this change reduces null build time with a hot server on
linux by about a quarter. We were running the linux sandbox twice on every
build, which takes about 70ms each (on my machine), with the total null build
time around 300ms.
PiperOrigin-RevId: 200045145
|
|
|
|
|
|
|
|
| |
Instead, add some simple APIs to ExecutorBuilder and inline all the previous
subclasses into their corresponding modules. This removes a bunch of
boilerplate.
PiperOrigin-RevId: 200017162
|
|
|
|
| |
PiperOrigin-RevId: 199732415
|
|
|
|
|
|
| |
(minor) ActionFS now implements MetadataProvider.getInput
PiperOrigin-RevId: 199575194
|
|
|
|
|
|
|
| |
each named worker.
RELNOTES: Allow --worker_max_instances to take MnemonicName=value to specify max for each worker.
PiperOrigin-RevId: 195244295
|
|
|
|
| |
PiperOrigin-RevId: 195100670
|
|
|
|
|
|
|
| |
Spawns can contain param files in the form of virtual action inputs. Sandboxed runners must write these prior to execution of the action.
RELNOTES: None
PiperOrigin-RevId: 194951208
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There is no need for the cache to be on disk. Originally, there was a
desire to share this cache with other tools... but this never happened.
And, actually, because Bazel is in control of what it runs, it can just
inject the "cached" values into those tools via flags.
Instead, just store the cache in-memory. This avoids having to open and
read the cache on every single action that is locally executed on a Mac.
Results when building a large iOS app from a clean slate show up to a
1% wall time improvement on my Mac Pro 2013 and a reduction in the
variance of the measurements.
This change also gets rid of the OS check from the action execution's
critical path. There is not much use in checking this: if we instantiate
this by mistake, the actual calls will fail. But sometimes we want to
actually run this code on non-macOS systems (e.g. for unit-testing with
mocked tools), so we should allow that. And this change also ensures
that XcodeLocalEnvProviderTest builds and runs...
RELNOTES: None.
PiperOrigin-RevId: 194681802
|
|
|
|
|
|
|
|
|
|
|
| |
When two or more instances of high priority workers are running, no other worker execution may start.
Tweak WorkerSpawnRunner. Only reserve resources for a worker after it has been obtained from the pool.
This allows us to block waiting for high priority workers to have finished before resources are reserved.
RELNOTES: Add --high_priority_workers flag.
PiperOrigin-RevId: 193672343
|
|
|
|
|
|
|
|
| |
This class will be used to tie a Spawn to a SpawnRunner, and isn't really a policy object. It will carry state such as the expanded inputs and expanded command line.
Currently a context can be passed between different SpawnRunners. This will be addressed independently, so a context is tied to a particular spawn runner.
PiperOrigin-RevId: 193501918
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
https://github.com/bazelbuild/bazel/commit/656a0bab1e025ff3c27d595284a4bf1c5a8d8028 with test (unknown commit) and fix.
Big round of sandbox fixes / performance improvements.
- The number of stat() syscalls in the SymlinkedSandboxedSpawn was way too high. Do less, feel better.
- When using --experimental_sandbox_base, ensure that symlinks in the path are resolved. Before this, you had to check whether on your system /dev/shm is a symlink to /run/shm and then use that instead. Now it no longer matters, as symlinks are resolved.
- Remove an unnecessary directory creation from each sandboxed invocation. Turns out that the "tmpdir" that we created was no longer used after some changes to Bazel's TMPDIR handling.
- Use simpler sandbox paths, by using the unique ID for each Spawn provided by SpawnExecutionPolicy instead of a randomly generated temp folder name. This also saves a round-trip from our VFS to NIO and back. Clean up the sandbox base before each build to ensure that the unique IDs are actually unique. ;)
- Use Java 8's Process#isAlive to check whether a process is alive instead of trying to get the exitcode and catching an exception.
Closes #4913.
PiperOrigin-RevId: 193031017
|
|
|
|
|
|
|
|
|
| |
BuildConfiguration.Fragment#compatibleWithStrategy().
The experiment did not work out.
RELNOTES: None.
PiperOrigin-RevId: 192567832
|
|
|
|
| |
PiperOrigin-RevId: 191642942
|
|
|
|
|
|
|
|
|
| |
These have all had a chance to be categorized with the OptionDocumentationCategory enum, and the help output already uses the enum-grouped format.
The "incompatible changes" category has meaning for --all_incompatible_changes and will be removed separately.
RELNOTES: None.
PiperOrigin-RevId: 190773778
|
|
|
|
|
|
| |
holds a variety of strategy/context maps.
PiperOrigin-RevId: 190491357
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- The number of stat() syscalls in the SymlinkedSandboxedSpawn was way too high. Do less, feel better.
- When using --experimental_sandbox_base, ensure that symlinks in the path are resolved. Before this, you had to check whether on your system /dev/shm is a symlink to /run/shm and then use that instead. Now it no longer matters, as symlinks are resolved.
- Remove an unnecessary directory creation from each sandboxed invocation. Turns out that the "tmpdir" that we created was no longer used after some changes to Bazel's TMPDIR handling.
- Use simpler sandbox paths, by using the unique ID for each Spawn provided by SpawnExecutionPolicy instead of a randomly generated temp folder name. This also saves a round-trip from our VFS to NIO and back. Clean up the sandbox base before each build to ensure that the unique IDs are actually unique. ;)
- Use Java 8's Process#isAlive to check whether a process is alive instead of trying to get the exitcode and catching an exception.
Closes #4913.
PiperOrigin-RevId: 190472170
|
|
|
|
|
|
| |
Fixes #4897.
PiperOrigin-RevId: 190045930
|
|
|
|
| |
PiperOrigin-RevId: 189244665
|
|
|
|
|
|
| |
Closes #4640.
PiperOrigin-RevId: 188022228
|
|
|
|
|
|
|
|
|
|
| |
just add it to the interface, and include it in the SpawnResult. This will be used to categorize/aggregate spawns executed by various runners.
Also, minor refinement to the cacheHit property of the SpawnResult with remote execution.
RELNOTES: None
TESTED=presubmit, next cl
PiperOrigin-RevId: 186637978
|
|
|
|
|
|
| |
...instead of passing it everywhere so we can rewrite the local env.
PiperOrigin-RevId: 186316859
|
|
|
|
|
|
|
|
|
| |
script.
Accomplished by creating an explicit attribute for both of them on test actions.
RELNOTES: None.
PiperOrigin-RevId: 185132460
|
|
|
|
|
|
|
| |
Fixes https://github.com/bazelbuild/bazel/issues/4376
Change-Id: Id78bb0930044626304e54f07735db4d4b2c84720
PiperOrigin-RevId: 181959528
|
|
|
|
|
|
| |
They need this to parse input manifests. Previously we would grab the exec root from the Root, but wish to unsupport this.
PiperOrigin-RevId: 181669143
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Create a PosixLocalEnvProvider and
WindowsLocalEnvProvider class, with singleton
instances for now.
This refactoring should not change functionality,
it's just a requirement for an upcoming change.
That upcoming change is for these classes to
respect the client environment's TMPDIR or
TMP/TEMP envvars.
See https://github.com/bazelbuild/bazel/issues/4376
Change-Id: I032bb6f18adf8af9e43e6bc543c09c58adae3863
PiperOrigin-RevId: 180799936
|
|
|
|
|
|
|
|
| |
This is in preparation for merging FileArtifactValue and FileStateValue.
Progress on #3360.
PiperOrigin-RevId: 179832948
|
|
|
|
|
|
|
|
|
| |
everywhere instead of duplicating process-wrapper --shell_arguments in Blaze.
To avoid a cyclic dependency, I broke up exec/local:local into exec/local:local and exec/local:options.
RELNOTES: None.
PiperOrigin-RevId: 177419268
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It turns out that the SUCCESS status is often misunderstood to mean "zero exit",
even though this is clearly documented. I've decided to add another status for
non-zero exit, and use success only for zero exit to avoid this pitfall.
Also, many of the status codes are set, but never used. I decided to reduce the
number of status codes to only those that are actually relevant, which
simplifies further processing. Instead, we should add a string message for the
error case when we need one - we're not using it right now, so I decided not to
add that yet.
PiperOrigin-RevId: 177129441
|
|
|
|
|
|
|
|
|
| |
Currently we don't care about the list order of SpawnResults. However, we may care about the list order later. Also, if the equals() method for SpawnResults is ever changed then it may be problematic to return SpawnResults in a Set.
Aside: ActionResults use SpawnResults to calculate cumulative execution times for Actions, and may provide other metrics in future.
RELNOTES: None.
PiperOrigin-RevId: 176579460
|
|
|
|
|
|
|
|
| |
Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own.
This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually.
PiperOrigin-RevId: 175033526
|
|
|
|
|
|
|
| |
and make cumulative execution times available in ActionResults.
RELNOTES: None
PiperOrigin-RevId: 174553272
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
These two close operations were added to work around #1708, but caused #2675.
We found the root cause of the hanging problem in #1708 is a race
condition when creating Windows processes:
When Bazel trys to create two processes, one for a local command
execution, one for starting the worker process. The worker process
might accidentally inherits handles opened when creating the local
command process, and it holds those handles as long as it lives.
Therefore, ReadFile function hangs when handles for the write end of
stdout/stderr pipes are released by the worker.
The solution is to make Bazel native createProcess JNI function
explicitly inheirts handles as needed, and use this function to start
worker process.
Related: http://support.microsoft.com/kb/315939
Fixed https://github.com/bazelbuild/bazel/issues/2675
Change-Id: I1c9b1ac3c9383ed2fd28ea92f528f19649693275
PiperOrigin-RevId: 173244832
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Every build and test action that creates a Spawn
will now have platform-specific environment
variables for temp directories:
- on Windows: TMP and TEMP
- on Linux/Darwin: TMPDIR
This is particularly important on Windows where
e.g. Java programs cannot create temp directories
unless there's a valid TMP or TEMP environment
variable set.
Fixes:
- https://github.com/bazelbuild/bazel/issues/1590
- https://github.com/bazelbuild/bazel/issues/2349
- https://github.com/bazelbuild/bazel/issues/2870
Change-Id: Ib758307daf6b3a51b0f71ae5e65e5bb564dad643
PiperOrigin-RevId: 172326371
|
|
|
|
|
|
|
|
|
| |
ActionContexts, etc., so that SpawnResult metadata is returned upwards.
Note that the TODOs mostly refer to changes that will appear in a subsequent CL (a CL to return SpawnResults, contained in ActionResults, from Actions/AbstractActions). I split off the remaining SpawnResult-related changes into this CL and kept the ActionResult-related changes separate.
RELNOTES: None.
PiperOrigin-RevId: 171355611
|
|
|
|
|
|
|
| |
build.lib.actions.SpawnActionContext can import SpawnResult without creating a cyclic dependency.
RELNOTES: None.
PiperOrigin-RevId: 169642267
|
|
|
|
|
|
|
|
|
| |
Split collect, concurrent, vfs, windows into package-level BUILD files.
Move clock classes out of "util", into their own Java package.
Move CompactHashSet into its own Java package to break a dependency cycle.
Give nestedset and inmemoryfs their own package-level BUILD files.
PiperOrigin-RevId: 167702127
|
|
|
|
|
|
|
|
| |
We must check that a given artifact is a file, before calling getDigest, according to the interface contract specified in actions/Metadata.java.
Add a regression test to bazel_worker_test.sh, too. It's enough to simply add a directory entry to the worker's data to trigger the original bug.
PiperOrigin-RevId: 166856512
|
|
|
|
| |
PiperOrigin-RevId: 165571541
|
|
|
|
|
|
| |
Fixes #3329.
PiperOrigin-RevId: 165443367
|
|
|
|
|
|
|
| |
disk have changed since it was launched, print *which* files have changed.
RELNOTES: Improved logging when workers have to be restarted due to its files having changed.
PiperOrigin-RevId: 165419664
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Change the persistent worker spawn strategy to extend
AbstractSpawnStrategy and put the actual logic into
WorkerSpawnRunner. WorkerTestStrategy is unaffected.
I had to extend SpawnPolicy with a speculating() method. Persistent
workers need to know if speculation is happening in order to require
sandboxing.
Additionally, I added java_test rules for the local runner tests and
worker tests. See https://github.com/bazelbuild/bazel/issues/3481.
NOTE: ulfjack@ made some changes to this change before merging:
- changed Reporter to EventHandler; added TODO about its usage
- reverted non-semantic indentation change in AbstractSpawnStrategy
- reverted a non-semantic indentation change in WorkerSpawnRunner
- updated some internal classes to match
- removed catch IOException in WorkerSpawnRunner in some cases,
removed verboseFailures flag from WorkerSpawnRunner, updated callers
- disable some tests on Windows; we were previously not running them,
now that we do, they fail :-(
Change-Id: I207b3938f0dc84d374ab052d5030020886451d47
PiperOrigin-RevId: 164965398
|
|
|
|
|
|
|
|
| |
These are depended upon by analysis code, so need to live in the same library
as lib.analysis. Moving them here makes it possible to split the build-base
library into separate libraries for analysis, execution, and rules.
PiperOrigin-RevId: 164847161
|
|
|
|
|
|
|
| |
The option filters proto dependency can be removed from the OptionsParser. This is in response to option parser users that want to avoid the bazel-internal proto file in their dependencies.
RELNOTES: None.
PiperOrigin-RevId: 162249778
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The main change here is to only catch SpawnExecException in
StandaloneTestStrategy, so all other exceptions simplify propagate up. As a
result, Bazel no longer retries tests that fail with an exception, we only
retry tests that actually ran, had a spawn result, and resulted in a
UserExecException. That is probably what we want.
Also do some cleanup:
- Remove ExecException.timedOut; nobody was calling it (but there's still
SpawnExecException.timedOut)
- Remove SpawnActionContext.shouldPropagateExecException; all exceptions
(except SpawnExecException) are now propagated by default
- Remote the SandboxOptions from the SandboxStrategies; all sandboxing options
are now handled by the underlying SpawnRunner implementations
I'll send a followup CL to remove the UserExecException and
EnvironmentalExecException types; the types don't do anything special, and
there are no catch blocks in production code that catch one of these more
specific types.
This should fix #3322 by removing a bunch of special handling.
PiperOrigin-RevId: 161960919
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Make use of existing abstractions like SpawnRunner and SpawnExecutionPolicy.
- Instead of having the *Strategy create a *Runner, and then call back into
SandboxStrategy, create a single SandboxContainer which contains the full
command line, environment, and everything needed to create and delete the
sandbox directory.
- Do all the work in SandboxStrategy, including creation and deletion of the
sandbox directory.
- Use SpawnResult instead of throwing, catching, and rethrowing.
- Simplify the control flow a bit.
PiperOrigin-RevId: 161644979
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The cachable flag was only set to false for flaky tests.
Before this CL, we did not cache flaky tests on subsequent runs, instead
rerunning them, even though this is both very expensive and inconsistent with
our normal handling, which is to cache passes but not errors.
If cache_test_results is enabled, we now also cached timed out test results.
If cache_test_results=auto (the default), we now also cache flaky tests.
We still do not cache failed tests; the TestRunnerAction checks if the previous
test result was marked as passed (which also applies to flaky tests, but not
to failed or timed-out tests).
Also add some unit tests.
PiperOrigin-RevId: 161187950
|
|
|
|
|
|
|
|
|
| |
Add a single getMetadata method (matching MetadataHandler), and rewrite
everything in those terms.
This is in preparation for merging ActionInputFileCache and MetadataHandler.
PiperOrigin-RevId: 161053535
|