| Commit message (Collapse) | Author | Age |
... | |
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 179646155
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
To reproduce: run a failing test with --experimental_remote_spawn_cache or with --spawn_strategy=remote and no executor. Expected: test log is uploaded.
Desired behavior:
- regardless of whether a spawn is cacheable or not, its artifacts should be uploaded to the remote cache.
- the spawn result should only be set if the spawn is cacheable *and* the action succeeded.
- when executing remotely, the do_not_cache field should be set for non-cacheable spawns, and the remote execution engine should respect it.
This CL contains multiple fixes to ensure the above behaviors, and adds a few tests, both end to end and unit tests. Important behavior change: it is no longer assumed that non-cacheable spawns should use a NO_CACHE SpawnCache! The appropriate test case was removed. Instead, an assumption was added that all implementations of SpawnCache should respect the Spawns.mayBeCached(spawn) property. Currently, only NO_CACHE and RemoteSpawnCache exist, and they (now) support it.
TESTED=remote build execution backend.
WANT_LGTM: philwo,buchgr
RELNOTES: None
PiperOrigin-RevId: 178617937
|
|
|
|
|
|
|
|
|
| |
everywhere instead of duplicating process-wrapper --shell_arguments in Blaze.
To avoid a cyclic dependency, I broke up exec/local:local into exec/local:local and exec/local:options.
RELNOTES: None.
PiperOrigin-RevId: 177419268
|
|
|
|
|
| |
RELNOTES: None
PiperOrigin-RevId: 177290508
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- remove BaseSpawn.Local; instead, all callers pass in the full set of
execution requirements they want to set
- disable caching and sandboxing for the symlink tree action - it does not
declare outputs, so it can't be cached or sandboxed (fixes #4041)
- centralize the existing execution requirements in the ExecutionRequirements
class
- centralize checking for execution requirements in the Spawn class
(it's possible that we may need a more decentralized, extensible design in
the future, but for now having them in a single place is simple and
effective)
- update the documentation
- forward the relevant tags to execution requirements in TargetUtils (progress
on #3960)
- this also contributes to #4153
PiperOrigin-RevId: 177288598
|
|
|
|
|
|
|
| |
Also rename ProcessWrapperRunner to ProcessWrapperUtil, since it isn't really a Runner (it doesn't run anything).
RELNOTES: None.
PiperOrigin-RevId: 177260951
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It turns out that the SUCCESS status is often misunderstood to mean "zero exit",
even though this is clearly documented. I've decided to add another status for
non-zero exit, and use success only for zero exit to avoid this pitfall.
Also, many of the status codes are set, but never used. I decided to reduce the
number of status codes to only those that are actually relevant, which
simplifies further processing. Instead, we should add a string message for the
error case when we need one - we're not using it right now, so I decided not to
add that yet.
PiperOrigin-RevId: 177129441
|
|
|
|
|
|
|
|
| |
Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own.
This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually.
PiperOrigin-RevId: 175033526
|
|
|
|
|
|
|
| |
and make cumulative execution times available in ActionResults.
RELNOTES: None
PiperOrigin-RevId: 174553272
|
|
|
|
|
|
| |
This requires a fairly large amount of changes to fundamental objects like BlazeRuntime, Executor, and so on, as well as changing a lot of test code to thread the file system through. I expect future CLs to be much smaller.
PiperOrigin-RevId: 173678144
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Every build and test action that creates a Spawn
will now have platform-specific environment
variables for temp directories:
- on Windows: TMP and TEMP
- on Linux/Darwin: TMPDIR
This is particularly important on Windows where
e.g. Java programs cannot create temp directories
unless there's a valid TMP or TEMP environment
variable set.
Fixes:
- https://github.com/bazelbuild/bazel/issues/1590
- https://github.com/bazelbuild/bazel/issues/2349
- https://github.com/bazelbuild/bazel/issues/2870
Change-Id: Ib758307daf6b3a51b0f71ae5e65e5bb564dad643
PiperOrigin-RevId: 172326371
|
|
|
|
|
|
|
| |
Whether or not there was a catastrophic error is stored in the SpawnResult, so
we can just use that instead of passing in an additional boolean.
PiperOrigin-RevId: 172083752
|
|
|
|
|
|
|
|
|
| |
ActionContexts, etc., so that SpawnResult metadata is returned upwards.
Note that the TODOs mostly refer to changes that will appear in a subsequent CL (a CL to return SpawnResults, contained in ActionResults, from Actions/AbstractActions). I split off the remaining SpawnResult-related changes into this CL and kept the ActionResult-related changes separate.
RELNOTES: None.
PiperOrigin-RevId: 171355611
|
|
|
|
|
|
|
| |
build.lib.actions.SpawnActionContext can import SpawnResult without creating a cyclic dependency.
RELNOTES: None.
PiperOrigin-RevId: 169642267
|
|
|
|
|
|
|
|
|
| |
Split collect, concurrent, vfs, windows into package-level BUILD files.
Move clock classes out of "util", into their own Java package.
Move CompactHashSet into its own Java package to break a dependency cycle.
Give nestedset and inmemoryfs their own package-level BUILD files.
PiperOrigin-RevId: 167702127
|
|
|
|
| |
PiperOrigin-RevId: 166849610
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
linux-sandbox has a useful option -R, that runs the spawn as fake
root. However, it's not exposed to Bazel rules. Here, we do that via
the "requires-fakeroot" tag.
One possible usecase: In combination with "block-network",
"requires-fakeroot" makes it possible to integration test services
that insist on listening on privileged ports.
Unsurprisingly, this is incompatible with --sandbox_fake_username.
Change-Id: I9e8ab4d4abf0e45626e005ff21f73e6c17de0788
PiperOrigin-RevId: 164961019
|
|
|
|
|
|
|
| |
...instead of injecting it through the constructor. Simplify all the callers
accordingly.
PiperOrigin-RevId: 164955391
|
|
|
|
|
|
|
|
| |
These are depended upon by analysis code, so need to live in the same library
as lib.analysis. Moving them here makes it possible to split the build-base
library into separate libraries for analysis, execution, and rules.
PiperOrigin-RevId: 164847161
|
|
|
|
|
|
|
| |
All prefetching now goes through AbstractSpawnStrategy's implementation of
SpawnExecutionPolicy. Make sure the sandbox runners also do this consistently.
PiperOrigin-RevId: 164836877
|
|
|
|
|
|
| |
Fixes #3521.
PiperOrigin-RevId: 164728178
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Important: the simplified API now defaults to forwarding interrupts to
subprocesses. I did audit all the call sites, and I think this is a safe change
to make.
- Properly support timeouts with all implementations
- Simplify the API
- only provide two flavours of blocking calls, which require no input and
forward interrupts; this is the most common usage
- provide a number of async calls, which optionally takes input, and a flag
whether to forward interrupts
- only support input streams, no byte arrays or other 'convenience features'
that are rarely needed and unnecessarily increase the surface area
- use java.time.Duration to specify timeout; for consistency, interpret a
timeout of <= 0 as no timeout (i.e., including rather than excluding 0)
- KillableObserver and subclasses are no longer part of the public API, but
still used to implement timeouts if the Subprocess.Factory does not support
them
- Update the documentation for Command
- Update all callers; most callers now use the simplified API
PiperOrigin-RevId: 164716782
|
|
|
|
| |
PiperOrigin-RevId: 164577062
|
|
|
|
|
|
| |
--test_env isn't moved in this CL since it's exposed to Skylark via BuildConfiguration, making it a somewhat riskier refactor.
PiperOrigin-RevId: 164266168
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Remove BuildRequest as parameter to sandboxed spawn runner
constructors. Previously, the build request was used to obtain some
options, but those can be extricated from a CommandEnvironment, which
is passed in, too.
Also, remove LinuxSandboxedSpawnRunner's aliased sandboxOptions member
variable. It can just use the superclass's.
Change-Id: I1ef1a45cbf7e800d0809f05673f097a148289740
PiperOrigin-RevId: 164257471
|
|
|
|
|
|
| |
Closes #3444.
PiperOrigin-RevId: 163047183
|
|
|
|
|
|
| |
This removes a bunch of code duplication that I previously introduced.
PiperOrigin-RevId: 162909430
|
|
|
|
|
|
| |
The plan is to add it to ActionExecutionContext, which is also there.
PiperOrigin-RevId: 162656835
|
|
|
|
|
|
|
|
|
|
|
|
| |
TestRunner actions are special because their action outputs are
different from their spawn outputs. If there's a spawn output that's
not an action output, SymlinkExecroot can't rely on the parent
directories for that output existing in the real execroot. Thus,
copyOutputs() must ensure the real execroot has the relevant ancestral
directories before copying the output over.
Change-Id: I84fd69cd51628c51de9c8993b6a4407bbff038a0
PiperOrigin-RevId: 162470058
|
|
|
|
|
|
| |
The sandbox execution strategies were not passing the timeout grace period specified via --local_termination_grace_seconds, resulting in process-wrapper defaulting to just 5 seconds and linux-sandbox not using any grace period at all, instead directly SIGKILLing the child on timeout.
PiperOrigin-RevId: 162349919
|
|
|
|
|
|
|
| |
The option filters proto dependency can be removed from the OptionsParser. This is in response to option parser users that want to avoid the bazel-internal proto file in their dependencies.
RELNOTES: None.
PiperOrigin-RevId: 162249778
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- add an id for logging; this allows us to correlate log entries for the same
spawn from multiple spawn runner implementations in the future
- add a prefetch method to the SpawnExecutionPolicy; better than relying on
the ActionInputPrefetcher being injected in the constructor
- add a name parameter to the report method; this is in preparation for a
single unified SpawnStrategy implementation - it's basically the last bit of
difference between SandboxStrategy and RemoteSpawnStrategy; they're otherwise
equivalent (if not identical)
PiperOrigin-RevId: 162194684
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The main change here is to only catch SpawnExecException in
StandaloneTestStrategy, so all other exceptions simplify propagate up. As a
result, Bazel no longer retries tests that fail with an exception, we only
retry tests that actually ran, had a spawn result, and resulted in a
UserExecException. That is probably what we want.
Also do some cleanup:
- Remove ExecException.timedOut; nobody was calling it (but there's still
SpawnExecException.timedOut)
- Remove SpawnActionContext.shouldPropagateExecException; all exceptions
(except SpawnExecException) are now propagated by default
- Remote the SandboxOptions from the SandboxStrategies; all sandboxing options
are now handled by the underlying SpawnRunner implementations
I'll send a followup CL to remove the UserExecException and
EnvironmentalExecException types; the types don't do anything special, and
there are no catch blocks in production code that catch one of these more
specific types.
This should fix #3322 by removing a bunch of special handling.
PiperOrigin-RevId: 161960919
|
|
|
|
|
|
|
|
| |
This adds a bunch of classes that only implement the SpawnRunner interface, and
will allow us to support remote caching in combination with local sandboxed
execution in a subsequent change.
PiperOrigin-RevId: 161664556
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Make use of existing abstractions like SpawnRunner and SpawnExecutionPolicy.
- Instead of having the *Strategy create a *Runner, and then call back into
SandboxStrategy, create a single SandboxContainer which contains the full
command line, environment, and everything needed to create and delete the
sandbox directory.
- Do all the work in SandboxStrategy, including creation and deletion of the
sandbox directory.
- Use SpawnResult instead of throwing, catching, and rethrowing.
- Simplify the control flow a bit.
PiperOrigin-RevId: 161644979
|
|
|
|
|
|
|
|
| |
Move the default from the annotation to every mention. This makes the incompleteness explicit. Will add the defaults to test targets in a separate change.
Once all dependencies are cleaned up, the Option annotation will no longer allow options without the documentationCategory or effectTag, to prevent new options being added without categories while we migrate to the new option categorization.
PiperOrigin-RevId: 160281252
|
|
|
|
|
|
|
|
| |
This will allow us to add new and optional flags like selecting a strategy used to spawn / wait for the child process.
No one except Bazel should be calling "process-wrapper" and I couldn't find any references, so this breaking change should be fine.
PiperOrigin-RevId: 159685867
|
|
|
|
| |
PiperOrigin-RevId: 159423459
|
|
|
|
|
|
|
| |
Move everything to ActionExecutionContext, and drop Executor whereever possible.
This clarifies the API, makes it simpler to test, and simplifies the code.
PiperOrigin-RevId: 159414816
|
|
|
|
| |
PiperOrigin-RevId: 159221067
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Also add an interface to allow injecting that logic into LocalSpawnRunner; this
is in preparation for rewriting StandaloneSpawnStrategy to use
LocalSpawnRunner.
At the same time, this reduces the dependencies from exec / standalone to
rules.apple, which is a prerequisite for micro-Bazel.
There's a small semantic change hidden here - we now only set the new
XCodeLocalEnvProvider if we're actually running on Darwin, so we no longer
fail execution on non-Darwin platforms if XCODE_VERSION_OVERRIDE or
APPLE_SDK_VERSION_OVERRIDE is set. As a result, I moved the corresponding test
from StandaloneSpawnStrategyTest to the new XCodeLocalEnvProviderTest.
While I'm at it, also open source DottedVersionTest and CacheManagerTest.
PiperOrigin-RevId: 158829077
|
|
|
|
|
|
|
|
| |
existing directory writable when running actions.
RELNOTES: Added a new flag --sandbox_writable_path, which asks the sandbox to
make an existing directory writable when running actions.
PiperOrigin-RevId: 157971858
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Roll forward of directory name change
*** Original change description ***
Automated g4 rollback of commit 1d9e1ac90197b1d3d7b137ba3c1ada67bb9ba31b.
*** Reason for rollback ***
Breaks //src/test/shell/integration:force_delete_output_test
*** Original change description ***
Symlink output directories to the correct directory name
If the workspace directory is /path/to/my/proj and the name in the WORKSPACE
file is "floop", this will symlink the output directories to
output_base/execroot/floop instead of output_base/execroot/proj.
More prep for #1262, fixes #1681.
PiperOrigin-RevId: 156892980
|
|
|
|
|
|
|
|
| |
By removing the now unnecessary call to Path#resolveSymbolicLinks we can save a
few stat's per action execution.
Change-Id: Iee157e941c1cd3515ff5ea3b7f410824c24cf44d
PiperOrigin-RevId: 155946544
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is basically a rollback of https://github.com/bazelbuild/bazel/commit/3e2329a73ffd5d60e5e2babe60ebe5bf322c07da, except this solves the
reason why the feature was removed in the first place. We now create
the helper files necessary to make files unreadable in Linux in Bazel's
Java code and manage their lifetime there.
Request was filed by a user here:
http://stackoverflow.com/questions/43849651/how-to-lock-down-the-bazel-filesystem-sandbox
PiperOrigin-RevId: 155913246
|
|
|
|
|
|
|
|
|
| |
This unifies our code to use just one standard implementation to get the
entire expanded input files for a Spawn, including from Filesets and
Runfiles.
Change-Id: I1e286508adf0a9aeddf70934b010e6fcc144c4a7
PiperOrigin-RevId: 155497273
|
|
|
|
|
|
|
|
|
|
|
| |
Hardlinks are problematic due to not working across filesystem
boundaries and causing Bazel to do lots of I/O because it has to create
a hardlink and a symlink for each input file.
This improves performance of Bazel building itself by 10% on my system.
Change-Id: I8acb77053de875160a046e38624735ed18375bed
PiperOrigin-RevId: 155493583
|
|
|
|
|
|
|
|
|
| |
This gives us much improved process management, because Bazel can now
reliably kill child processes of spawns via their process group and wait
for them to exit.
Change-Id: Ib3cb20725b3c569aa5b317a69d7682f5774707b0
PiperOrigin-RevId: 155493511
|
|
|
|
|
| |
Change-Id: Idc023f3a8c1c3b60d3f3f23a579a5eccb92d074d
PiperOrigin-RevId: 155487527
|
|
|
|
|
| |
Change-Id: I1522c364a157ee0a144ab83eca54e419142c03b1
PiperOrigin-RevId: 155484109
|