| Commit message (Collapse) | Author | Age |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bazel now adds env[TMPDIR] to the set of
sandbox-writable paths, instead of adding the
caller-defined `tmpDir` as it used to.
Since every caller of getWritableDirs passes the
LocalEnvProvider-processed environment to
getWritableDirs, and because all such callers use
either PosixLocalEnvProvider or
XCodeLocalEnvProvider, we can be sure that the
environment has an entry for TMPDIR.
Change-Id: Ia89544a009e56d9cc922ab56823d16d20465545e
PiperOrigin-RevId: 181595606
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When Bazel creates the sandbox for an action,
Bazel collects a set of paths that the action may
write to.
The action needs write access to its temp
directory, so Bazel needs to add it to the
writable paths.
See https://github.com/bazelbuild/bazel/issues/4376
Change-Id: Ifd3c482aa67ff8a2070045356abad8b39c808db8
PiperOrigin-RevId: 181591520
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When Bazel creates the sandbox, it will allow
making non-existent paths writable, as long as the
path is under the sandbox root.
As Bazel adds entries to the sandbox's set of
writable paths, Bazel needs to make sure that it's
not adding symlinks, because doing so would make
the symlink writable, not what the link points to.
If the path is under the sandbox root, then at the
time of setting up the sandbox's writable paths
the path surely doesn't exist yet, but that's OK,
because at that time Bazel didn't yet create the
sandbox root.
If the path is not under the sandbox root, then
Bazel needs to resolve all symlinks on this path,
which is only possible if the path exists,
therefore Bazel checks for the path's existence.
Change-Id: Ic7d99a81905e7401455286c0b375d69b85ece1d5
PiperOrigin-RevId: 181325749
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
As part of setting up a sandbox, Bazel creates the
list of writable paths. If the action's
environment defines $TEST_TMPDIR, then it's a test
action and the sandbox must allow writing to that
path, therefore Bazel must add $TEST_TMPDIR to the
writable paths.
Bazel must resolve symlinks in that path though,
at least on the last path segment, because in case
the path points to a symlink, the action would be
allowed to modify the symlink itself, and not
access what the link points to.
However the path must exist for Bazel to
successfully resolve symlinks, therefore this
commit adds a check for that.
Given that the code was there since at least July
2017, and I'm not aware of bugs caused by it, I
conclude that this code path either never runs or
nobody ever triggered it. Either way, adding the
check is the right thing to do.
Change-Id: I87a5d3fc3fe7878a918ed318c71e8d135f10f1b8
PiperOrigin-RevId: 180931382
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
that use either the LinuxSandboxedSpawnRunner or the ProcessWrapperSandboxedSpawnRunner.
In particular, record metrics for user and system CPU execution time, block I/O and involuntary context switches.
This feature is guarded behind a new option, --experimental_collect_local_sandbox_action_metrics.
Note: We still need to enable execution statistics for the DarwinSandboxedSpawnRunner in a later change.
RELNOTES: None.
PiperOrigin-RevId: 179976217
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
To reproduce: run a failing test with --experimental_remote_spawn_cache or with --spawn_strategy=remote and no executor. Expected: test log is uploaded.
Desired behavior:
- regardless of whether a spawn is cacheable or not, its artifacts should be uploaded to the remote cache.
- the spawn result should only be set if the spawn is cacheable *and* the action succeeded.
- when executing remotely, the do_not_cache field should be set for non-cacheable spawns, and the remote execution engine should respect it.
This CL contains multiple fixes to ensure the above behaviors, and adds a few tests, both end to end and unit tests. Important behavior change: it is no longer assumed that non-cacheable spawns should use a NO_CACHE SpawnCache! The appropriate test case was removed. Instead, an assumption was added that all implementations of SpawnCache should respect the Spawns.mayBeCached(spawn) property. Currently, only NO_CACHE and RemoteSpawnCache exist, and they (now) support it.
TESTED=remote build execution backend.
WANT_LGTM: philwo,buchgr
RELNOTES: None
PiperOrigin-RevId: 178617937
|
| |
|
|
|
| |
RELNOTES: None
PiperOrigin-RevId: 177290508
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
It turns out that the SUCCESS status is often misunderstood to mean "zero exit",
even though this is clearly documented. I've decided to add another status for
non-zero exit, and use success only for zero exit to avoid this pitfall.
Also, many of the status codes are set, but never used. I decided to reduce the
number of status codes to only those that are actually relevant, which
simplifies further processing. Instead, we should add a string message for the
error case when we need one - we're not using it right now, so I decided not to
add that yet.
PiperOrigin-RevId: 177129441
|
| |
|
|
|
|
|
| |
and make cumulative execution times available in ActionResults.
RELNOTES: None
PiperOrigin-RevId: 174553272
|
| |
|
|
|
|
| |
This requires a fairly large amount of changes to fundamental objects like BlazeRuntime, Executor, and so on, as well as changing a lot of test code to thread the file system through. I expect future CLs to be much smaller.
PiperOrigin-RevId: 173678144
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Every build and test action that creates a Spawn
will now have platform-specific environment
variables for temp directories:
- on Windows: TMP and TEMP
- on Linux/Darwin: TMPDIR
This is particularly important on Windows where
e.g. Java programs cannot create temp directories
unless there's a valid TMP or TEMP environment
variable set.
Fixes:
- https://github.com/bazelbuild/bazel/issues/1590
- https://github.com/bazelbuild/bazel/issues/2349
- https://github.com/bazelbuild/bazel/issues/2870
Change-Id: Ib758307daf6b3a51b0f71ae5e65e5bb564dad643
PiperOrigin-RevId: 172326371
|
| |
|
|
|
|
|
| |
Whether or not there was a catastrophic error is stored in the SpawnResult, so
we can just use that instead of passing in an additional boolean.
PiperOrigin-RevId: 172083752
|
| |
|
|
|
|
|
| |
build.lib.actions.SpawnActionContext can import SpawnResult without creating a cyclic dependency.
RELNOTES: None.
PiperOrigin-RevId: 169642267
|
| |
|
|
|
|
|
| |
All prefetching now goes through AbstractSpawnStrategy's implementation of
SpawnExecutionPolicy. Make sure the sandbox runners also do this consistently.
PiperOrigin-RevId: 164836877
|
| |
|
|
|
|
| |
Fixes #3521.
PiperOrigin-RevId: 164728178
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Important: the simplified API now defaults to forwarding interrupts to
subprocesses. I did audit all the call sites, and I think this is a safe change
to make.
- Properly support timeouts with all implementations
- Simplify the API
- only provide two flavours of blocking calls, which require no input and
forward interrupts; this is the most common usage
- provide a number of async calls, which optionally takes input, and a flag
whether to forward interrupts
- only support input streams, no byte arrays or other 'convenience features'
that are rarely needed and unnecessarily increase the surface area
- use java.time.Duration to specify timeout; for consistency, interpret a
timeout of <= 0 as no timeout (i.e., including rather than excluding 0)
- KillableObserver and subclasses are no longer part of the public API, but
still used to implement timeouts if the Subprocess.Factory does not support
them
- Update the documentation for Command
- Update all callers; most callers now use the simplified API
PiperOrigin-RevId: 164716782
|
| |
|
|
| |
PiperOrigin-RevId: 164577062
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Remove BuildRequest as parameter to sandboxed spawn runner
constructors. Previously, the build request was used to obtain some
options, but those can be extricated from a CommandEnvironment, which
is passed in, too.
Also, remove LinuxSandboxedSpawnRunner's aliased sandboxOptions member
variable. It can just use the superclass's.
Change-Id: I1ef1a45cbf7e800d0809f05673f097a148289740
PiperOrigin-RevId: 164257471
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
- add an id for logging; this allows us to correlate log entries for the same
spawn from multiple spawn runner implementations in the future
- add a prefetch method to the SpawnExecutionPolicy; better than relying on
the ActionInputPrefetcher being injected in the constructor
- add a name parameter to the report method; this is in preparation for a
single unified SpawnStrategy implementation - it's basically the last bit of
difference between SandboxStrategy and RemoteSpawnStrategy; they're otherwise
equivalent (if not identical)
PiperOrigin-RevId: 162194684
|
|
|
This adds a bunch of classes that only implement the SpawnRunner interface, and
will allow us to support remote caching in combination with local sandboxed
execution in a subsequent change.
PiperOrigin-RevId: 161664556
|
Laszlo Csomor
ruperts
olaola
ulfjack
tomlu
Benjamin Peterson