| Commit message (Collapse) | Author | Age |
|
|
|
|
|
|
|
|
|
| |
just add it to the interface, and include it in the SpawnResult. This will be used to categorize/aggregate spawns executed by various runners.
Also, minor refinement to the cacheHit property of the SpawnResult with remote execution.
RELNOTES: None
TESTED=presubmit, next cl
PiperOrigin-RevId: 186637978
|
|
|
|
|
|
| |
...instead of passing it everywhere so we can rewrite the local env.
PiperOrigin-RevId: 186316859
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
There's already a --test_tmpdir flag, and Java tests don't pick up this new one. More info: https://github.com/bazelbuild/bazel/issues/4621#issuecomment-366217321
*** Original change description ***
tmpdir,local-exec: implement --local_tmp_root
Add new flag called `--local_tmp_root`, which (if
specified) tells Bazel what temp directory should
locally executed actions use.
Fixes https://github.com/bazelbuild/bazel/issues/4621
Related to https://github.com/bazelbuild/bazel/issues/3215
RELNOTES[NEW]: The new "--local_tmp_root=<path>" flag allows specifying the temp directory for locally executed actions.
Change-Id: Ice69a5e63d0bf4d3b5c9ef4dbdd1ed1c5025f85e
PiperOrigin-RevId: 185982705
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add new flag called `--local_tmp_root`, which (if
specified) tells Bazel what temp directory should
locally executed actions use.
Fixes https://github.com/bazelbuild/bazel/issues/4621
Related to https://github.com/bazelbuild/bazel/issues/3215
RELNOTES[NEW]: The new "--local_tmp_root=<path>" flag allows specifying the temp directory for locally executed actions.
Change-Id: Ice69a5e63d0bf4d3b5c9ef4dbdd1ed1c5025f85e
PiperOrigin-RevId: 185509555
|
|
|
|
|
|
|
| |
rule for the execution statistics path.
RELNOTES: None.
PiperOrigin-RevId: 184534589
|
|
|
|
|
|
|
| |
Fixes https://github.com/bazelbuild/bazel/issues/4376
Change-Id: Id78bb0930044626304e54f07735db4d4b2c84720
PiperOrigin-RevId: 181959528
|
|
|
|
|
|
| |
They need this to parse input manifests. Previously we would grab the exec root from the Root, but wish to unsupport this.
PiperOrigin-RevId: 181669143
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bazel now adds env[TMPDIR] to the set of
sandbox-writable paths, instead of adding the
caller-defined `tmpDir` as it used to.
Since every caller of getWritableDirs passes the
LocalEnvProvider-processed environment to
getWritableDirs, and because all such callers use
either PosixLocalEnvProvider or
XCodeLocalEnvProvider, we can be sure that the
environment has an entry for TMPDIR.
Change-Id: Ia89544a009e56d9cc922ab56823d16d20465545e
PiperOrigin-RevId: 181595606
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When Bazel creates the sandbox for an action,
Bazel collects a set of paths that the action may
write to.
The action needs write access to its temp
directory, so Bazel needs to add it to the
writable paths.
See https://github.com/bazelbuild/bazel/issues/4376
Change-Id: Ifd3c482aa67ff8a2070045356abad8b39c808db8
PiperOrigin-RevId: 181591520
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When Bazel creates the sandbox, it will allow
making non-existent paths writable, as long as the
path is under the sandbox root.
As Bazel adds entries to the sandbox's set of
writable paths, Bazel needs to make sure that it's
not adding symlinks, because doing so would make
the symlink writable, not what the link points to.
If the path is under the sandbox root, then at the
time of setting up the sandbox's writable paths
the path surely doesn't exist yet, but that's OK,
because at that time Bazel didn't yet create the
sandbox root.
If the path is not under the sandbox root, then
Bazel needs to resolve all symlinks on this path,
which is only possible if the path exists,
therefore Bazel checks for the path's existence.
Change-Id: Ic7d99a81905e7401455286c0b375d69b85ece1d5
PiperOrigin-RevId: 181325749
|
|
|
|
|
|
|
|
| |
This simplifies some spawn runners, which no longer have to specially handle
null; unfortunately, the sandbox runners do not support VirtualActionInput,
so they still have to special-case it.
PiperOrigin-RevId: 181175408
|
|
|
|
|
|
|
|
|
|
|
| |
The sandboxed Spawn runners already used to call
LocalEnvProvider.rewriteEnvironment in order to
add TMPDIR for example, but didn't consistently
use this rewritten environment everywhere, and
used the Spawn's original environment instead.
Change-Id: Ifbede6ac0f14b83c617dcbcd85edfae88718b157
PiperOrigin-RevId: 181130506
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
As part of setting up a sandbox, Bazel creates the
list of writable paths. If the action's
environment defines $TEST_TMPDIR, then it's a test
action and the sandbox must allow writing to that
path, therefore Bazel must add $TEST_TMPDIR to the
writable paths.
Bazel must resolve symlinks in that path though,
at least on the last path segment, because in case
the path points to a symlink, the action would be
allowed to modify the symlink itself, and not
access what the link points to.
However the path must exist for Bazel to
successfully resolve symlinks, therefore this
commit adds a check for that.
Given that the code was there since at least July
2017, and I'm not aware of bugs caused by it, I
conclude that this code path either never runs or
nobody ever triggered it. Either way, adding the
check is the right thing to do.
Change-Id: I87a5d3fc3fe7878a918ed318c71e8d135f10f1b8
PiperOrigin-RevId: 180931382
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Create a PosixLocalEnvProvider and
WindowsLocalEnvProvider class, with singleton
instances for now.
This refactoring should not change functionality,
it's just a requirement for an upcoming change.
That upcoming change is for these classes to
respect the client environment's TMPDIR or
TMP/TEMP envvars.
See https://github.com/bazelbuild/bazel/issues/4376
Change-Id: I032bb6f18adf8af9e43e6bc543c09c58adae3863
PiperOrigin-RevId: 180799936
|
|
|
|
|
|
|
|
|
|
|
| |
that use the DarwinSandboxedSpawnRunner.
In particular, record metrics for user and system CPU execution time, block I/O and involuntary context switches.
This feature is guarded behind a new option, --experimental_collect_local_sandbox_action_metrics.
RELNOTES: None.
PiperOrigin-RevId: 179980734
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
that use either the LinuxSandboxedSpawnRunner or the ProcessWrapperSandboxedSpawnRunner.
In particular, record metrics for user and system CPU execution time, block I/O and involuntary context switches.
This feature is guarded behind a new option, --experimental_collect_local_sandbox_action_metrics.
Note: We still need to enable execution statistics for the DarwinSandboxedSpawnRunner in a later change.
RELNOTES: None.
PiperOrigin-RevId: 179976217
|
|
|
|
|
|
| |
Only the sandbox uses the cached version. We can let it do its own caching of the top-level directory. It won't help with higher-level directories, but if there are a lot of inputs in the same directory we'll avoid I/O there.
PiperOrigin-RevId: 179830651
|
|
|
|
|
|
|
| |
be consistent with LinuxSandboxUtil.
RELNOTES: None.
PiperOrigin-RevId: 179758847
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 179705357
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Breaks //src/test/shell/bazel:bazel_sandboxing_test
*** Original change description ***
Use linux-sandbox via the (new) LinuxSandboxUtil.
RELNOTES: None.
PiperOrigin-RevId: 179676894
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 179646155
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
To reproduce: run a failing test with --experimental_remote_spawn_cache or with --spawn_strategy=remote and no executor. Expected: test log is uploaded.
Desired behavior:
- regardless of whether a spawn is cacheable or not, its artifacts should be uploaded to the remote cache.
- the spawn result should only be set if the spawn is cacheable *and* the action succeeded.
- when executing remotely, the do_not_cache field should be set for non-cacheable spawns, and the remote execution engine should respect it.
This CL contains multiple fixes to ensure the above behaviors, and adds a few tests, both end to end and unit tests. Important behavior change: it is no longer assumed that non-cacheable spawns should use a NO_CACHE SpawnCache! The appropriate test case was removed. Instead, an assumption was added that all implementations of SpawnCache should respect the Spawns.mayBeCached(spawn) property. Currently, only NO_CACHE and RemoteSpawnCache exist, and they (now) support it.
TESTED=remote build execution backend.
WANT_LGTM: philwo,buchgr
RELNOTES: None
PiperOrigin-RevId: 178617937
|
|
|
|
|
|
|
|
|
| |
everywhere instead of duplicating process-wrapper --shell_arguments in Blaze.
To avoid a cyclic dependency, I broke up exec/local:local into exec/local:local and exec/local:options.
RELNOTES: None.
PiperOrigin-RevId: 177419268
|
|
|
|
|
| |
RELNOTES: None
PiperOrigin-RevId: 177290508
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- remove BaseSpawn.Local; instead, all callers pass in the full set of
execution requirements they want to set
- disable caching and sandboxing for the symlink tree action - it does not
declare outputs, so it can't be cached or sandboxed (fixes #4041)
- centralize the existing execution requirements in the ExecutionRequirements
class
- centralize checking for execution requirements in the Spawn class
(it's possible that we may need a more decentralized, extensible design in
the future, but for now having them in a single place is simple and
effective)
- update the documentation
- forward the relevant tags to execution requirements in TargetUtils (progress
on #3960)
- this also contributes to #4153
PiperOrigin-RevId: 177288598
|
|
|
|
|
|
|
| |
Also rename ProcessWrapperRunner to ProcessWrapperUtil, since it isn't really a Runner (it doesn't run anything).
RELNOTES: None.
PiperOrigin-RevId: 177260951
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It turns out that the SUCCESS status is often misunderstood to mean "zero exit",
even though this is clearly documented. I've decided to add another status for
non-zero exit, and use success only for zero exit to avoid this pitfall.
Also, many of the status codes are set, but never used. I decided to reduce the
number of status codes to only those that are actually relevant, which
simplifies further processing. Instead, we should add a string message for the
error case when we need one - we're not using it right now, so I decided not to
add that yet.
PiperOrigin-RevId: 177129441
|
|
|
|
|
|
|
|
| |
Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own.
This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually.
PiperOrigin-RevId: 175033526
|
|
|
|
|
|
|
| |
and make cumulative execution times available in ActionResults.
RELNOTES: None
PiperOrigin-RevId: 174553272
|
|
|
|
|
|
| |
This requires a fairly large amount of changes to fundamental objects like BlazeRuntime, Executor, and so on, as well as changing a lot of test code to thread the file system through. I expect future CLs to be much smaller.
PiperOrigin-RevId: 173678144
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Every build and test action that creates a Spawn
will now have platform-specific environment
variables for temp directories:
- on Windows: TMP and TEMP
- on Linux/Darwin: TMPDIR
This is particularly important on Windows where
e.g. Java programs cannot create temp directories
unless there's a valid TMP or TEMP environment
variable set.
Fixes:
- https://github.com/bazelbuild/bazel/issues/1590
- https://github.com/bazelbuild/bazel/issues/2349
- https://github.com/bazelbuild/bazel/issues/2870
Change-Id: Ib758307daf6b3a51b0f71ae5e65e5bb564dad643
PiperOrigin-RevId: 172326371
|
|
|
|
|
|
|
| |
Whether or not there was a catastrophic error is stored in the SpawnResult, so
we can just use that instead of passing in an additional boolean.
PiperOrigin-RevId: 172083752
|
|
|
|
|
|
|
|
|
| |
ActionContexts, etc., so that SpawnResult metadata is returned upwards.
Note that the TODOs mostly refer to changes that will appear in a subsequent CL (a CL to return SpawnResults, contained in ActionResults, from Actions/AbstractActions). I split off the remaining SpawnResult-related changes into this CL and kept the ActionResult-related changes separate.
RELNOTES: None.
PiperOrigin-RevId: 171355611
|
|
|
|
|
|
|
| |
build.lib.actions.SpawnActionContext can import SpawnResult without creating a cyclic dependency.
RELNOTES: None.
PiperOrigin-RevId: 169642267
|
|
|
|
|
|
|
|
|
| |
Split collect, concurrent, vfs, windows into package-level BUILD files.
Move clock classes out of "util", into their own Java package.
Move CompactHashSet into its own Java package to break a dependency cycle.
Give nestedset and inmemoryfs their own package-level BUILD files.
PiperOrigin-RevId: 167702127
|
|
|
|
| |
PiperOrigin-RevId: 166849610
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
linux-sandbox has a useful option -R, that runs the spawn as fake
root. However, it's not exposed to Bazel rules. Here, we do that via
the "requires-fakeroot" tag.
One possible usecase: In combination with "block-network",
"requires-fakeroot" makes it possible to integration test services
that insist on listening on privileged ports.
Unsurprisingly, this is incompatible with --sandbox_fake_username.
Change-Id: I9e8ab4d4abf0e45626e005ff21f73e6c17de0788
PiperOrigin-RevId: 164961019
|
|
|
|
|
|
|
| |
...instead of injecting it through the constructor. Simplify all the callers
accordingly.
PiperOrigin-RevId: 164955391
|
|
|
|
|
|
|
|
| |
These are depended upon by analysis code, so need to live in the same library
as lib.analysis. Moving them here makes it possible to split the build-base
library into separate libraries for analysis, execution, and rules.
PiperOrigin-RevId: 164847161
|
|
|
|
|
|
|
| |
All prefetching now goes through AbstractSpawnStrategy's implementation of
SpawnExecutionPolicy. Make sure the sandbox runners also do this consistently.
PiperOrigin-RevId: 164836877
|
|
|
|
|
|
| |
Fixes #3521.
PiperOrigin-RevId: 164728178
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Important: the simplified API now defaults to forwarding interrupts to
subprocesses. I did audit all the call sites, and I think this is a safe change
to make.
- Properly support timeouts with all implementations
- Simplify the API
- only provide two flavours of blocking calls, which require no input and
forward interrupts; this is the most common usage
- provide a number of async calls, which optionally takes input, and a flag
whether to forward interrupts
- only support input streams, no byte arrays or other 'convenience features'
that are rarely needed and unnecessarily increase the surface area
- use java.time.Duration to specify timeout; for consistency, interpret a
timeout of <= 0 as no timeout (i.e., including rather than excluding 0)
- KillableObserver and subclasses are no longer part of the public API, but
still used to implement timeouts if the Subprocess.Factory does not support
them
- Update the documentation for Command
- Update all callers; most callers now use the simplified API
PiperOrigin-RevId: 164716782
|
|
|
|
| |
PiperOrigin-RevId: 164577062
|
|
|
|
|
|
| |
--test_env isn't moved in this CL since it's exposed to Skylark via BuildConfiguration, making it a somewhat riskier refactor.
PiperOrigin-RevId: 164266168
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Remove BuildRequest as parameter to sandboxed spawn runner
constructors. Previously, the build request was used to obtain some
options, but those can be extricated from a CommandEnvironment, which
is passed in, too.
Also, remove LinuxSandboxedSpawnRunner's aliased sandboxOptions member
variable. It can just use the superclass's.
Change-Id: I1ef1a45cbf7e800d0809f05673f097a148289740
PiperOrigin-RevId: 164257471
|
|
|
|
|
|
| |
Closes #3444.
PiperOrigin-RevId: 163047183
|
|
|
|
|
|
| |
This removes a bunch of code duplication that I previously introduced.
PiperOrigin-RevId: 162909430
|
|
|
|
|
|
| |
The plan is to add it to ActionExecutionContext, which is also there.
PiperOrigin-RevId: 162656835
|
|
|
|
|
|
|
|
|
|
|
|
| |
TestRunner actions are special because their action outputs are
different from their spawn outputs. If there's a spawn output that's
not an action output, SymlinkExecroot can't rely on the parent
directories for that output existing in the real execroot. Thus,
copyOutputs() must ensure the real execroot has the relevant ancestral
directories before copying the output over.
Change-Id: I84fd69cd51628c51de9c8993b6a4407bbff038a0
PiperOrigin-RevId: 162470058
|
|
|
|
|
|
| |
The sandbox execution strategies were not passing the timeout grace period specified via --local_termination_grace_seconds, resulting in process-wrapper defaulting to just 5 seconds and linux-sandbox not using any grace period at all, instead directly SIGKILLing the child on timeout.
PiperOrigin-RevId: 162349919
|