| Commit message (Collapse) | Author | Age |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The main change here is to only catch SpawnExecException in
StandaloneTestStrategy, so all other exceptions simplify propagate up. As a
result, Bazel no longer retries tests that fail with an exception, we only
retry tests that actually ran, had a spawn result, and resulted in a
UserExecException. That is probably what we want.
Also do some cleanup:
- Remove ExecException.timedOut; nobody was calling it (but there's still
SpawnExecException.timedOut)
- Remove SpawnActionContext.shouldPropagateExecException; all exceptions
(except SpawnExecException) are now propagated by default
- Remote the SandboxOptions from the SandboxStrategies; all sandboxing options
are now handled by the underlying SpawnRunner implementations
I'll send a followup CL to remove the UserExecException and
EnvironmentalExecException types; the types don't do anything special, and
there are no catch blocks in production code that catch one of these more
specific types.
This should fix #3322 by removing a bunch of special handling.
PiperOrigin-RevId: 161960919
|
|
|
|
|
|
|
|
| |
This adds a bunch of classes that only implement the SpawnRunner interface, and
will allow us to support remote caching in combination with local sandboxed
execution in a subsequent change.
PiperOrigin-RevId: 161664556
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Make use of existing abstractions like SpawnRunner and SpawnExecutionPolicy.
- Instead of having the *Strategy create a *Runner, and then call back into
SandboxStrategy, create a single SandboxContainer which contains the full
command line, environment, and everything needed to create and delete the
sandbox directory.
- Do all the work in SandboxStrategy, including creation and deletion of the
sandbox directory.
- Use SpawnResult instead of throwing, catching, and rethrowing.
- Simplify the control flow a bit.
PiperOrigin-RevId: 161644979
|
|
|
|
|
|
|
|
| |
Move the default from the annotation to every mention. This makes the incompleteness explicit. Will add the defaults to test targets in a separate change.
Once all dependencies are cleaned up, the Option annotation will no longer allow options without the documentationCategory or effectTag, to prevent new options being added without categories while we migrate to the new option categorization.
PiperOrigin-RevId: 160281252
|
|
|
|
|
|
|
|
| |
This will allow us to add new and optional flags like selecting a strategy used to spawn / wait for the child process.
No one except Bazel should be calling "process-wrapper" and I couldn't find any references, so this breaking change should be fine.
PiperOrigin-RevId: 159685867
|
|
|
|
| |
PiperOrigin-RevId: 159423459
|
|
|
|
|
|
|
| |
Move everything to ActionExecutionContext, and drop Executor whereever possible.
This clarifies the API, makes it simpler to test, and simplifies the code.
PiperOrigin-RevId: 159414816
|
|
|
|
| |
PiperOrigin-RevId: 159221067
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Also add an interface to allow injecting that logic into LocalSpawnRunner; this
is in preparation for rewriting StandaloneSpawnStrategy to use
LocalSpawnRunner.
At the same time, this reduces the dependencies from exec / standalone to
rules.apple, which is a prerequisite for micro-Bazel.
There's a small semantic change hidden here - we now only set the new
XCodeLocalEnvProvider if we're actually running on Darwin, so we no longer
fail execution on non-Darwin platforms if XCODE_VERSION_OVERRIDE or
APPLE_SDK_VERSION_OVERRIDE is set. As a result, I moved the corresponding test
from StandaloneSpawnStrategyTest to the new XCodeLocalEnvProviderTest.
While I'm at it, also open source DottedVersionTest and CacheManagerTest.
PiperOrigin-RevId: 158829077
|
|
|
|
|
|
|
|
| |
existing directory writable when running actions.
RELNOTES: Added a new flag --sandbox_writable_path, which asks the sandbox to
make an existing directory writable when running actions.
PiperOrigin-RevId: 157971858
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Roll forward of directory name change
*** Original change description ***
Automated g4 rollback of commit 1d9e1ac90197b1d3d7b137ba3c1ada67bb9ba31b.
*** Reason for rollback ***
Breaks //src/test/shell/integration:force_delete_output_test
*** Original change description ***
Symlink output directories to the correct directory name
If the workspace directory is /path/to/my/proj and the name in the WORKSPACE
file is "floop", this will symlink the output directories to
output_base/execroot/floop instead of output_base/execroot/proj.
More prep for #1262, fixes #1681.
PiperOrigin-RevId: 156892980
|
|
|
|
|
|
|
|
| |
By removing the now unnecessary call to Path#resolveSymbolicLinks we can save a
few stat's per action execution.
Change-Id: Iee157e941c1cd3515ff5ea3b7f410824c24cf44d
PiperOrigin-RevId: 155946544
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is basically a rollback of https://github.com/bazelbuild/bazel/commit/3e2329a73ffd5d60e5e2babe60ebe5bf322c07da, except this solves the
reason why the feature was removed in the first place. We now create
the helper files necessary to make files unreadable in Linux in Bazel's
Java code and manage their lifetime there.
Request was filed by a user here:
http://stackoverflow.com/questions/43849651/how-to-lock-down-the-bazel-filesystem-sandbox
PiperOrigin-RevId: 155913246
|
|
|
|
|
|
|
|
|
| |
This unifies our code to use just one standard implementation to get the
entire expanded input files for a Spawn, including from Filesets and
Runfiles.
Change-Id: I1e286508adf0a9aeddf70934b010e6fcc144c4a7
PiperOrigin-RevId: 155497273
|
|
|
|
|
|
|
|
|
|
|
| |
Hardlinks are problematic due to not working across filesystem
boundaries and causing Bazel to do lots of I/O because it has to create
a hardlink and a symlink for each input file.
This improves performance of Bazel building itself by 10% on my system.
Change-Id: I8acb77053de875160a046e38624735ed18375bed
PiperOrigin-RevId: 155493583
|
|
|
|
|
|
|
|
|
| |
This gives us much improved process management, because Bazel can now
reliably kill child processes of spawns via their process group and wait
for them to exit.
Change-Id: Ib3cb20725b3c569aa5b317a69d7682f5774707b0
PiperOrigin-RevId: 155493511
|
|
|
|
|
| |
Change-Id: Idc023f3a8c1c3b60d3f3f23a579a5eccb92d074d
PiperOrigin-RevId: 155487527
|
|
|
|
|
| |
Change-Id: I1522c364a157ee0a144ab83eca54e419142c03b1
PiperOrigin-RevId: 155484109
|
|
|
|
|
|
|
|
| |
There's no need to make it explicitly readable, because the entire host
filesystem is readable anyway.
Change-Id: I6a63cc93b600250c1c8828ef8d1c9d6133b671d7
PiperOrigin-RevId: 155477093
|
|
|
|
|
| |
Change-Id: I1bc1901ea7cd9a5b93c280ec0ff8ac0d10959a09
PiperOrigin-RevId: 155381163
|
|
|
|
|
|
|
|
| |
There's no need to check for the OS version, as we can just try to use
sandbox-exec and if it works, we're good.
Change-Id: I7fe9a0b55856c646da915a2872531f050a25b110
PiperOrigin-RevId: 155368707
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(fallback only).
On macOS the processwrapper-sandbox will be used when the darwin-sandbox doesn't
work. Most notably this is the case for nested sandboxing, e.g. Bazel running
Bazel inside an integration test.
Also includes a fix to pull in some extra environment vars on macOS, similar to
what DarwinSandboxedStrategy and StandaloneSpawnStrategy already do. Without
this the processwrapper-sandbox seems to occasionally cause ObjC builds (and two
of our tests) to fail.
Change-Id: Ic7462080caf56d9bb98e2f3765bd37853b01632b
RELNOTES: Sandboxing is now enabled by default on FreeBSD (via processwrapper-sandbox).
PiperOrigin-RevId: 155366728
|
|
|
|
|
| |
Change-Id: I1355c2448cb6cbbcdbace81051a7beb8659f1f00
PiperOrigin-RevId: 155366727
|
|
|
|
|
| |
Change-Id: I43dfd979aee0c510ec18b479f2a6bd55562b3fc0
PiperOrigin-RevId: 155361450
|
|
|
|
|
|
|
|
|
|
|
| |
selected even if they're not the preferred one on a platform.
Simplify the SandboxActionContextProvider and remove the warning about
sandboxing being unsupported. With the ProcessWrapperSandboxedStrategy
now being reliable enough and the strategies printing their real name in
the UI, this is overall a better UX.
PiperOrigin-RevId: 153825986
|
|
|
|
|
|
| |
With the process-wrapper improvements and the additional deletion of the sandbox base in the SandboxModule in, this should be reliable enough. The warning was also not actionable for users and annoyed them, so let's get rid of it.
PiperOrigin-RevId: 153823045
|
|
|
|
|
|
|
|
|
| |
macOS version strings that have only two components, e.g. "10.12", were
always failing the check that the macOS version is >= 10.11, causing
Bazel to erroneously think sandboxing isn't supported
Change-Id: Ifa4a01fc304e7620502d3f0f9f70c3b500d23864
PiperOrigin-RevId: 152493682
|
|
|
|
|
|
|
|
| |
Usually, Bazel creates the sandbox directories underneath its
output_base. With --experimental_sandbox_base you can specify a
different parent directory for this, e.g. /dev/shm to run all sandboxed
actions on a memory-backed filesystem.
PiperOrigin-RevId: 152490815
|
|
|
|
|
|
|
|
|
|
|
|
| |
'create' method.
This paves the way for changing PathFragment to e.g. an abstract class with multiple subclasses. This way we can split out the windows-specific stuff into one of these concrete classes, making the code more readable and also saving memory (since the shallow heap size of the NonWindowsPathFragment subclass will hopefully be smaller than that of the current PathFragment).
This also lets us pursue gc churn optimizations. We can now do interning in PathFragment#create and can also get rid of unnecessary intermediate PathFragment allocations.
RELNOTES: None
PiperOrigin-RevId: 152145768
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Breaks //src/test/shell/integration:force_delete_output_test
*** Original change description ***
Symlink output directories to the correct directory name
If the workspace directory is /path/to/my/proj and the name in the WORKSPACE
file is "floop", this will symlink the output directories to
output_base/execroot/floop instead of output_base/execroot/proj.
More prep for #1262, fixes #1681.
PiperOrigin-RevId: 152126545
|
|
|
|
|
|
|
|
|
|
| |
If the workspace directory is /path/to/my/proj and the name in the WORKSPACE
file is "floop", this will symlink the output directories to
output_base/execroot/floop instead of output_base/execroot/proj.
More prep for #1262, fixes #1681.
PiperOrigin-RevId: 151712384
|
|
|
|
|
|
|
|
|
|
|
| |
The linux-sandbox no longer requires extensive configuration via flags,
so the command-line arguments will easily fit into the allowed length.
This simplifies the code and gets rid of some I/O.
--
PiperOrigin-RevId: 151176551
MOS_MIGRATED_REVID=151176551
|
|
|
|
|
|
|
|
|
|
|
| |
Extract the process-wrapper + symlink tree sandbox strategy into its own
class and allow its use in FreeBSD.
RELNOTES: Bazel can now use the process-wrapper + symlink tree based sandbox implementation in FreeBSD.
--
PiperOrigin-RevId: 151171652
MOS_MIGRATED_REVID=151171652
|
|
|
|
|
|
|
|
|
|
| |
This allows us to see for example whether an action ran using the
"process wrapper + symlink tree" sandbox or the real "PID and mount
namespaces" Linux sandbox.
--
PiperOrigin-RevId: 151165170
MOS_MIGRATED_REVID=151165170
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 151160662
MOS_MIGRATED_REVID=151160662
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 151130566
MOS_MIGRATED_REVID=151130566
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Also refactor the way we compute writable dirs so that they're computed
only once per running action, not twice.
Fixes #2056, fixes #1973, fixes #1460.
RELNOTES: /tmp and /dev/shm are now writable by default inside the
Linux sandbox.
--
PiperOrigin-RevId: 151123543
MOS_MIGRATED_REVID=151123543
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Try to run /bin/true as a test of whether the Linux sandbox works,
instead of just trying to create a bunch of namespaces as a proxy.
This helps resolve issues on Linux distros where the earlier check
worked, but then the sandbox ultimately failed due to other operations
being unsupported.
As an example, Debian Jessie and certain Docker versions seem to allow
the creation of PID namespaces, but forbid mounting a new proc on top of
/proc (see #1972). This resulted in Bazel thinking that sandboxing works
fine, when it actually didn't. The improved check correctly catches this
situation and disabled sandboxing.
--
PiperOrigin-RevId: 151116894
MOS_MIGRATED_REVID=151116894
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This can be reactivated by passing the --sandbox_fake_username flag
to Bazel.
Reasoning: 'nobody' has a non-existent home directory on many Linux
distros, leading to issues when tools try to stat / read / write to the
home directory.
Related to #2688.
RELNOTES: The Linux sandbox no longer changes the user to 'nobody' by
default, instead the current user is used as is. The old behavior can be
restored via the --sandbox_fake_username flag.
--
PiperOrigin-RevId: 151115218
MOS_MIGRATED_REVID=151115218
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
By removing the --sandbox_block_path feature in an earlier change and
taking advantage of the fact that in a mount namespace we can actually
"remount" mount points to be read-only without bind mounting them to
some other place beforehand, this is no longer necessary. The code
becomes much simpler due to this, for example we no longer need to
chroot.
--
PiperOrigin-RevId: 151111360
MOS_MIGRATED_REVID=151111360
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It is in the way of optimizing the performance of the sandbox, because
it requires us to create two helper files (an unreadable file and an
unreadable directory) which are bind-mounted on top of paths specified
via this flag. These two helper files were created on a tmpfs mounted by
the sandbox until now, which ensured that they were automatically
deleted on exit. However, mounting tmpfs on /dev/shm or /tmp causes
issues like #2686 or #1882.
By removing this flag, we can get rid of the two helper files, which
means we can also remove the reliance on a "sandbox temp directory"
completely in the next change.
--
PiperOrigin-RevId: 151107496
MOS_MIGRATED_REVID=151107496
|
|
|
|
|
|
|
|
|
|
|
| |
Using /dev/null was causing it to create symlinks to /dev/null, which
breaks Python programs. I didn't catch this earlier, because my machine had
an old linux kernel installed that didn't support sandboxing. There's an
existing integration test, which just broke in our CI.
--
PiperOrigin-RevId: 150750032
MOS_MIGRATED_REVID=150750032
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This new class is a combination of SpawnHelper and our internal code; the
plan is to migrate all spawn strategies to the new class. The strict flag
should be enabled by default, but that's a breaking change, so we need to do
it later.
- Use it in SandboxStrategy.
- Add ActionInput.getExecPath to return a PathFragment; this avoids lots of
back and forth between path fragments and strings.
This is a step towards #1593.
The previous attempt was missing a one-line patch in StandaloneTestStrategy,
which broke all tests with sandboxing. StandaloneTestStrategy was fixed in a
separate change, so this should be safe now.
--
PiperOrigin-RevId: 150733457
MOS_MIGRATED_REVID=150733457
|
|
|
|
|
|
|
|
|
|
| |
Add preconditions to enforce this and remove some now unnecessary code.
A small step towards #1593.
--
PiperOrigin-RevId: 150625693
MOS_MIGRATED_REVID=150625693
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Break bazel-tests and many other jobs on CI.
http://ci.bazel.io/job/bazel-tests/BAZEL_VERSION=HEAD,PLATFORM_NAME=linux-x86_64/651/console
*** Original change description ***
Add SpawnInputExpander helper class to arrange runfiles for spawn strategies
This new class is a combination of SpawnHelper and our internal code; the
plan is to migrate all spawn strategies to the new class. The strict flag
should be enabled by default, but that's a breaking change, so we need to do
it later.
- Use it in SandboxStrategy.
- Add ActionInput.getExecPath to return a PathFragment; this avoids lots of
back and forth between path fragments and strings.
This is a step towards #159...
***
--
PiperOrigin-RevId: 150610616
MOS_MIGRATED_REVID=150610616
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This new class is a combination of SpawnHelper and our internal code; the
plan is to migrate all spawn strategies to the new class. The strict flag
should be enabled by default, but that's a breaking change, so we need to do
it later.
- Use it in SandboxStrategy.
- Add ActionInput.getExecPath to return a PathFragment; this avoids lots of
back and forth between path fragments and strings.
This is a step towards #1593.
--
PiperOrigin-RevId: 150427021
MOS_MIGRATED_REVID=150427021
|
|
|
|
|
|
|
|
|
| |
- use SimpleSpawn in SpawnGccStrategy
- set PWD in CppCompileAction for consistency
--
PiperOrigin-RevId: 149745059
MOS_MIGRATED_REVID=149745059
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
All spawn strategies already treat all normal outputs as optional. Bazel checks
at the action level whether all action outputs are created, but does not check
at the spawn level. Spawn.getOptionalOutputs is therefore unnecessary, and
removed in this change.
The only place where this was set was in StandaloneTestStrategy, which now
specifies the full set of outputs, which is now computed by TestRunnerAction.
The internal test strategy implementations are also updated in this change.
While I'm at it, also remove the use of BaseSpawn and use SimpleSpawn instead.
This may go some way towards fixing #1413 and #942.
--
PiperOrigin-RevId: 149397100
MOS_MIGRATED_REVID=149397100
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 149110466
MOS_MIGRATED_REVID=149110466
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Move all local resource acquisition to where local execution actually happens.
Don't attempt to acquire resources per action, but only for individual spawns.
This significantly simplifies the code.
The downside is that we don't account for action-level work anymore. In
general, actions should not perform any process execution themselves, but
always delegate such work to a SpawnStrategy implementation.
This change makes sure that every Spawn has local resources set in a way that
is consistent with the previous state.
However, there are two actions - Fileset and FileWrite -, which are not spawns,
and so we now don't limit their concurrent execution anymore. For Fileset, all
work is done in a custom Fileset-specific thread pool, so this shouldn't be a
problem. I'm not sure about FileWriteAction.
--
PiperOrigin-RevId: 149012600
MOS_MIGRATED_REVID=149012600
|