| Commit message (Collapse) | Author | Age |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
https://github.com/bazelbuild/bazel/commit/656a0bab1e025ff3c27d595284a4bf1c5a8d8028 with test (unknown commit) and fix.
Big round of sandbox fixes / performance improvements.
- The number of stat() syscalls in the SymlinkedSandboxedSpawn was way too high. Do less, feel better.
- When using --experimental_sandbox_base, ensure that symlinks in the path are resolved. Before this, you had to check whether on your system /dev/shm is a symlink to /run/shm and then use that instead. Now it no longer matters, as symlinks are resolved.
- Remove an unnecessary directory creation from each sandboxed invocation. Turns out that the "tmpdir" that we created was no longer used after some changes to Bazel's TMPDIR handling.
- Use simpler sandbox paths, by using the unique ID for each Spawn provided by SpawnExecutionPolicy instead of a randomly generated temp folder name. This also saves a round-trip from our VFS to NIO and back. Clean up the sandbox base before each build to ensure that the unique IDs are actually unique. ;)
- Use Java 8's Process#isAlive to check whether a process is alive instead of trying to get the exitcode and catching an exception.
Closes #4913.
PiperOrigin-RevId: 193031017
|
| |
|
|
| |
PiperOrigin-RevId: 191642942
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- The number of stat() syscalls in the SymlinkedSandboxedSpawn was way too high. Do less, feel better.
- When using --experimental_sandbox_base, ensure that symlinks in the path are resolved. Before this, you had to check whether on your system /dev/shm is a symlink to /run/shm and then use that instead. Now it no longer matters, as symlinks are resolved.
- Remove an unnecessary directory creation from each sandboxed invocation. Turns out that the "tmpdir" that we created was no longer used after some changes to Bazel's TMPDIR handling.
- Use simpler sandbox paths, by using the unique ID for each Spawn provided by SpawnExecutionPolicy instead of a randomly generated temp folder name. This also saves a round-trip from our VFS to NIO and back. Clean up the sandbox base before each build to ensure that the unique IDs are actually unique. ;)
- Use Java 8's Process#isAlive to check whether a process is alive instead of trying to get the exitcode and catching an exception.
Closes #4913.
PiperOrigin-RevId: 190472170
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
- Remove Optional<> where it's not needed. It's nice for return values, but IMHO it was overused in this code (e.g. Optional<List<X>> is an anti-pattern, as the list itself can already signal that it is empty).
- Use Bazel's own Path class when dealing with paths, not String or java.io.File.
- Move LinuxSandboxUtil into the "sandbox" package.
- Remove dead code and unused fields.
- Migrate deprecated VFS method calls to their replacements.
- Fix a bug in ExecutionStatistics where a FileInputStream was not closed.
Closes #4868.
PiperOrigin-RevId: 190217476
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
This introduces user-facing options to enable the experimental sandboxfs
support and, when enabled, mounts a sandboxfs instance throughout the
build. The sandboxfs' process handle is passed to the
SandboxActionContextProvider so that the SpawnRunners can later consume it.
Note that this does NOT yet provide sandboxfs support for the builds as the
SpawnRunners are untouched.
RELNOTES: None.
PiperOrigin-RevId: 189678732
|
| |
|
|
|
|
| |
This requires a fairly large amount of changes to fundamental objects like BlazeRuntime, Executor, and so on, as well as changing a lot of test code to thread the file system through. I expect future CLs to be much smaller.
PiperOrigin-RevId: 173678144
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Remove BuildRequest as parameter to sandboxed spawn runner
constructors. Previously, the build request was used to obtain some
options, but those can be extricated from a CommandEnvironment, which
is passed in, too.
Also, remove LinuxSandboxedSpawnRunner's aliased sandboxOptions member
variable. It can just use the superclass's.
Change-Id: I1ef1a45cbf7e800d0809f05673f097a148289740
PiperOrigin-RevId: 164257471
|
| |
|
|
|
| |
Change-Id: I1bc1901ea7cd9a5b93c280ec0ff8ac0d10959a09
PiperOrigin-RevId: 155381163
|
| |
|
|
|
|
|
|
| |
Usually, Bazel creates the sandbox directories underneath its
output_base. With --experimental_sandbox_base you can specify a
different parent directory for this, e.g. /dev/shm to run all sandboxed
actions on a memory-backed filesystem.
PiperOrigin-RevId: 152490815
|
| |
|
|
|
|
|
|
|
|
|
|
| |
This change is part of the mu-bazel effort, which aims to build a minimally
useful Bazel binary with most extraneous functionality removed. As part of
that, we want to enforce layering of packages. In particular, lib.actions must
not depend on lib.rules or lib.exec. lib.rules must not depend on lib.exec.
Moving these classes is a necessary step to enforce that layering.
--
PiperOrigin-RevId: 142668172
MOS_MIGRATED_REVID=142668172
|
| |
|
|
|
|
|
| |
This significantly simplifies several of our modules.
--
MOS_MIGRATED_REVID=137713119
|
| |
|
|
|
|
|
|
|
| |
simple synchronous cleanup.
Tested with bazel building itself that this does not result in a performance degradation.
--
MOS_MIGRATED_REVID=134766597
|
| |
|
|
|
|
|
| |
sandbox directories could not be cleaned up.
--
MOS_MIGRATED_REVID=134286101
|
| |
|
|
|
| |
--
MOS_MIGRATED_REVID=131817068
|
| |
|
|
|
|
|
| |
than "pool-X-thread-Y".
--
MOS_MIGRATED_REVID=131150010
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This has the following improvements upon the older one:
- Uses PID namespaces, PR_SET_PDEATHSIG and a number of other tricks for
further process isolation and 100% reliable killing of child processes.
- Uses clone() instead of unshare() to work around a Linux kernel bug that
made creating a sandbox unreliable.
- Instead of mounting a hardcoded list of paths + whatever you add with
--sandbox_add_path, this sandbox instead mounts all of /, except for what
you make inaccessible via --sandbox_block_path. This should solve the
majority of "Sandboxing breaks my build, because my compiler is installed
in /opt or /usr/local" issues that users have seen.
- Instead of doing magic with bind mounts, we create a separate execroot for
each process containing symlinks to the input files. This is simpler and
gives more predictable performance.
- Actually makes everything except the working directory read-only
(fixes #1364). This means that a running process can no longer accidentally
modify your source code (yay!).
- Prevents a number of additional "attacks" or leaks, like accidentally
inheriting file handles from the parent.
- Simpler command-line interface.
- We can provide the same semantics in a Mac OS X sandbox, which will come in
a separate code review from yueg@.
It has the following caveats / known issues:
- The "fallback to /bin/bash on error" feature is gone, but now that the
sandbox mounts everything by default, the main use-case for this is no
longer needed.
The following improvements are planned:
- Use a FUSE filesystem if possible for the new execroot, instead of creating
symlinks.
- Mount a base image instead of "/".
FAQ:
Q: Why is mounting all of "/" okay, doesn't this make the whole sandbox
useless?
A: This is still a reasonable behavior, because the sandbox never tried to
isolate your build from the operating system it runs in. Instead it is
supposed to protect your data from a test running "rm -rf $HOME" and to
make it difficult / impossible for actions to use input files that are not
declared dependencies. For even more isolation the sandbox will support
mounting a base image as its root in a future version (similar to Docker
images).
Q: Let's say my process-specific execroot contains a symlink to an input file
"good.h", can't the process just resolve the symlink, strip off the file
name and then look around in the workspace?
A: Yes. Unfortunately we could not find any way on Linux to make a file appear
in a different directory with *all* of the semantics we would like. The
options investigated were:
1) Copying input files, which is much too slow.
2) Hard linking input files, which is fast, but doesn't work cross-
filesystems and it's also not possible to make them read-only.
3) Bind mounts, which don't scale once you're up in the thousands of input
files (across all actions) - it seems like the kernel has some
non-linear performance behavior when the mount table grows too much,
resulting in the mount syscall taking more time the more mounts you
have.
4) FUSE filesystem, good in theory, but wasn't ready for the first
iteration.
RELNOTES: New sandboxing implementation for Linux in which all actions run in a separate execroot that contains input files as symlinks back to the originals in the workspace. The running action now has read-write access to its execroot and /tmp only and can no longer write in arbitrary other places in the file system.
--
Change-Id: Ic91386fc92f8eef727ed6d22e6bd0f357d145063
Reviewed-on: https://bazel-review.googlesource.com/#/c/4053
MOS_MIGRATED_REVID=130638204
|
| |
|
|
|
|
|
| |
--
Change-Id: Idf232f3dce3a3221d9a35c89dcef13437b0c25ba
Reviewed-on: https://bazel-review.googlesource.com/#/c/3905/
MOS_MIGRATED_REVID=129620348
|
| |
|
|
|
| |
--
MOS_MIGRATED_REVID=127538990
|
| |
|
|
|
|
|
|
|
|
|
| |
Several modules now explicitly add common command options. Of the remaining
ones, most add options to the build command, except one, which adds options
to query. They now all use the canonical implementation.
Also updated the documentation to clarify what this method actually does.
--
MOS_MIGRATED_REVID=125560058
|
| |
|
|
|
| |
--
MOS_MIGRATED_REVID=118561661
|
| |
|
|
|
|
|
| |
Reduces garbage.
--
MOS_MIGRATED_REVID=109914243
|
| |
|
|
|
|
|
| |
The "clean" command that is run after each shell test finishes (in the tear_down method of test-setup.sh) tries to remove all subdirectories of the output base. This includes the bazel-sandboxes directory. This, however is already being cleaned up continuously in the background by a thread pool setup in SandboxModule. By waiting for the pool to finish before the clean command runs, the race condition can be solved.
--
MOS_MIGRATED_REVID=107578302
|
| |
|
|
|
|
|
|
|
|
|
| |
The headers were modified with
`find . -type f -exec 'sed' '-Ei' 's|Copyright 201([45]) Google|Copyright 201\1 The Bazel Authors|' '{}' ';'`
And manual edit for not Google owned copyright. Because of the nature of ijar, I did not modified the header of file owned by Alan Donovan.
The list of authors were extracted from the git log. It is missing older Google contributors that can be added on-demand.
--
MOS_MIGRATED_REVID=103938715
|
| |
|
|
|
| |
--
MOS_MIGRATED_REVID=103269584
|
| |
|
|
|
|
|
|
| |
The very long term plan is to make BlazeRuntime immutable, so that we can
run multiple commands in the same server.
--
MOS_MIGRATED_REVID=103080946
|
| |
|
|
|
|
|
| |
unavailable sandboxing.
--
MOS_MIGRATED_REVID=102042098
|
| |
|
|
|
|
|
|
|
| |
is Linux, do real auto-detection whether it is supported on the host or not and enable / disable it based on the result.
The warning that is printed when the Linux kernel is too old to support sandboxing can be disabled via a flag.
--
MOS_MIGRATED_REVID=101461120
|
| |
|
|
|
|
|
|
|
|
|
|
| |
use non-declared inputs) and safety (spawns can no longer affect the host system, e.g. accidentally wipe your home directory). This implementation works on Linux only and uses Linux containers ("namespaces").
The strategy works with all actions that Bazel supports (C++ / Java compilation, genrules, test execution, Skylark-based rules, ...) and in tests, Bazel could successfully bootstrap itself and pass the whole test suite using sandboxed execution.
This is not the default behavior yet, but can be activated explicitly by using:
bazel build --genrule_strategy=sandboxed --spawn_strategy=sandboxed //my:stuff
--
MOS_MIGRATED_REVID=101457297
|
|
|
--
MOS_MIGRATED_REVID=97126283
|
philwo
dannark
jmmv
tomlu
Benjamin Peterson
Ulf Adams
Lukacs Berki
Yue Gan
Mark Schaller
Damien Martin-Guillerez