| Commit message (Collapse) | Author | Age |
|
|
|
|
| |
Change-Id: Idbbf4662b96508c40c6f530ce5609faa079f0976
PiperOrigin-RevId: 162218592
|
|
|
|
|
|
|
|
| |
This will allow us to add new and optional flags like selecting a strategy used to spawn / wait for the child process.
No one except Bazel should be calling "process-wrapper" and I couldn't find any references, so this breaking change should be fine.
PiperOrigin-RevId: 159685867
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Rollforward with fix for FreeBSD. I added the missing headers and verified that it now builds fine on FreeBSD 11, too.
*** Original change description ***
Automated g4 rollback of commit 7f520a8286c39c5145b6d816cd0be5a6b7b18250.
*** Reason for rollback ***
This broke Bazel CI on freebsd:
http://ci.bazel.io/view/Dashboard/job/Bazel/JAVA_VERSION=1.8,PLATFORM_NAME=freebsd-11/1516/console#
*** Original change description ***
Refactor process-wrapper code so the spawn/wait code is pluggable.
In an upcoming change I'll reintroduce the new platform-specific implementations that can kill and wait for all descendant processes spawned by the wrapped process.
This has no functional changes.
PiperOrigin-RevId: 158133159
|
|
|
|
|
|
|
|
|
|
|
| |
If a directory that is being bind mounted has a subdirectory that is a mount
point (e.g.: tmpfs on /tmp/tmp), then the MS_BIND mount() call will fail with
invalid argument.
Fixes #3064.
RELNOTES: None.
PiperOrigin-RevId: 157973469
|
|
|
|
| |
PiperOrigin-RevId: 157479936
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
After this change, with a Bazel having MSVC as default toolchain,
the command to build a MSVC Bazel on Windows is simply:
bazel build //src:bazel
although bazel build --cpu=x64_windows_msvc //src:bazel is still supported.
And the command to build a MSYS Bazel on Windows is:
bazel build --cpu=x64_windows_msys //src:bazel
Also made //src/test/cpp:blaze_util_test pass without
--cpu=x64_windows_msvc
Change-Id: Iaf37513c778768d06fb5700442d5229a5f348964
PiperOrigin-RevId: 157446905
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
linux-sandbox expects InstallSignalHandler to fail silently when called
for a signal that doesn't allow one to install a signal handler. This
behavior was accidentally lost and changed into a DIE() on sigaction
failure when unifying the signal handling code with process-wrapper in
https://github.com/bazelbuild/bazel/commit/f5900474b8bce417c3ef4c3e06af6da5ed57b929. This CL restores the previous behavior.
The issue results in this failure:
$ linux-sandbox -- /bin/true
third_party/bazel/src/main/tools/process-tools.cc:112: "sigaction": Invalid argument
We do have tests for linux-sandbox and Bazel's usage of the linux-sandbox
strategy. None of these actually started failing when the linux-sandbox
stopped working, due to our automatic fallback logic that we added in
order to not annoy users / CI on platforms that don't support the Linux
sandbox :( Bazel silently falls back to the processwrapper-sandbox
strategy and disables the entire linux-sandbox test suite, but signals
to CI that it passed (because we don't support a "Skipped" test status).
Well, congrats. What an epic fail. I will have to rework this next week.
PiperOrigin-RevId: 157021455
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
This broke Bazel CI on freebsd:
http://ci.bazel.io/view/Dashboard/job/Bazel/JAVA_VERSION=1.8,PLATFORM_NAME=freebsd-11/1516/console#
*** Original change description ***
Refactor process-wrapper code so the spawn/wait code is pluggable.
In an upcoming change I'll reintroduce the new platform-specific implementations that can kill and wait for all descendant processes spawned by the wrapped process.
This has no functional changes.
PiperOrigin-RevId: 156884488
|
|
|
|
|
|
|
|
| |
In an upcoming change I'll reintroduce the new platform-specific implementations that can kill and wait for all descendant processes spawned by the wrapped process.
This has no functional changes.
PiperOrigin-RevId: 156849610
|
|
|
|
| |
PiperOrigin-RevId: 156347327
|
|
|
|
|
|
|
|
|
| |
- Refactoring to share more code between the two programs.
- Remove setuid() call in linux-sandbox. It was added due to a wrong understanding of what process-wrapper did in the beginning and unless someone installed linux-sandbox as a setuid binary, it was a no-op.
- Switch to a new process group in linux-sandbox to avoid accidentally killing our parent.
RELNOTES: None.
PiperOrigin-RevId: 156332503
|
|
|
|
|
|
|
| |
No functional changes.
Change-Id: Ia87c19b70dd1ff8fa7465ad90c499cf351b9687b
PiperOrigin-RevId: 156188343
|
|
|
|
| |
PiperOrigin-RevId: 156092500
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Likely cause for b/38172480 ("blaze now waits for all processes spawned by local tests to terminate") and b/38194553 ("Server terminated abruptly (error code: 14, error message: 'Endpoint read failed'").
I have a fix almost ready, but it consists of many lines of new code - we shouldn't rush that into Bazel's 0.5.0 release. Instead, let's roll this back, do a release using the known good older process-wrapper and then go forward in 0.5.1 with a better and well tested new version of this.
*** Original change description ***
process-wrapper: Wait for all (grand)children before exiting.
This uses Linux's PR_SET_CHILD_SUBREAPER and FreeBSD's PROC_REAP_ACQUIRE features to become an init-like process for all (grand)children spawned by process-wrapper, which allows us to a) kill them reliably and then b) wait for them reliably. Before this change, we only killed the main child, waited for it, then fired off a kill -9 on the process group, without waiting for it. This led to a race condition where Bazel would try to use...
***
PiperOrigin-RevId: 156068188
|
|
|
|
|
|
|
|
|
|
|
| |
if (!source_specified) {
^
src/main/tools/linux-sandbox-options.cc:126:8: note: 'source_specified' was declared here bool source_specified;
^
Closes #2735.
PiperOrigin-RevId: 155067584
|
|
|
|
|
|
|
|
|
| |
in order to degrade gracefully on older Linux versions that don't support this feature yet.
Fixes broken Bazel CI:
http://ci.bazel.io/job/bazel-docker-tests/532/BAZEL_VERSION=latest,PLATFORM_NAME=docker/console
PiperOrigin-RevId: 154190829
|
|
|
|
|
|
|
|
| |
This uses Linux's PR_SET_CHILD_SUBREAPER and FreeBSD's PROC_REAP_ACQUIRE features to become an init-like process for all (grand)children spawned by process-wrapper, which allows us to a) kill them reliably and then b) wait for them reliably. Before this change, we only killed the main child, waited for it, then fired off a kill -9 on the process group, without waiting for it. This led to a race condition where Bazel would try to use or delete files that were still helt open by children of the main child and thus to bugs like #2371.
This means we now have reliable process management on Linux, FreeBSD and Windows. Unfortunately I couldn't find any feature like this on macOS, so this is the only OS that will still have this race condition.
PiperOrigin-RevId: 153817210
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add jdk as runfiles for java, jar, and javac in jdk.BUILD. Done via data
attribute of filegroup.
This should 'mostly' work if you handwave around the differences between
default_runfiles and data_runfiles. The jdk runfiles will always be
accessed via data_runfiles, and thus not platform-targeted. Therefore, my
expectation is the target machine will be sent the host JDK.
However, since the jdk is itself composed of binary imports anyway, it
seems like the JDK already has issues being platform-targeted.
To reviewers:
I didn't do a thorough audit of the jdk.BUILD file. More specifically,
unsure if xjc or wsimport need a jdk as well.
Still investigating a good testing strategy.
Change-Id: I138b95b8cee2808c89e4202b822ada6a6577acce
PiperOrigin-RevId: 152290935
|
|
|
|
|
|
|
|
|
|
|
| |
If needed you can restore the old behavior by passing the flag
--sandbox_tmpfs_path=/tmp to bazel.
Fixes #2508.
--
PiperOrigin-RevId: 151127924
MOS_MIGRATED_REVID=151127924
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 151120717
MOS_MIGRATED_REVID=151120717
|
|
|
|
|
|
|
|
|
|
|
|
| |
Instead of trying to detect overlapping mount points, just ignore any
ENOENT errors during remount. If this error happens, the mount point
wouldn't be accessible anyway, so there's no harm if the remount fails.
Fixes #1948.
--
PiperOrigin-RevId: 151118726
MOS_MIGRATED_REVID=151118726
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Try to run /bin/true as a test of whether the Linux sandbox works,
instead of just trying to create a bunch of namespaces as a proxy.
This helps resolve issues on Linux distros where the earlier check
worked, but then the sandbox ultimately failed due to other operations
being unsupported.
As an example, Debian Jessie and certain Docker versions seem to allow
the creation of PID namespaces, but forbid mounting a new proc on top of
/proc (see #1972). This resulted in Bazel thinking that sandboxing works
fine, when it actually didn't. The improved check correctly catches this
situation and disabled sandboxing.
--
PiperOrigin-RevId: 151116894
MOS_MIGRATED_REVID=151116894
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This can be reactivated by passing the --sandbox_fake_username flag
to Bazel.
Reasoning: 'nobody' has a non-existent home directory on many Linux
distros, leading to issues when tools try to stat / read / write to the
home directory.
Related to #2688.
RELNOTES: The Linux sandbox no longer changes the user to 'nobody' by
default, instead the current user is used as is. The old behavior can be
restored via the --sandbox_fake_username flag.
--
PiperOrigin-RevId: 151115218
MOS_MIGRATED_REVID=151115218
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
By removing the --sandbox_block_path feature in an earlier change and
taking advantage of the fact that in a mount namespace we can actually
"remount" mount points to be read-only without bind mounting them to
some other place beforehand, this is no longer necessary. The code
becomes much simpler due to this, for example we no longer need to
chroot.
--
PiperOrigin-RevId: 151111360
MOS_MIGRATED_REVID=151111360
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It is in the way of optimizing the performance of the sandbox, because
it requires us to create two helper files (an unreadable file and an
unreadable directory) which are bind-mounted on top of paths specified
via this flag. These two helper files were created on a tmpfs mounted by
the sandbox until now, which ensured that they were automatically
deleted on exit. However, mounting tmpfs on /dev/shm or /tmp causes
issues like #2686 or #1882.
By removing this flag, we can get rid of the two helper files, which
means we can also remove the reliance on a "sandbox temp directory"
completely in the next change.
--
PiperOrigin-RevId: 151107496
MOS_MIGRATED_REVID=151107496
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
After this change, a msys bazel can be built with
a MSVC-default Bazel by adding --cpu=x64_windows_msys --host=x64_windows_msys
See https://github.com/bazelbuild/bazel/issues/2627
--
Change-Id: Iaa82bf4dd911c5740b98d3b2739dfccca6203f79
Reviewed-on: https://cr.bazel.build/9293
PiperOrigin-RevId: 149532274
MOS_MIGRATED_REVID=149532274
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The default C++ CROSSTOOL on Windows now becomes MSVC,
--cpu=x64_windows_msvc is still supported
To use MSYS toolcahin, add --cpu=x64_windows_msys and
--host_cpu=x64_windows_msys for host compilation
See https://github.com/bazelbuild/bazel/issues/2627
--
Change-Id: Ie788a39cb5ffbc9fc956ccfd51a3cc816c74543a
Reviewed-on: https://cr.bazel.build/9292
PiperOrigin-RevId: 149530250
MOS_MIGRATED_REVID=149530250
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This rule can be used in building JNI shared libraries for Windows.
For example, see TensorFlow usage of these targets in jdk.BUILD:
https://github.com/tensorflow/tensorflow/blo[]a98083a6c16f263d668271889863596efbeb84/tensorflow/java/src/main/native/BUILD#L68
Closes #2599.
--
Reviewed-on: https://github.com/bazelbuild/bazel/pull/2599
PiperOrigin-RevId: 149527656
MOS_MIGRATED_REVID=149527656
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Broke tests on CI: http://ci.bazel.io/job/bazel-tests/570/
*** Original change description ***
Roll forward execroot change
RELNOTES[INC]: Previously, an external repository would be symlinked into the
execution root at execroot/local_repo/external/remote_repo. This changes it to
be at execroot/remote_repo. This may break genrules/Skylark actions that
hardcode execution root paths. If this causes breakages for you, ensure that
genrules are using $(location :target) to access files and Skylark rules are
using http://bazel.io/docs/skylark/lib/File.html's path, dirname, etc.
functions. Cust...
--
PiperOrigin-RevId: 147833177
MOS_MIGRATED_REVID=147833177
|
|
|
|
|
|
|
|
| |
magic artifact.
--
PiperOrigin-RevId: 147830857
MOS_MIGRATED_REVID=147830857
|
|
|
|
|
|
|
|
|
| |
The targets contain the entries in the extdir and are used to
construct a classpath, so the label is not actually a directory.
--
PiperOrigin-RevId: 147798672
MOS_MIGRATED_REVID=147798672
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 147727032
MOS_MIGRATED_REVID=147727032
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
RELNOTES[INC]: Previously, an external repository would be symlinked into the
execution root at execroot/local_repo/external/remote_repo. This changes it to
be at execroot/remote_repo. This may break genrules/Skylark actions that
hardcode execution root paths. If this causes breakages for you, ensure that
genrules are using $(location :target) to access files and Skylark rules are
using http://bazel.io/docs/skylark/lib/File.html's path, dirname, etc.
functions. Custom crosstools that hardcode external/<repo> paths will have to
be updated.
Issue #1262.
--
PiperOrigin-RevId: 147726370
MOS_MIGRATED_REVID=147726370
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 146469548
MOS_MIGRATED_REVID=146469548
|
|
|
|
|
|
|
|
| |
Seems like on Ubuntu 16.04 we have to ignore EPERM on failed remounts, too.
--
PiperOrigin-RevId: 146354561
MOS_MIGRATED_REVID=146354561
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The behavior of the Linux sandbox was changed to not hide the local hostname by default.
It is now only hidden when the --sandbox_fake_hostname flag is specified.
Also, instead of using the hostname "sandbox" in this case, it now uses "localhost", which fixes the issue of sandboxed processes not being able to resolve their local hostname.
RELNOTES: For increased compatibility with environments where UTS namespaces are not available, the Linux sandbox no longer hides the hostname of the local machine by default. Use --sandbox_fake_hostname to re-enable this feature.
--
PiperOrigin-RevId: 146244268
MOS_MIGRATED_REVID=146244268
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
bazel build //src:bazel --cpu=x64_windows_msvc
now succeeds, hooray!
This change adds empty implementations for
build-runfiles and process-wrapper to make that
possible.
This means we can now build a bazel binary that
doesn't depend on MSYS.
The resulting binary is not yet functional because
many methods are still to be implemented, and they
just write "TODO: implement" or something similar.
But still this is great news, because now we can
add compile tests to the CI for MSVC!
See https://github.com/bazelbuild/bazel/issues/2107
--
PiperOrigin-RevId: 146106178
MOS_MIGRATED_REVID=146106178
|
|
|
|
|
|
|
|
|
|
| |
RELNOTES: New flag --sandbox_add_mount_pair to specify customized source:target path pairs to bind mount inside the sandbox.
--
Change-Id: Ifbacfc0e16bbaedcf5b6d3937799710f2cfa3d58
Reviewed-on: https://cr.bazel.build/7150
PiperOrigin-RevId: 142542381
MOS_MIGRATED_REVID=142542381
|
|
|
|
|
|
|
| |
These may happen when a broken NFS mount is present on the user system. Ideally, that mount should just be fixed or removed, but where this is not possible, it's probably safe to ignore it.
--
MOS_MIGRATED_REVID=140476478
|
|
|
|
|
|
|
|
|
|
|
|
| |
Skip remounting such mount points because they are not actually
visible in the sandbox after mounting new tmpfs.
Fixes https://github.com/bazelbuild/bazel/issues/1959
--
Change-Id: Ia1361559966ffb05ea1ddbeaee1ed7d3ebdb9e15
Reviewed-on: https://bazel-review.googlesource.com/#/c/6970/
MOS_MIGRATED_REVID=137397312
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=136338300
|
|
|
|
|
|
|
|
|
| |
Tested with the alpine:3.4 Docker image.
--
Change-Id: I8f3e585051988d8fd437ccd69c9c7bd009fd45d2
Reviewed-on: https://bazel-review.googlesource.com/#/c/5790/
MOS_MIGRATED_REVID=135468656
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is a semantic change for Bazel, which now observes the --cpu flag when
looking up a Jvm. Use "-default" as a fallback to keep the change backwards
compatible.
RELNOTES[INC]: Bazel now uses the --cpu flag to look up Jvms; it falls back
to "default" if it can't find a Jvm matching the CPU value.
--
MOS_MIGRATED_REVID=135333759
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Update `select` statements in BUILD files with the
new config_setting.
This is a first step on a long path that leads to
us being able to compile bazel on Windows with
--cpu=x64_windows_msvc. Needless to say, we're not
there yet.
Tested: on Linux, Darwin, Windows/MSYS
--
MOS_MIGRATED_REVID=134534613
|
|
|
|
|
|
|
| |
occasional hangs seen during builds and on Bazel CI.
--
MOS_MIGRATED_REVID=134279208
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
/tmp.
Add "-b" option to linux-sandbox to explicitly bind mount files / directories into the sandbox. This is used to pull in the workspace and output base of Bazel even when they're located in /tmp and would thus be hidden by the tmpfs we mount on the /tmp directory in the sandbox.
Add "-S" option to linux-sandbox to explicitly specify a temporary directory to be used to contain the sandbox. This can be created by Bazel and then removed more reliably, compared to the earlier behavior where the sandbox would create its own temporary root directory in /tmp/sandbox.XXXXXX (and fail to delete it in case it gets killed by a signal).
Fix spurious empty.XXXXXX files and directories not being deleted from /tmp.
--
MOS_MIGRATED_REVID=133695992
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This has the following improvements upon the older one:
- Uses PID namespaces, PR_SET_PDEATHSIG and a number of other tricks for
further process isolation and 100% reliable killing of child processes.
- Uses clone() instead of unshare() to work around a Linux kernel bug that
made creating a sandbox unreliable.
- Instead of mounting a hardcoded list of paths + whatever you add with
--sandbox_add_path, this sandbox instead mounts all of /, except for what
you make inaccessible via --sandbox_block_path. This should solve the
majority of "Sandboxing breaks my build, because my compiler is installed
in /opt or /usr/local" issues that users have seen.
- Instead of doing magic with bind mounts, we create a separate execroot for
each process containing symlinks to the input files. This is simpler and
gives more predictable performance.
- Actually makes everything except the working directory read-only
(fixes #1364). This means that a running process can no longer accidentally
modify your source code (yay!).
- Prevents a number of additional "attacks" or leaks, like accidentally
inheriting file handles from the parent.
- Simpler command-line interface.
- We can provide the same semantics in a Mac OS X sandbox, which will come in
a separate code review from yueg@.
It has the following caveats / known issues:
- The "fallback to /bin/bash on error" feature is gone, but now that the
sandbox mounts everything by default, the main use-case for this is no
longer needed.
The following improvements are planned:
- Use a FUSE filesystem if possible for the new execroot, instead of creating
symlinks.
- Mount a base image instead of "/".
FAQ:
Q: Why is mounting all of "/" okay, doesn't this make the whole sandbox
useless?
A: This is still a reasonable behavior, because the sandbox never tried to
isolate your build from the operating system it runs in. Instead it is
supposed to protect your data from a test running "rm -rf $HOME" and to
make it difficult / impossible for actions to use input files that are not
declared dependencies. For even more isolation the sandbox will support
mounting a base image as its root in a future version (similar to Docker
images).
Q: Let's say my process-specific execroot contains a symlink to an input file
"good.h", can't the process just resolve the symlink, strip off the file
name and then look around in the workspace?
A: Yes. Unfortunately we could not find any way on Linux to make a file appear
in a different directory with *all* of the semantics we would like. The
options investigated were:
1) Copying input files, which is much too slow.
2) Hard linking input files, which is fast, but doesn't work cross-
filesystems and it's also not possible to make them read-only.
3) Bind mounts, which don't scale once you're up in the thousands of input
files (across all actions) - it seems like the kernel has some
non-linear performance behavior when the mount table grows too much,
resulting in the mount syscall taking more time the more mounts you
have.
4) FUSE filesystem, good in theory, but wasn't ready for the first
iteration.
RELNOTES: New sandboxing implementation for Linux in which all actions run in a separate execroot that contains input files as symlinks back to the originals in the workspace. The running action now has read-write access to its execroot and /tmp only and can no longer write in arbitrary other places in the file system.
--
Change-Id: Ic91386fc92f8eef727ed6d22e6bd0f357d145063
Reviewed-on: https://bazel-review.googlesource.com/#/c/4053
MOS_MIGRATED_REVID=130638204
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Please don't merge before I8b8c3596fd2a4c125071406eefd407ca402099dd. The
test will fail then.
I've seen two issues with this functionality breaking: #481 and #1637.
Seems like it's time to add a test which catches the issue on the
Debian/Ubuntu machines people commonly test on. The test fails on my
Debian system before I8b8c3596fd2a4c125071406eefd407ca402099dd, and
passes with that change applied.
--
Change-Id: Ib785c874cdb9192920f9935b696bfd6c9c0e5f4f
Reviewed-on: https://bazel-review.git.corp.google.com/#/c/5451/
MOS_MIGRATED_REVID=130635565
|
|
|
|
|
|
|
|
|
| |
fixes #1637 (https://github.com/bazelbuild/bazel/issues/1637)
--
Change-Id: I8b8c3596fd2a4c125071406eefd407ca402099dd
Reviewed-on: https://bazel-review.git.corp.google.com/#/c/5410/2
MOS_MIGRATED_REVID=130633667
|
|
|
|
|
|
|
| |
on Windows by default)
--
MOS_MIGRATED_REVID=129319018
|