aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/sandbox
Commit message (Collapse)AuthorAge
* Automated g4 rollback of commit c4134802dd15d6ef5cca6521f6bf6aac395ee2ad.Gravatar kchodorow2017-05-23
| | | | | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Roll forward of directory name change *** Original change description *** Automated g4 rollback of commit 1d9e1ac90197b1d3d7b137ba3c1ada67bb9ba31b. *** Reason for rollback *** Breaks //src/test/shell/integration:force_delete_output_test *** Original change description *** Symlink output directories to the correct directory name If the workspace directory is /path/to/my/proj and the name in the WORKSPACE file is "floop", this will symlink the output directories to output_base/execroot/floop instead of output_base/execroot/proj. More prep for #1262, fixes #1681. PiperOrigin-RevId: 156892980
* sandbox: Create more compact sandbox profiles faster.Gravatar Philipp Wollermann2017-05-15
| | | | | | | | By removing the now unnecessary call to Path#resolveSymbolicLinks we can save a few stat's per action execution. Change-Id: Iee157e941c1cd3515ff5ea3b7f410824c24cf44d PiperOrigin-RevId: 155946544
* Bring back --sandbox_block_path.Gravatar philwo2017-05-15
| | | | | | | | | | | | This is basically a rollback of https://github.com/bazelbuild/bazel/commit/3e2329a73ffd5d60e5e2babe60ebe5bf322c07da, except this solves the reason why the feature was removed in the first place. We now create the helper files necessary to make files unreadable in Linux in Bazel's Java code and manage their lifetime there. Request was filed by a user here: http://stackoverflow.com/questions/43849651/how-to-lock-down-the-bazel-filesystem-sandbox PiperOrigin-RevId: 155913246
* sandbox: Use the SpawnInputExpander everywhere and delete SpawnHelpers.Gravatar Philipp Wollermann2017-05-09
| | | | | | | | | This unifies our code to use just one standard implementation to get the entire expanded input files for a Spawn, including from Filesets and Runfiles. Change-Id: I1e286508adf0a9aeddf70934b010e6fcc144c4a7 PiperOrigin-RevId: 155497273
* sandbox: Switch Darwin sandbox to the SymlinkedExecRoot strategy.Gravatar Philipp Wollermann2017-05-09
| | | | | | | | | | | Hardlinks are problematic due to not working across filesystem boundaries and causing Bazel to do lots of I/O because it has to create a hardlink and a symlink for each input file. This improves performance of Bazel building itself by 10% on my system. Change-Id: I8acb77053de875160a046e38624735ed18375bed PiperOrigin-RevId: 155493583
* sandbox: Use process-wrapper in addition to sandbox-exec on macOS.Gravatar Philipp Wollermann2017-05-09
| | | | | | | | | This gives us much improved process management, because Bazel can now reliably kill child processes of spawns via their process group and wait for them to exit. Change-Id: Ib3cb20725b3c569aa5b317a69d7682f5774707b0 PiperOrigin-RevId: 155493511
* sandbox: Remove debug logging in DarwinSandboxedStrategy.Gravatar Philipp Wollermann2017-05-09
| | | | | Change-Id: Idc023f3a8c1c3b60d3f3f23a579a5eccb92d074d PiperOrigin-RevId: 155487527
* sandbox: Precache writable directories in DarwinSandboxedStrategy.Gravatar Philipp Wollermann2017-05-09
| | | | | Change-Id: I1522c364a157ee0a144ab83eca54e419142c03b1 PiperOrigin-RevId: 155484109
* sandbox: Remove special treatment for runUnderPath from Darwin sandbox.Gravatar Philipp Wollermann2017-05-09
| | | | | | | | There's no need to make it explicitly readable, because the entire host filesystem is readable anyway. Change-Id: I6a63cc93b600250c1c8828ef8d1c9d6133b671d7 PiperOrigin-RevId: 155477093
* sandbox: Use CommandEnvironment for more robust look-up of embedded binaries.Gravatar Philipp Wollermann2017-05-09
| | | | | Change-Id: I1bc1901ea7cd9a5b93c280ec0ff8ac0d10959a09 PiperOrigin-RevId: 155381163
* sandbox: Simplify DarwinSandboxRunner#isSupported.Gravatar Philipp Wollermann2017-05-08
| | | | | | | | There's no need to check for the OS version, as we can just try to use sandbox-exec and if it works, we're good. Change-Id: I7fe9a0b55856c646da915a2872531f050a25b110 PiperOrigin-RevId: 155368707
* sandbox: Use the processwrapper-sandbox on FreeBSD (default) and macOS ↵Gravatar Philipp Wollermann2017-05-08
| | | | | | | | | | | | | | | | | (fallback only). On macOS the processwrapper-sandbox will be used when the darwin-sandbox doesn't work. Most notably this is the case for nested sandboxing, e.g. Bazel running Bazel inside an integration test. Also includes a fix to pull in some extra environment vars on macOS, similar to what DarwinSandboxedStrategy and StandaloneSpawnStrategy already do. Without this the processwrapper-sandbox seems to occasionally cause ObjC builds (and two of our tests) to fail. Change-Id: Ic7462080caf56d9bb98e2f3765bd37853b01632b RELNOTES: Sandboxing is now enabled by default on FreeBSD (via processwrapper-sandbox). PiperOrigin-RevId: 155366728
* sandbox: Some refactoring and small improvements.Gravatar Philipp Wollermann2017-05-08
| | | | | Change-Id: I1355c2448cb6cbbcdbace81051a7beb8659f1f00 PiperOrigin-RevId: 155366727
* sandbox: Use absolute path to getconf.Gravatar Philipp Wollermann2017-05-08
| | | | | Change-Id: I43dfd979aee0c510ec18b479f2a6bd55562b3fc0 PiperOrigin-RevId: 155361450
* Add explicit names to the sandboxing strategies so that they can beGravatar philwo2017-04-24
| | | | | | | | | | | selected even if they're not the preferred one on a platform. Simplify the SandboxActionContextProvider and remove the warning about sandboxing being unsupported. With the ProcessWrapperSandboxedStrategy now being reliable enough and the strategies printing their real name in the UI, this is overall a better UX. PiperOrigin-RevId: 153825986
* No longer print a warning when a sandbox directory couldn't be deleted.Gravatar philwo2017-04-24
| | | | | | With the process-wrapper improvements and the additional deletion of the sandbox base in the SandboxModule in, this should be reliable enough. The warning was also not actionable for users and annoyed them, so let's get rid of it. PiperOrigin-RevId: 153823045
* Fix bug breaking sandboxing on macOS 10.12Gravatar Ty Book2017-04-07
| | | | | | | | | macOS version strings that have only two components, e.g. "10.12", were always failing the check that the macOS version is >= 10.11, causing Bazel to erroneously think sandboxing isn't supported Change-Id: Ifa4a01fc304e7620502d3f0f9f70c3b500d23864 PiperOrigin-RevId: 152493682
* sandbox: Add new flag --experimental_sandbox_base.Gravatar philwo2017-04-07
| | | | | | | | Usually, Bazel creates the sandbox directories underneath its output_base. With --experimental_sandbox_base you can specify a different parent directory for this, e.g. /dev/shm to run all sandboxed actions on a memory-backed filesystem. PiperOrigin-RevId: 152490815
* Refactor all ctor callsites of PathFragment to instead call a static ↵Gravatar nharmata2017-04-05
| | | | | | | | | | | | 'create' method. This paves the way for changing PathFragment to e.g. an abstract class with multiple subclasses. This way we can split out the windows-specific stuff into one of these concrete classes, making the code more readable and also saving memory (since the shallow heap size of the NonWindowsPathFragment subclass will hopefully be smaller than that of the current PathFragment). This also lets us pursue gc churn optimizations. We can now do interning in PathFragment#create and can also get rid of unnecessary intermediate PathFragment allocations. RELNOTES: None PiperOrigin-RevId: 152145768
* Automated g4 rollback of commit 1d9e1ac90197b1d3d7b137ba3c1ada67bb9ba31b.Gravatar hlopko2017-04-04
| | | | | | | | | | | | | | | | | | *** Reason for rollback *** Breaks //src/test/shell/integration:force_delete_output_test *** Original change description *** Symlink output directories to the correct directory name If the workspace directory is /path/to/my/proj and the name in the WORKSPACE file is "floop", this will symlink the output directories to output_base/execroot/floop instead of output_base/execroot/proj. More prep for #1262, fixes #1681. PiperOrigin-RevId: 152126545
* Symlink output directories to the correct directory nameGravatar kchodorow2017-03-31
| | | | | | | | | | If the workspace directory is /path/to/my/proj and the name in the WORKSPACE file is "floop", this will symlink the output directories to output_base/execroot/floop instead of output_base/execroot/proj. More prep for #1262, fixes #1681. PiperOrigin-RevId: 151712384
* sandbox: Pass config flags to linux-sandbox on the command-line.Gravatar Philipp Wollermann2017-03-27
| | | | | | | | | | | The linux-sandbox no longer requires extensive configuration via flags, so the command-line arguments will easily fit into the allowed length. This simplifies the code and gets rid of some I/O. -- PiperOrigin-RevId: 151176551 MOS_MIGRATED_REVID=151176551
* sandbox: Enable the process-wrapper + symlink tree sandbox on FreeBSD.Gravatar Philipp Wollermann2017-03-27
| | | | | | | | | | | Extract the process-wrapper + symlink tree sandbox strategy into its own class and allow its use in FreeBSD. RELNOTES: Bazel can now use the process-wrapper + symlink tree based sandbox implementation in FreeBSD. -- PiperOrigin-RevId: 151171652 MOS_MIGRATED_REVID=151171652
* sandbox: Show the actual running sandbox strategy for actions.Gravatar Philipp Wollermann2017-03-27
| | | | | | | | | | This allows us to see for example whether an action ran using the "process wrapper + symlink tree" sandbox or the real "PID and mount namespaces" Linux sandbox. -- PiperOrigin-RevId: 151165170 MOS_MIGRATED_REVID=151165170
* sandbox: Clean up comments / code in SandboxRunner#run.Gravatar Philipp Wollermann2017-03-27
| | | | | | -- PiperOrigin-RevId: 151160662 MOS_MIGRATED_REVID=151160662
* sandbox: Refactor sandbox strategies to use a common exec method.Gravatar Philipp Wollermann2017-03-27
| | | | | | -- PiperOrigin-RevId: 151130566 MOS_MIGRATED_REVID=151130566
* sandbox: Make /tmp and /dev/shm writable by default on Linux.Gravatar Philipp Wollermann2017-03-27
| | | | | | | | | | | | | | Also refactor the way we compute writable dirs so that they're computed only once per running action, not twice. Fixes #2056, fixes #1973, fixes #1460. RELNOTES: /tmp and /dev/shm are now writable by default inside the Linux sandbox. -- PiperOrigin-RevId: 151123543 MOS_MIGRATED_REVID=151123543
* sandbox: Improve the check whether the Linux sandbox is supported.Gravatar Philipp Wollermann2017-03-27
| | | | | | | | | | | | | | | | | | | Try to run /bin/true as a test of whether the Linux sandbox works, instead of just trying to create a bunch of namespaces as a proxy. This helps resolve issues on Linux distros where the earlier check worked, but then the sandbox ultimately failed due to other operations being unsupported. As an example, Debian Jessie and certain Docker versions seem to allow the creation of PID namespaces, but forbid mounting a new proc on top of /proc (see #1972). This resulted in Bazel thinking that sandboxing works fine, when it actually didn't. The improved check correctly catches this situation and disabled sandboxing. -- PiperOrigin-RevId: 151116894 MOS_MIGRATED_REVID=151116894
* sandbox: No longer change the user to 'nobody' by default.Gravatar Philipp Wollermann2017-03-24
| | | | | | | | | | | | | | | | | | | This can be reactivated by passing the --sandbox_fake_username flag to Bazel. Reasoning: 'nobody' has a non-existent home directory on many Linux distros, leading to issues when tools try to stat / read / write to the home directory. Related to #2688. RELNOTES: The Linux sandbox no longer changes the user to 'nobody' by default, instead the current user is used as is. The old behavior can be restored via the --sandbox_fake_username flag. -- PiperOrigin-RevId: 151115218 MOS_MIGRATED_REVID=151115218
* sandbox: No longer require a temporary "sandbox" directory.Gravatar Philipp Wollermann2017-03-24
| | | | | | | | | | | | | By removing the --sandbox_block_path feature in an earlier change and taking advantage of the fact that in a mount namespace we can actually "remount" mount points to be read-only without bind mounting them to some other place beforehand, this is no longer necessary. The code becomes much simpler due to this, for example we no longer need to chroot. -- PiperOrigin-RevId: 151111360 MOS_MIGRATED_REVID=151111360
* sandbox: Remove the flag --sandbox_block_path.Gravatar Philipp Wollermann2017-03-24
| | | | | | | | | | | | | | | | | | It is in the way of optimizing the performance of the sandbox, because it requires us to create two helper files (an unreadable file and an unreadable directory) which are bind-mounted on top of paths specified via this flag. These two helper files were created on a tmpfs mounted by the sandbox until now, which ensured that they were automatically deleted on exit. However, mounting tmpfs on /dev/shm or /tmp causes issues like #2686 or #1882. By removing this flag, we can get rid of the two helper files, which means we can also remove the reliance on a "sandbox temp directory" completely in the next change. -- PiperOrigin-RevId: 151107496 MOS_MIGRATED_REVID=151107496
* Fix linux sandbox to create empty files correctly.Gravatar Ulf Adams2017-03-22
| | | | | | | | | | | Using /dev/null was causing it to create symlinks to /dev/null, which breaks Python programs. I didn't catch this earlier, because my machine had an old linux kernel installed that didn't support sandboxing. There's an existing integration test, which just broke in our CI. -- PiperOrigin-RevId: 150750032 MOS_MIGRATED_REVID=150750032
* Add SpawnInputExpander helper class to arrange runfiles for spawn strategiesGravatar Ulf Adams2017-03-21
| | | | | | | | | | | | | | | | | | | | | This new class is a combination of SpawnHelper and our internal code; the plan is to migrate all spawn strategies to the new class. The strict flag should be enabled by default, but that's a breaking change, so we need to do it later. - Use it in SandboxStrategy. - Add ActionInput.getExecPath to return a PathFragment; this avoids lots of back and forth between path fragments and strings. This is a step towards #1593. The previous attempt was missing a one-line patch in StandaloneTestStrategy, which broke all tests with sandboxing. StandaloneTestStrategy was fixed in a separate change, so this should be safe now. -- PiperOrigin-RevId: 150733457 MOS_MIGRATED_REVID=150733457
* Use a relative path for the runfiles treeGravatar Ulf Adams2017-03-21
| | | | | | | | | | Add preconditions to enforce this and remove some now unnecessary code. A small step towards #1593. -- PiperOrigin-RevId: 150625693 MOS_MIGRATED_REVID=150625693
* Rollback of commit 59180a4ea66b1395b5b85defd732859ecae919ea.Gravatar Yue Gan2017-03-20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Break bazel-tests and many other jobs on CI. http://ci.bazel.io/job/bazel-tests/BAZEL_VERSION=HEAD,PLATFORM_NAME=linux-x86_64/651/console *** Original change description *** Add SpawnInputExpander helper class to arrange runfiles for spawn strategies This new class is a combination of SpawnHelper and our internal code; the plan is to migrate all spawn strategies to the new class. The strict flag should be enabled by default, but that's a breaking change, so we need to do it later. - Use it in SandboxStrategy. - Add ActionInput.getExecPath to return a PathFragment; this avoids lots of back and forth between path fragments and strings. This is a step towards #159... *** -- PiperOrigin-RevId: 150610616 MOS_MIGRATED_REVID=150610616
* Add SpawnInputExpander helper class to arrange runfiles for spawn strategiesGravatar Ulf Adams2017-03-17
| | | | | | | | | | | | | | | | | This new class is a combination of SpawnHelper and our internal code; the plan is to migrate all spawn strategies to the new class. The strict flag should be enabled by default, but that's a breaking change, so we need to do it later. - Use it in SandboxStrategy. - Add ActionInput.getExecPath to return a PathFragment; this avoids lots of back and forth between path fragments and strings. This is a step towards #1593. -- PiperOrigin-RevId: 150427021 MOS_MIGRATED_REVID=150427021
* Clean up *GccStrategyGravatar Ulf Adams2017-03-10
| | | | | | | | | - use SimpleSpawn in SpawnGccStrategy - set PWD in CppCompileAction for consistency -- PiperOrigin-RevId: 149745059 MOS_MIGRATED_REVID=149745059
* StandaloneTestStrategy sets the full list of outputs on the test spawnGravatar Ulf Adams2017-03-07
| | | | | | | | | | | | | | | | | | | All spawn strategies already treat all normal outputs as optional. Bazel checks at the action level whether all action outputs are created, but does not check at the spawn level. Spawn.getOptionalOutputs is therefore unnecessary, and removed in this change. The only place where this was set was in StandaloneTestStrategy, which now specifies the full set of outputs, which is now computed by TestRunnerAction. The internal test strategy implementations are also updated in this change. While I'm at it, also remove the use of BaseSpawn and use SimpleSpawn instead. This may go some way towards fixing #1413 and #942. -- PiperOrigin-RevId: 149397100 MOS_MIGRATED_REVID=149397100
* Remove all the action resource estimation codeGravatar Ulf Adams2017-03-06
| | | | | | -- PiperOrigin-RevId: 149110466 MOS_MIGRATED_REVID=149110466
* Rationalize local resource acquisitionGravatar Ulf Adams2017-03-03
| | | | | | | | | | | | | | | | | | | | | | Move all local resource acquisition to where local execution actually happens. Don't attempt to acquire resources per action, but only for individual spawns. This significantly simplifies the code. The downside is that we don't account for action-level work anymore. In general, actions should not perform any process execution themselves, but always delegate such work to a SpawnStrategy implementation. This change makes sure that every Spawn has local resources set in a way that is consistent with the previous state. However, there are two actions - Fileset and FileWrite -, which are not spawns, and so we now don't limit their concurrent execution anymore. For Fileset, all work is done in a custom Fileset-specific thread pool, so this shouldn't be a problem. I'm not sure about FileWriteAction. -- PiperOrigin-RevId: 149012600 MOS_MIGRATED_REVID=149012600
* Make HardlinkedExecRoot correctly mount empty directory inputs.Gravatar Adam Michael2017-03-03
| | | | | | | | Fixes #2597 -- PiperOrigin-RevId: 148999160 MOS_MIGRATED_REVID=148999160
* Move UnixFileSystem to lib.unix, WindowsFileSystem to lib.windowsGravatar Ulf Adams2017-02-28
| | | | | | -- PiperOrigin-RevId: 148749485 MOS_MIGRATED_REVID=148749485
* sandbox should create regular empty files, not symlink to /dev/null.Gravatar Philipp Wollermann2017-02-28
| | | | | | | | | | | | | | Adds a test based on our Python rules that makes sure that this actually fixes the issue. Thanks to @duggelz for the suggestion. It seems like our Python rules are the only place that actually provides an EmptyFilesSupplier to Runfiles, so there's probably no simpler way to test this behavior in an integration test. Fix #1458. Fix #2394. -- PiperOrigin-RevId: 148656193 MOS_MIGRATED_REVID=148656193
* Rollback of commit 4b73e972d909bcd533f2f9940f95a00b9b73bdde.Gravatar Dmitry Lomov2017-02-17
| | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Broke tests on CI: http://ci.bazel.io/job/bazel-tests/570/ *** Original change description *** Roll forward execroot change RELNOTES[INC]: Previously, an external repository would be symlinked into the execution root at execroot/local_repo/external/remote_repo. This changes it to be at execroot/remote_repo. This may break genrules/Skylark actions that hardcode execution root paths. If this causes breakages for you, ensure that genrules are using $(location :target) to access files and Skylark rules are using http://bazel.io/docs/skylark/lib/File.html's path, dirname, etc. functions. Cust... -- PiperOrigin-RevId: 147833177 MOS_MIGRATED_REVID=147833177
* Roll forward execroot changeGravatar Kristina Chodorow2017-02-16
| | | | | | | | | | | | | | | | | RELNOTES[INC]: Previously, an external repository would be symlinked into the execution root at execroot/local_repo/external/remote_repo. This changes it to be at execroot/remote_repo. This may break genrules/Skylark actions that hardcode execution root paths. If this causes breakages for you, ensure that genrules are using $(location :target) to access files and Skylark rules are using http://bazel.io/docs/skylark/lib/File.html's path, dirname, etc. functions. Custom crosstools that hardcode external/<repo> paths will have to be updated. Issue #1262. -- PiperOrigin-RevId: 147726370 MOS_MIGRATED_REVID=147726370
* Fix #2018: Can't access localhost in bazel 0.3.2.Gravatar Philipp Wollermann2017-02-01
| | | | | | | | | | | | | | The behavior of the Linux sandbox was changed to not hide the local hostname by default. It is now only hidden when the --sandbox_fake_hostname flag is specified. Also, instead of using the hostname "sandbox" in this case, it now uses "localhost", which fixes the issue of sandboxed processes not being able to resolve their local hostname. RELNOTES: For increased compatibility with environments where UTS namespaces are not available, the Linux sandbox no longer hides the hostname of the local machine by default. Use --sandbox_fake_hostname to re-enable this feature. -- PiperOrigin-RevId: 146244268 MOS_MIGRATED_REVID=146244268
* Simplify the Spawn interface, update Executor interfaceGravatar Ulf Adams2017-02-01
| | | | | | | | | | | | | | | | | | | | Drop the Spawn.getOwner() method, which was duplicating functionality already available through Spawn.getResourceOwner(), in favor of the latter. In order to do that, change Executor.reportSubcommand to take a Spawn instance instead, which in turn requires updating all call sites. This is part of establishing Spawn as an abstraction for lower-level local and remote execution in all cases instead of passing args, env, input/output files and possibly other metadata as individual method parameters to the underlying implementations. In order for that to be maintainable, Spawn should be light-weight and easy to construct correctly, and not contain too much unnecessary data. -- PiperOrigin-RevId: 146224914 MOS_MIGRATED_REVID=146224914
* Simplify the Spawn interfaceGravatar Ulf Adams2017-02-01
| | | | | | | | | | | | | | | | | | | | | | | | Remove getSpawnInfo and asShellCommand, which are never really overridden in a useful way. asShellCommand moves to the Spawns class, and getSpawnInfo is only ever called by SpawnAction, and the implementation moves there. I'm considering using Spawn as the general lower-level abstraction for both local and remote execution. It sort of is that already, except it's not used consistently - we often pass a tuple of (args, env) plus possibly input and output files through parameter-heavy method call hierarchies instead of using this existing abstraction. However, I'm concerned about the amount of baggage it's carrying as well as the number of implementations for what is supposed to be a simple interface (or possibly even a simple value class), and this is an attempt to slim it down a bit. This should have no visible effects on builds. -- PiperOrigin-RevId: 146109838 MOS_MIGRATED_REVID=146109838
* Switch to RunfilesSuppliers for communicating runfilesGravatar Michajlo Matijkiw2017-01-30
| | | | | | | | | | | | | | | | | | | | | | | ActionSpawn/SpawnAction now deal exclusively in RunfilesSuppliers, manifests maps are no more. There is some lingering awkwardness, in particular: - Manifests still need to be tracked in some places, we can work out if this is still necessary on a case by case basis. - Skylark makes actions' runfiles available via 'resolve_command' where they are consumed by 'action'. I've updated the documentation, though the name isn't entirely accurate anymore. That being said these interfaces _are_ marked as experimental, so we _should_ be able to be flexible here. Overall, I think the benefits consolidating runfiles into suppliers, from both code cleanliness and performance perspectives (no longer needing to parse manifests), outweights the awkwardnesses. RELNOTES: resolve_command/action's input_manifest return/parameter is now list -- PiperOrigin-RevId: 145817429 MOS_MIGRATED_REVID=145817429
* Make include pruning work in Bazel.Gravatar Lukacs Berki2017-01-25
| | | | | | | | Fixes #2372. -- PiperOrigin-RevId: 145539067 MOS_MIGRATED_REVID=145539067
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-05 01:11:20 +0000