aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/sandbox/DarwinSandboxedStrategy.java
Commit message (Collapse)AuthorAge
* Extract a common AbstractSpawnStrategy parent classGravatar ulfjack2017-07-24
| | | | | | This removes a bunch of code duplication that I previously introduced. PiperOrigin-RevId: 162909430
* Fix #1027: XML test output missing for tests that timeoutGravatar philwo2017-07-19
| | | | | | The sandbox execution strategies were not passing the timeout grace period specified via --local_termination_grace_seconds, resulting in process-wrapper defaulting to just 5 seconds and linux-sandbox not using any grace period at all, instead directly SIGKILLing the child on timeout. PiperOrigin-RevId: 162349919
* Simplify exception handling in spawn strategiesGravatar ulfjack2017-07-17
| | | | | | | | | | | | | | | | | | | | | | | | | The main change here is to only catch SpawnExecException in StandaloneTestStrategy, so all other exceptions simplify propagate up. As a result, Bazel no longer retries tests that fail with an exception, we only retry tests that actually ran, had a spawn result, and resulted in a UserExecException. That is probably what we want. Also do some cleanup: - Remove ExecException.timedOut; nobody was calling it (but there's still SpawnExecException.timedOut) - Remove SpawnActionContext.shouldPropagateExecException; all exceptions (except SpawnExecException) are now propagated by default - Remote the SandboxOptions from the SandboxStrategies; all sandboxing options are now handled by the underlying SpawnRunner implementations I'll send a followup CL to remove the UserExecException and EnvironmentalExecException types; the types don't do anything special, and there are no catch blocks in production code that catch one of these more specific types. This should fix #3322 by removing a bunch of special handling. PiperOrigin-RevId: 161960919
* Split the sandbox strategies into SpawnRunner implementations and strategiesGravatar ulfjack2017-07-13
| | | | | | | | This adds a bunch of classes that only implement the SpawnRunner interface, and will allow us to support remote caching in combination with local sandboxed execution in a subsequent change. PiperOrigin-RevId: 161664556
* Rewrite all the sandbox strategy implementationsGravatar ulfjack2017-07-12
| | | | | | | | | | | | | | - Make use of existing abstractions like SpawnRunner and SpawnExecutionPolicy. - Instead of having the *Strategy create a *Runner, and then call back into SandboxStrategy, create a single SandboxContainer which contains the full command line, environment, and everything needed to create and delete the sandbox directory. - Do all the work in SandboxStrategy, including creation and deletion of the sandbox directory. - Use SpawnResult instead of throwing, catching, and rethrowing. - Simplify the control flow a bit. PiperOrigin-RevId: 161644979
* Use getopt to parse process-wrapper's command-line.Gravatar philwo2017-06-22
| | | | | | | | This will allow us to add new and optional flags like selecting a strategy used to spawn / wait for the child process. No one except Bazel should be calling "process-wrapper" and I couldn't find any references, so this breaking change should be fine. PiperOrigin-RevId: 159685867
* Rewrite the Executor/ActionExecutionContext splitGravatar ulfjack2017-06-19
| | | | | | | Move everything to ActionExecutionContext, and drop Executor whereever possible. This clarifies the API, makes it simpler to test, and simplifies the code. PiperOrigin-RevId: 159414816
* Extract the MacOS/XCode env rewrite logic into lib.exec.appleGravatar ulfjack2017-06-13
| | | | | | | | | | | | | | | | | | | Also add an interface to allow injecting that logic into LocalSpawnRunner; this is in preparation for rewriting StandaloneSpawnStrategy to use LocalSpawnRunner. At the same time, this reduces the dependencies from exec / standalone to rules.apple, which is a prerequisite for micro-Bazel. There's a small semantic change hidden here - we now only set the new XCodeLocalEnvProvider if we're actually running on Darwin, so we no longer fail execution on non-Darwin platforms if XCODE_VERSION_OVERRIDE or APPLE_SDK_VERSION_OVERRIDE is set. As a result, I moved the corresponding test from StandaloneSpawnStrategyTest to the new XCodeLocalEnvProviderTest. While I'm at it, also open source DottedVersionTest and CacheManagerTest. PiperOrigin-RevId: 158829077
* sandbox: Create more compact sandbox profiles faster.Gravatar Philipp Wollermann2017-05-15
| | | | | | | | By removing the now unnecessary call to Path#resolveSymbolicLinks we can save a few stat's per action execution. Change-Id: Iee157e941c1cd3515ff5ea3b7f410824c24cf44d PiperOrigin-RevId: 155946544
* Bring back --sandbox_block_path.Gravatar philwo2017-05-15
| | | | | | | | | | | | This is basically a rollback of https://github.com/bazelbuild/bazel/commit/3e2329a73ffd5d60e5e2babe60ebe5bf322c07da, except this solves the reason why the feature was removed in the first place. We now create the helper files necessary to make files unreadable in Linux in Bazel's Java code and manage their lifetime there. Request was filed by a user here: http://stackoverflow.com/questions/43849651/how-to-lock-down-the-bazel-filesystem-sandbox PiperOrigin-RevId: 155913246
* sandbox: Use the SpawnInputExpander everywhere and delete SpawnHelpers.Gravatar Philipp Wollermann2017-05-09
| | | | | | | | | This unifies our code to use just one standard implementation to get the entire expanded input files for a Spawn, including from Filesets and Runfiles. Change-Id: I1e286508adf0a9aeddf70934b010e6fcc144c4a7 PiperOrigin-RevId: 155497273
* sandbox: Switch Darwin sandbox to the SymlinkedExecRoot strategy.Gravatar Philipp Wollermann2017-05-09
| | | | | | | | | | | Hardlinks are problematic due to not working across filesystem boundaries and causing Bazel to do lots of I/O because it has to create a hardlink and a symlink for each input file. This improves performance of Bazel building itself by 10% on my system. Change-Id: I8acb77053de875160a046e38624735ed18375bed PiperOrigin-RevId: 155493583
* sandbox: Remove debug logging in DarwinSandboxedStrategy.Gravatar Philipp Wollermann2017-05-09
| | | | | Change-Id: Idc023f3a8c1c3b60d3f3f23a579a5eccb92d074d PiperOrigin-RevId: 155487527
* sandbox: Precache writable directories in DarwinSandboxedStrategy.Gravatar Philipp Wollermann2017-05-09
| | | | | Change-Id: I1522c364a157ee0a144ab83eca54e419142c03b1 PiperOrigin-RevId: 155484109
* sandbox: Remove special treatment for runUnderPath from Darwin sandbox.Gravatar Philipp Wollermann2017-05-09
| | | | | | | | There's no need to make it explicitly readable, because the entire host filesystem is readable anyway. Change-Id: I6a63cc93b600250c1c8828ef8d1c9d6133b671d7 PiperOrigin-RevId: 155477093
* sandbox: Use CommandEnvironment for more robust look-up of embedded binaries.Gravatar Philipp Wollermann2017-05-09
| | | | | Change-Id: I1bc1901ea7cd9a5b93c280ec0ff8ac0d10959a09 PiperOrigin-RevId: 155381163
* sandbox: Use absolute path to getconf.Gravatar Philipp Wollermann2017-05-08
| | | | | Change-Id: I43dfd979aee0c510ec18b479f2a6bd55562b3fc0 PiperOrigin-RevId: 155361450
* Add explicit names to the sandboxing strategies so that they can beGravatar philwo2017-04-24
| | | | | | | | | | | selected even if they're not the preferred one on a platform. Simplify the SandboxActionContextProvider and remove the warning about sandboxing being unsupported. With the ProcessWrapperSandboxedStrategy now being reliable enough and the strategies printing their real name in the UI, this is overall a better UX. PiperOrigin-RevId: 153825986
* No longer print a warning when a sandbox directory couldn't be deleted.Gravatar philwo2017-04-24
| | | | | | With the process-wrapper improvements and the additional deletion of the sandbox base in the SandboxModule in, this should be reliable enough. The warning was also not actionable for users and annoyed them, so let's get rid of it. PiperOrigin-RevId: 153823045
* sandbox: Add new flag --experimental_sandbox_base.Gravatar philwo2017-04-07
| | | | | | | | Usually, Bazel creates the sandbox directories underneath its output_base. With --experimental_sandbox_base you can specify a different parent directory for this, e.g. /dev/shm to run all sandboxed actions on a memory-backed filesystem. PiperOrigin-RevId: 152490815
* Refactor all ctor callsites of PathFragment to instead call a static ↵Gravatar nharmata2017-04-05
| | | | | | | | | | | | 'create' method. This paves the way for changing PathFragment to e.g. an abstract class with multiple subclasses. This way we can split out the windows-specific stuff into one of these concrete classes, making the code more readable and also saving memory (since the shallow heap size of the NonWindowsPathFragment subclass will hopefully be smaller than that of the current PathFragment). This also lets us pursue gc churn optimizations. We can now do interning in PathFragment#create and can also get rid of unnecessary intermediate PathFragment allocations. RELNOTES: None PiperOrigin-RevId: 152145768
* Automated g4 rollback of commit 1d9e1ac90197b1d3d7b137ba3c1ada67bb9ba31b.Gravatar hlopko2017-04-04
| | | | | | | | | | | | | | | | | | *** Reason for rollback *** Breaks //src/test/shell/integration:force_delete_output_test *** Original change description *** Symlink output directories to the correct directory name If the workspace directory is /path/to/my/proj and the name in the WORKSPACE file is "floop", this will symlink the output directories to output_base/execroot/floop instead of output_base/execroot/proj. More prep for #1262, fixes #1681. PiperOrigin-RevId: 152126545
* Symlink output directories to the correct directory nameGravatar kchodorow2017-03-31
| | | | | | | | | | If the workspace directory is /path/to/my/proj and the name in the WORKSPACE file is "floop", this will symlink the output directories to output_base/execroot/floop instead of output_base/execroot/proj. More prep for #1262, fixes #1681. PiperOrigin-RevId: 151712384
* sandbox: Show the actual running sandbox strategy for actions.Gravatar Philipp Wollermann2017-03-27
| | | | | | | | | | This allows us to see for example whether an action ran using the "process wrapper + symlink tree" sandbox or the real "PID and mount namespaces" Linux sandbox. -- PiperOrigin-RevId: 151165170 MOS_MIGRATED_REVID=151165170
* sandbox: Refactor sandbox strategies to use a common exec method.Gravatar Philipp Wollermann2017-03-27
| | | | | | -- PiperOrigin-RevId: 151130566 MOS_MIGRATED_REVID=151130566
* sandbox: Make /tmp and /dev/shm writable by default on Linux.Gravatar Philipp Wollermann2017-03-27
| | | | | | | | | | | | | | Also refactor the way we compute writable dirs so that they're computed only once per running action, not twice. Fixes #2056, fixes #1973, fixes #1460. RELNOTES: /tmp and /dev/shm are now writable by default inside the Linux sandbox. -- PiperOrigin-RevId: 151123543 MOS_MIGRATED_REVID=151123543
* sandbox: Remove the flag --sandbox_block_path.Gravatar Philipp Wollermann2017-03-24
| | | | | | | | | | | | | | | | | | It is in the way of optimizing the performance of the sandbox, because it requires us to create two helper files (an unreadable file and an unreadable directory) which are bind-mounted on top of paths specified via this flag. These two helper files were created on a tmpfs mounted by the sandbox until now, which ensured that they were automatically deleted on exit. However, mounting tmpfs on /dev/shm or /tmp causes issues like #2686 or #1882. By removing this flag, we can get rid of the two helper files, which means we can also remove the reliance on a "sandbox temp directory" completely in the next change. -- PiperOrigin-RevId: 151107496 MOS_MIGRATED_REVID=151107496
* Rationalize local resource acquisitionGravatar Ulf Adams2017-03-03
| | | | | | | | | | | | | | | | | | | | | | Move all local resource acquisition to where local execution actually happens. Don't attempt to acquire resources per action, but only for individual spawns. This significantly simplifies the code. The downside is that we don't account for action-level work anymore. In general, actions should not perform any process execution themselves, but always delegate such work to a SpawnStrategy implementation. This change makes sure that every Spawn has local resources set in a way that is consistent with the previous state. However, there are two actions - Fileset and FileWrite -, which are not spawns, and so we now don't limit their concurrent execution anymore. For Fileset, all work is done in a custom Fileset-specific thread pool, so this shouldn't be a problem. I'm not sure about FileWriteAction. -- PiperOrigin-RevId: 149012600 MOS_MIGRATED_REVID=149012600
* sandbox should create regular empty files, not symlink to /dev/null.Gravatar Philipp Wollermann2017-02-28
| | | | | | | | | | | | | | Adds a test based on our Python rules that makes sure that this actually fixes the issue. Thanks to @duggelz for the suggestion. It seems like our Python rules are the only place that actually provides an EmptyFilesSupplier to Runfiles, so there's probably no simpler way to test this behavior in an integration test. Fix #1458. Fix #2394. -- PiperOrigin-RevId: 148656193 MOS_MIGRATED_REVID=148656193
* Rollback of commit 4b73e972d909bcd533f2f9940f95a00b9b73bdde.Gravatar Dmitry Lomov2017-02-17
| | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Broke tests on CI: http://ci.bazel.io/job/bazel-tests/570/ *** Original change description *** Roll forward execroot change RELNOTES[INC]: Previously, an external repository would be symlinked into the execution root at execroot/local_repo/external/remote_repo. This changes it to be at execroot/remote_repo. This may break genrules/Skylark actions that hardcode execution root paths. If this causes breakages for you, ensure that genrules are using $(location :target) to access files and Skylark rules are using http://bazel.io/docs/skylark/lib/File.html's path, dirname, etc. functions. Cust... -- PiperOrigin-RevId: 147833177 MOS_MIGRATED_REVID=147833177
* Roll forward execroot changeGravatar Kristina Chodorow2017-02-16
| | | | | | | | | | | | | | | | | RELNOTES[INC]: Previously, an external repository would be symlinked into the execution root at execroot/local_repo/external/remote_repo. This changes it to be at execroot/remote_repo. This may break genrules/Skylark actions that hardcode execution root paths. If this causes breakages for you, ensure that genrules are using $(location :target) to access files and Skylark rules are using http://bazel.io/docs/skylark/lib/File.html's path, dirname, etc. functions. Custom crosstools that hardcode external/<repo> paths will have to be updated. Issue #1262. -- PiperOrigin-RevId: 147726370 MOS_MIGRATED_REVID=147726370
* Switch to RunfilesSuppliers for communicating runfilesGravatar Michajlo Matijkiw2017-01-30
| | | | | | | | | | | | | | | | | | | | | | | ActionSpawn/SpawnAction now deal exclusively in RunfilesSuppliers, manifests maps are no more. There is some lingering awkwardness, in particular: - Manifests still need to be tracked in some places, we can work out if this is still necessary on a case by case basis. - Skylark makes actions' runfiles available via 'resolve_command' where they are consumed by 'action'. I've updated the documentation, though the name isn't entirely accurate anymore. That being said these interfaces _are_ marked as experimental, so we _should_ be able to be flexible here. Overall, I think the benefits consolidating runfiles into suppliers, from both code cleanliness and performance perspectives (no longer needing to parse manifests), outweights the awkwardnesses. RELNOTES: resolve_command/action's input_manifest return/parameter is now list -- PiperOrigin-RevId: 145817429 MOS_MIGRATED_REVID=145817429
* sandbox: Make ~/Library/{Cache,Logs} writable on macOS.Gravatar Philipp Wollermann2016-12-16
| | | | | | | | Fixes #2231. -- PiperOrigin-RevId: 142266332 MOS_MIGRATED_REVID=142266332
* Gives warning instead of error when fail to delete sandbox directory.Gravatar Yue Gan2016-10-24
| | | | | -- MOS_MIGRATED_REVID=137012950
* sandbox: Replace the error-prone lazy cleanup of sandbox directories by a ↵Gravatar Philipp Wollermann2016-09-30
| | | | | | | | | simple synchronous cleanup. Tested with bazel building itself that this does not result in a performance degradation. -- MOS_MIGRATED_REVID=134766597
* sandbox: Fix an issue where an IOException happening during ↵Gravatar Philipp Wollermann2016-09-26
| | | | | | | SandboxExecRoot#copyOutputs could hide an earlier ExecException from SandboxRunner#run in the Darwin and Linux sandbox strategies. -- MOS_MIGRATED_REVID=134273806
* Add a SandboxedSpawnActionContext interface and implementations for our ↵Gravatar Philipp Wollermann2016-09-23
| | | | | | | sandboxed execution strategies. -- MOS_MIGRATED_REVID=134054610
* Rollback of commit 82d43279f93d95e4c41b4bc598a3cc05ddd1ae1a.Gravatar Laszlo Csomor2016-09-21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Breaks TensorFlow and other Bazel jobs on ci.bazel.io *** Original change description *** Change execution root for external repositories to be ../repo Some of the important aspect of this change: * Remote repos in the execution root are under output_base/execroot/repo_name, so the prefix is ../repo_name (to escape the local workspace name). * Package roots for external repos were previously "output_base/", they are now output_base/external/repo_name (which means source artifacts always have a relative path from their repository). * Outputs are under bazel-bin/external/repo_name/ (or similarly under genfiles). Note that this is a bit of a change from how this was implemented in the previous cl. Fixes #1262. RELNOTES[INC]: Previously, an external repository would be symlinked into the execution root at execroot/local_repo/external/remote_repo. This changes it to be at execroot/remote_repo. This may break genrules/Skylark actions that hardcode execution root paths. If this causes breakages for you, ensure that genrules are using $(location :target) to access files and Skylark rules are using http://bazel.io/docs/skylark/lib/File.html's path, dirname, etc. functions. Roll forward of bdfd58a. -- MOS_MIGRATED_REVID=133709658
* Add flag --nosandbox/--no_sandbox/--no-sandbox to disable sandbox.Gravatar Yue Gan2016-09-21
| | | | | -- MOS_MIGRATED_REVID=133697962
* Change execution root for external repositories to be ../repoGravatar Kristina Chodorow2016-09-20
| | | | | | | | | | | | | | | | | | | | | | | Some of the important aspect of this change: * Remote repos in the execution root are under output_base/execroot/repo_name, so the prefix is ../repo_name (to escape the local workspace name). * Package roots for external repos were previously "output_base/", they are now output_base/external/repo_name (which means source artifacts always have a relative path from their repository). * Outputs are under bazel-bin/external/repo_name/ (or similarly under genfiles). Note that this is a bit of a change from how this was implemented in the previous cl. Fixes #1262. RELNOTES[INC]: Previously, an external repository would be symlinked into the execution root at execroot/local_repo/external/remote_repo. This changes it to be at execroot/remote_repo. This may break genrules/Skylark actions that hardcode execution root paths. If this causes breakages for you, ensure that genrules are using $(location :target) to access files and Skylark rules are using http://bazel.io/docs/skylark/lib/File.html's path, dirname, etc. functions. Roll forward of bdfd58a. -- MOS_MIGRATED_REVID=133606309
* Set workspace name for sandbox dirsGravatar Kristina Chodorow2016-09-09
| | | | | | | | | | | This is required for #1262, as execution roots for external repos will be accessed via ../reponame under the exec root. I'm trying to break up the change into several small CLs. This should have no impact on the sandbox's behavior. -- MOS_MIGRATED_REVID=132671034
* sandbox: Extract the common methods that get the input files from a Spawn ↵Gravatar Philipp Wollermann2016-09-08
| | | | | | | into a shared helper class. -- MOS_MIGRATED_REVID=132546638
* Refactoring getTimeoutSeconds of a Spawn into Spaws.java, becaused it is usedGravatar Googler2016-09-08
| | | | | | | in places other than the sandbox code. -- MOS_MIGRATED_REVID=132436150
* Fix missing debug log lines from --sandbox_debug on OS X.Gravatar Philipp Wollermann2016-09-06
| | | | | -- MOS_MIGRATED_REVID=132051338
* Fix #1709: Print debug messages on OS X when --sandbox_debug is specified.Gravatar Philipp Wollermann2016-09-01
| | | | | -- MOS_MIGRATED_REVID=131941585
* fix run_under commandGravatar Yue Gan2016-09-01
| | | | | | | -- Change-Id: I765eaa6f0ecb31508eaf41f88be989f8e1169c51 Reviewed-on: https://bazel-review.googlesource.com/#/c/5711 MOS_MIGRATED_REVID=131934871
* Refactor our sandboxing code.Gravatar Philipp Wollermann2016-08-31
| | | | | -- MOS_MIGRATED_REVID=131817068
* modify sandbox script to allow defaultGravatar Yue Gan2016-08-29
| | | | | | | -- Change-Id: I48611500044e05c177d3a044d5d335c4a98135f4 Reviewed-on: https://bazel-review.googlesource.com/#/c/5530 MOS_MIGRATED_REVID=131593454
* sandbox: Allow network access for builds by default.Gravatar Philipp Wollermann2016-08-26
| | | | | | | | | | | | | This solves a performance issue that slowed down builds by about 40% at least on Linux, due to clone() with CLONE_NEWNET becoming extremely slow (>1 second) for highly parallel builds. See this thread for a discussion: https://lkml.org/lkml/2014/8/20/40 For the sake of consistency, we apply the same policy on OS X, too. If we find a better way to block network access for processes on Linux that doesn't have this performance hit, we will revisit this. RELNOTES: Sandboxed builds allow network access for builds by default. Tests will still be run without networking, unless "requires-network" is specified as a tag. -- MOS_MIGRATED_REVID=131393514
* Sandbox 2.0 for Mac OS X.Gravatar Yue Gan2016-08-08
-- Change-Id: Idf232f3dce3a3221d9a35c89dcef13437b0c25ba Reviewed-on: https://bazel-review.googlesource.com/#/c/3905/ MOS_MIGRATED_REVID=129620348
generated by cgit v1.2.3 (git 2.39.1) at 2024-08-03 09:50:21 +0000