aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/sandbox
Commit message (Collapse)AuthorAge
* Add easy access to MessageDigests in DigestHashFunctionGravatar ccalvarin2018-08-13
| | | | | | | Move the message-digest cloning to DigestHashFunction and out of Fingerprint, to make it possible to configure Fingerprint to use different hash functions. We keep the default MD5 for now, we'd like it to use the global default but want to isolate the configuration change from any change adding potential contention. RELNOTES: None. PiperOrigin-RevId: 208502993
* Docker sandbox: Pass client environment to "docker" when calling it.Gravatar Philipp Wollermann2018-08-02
| | | | | | | | Otherwise Docker will not be able to read the configuration of the current user and might fail to download containers from registries that require authentication. Closes #5721. PiperOrigin-RevId: 207094268
* Don't check if sandbox is supported over and over again.Gravatar twerth2018-07-06
| | | | | RELNOTES: None PiperOrigin-RevId: 203454741
* Add a bit more profiler coverageGravatar ulfjack2018-06-15
| | | | | | | | | | | | | | | | | | The intent is that the main thread has ~zero gaps in the profile (though there may still be small gaps due to the time between one try block and an immediately subsequent try block). We need to be careful not to wrap markPhase calls (or methods that call markPhase) in try blocks for the profiler - the Profiler requires that all markPhase calls happen at top level, and throws an exception if not. This should not have any performance impact - all of these are once per build, or at most once per module per build, and we don't expect a very large number of modules (and if we see an increasing number, we need to change the module API to not have to call every single module, but only those that are actually interested in certain events, maybe with an EventBus-based setup). PiperOrigin-RevId: 200712677
* Inline some ActionContextProvider classes into their modulesGravatar ulfjack2018-06-11
| | | | | | | | | | | | | | | This also gets rid of some boilerplate. The ExecutionTool.addActionContext method has been around for a while, but is underused. There are still a few ActionContextProvider implementations left, which are implementing other functionality besides adding action contexts. As a side effect, this change reduces null build time with a hot server on linux by about a quarter. We were running the linux sandbox twice on every build, which takes about 70ms each (on my machine), with the total null build time around 300ms. PiperOrigin-RevId: 200045145
* Remove ActionContextConsumerGravatar ulfjack2018-06-11
| | | | | | | | Instead, add some simple APIs to ExecutorBuilder and inline all the previous subclasses into their corresponding modules. This removes a bunch of boilerplate. PiperOrigin-RevId: 200017162
* Add option to enable Docker sandboxing.Gravatar lberki2018-06-06
| | | | | RELNOTES: None. PiperOrigin-RevId: 199467128
* uid related fixes in docker sandboxGravatar Siddhartha Bagaria2018-05-17
| | | | | | | | uid and gid of -1 (set when not linux) can not really be used anywhere. Closes #5165. PiperOrigin-RevId: 196974660
* Clean up code that directly imports nested classes like Builder, Entry, etc.Gravatar jcater2018-05-02
| | | | PiperOrigin-RevId: 195100670
* Add param file support to sandboxed spawn runners.Gravatar tomlu2018-05-01
| | | | | | | Spawns can contain param files in the form of virtual action inputs. Sandboxed runners must write these prior to execution of the action. RELNOTES: None PiperOrigin-RevId: 194951208
* Remove the on-disk caching of Xcode locations.Gravatar jmmv2018-04-28
| | | | | | | | | | | | | | | | | | | | | | | There is no need for the cache to be on disk. Originally, there was a desire to share this cache with other tools... but this never happened. And, actually, because Bazel is in control of what it runs, it can just inject the "cached" values into those tools via flags. Instead, just store the cache in-memory. This avoids having to open and read the cache on every single action that is locally executed on a Mac. Results when building a large iOS app from a clean slate show up to a 1% wall time improvement on my Mac Pro 2013 and a reduction in the variance of the measurements. This change also gets rid of the OS check from the action execution's critical path. There is not much use in checking this: if we instantiate this by mistake, the actual calls will fail. But sometimes we want to actually run this code on non-macOS systems (e.g. for unit-testing with mocked tools), so we should allow that. And this change also ensures that XcodeLocalEnvProviderTest builds and runs... RELNOTES: None. PiperOrigin-RevId: 194681802
* sandbox: Add support for running actions inside Docker containers.Gravatar philwo2018-04-27
| | | | | | | RELNOTES: Bazel now supports running actions inside Docker containers. To use this feature, run "bazel build --spawn_strategy=docker --experimental_docker_image=myimage:latest". PiperOrigin-RevId: 194582691
* sandbox: Generate actually unique sandbox paths.Gravatar philwo2018-04-25
| | | | | | | | | | | | | Turns out the "unique" id returned by SpawnExecutionContext is only unique within one SpawnRunner, but not across multiple SpawnRunner classes. We have to prefix the generated path with the name of the current strategy in order to not run into conflicts when a build happens to use multiple different sandbox strategies. RELNOTES: None. PiperOrigin-RevId: 194230475
* sandbox: Create base class for sandbox building strategies.Gravatar philwo2018-04-24
| | | | | | | | | The CopyingSandboxedSpawn will be used by the upcoming Docker sandbox. This is otherwise a no-op change. RELNOTES: None. PiperOrigin-RevId: 194096413
* Remove use of bare Immutable{List,Map,Set} Builder classes.Gravatar jcater2018-04-20
| | | | | | Always use the more-qualified class name for clarity at the site of use. There are too many classes named Builder. PiperOrigin-RevId: 193649193
* Rename SpawnExecutionPolicy -> SpawnExecutionContext.Gravatar tomlu2018-04-19
| | | | | | | | This class will be used to tie a Spawn to a SpawnRunner, and isn't really a policy object. It will carry state such as the expanded inputs and expanded command line. Currently a context can be passed between different SpawnRunners. This will be addressed independently, so a context is tied to a particular spawn runner. PiperOrigin-RevId: 193501918
* Roll forward of ↵Gravatar philwo2018-04-16
| | | | | | | | | | | | | | | | | | | | https://github.com/bazelbuild/bazel/commit/656a0bab1e025ff3c27d595284a4bf1c5a8d8028 with test (unknown commit) and fix. Big round of sandbox fixes / performance improvements. - The number of stat() syscalls in the SymlinkedSandboxedSpawn was way too high. Do less, feel better. - When using --experimental_sandbox_base, ensure that symlinks in the path are resolved. Before this, you had to check whether on your system /dev/shm is a symlink to /run/shm and then use that instead. Now it no longer matters, as symlinks are resolved. - Remove an unnecessary directory creation from each sandboxed invocation. Turns out that the "tmpdir" that we created was no longer used after some changes to Bazel's TMPDIR handling. - Use simpler sandbox paths, by using the unique ID for each Spawn provided by SpawnExecutionPolicy instead of a randomly generated temp folder name. This also saves a round-trip from our VFS to NIO and back. Clean up the sandbox base before each build to ensure that the unique IDs are actually unique. ;) - Use Java 8's Process#isAlive to check whether a process is alive instead of trying to get the exitcode and catching an exception. Closes #4913. PiperOrigin-RevId: 193031017
* Use a regular file when exposing empty files instead of /dev/null.Gravatar jmmv2018-04-13
| | | | | | | | | Exposing devices through a FUSE mount point is possible but accessing them requires the file system to be mounted -odev, which needs root privileges for security reasons. Because we cannot ask sandboxfs to be mounted this way, change our code to instead use a regular empty file when necessary. PiperOrigin-RevId: 192761352
* Make FileSystemUtils.moveFile always preserve symlinks and use it in ↵Gravatar Benjamin Peterson2018-04-12
| | | | | | | | | | | | | | | | | | | | | | | SandboxedSpawn.copyOutputs. Previously, if moveFile fell back to copying from a true rename (e.g., if the move is across file systems), it would fully dereference the source and produce a regular file at the output location. This CL fixes moveFile to properly copy symlinks. Note we don't preserve metadata in the symlink case mostly because the Bazel VFS has no API to write metadata without dereferencing symlinks. (But, also, it's not important for the current use cases.) This breaks the backward-compatibility of FileSystemUtils.moveFile and FileSystemUtils.moveTreeBelow. This seems okay because the new behavior makes more sense, and sandbox is the only consumer of these APIs. Switching SandboxedSpawn.copyOutputs to use FileSystemUtils.moveFile rather than Guava's Files.move fixes https://github.com/bazelbuild/bazel/issues/4987. Closes #4989. Change-Id: I0283e8aa11fadff5b0afd7bdfd0490aca12a1f6b PiperOrigin-RevId: 192611227
* Add support for sandboxfs-based sandboxing to Linux.Gravatar jmmv2018-04-10
| | | | | | | This is essentially the same as https://github.com/bazelbuild/bazel/commit/3a7b8bc2abeaf8b8647c037bed1dd5fd73b8392b. RELNOTES: none. PiperOrigin-RevId: 192342039
* Fix handling of relative symlinks within sandboxfs.Gravatar jmmv2018-04-10
| | | | | | | | | | | | | | | | | | If an action expresses a symlink as an input, the target of the symlink does not necessarily appear as a file to map within the sandbox. This is a problem when the target of the symlink is relative because sandboxfs would expose the link verbatim and the target would be missing later on during resolution. To fix this, special-case the handling of symlinks: when trying to expose them via a sandboxfs mount point, resolve their final target instead of respecting the original contents. This loses the fact that the file was a symlink when running within the sandboxfs sandbox, but is easier to implement and slightly faster at runtime. We can reconsider this choice if this causes problems. RELNOTES: None. PiperOrigin-RevId: 192325932
* Internal changeGravatar dannark2018-04-04
| | | | PiperOrigin-RevId: 191642942
* Remove categories from Bazel options.Gravatar ccalvarin2018-03-28
| | | | | | | | | These have all had a chance to be categorized with the OptionDocumentationCategory enum, and the help output already uses the enum-grouped format. The "incompatible changes" category has meaning for --all_incompatible_changes and will be removed separately. RELNOTES: None. PiperOrigin-RevId: 190773778
* Simplified ActionContextConsumer by having it operate on a new class which ↵Gravatar Googler2018-03-26
| | | | | | holds a variety of strategy/context maps. PiperOrigin-RevId: 190491357
* Big round of sandbox fixes / performance improvements.Gravatar Philipp Wollermann2018-03-26
| | | | | | | | | | | | | | | | - The number of stat() syscalls in the SymlinkedSandboxedSpawn was way too high. Do less, feel better. - When using --experimental_sandbox_base, ensure that symlinks in the path are resolved. Before this, you had to check whether on your system /dev/shm is a symlink to /run/shm and then use that instead. Now it no longer matters, as symlinks are resolved. - Remove an unnecessary directory creation from each sandboxed invocation. Turns out that the "tmpdir" that we created was no longer used after some changes to Bazel's TMPDIR handling. - Use simpler sandbox paths, by using the unique ID for each Spawn provided by SpawnExecutionPolicy instead of a randomly generated temp folder name. This also saves a round-trip from our VFS to NIO and back. Clean up the sandbox base before each build to ensure that the unique IDs are actually unique. ;) - Use Java 8's Process#isAlive to check whether a process is alive instead of trying to get the exitcode and catching an exception. Closes #4913. PiperOrigin-RevId: 190472170
* Refactor and cleanup the sandboxing code.Gravatar Philipp Wollermann2018-03-23
| | | | | | | | | | | | | - Remove Optional<> where it's not needed. It's nice for return values, but IMHO it was overused in this code (e.g. Optional<List<X>> is an anti-pattern, as the list itself can already signal that it is empty). - Use Bazel's own Path class when dealing with paths, not String or java.io.File. - Move LinuxSandboxUtil into the "sandbox" package. - Remove dead code and unused fields. - Migrate deprecated VFS method calls to their replacements. - Fix a bug in ExecutionStatistics where a FileInputStream was not closed. Closes #4868. PiperOrigin-RevId: 190217476
* Remove unnecessary product name and constructors from sandboxed spawn runners.Gravatar jmmv2018-03-23
| | | | | | | | | | The product name was previously propagated to all sandboxed spawn runners unconditionally, even when the majority did not need it. Also, these same spawn runners had obsolete constructors that were not used at all or were only used in tests. Remove both for simplicity. RELNOTES: None. PiperOrigin-RevId: 190195305
* Add support for sandboxfs-based sandboxing to macOS.Gravatar jmmv2018-03-22
| | | | | | | | | | | | | | | | This only modifies the DarwinSandboxedSpawnRunner for now and leaves behind the Linux version. The latter is a bit more complex to do because of the need to deal with tmpfs and bind-mount interactions. We expect sandboxfs to have the biggest impact on macOS anyway, not Linux, so this can wait until this has proven that it deserves staying for the long-term. Tested: Manually ran a self-build of Bazel on a version of Bazel built with these changes and the build passed. RELNOTES: None. PiperOrigin-RevId: 190133429
* Add a new SandboxfsSandboxedSpawn to spawn actions using sandboxfs.Gravatar jmmv2018-03-22
| | | | | RELNOTES: None. PiperOrigin-RevId: 190062172
* Plumb support for mounting a sandboxfs instance during a build.Gravatar jmmv2018-03-19
| | | | | | | | | | | | | This introduces user-facing options to enable the experimental sandboxfs support and, when enabled, mounts a sandboxfs instance throughout the build. The sandboxfs' process handle is passed to the SandboxActionContextProvider so that the SpawnRunners can later consume it. Note that this does NOT yet provide sandboxfs support for the builds as the SpawnRunners are untouched. RELNOTES: None. PiperOrigin-RevId: 189678732
* Add an interface to interact with sandboxfs.Gravatar jmmv2018-03-08
| | | | | | | | | | | | | | | | The new SandboxfsProcess interface allows interacting with sandboxfs. There are two implementations: RealSandboxfsProcess, which spawns the sandboxfs binary, and FakeSandboxfsProcess, which mimics what sandboxfs does but using symlinks and is intended for testing purposes only. The RealSandboxfsProcess implementation works but still carries many TODOs. The most "painful" one may be that the test requires manual invocation because we do not yet have an easy way to integrate with sandboxfs. That will be solved later on; for now this is sufficient for initial testing. RELNOTES: None. PiperOrigin-RevId: 188347393
* Replace instances of XCode with Xcode which is the proper casing for the name.Gravatar Sergio Campama2018-03-06
| | | | | | Closes #4640. PiperOrigin-RevId: 188022228
* Adding a property name to the SpawnRunner. Most runners already had it, I ↵Gravatar olaola2018-02-22
| | | | | | | | | | just add it to the interface, and include it in the SpawnResult. This will be used to categorize/aggregate spawns executed by various runners. Also, minor refinement to the cacheHit property of the SpawnResult with remote execution. RELNOTES: None TESTED=presubmit, next cl PiperOrigin-RevId: 186637978
* Inject the product name in the constructor where neededGravatar ulfjack2018-02-20
| | | | | | ...instead of passing it everywhere so we can rewrite the local env. PiperOrigin-RevId: 186316859
* Automated rollback of commit 04757dba0174d22c0a695a7ed5fe511fd13df008.Gravatar laszlocsomor2018-02-16
| | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** There's already a --test_tmpdir flag, and Java tests don't pick up this new one. More info: https://github.com/bazelbuild/bazel/issues/4621#issuecomment-366217321 *** Original change description *** tmpdir,local-exec: implement --local_tmp_root Add new flag called `--local_tmp_root`, which (if specified) tells Bazel what temp directory should locally executed actions use. Fixes https://github.com/bazelbuild/bazel/issues/4621 Related to https://github.com/bazelbuild/bazel/issues/3215 RELNOTES[NEW]: The new "--local_tmp_root=<path>" flag allows specifying the temp directory for locally executed actions. Change-Id: Ice69a5e63d0bf4d3b5c9ef4dbdd1ed1c5025f85e PiperOrigin-RevId: 185982705
* tmpdir,local-exec: implement --local_tmp_rootGravatar Laszlo Csomor2018-02-13
| | | | | | | | | | | | | | Add new flag called `--local_tmp_root`, which (if specified) tells Bazel what temp directory should locally executed actions use. Fixes https://github.com/bazelbuild/bazel/issues/4621 Related to https://github.com/bazelbuild/bazel/issues/3215 RELNOTES[NEW]: The new "--local_tmp_root=<path>" flag allows specifying the temp directory for locally executed actions. Change-Id: Ice69a5e63d0bf4d3b5c9ef4dbdd1ed1c5025f85e PiperOrigin-RevId: 185509555
* Update DarwinSandboxedSpawnRunner to correctly set the sandbox configuration ↵Gravatar ruperts2018-02-05
| | | | | | | rule for the execution statistics path. RELNOTES: None. PiperOrigin-RevId: 184534589
* actions,temp: respect TMPDIR envvarGravatar Laszlo Csomor2018-01-15
| | | | | | | Fixes https://github.com/bazelbuild/bazel/issues/4376 Change-Id: Id78bb0930044626304e54f07735db4d4b2c84720 PiperOrigin-RevId: 181959528
* Plumb exec root through to all spawn runners.Gravatar tomlu2018-01-11
| | | | | | They need this to parse input manifests. Previously we would grab the exec root from the Root, but wish to unsupport this. PiperOrigin-RevId: 181669143
* sandbox: add env[TMPDIR] instead of `tmpDir`Gravatar Laszlo Csomor2018-01-11
| | | | | | | | | | | | | | | | Bazel now adds env[TMPDIR] to the set of sandbox-writable paths, instead of adding the caller-defined `tmpDir` as it used to. Since every caller of getWritableDirs passes the LocalEnvProvider-processed environment to getWritableDirs, and because all such callers use either PosixLocalEnvProvider or XCodeLocalEnvProvider, we can be sure that the environment has an entry for TMPDIR. Change-Id: Ia89544a009e56d9cc922ab56823d16d20465545e PiperOrigin-RevId: 181595606
* sandbox: properly add `tmpDir` to `writablePaths`Gravatar Laszlo Csomor2018-01-11
| | | | | | | | | | | | | | | When Bazel creates the sandbox for an action, Bazel collects a set of paths that the action may write to. The action needs write access to its temp directory, so Bazel needs to add it to the writable paths. See https://github.com/bazelbuild/bazel/issues/4376 Change-Id: Ifd3c482aa67ff8a2070045356abad8b39c808db8 PiperOrigin-RevId: 181591520
* sandbox: allow adding some non-existent pathsGravatar Laszlo Csomor2018-01-09
| | | | | | | | | | | | | | | | | | | | | | | | | When Bazel creates the sandbox, it will allow making non-existent paths writable, as long as the path is under the sandbox root. As Bazel adds entries to the sandbox's set of writable paths, Bazel needs to make sure that it's not adding symlinks, because doing so would make the symlink writable, not what the link points to. If the path is under the sandbox root, then at the time of setting up the sandbox's writable paths the path surely doesn't exist yet, but that's OK, because at that time Bazel didn't yet create the sandbox root. If the path is not under the sandbox root, then Bazel needs to resolve all symlinks on this path, which is only possible if the path exists, therefore Bazel checks for the path's existence. Change-Id: Ic7d99a81905e7401455286c0b375d69b85ece1d5 PiperOrigin-RevId: 181325749
* Use EmptyActionInput instead of null in SpawnInputExpanderGravatar ulfjack2018-01-08
| | | | | | | | This simplifies some spawn runners, which no longer have to specially handle null; unfortunately, the sandbox runners do not support VirtualActionInput, so they still have to special-case it. PiperOrigin-RevId: 181175408
* sandbox runners: always use updated environmentGravatar Laszlo Csomor2018-01-08
| | | | | | | | | | | The sandboxed Spawn runners already used to call LocalEnvProvider.rewriteEnvironment in order to add TMPDIR for example, but didn't consistently use this rewritten environment everywhere, and used the Spawn's original environment instead. Change-Id: Ifbede6ac0f14b83c617dcbcd85edfae88718b157 PiperOrigin-RevId: 181130506
* sandbox: error out if $TEST_TMPDIR doesn't existGravatar Laszlo Csomor2018-01-05
| | | | | | | | | | | | | | | | | | | | | | | | | | | | As part of setting up a sandbox, Bazel creates the list of writable paths. If the action's environment defines $TEST_TMPDIR, then it's a test action and the sandbox must allow writing to that path, therefore Bazel must add $TEST_TMPDIR to the writable paths. Bazel must resolve symlinks in that path though, at least on the last path segment, because in case the path points to a symlink, the action would be allowed to modify the symlink itself, and not access what the link points to. However the path must exist for Bazel to successfully resolve symlinks, therefore this commit adds a check for that. Given that the code was there since at least July 2017, and I'm not aware of bugs caused by it, I conclude that this code path either never runs or nobody ever triggered it. Either way, adding the check is the right thing to do. Change-Id: I87a5d3fc3fe7878a918ed318c71e8d135f10f1b8 PiperOrigin-RevId: 180931382
* Extract LocalEnvProvider members to classes.Gravatar Laszlo Csomor2018-01-04
| | | | | | | | | | | | | | | | | | Create a PosixLocalEnvProvider and WindowsLocalEnvProvider class, with singleton instances for now. This refactoring should not change functionality, it's just a requirement for an upcoming change. That upcoming change is for these classes to respect the client environment's TMPDIR or TMP/TEMP envvars. See https://github.com/bazelbuild/bazel/issues/4376 Change-Id: I032bb6f18adf8af9e43e6bc543c09c58adae3863 PiperOrigin-RevId: 180799936
* Enable local action execution statistics collection for sandboxed actions ↵Gravatar ruperts2017-12-22
| | | | | | | | | | | that use the DarwinSandboxedSpawnRunner. In particular, record metrics for user and system CPU execution time, block I/O and involuntary context switches. This feature is guarded behind a new option, --experimental_collect_local_sandbox_action_metrics. RELNOTES: None. PiperOrigin-RevId: 179980734
* Enable local action execution statistics collection for sandboxed actions ↵Gravatar ruperts2017-12-22
| | | | | | | | | | | | | that use either the LinuxSandboxedSpawnRunner or the ProcessWrapperSandboxedSpawnRunner. In particular, record metrics for user and system CPU execution time, block I/O and involuntary context switches. This feature is guarded behind a new option, --experimental_collect_local_sandbox_action_metrics. Note: We still need to enable execution statistics for the DarwinSandboxedSpawnRunner in a later change. RELNOTES: None. PiperOrigin-RevId: 179976217
* Remove cached version of FileSystemUtils#createDirectoryAndParents.Gravatar tomlu2017-12-21
| | | | | | Only the sandbox uses the cached version. We can let it do its own caching of the top-level directory. It won't help with higher-level directories, but if there are a lot of inputs in the same directory we'll avoid I/O there. PiperOrigin-RevId: 179830651
* Move compulsory ProcessWrapperUtil parameters into its builder() method, to ↵Gravatar ruperts2017-12-20
| | | | | | | be consistent with LinuxSandboxUtil. RELNOTES: None. PiperOrigin-RevId: 179758847
generated by cgit v1.2.3 (git 2.39.1) at 2024-06-23 11:26:24 +0000