aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/sandbox
Commit message (Collapse)AuthorAge
* Add an interface to interact with sandboxfs.Gravatar jmmv2018-03-08
| | | | | | | | | | | | | | | | The new SandboxfsProcess interface allows interacting with sandboxfs. There are two implementations: RealSandboxfsProcess, which spawns the sandboxfs binary, and FakeSandboxfsProcess, which mimics what sandboxfs does but using symlinks and is intended for testing purposes only. The RealSandboxfsProcess implementation works but still carries many TODOs. The most "painful" one may be that the test requires manual invocation because we do not yet have an easy way to integrate with sandboxfs. That will be solved later on; for now this is sufficient for initial testing. RELNOTES: None. PiperOrigin-RevId: 188347393
* Replace instances of XCode with Xcode which is the proper casing for the name.Gravatar Sergio Campama2018-03-06
| | | | | | Closes #4640. PiperOrigin-RevId: 188022228
* Adding a property name to the SpawnRunner. Most runners already had it, I ↵Gravatar olaola2018-02-22
| | | | | | | | | | just add it to the interface, and include it in the SpawnResult. This will be used to categorize/aggregate spawns executed by various runners. Also, minor refinement to the cacheHit property of the SpawnResult with remote execution. RELNOTES: None TESTED=presubmit, next cl PiperOrigin-RevId: 186637978
* Inject the product name in the constructor where neededGravatar ulfjack2018-02-20
| | | | | | ...instead of passing it everywhere so we can rewrite the local env. PiperOrigin-RevId: 186316859
* Automated rollback of commit 04757dba0174d22c0a695a7ed5fe511fd13df008.Gravatar laszlocsomor2018-02-16
| | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** There's already a --test_tmpdir flag, and Java tests don't pick up this new one. More info: https://github.com/bazelbuild/bazel/issues/4621#issuecomment-366217321 *** Original change description *** tmpdir,local-exec: implement --local_tmp_root Add new flag called `--local_tmp_root`, which (if specified) tells Bazel what temp directory should locally executed actions use. Fixes https://github.com/bazelbuild/bazel/issues/4621 Related to https://github.com/bazelbuild/bazel/issues/3215 RELNOTES[NEW]: The new "--local_tmp_root=<path>" flag allows specifying the temp directory for locally executed actions. Change-Id: Ice69a5e63d0bf4d3b5c9ef4dbdd1ed1c5025f85e PiperOrigin-RevId: 185982705
* tmpdir,local-exec: implement --local_tmp_rootGravatar Laszlo Csomor2018-02-13
| | | | | | | | | | | | | | Add new flag called `--local_tmp_root`, which (if specified) tells Bazel what temp directory should locally executed actions use. Fixes https://github.com/bazelbuild/bazel/issues/4621 Related to https://github.com/bazelbuild/bazel/issues/3215 RELNOTES[NEW]: The new "--local_tmp_root=<path>" flag allows specifying the temp directory for locally executed actions. Change-Id: Ice69a5e63d0bf4d3b5c9ef4dbdd1ed1c5025f85e PiperOrigin-RevId: 185509555
* Update DarwinSandboxedSpawnRunner to correctly set the sandbox configuration ↵Gravatar ruperts2018-02-05
| | | | | | | rule for the execution statistics path. RELNOTES: None. PiperOrigin-RevId: 184534589
* actions,temp: respect TMPDIR envvarGravatar Laszlo Csomor2018-01-15
| | | | | | | Fixes https://github.com/bazelbuild/bazel/issues/4376 Change-Id: Id78bb0930044626304e54f07735db4d4b2c84720 PiperOrigin-RevId: 181959528
* Plumb exec root through to all spawn runners.Gravatar tomlu2018-01-11
| | | | | | They need this to parse input manifests. Previously we would grab the exec root from the Root, but wish to unsupport this. PiperOrigin-RevId: 181669143
* sandbox: add env[TMPDIR] instead of `tmpDir`Gravatar Laszlo Csomor2018-01-11
| | | | | | | | | | | | | | | | Bazel now adds env[TMPDIR] to the set of sandbox-writable paths, instead of adding the caller-defined `tmpDir` as it used to. Since every caller of getWritableDirs passes the LocalEnvProvider-processed environment to getWritableDirs, and because all such callers use either PosixLocalEnvProvider or XCodeLocalEnvProvider, we can be sure that the environment has an entry for TMPDIR. Change-Id: Ia89544a009e56d9cc922ab56823d16d20465545e PiperOrigin-RevId: 181595606
* sandbox: properly add `tmpDir` to `writablePaths`Gravatar Laszlo Csomor2018-01-11
| | | | | | | | | | | | | | | When Bazel creates the sandbox for an action, Bazel collects a set of paths that the action may write to. The action needs write access to its temp directory, so Bazel needs to add it to the writable paths. See https://github.com/bazelbuild/bazel/issues/4376 Change-Id: Ifd3c482aa67ff8a2070045356abad8b39c808db8 PiperOrigin-RevId: 181591520
* sandbox: allow adding some non-existent pathsGravatar Laszlo Csomor2018-01-09
| | | | | | | | | | | | | | | | | | | | | | | | | When Bazel creates the sandbox, it will allow making non-existent paths writable, as long as the path is under the sandbox root. As Bazel adds entries to the sandbox's set of writable paths, Bazel needs to make sure that it's not adding symlinks, because doing so would make the symlink writable, not what the link points to. If the path is under the sandbox root, then at the time of setting up the sandbox's writable paths the path surely doesn't exist yet, but that's OK, because at that time Bazel didn't yet create the sandbox root. If the path is not under the sandbox root, then Bazel needs to resolve all symlinks on this path, which is only possible if the path exists, therefore Bazel checks for the path's existence. Change-Id: Ic7d99a81905e7401455286c0b375d69b85ece1d5 PiperOrigin-RevId: 181325749
* Use EmptyActionInput instead of null in SpawnInputExpanderGravatar ulfjack2018-01-08
| | | | | | | | This simplifies some spawn runners, which no longer have to specially handle null; unfortunately, the sandbox runners do not support VirtualActionInput, so they still have to special-case it. PiperOrigin-RevId: 181175408
* sandbox runners: always use updated environmentGravatar Laszlo Csomor2018-01-08
| | | | | | | | | | | The sandboxed Spawn runners already used to call LocalEnvProvider.rewriteEnvironment in order to add TMPDIR for example, but didn't consistently use this rewritten environment everywhere, and used the Spawn's original environment instead. Change-Id: Ifbede6ac0f14b83c617dcbcd85edfae88718b157 PiperOrigin-RevId: 181130506
* sandbox: error out if $TEST_TMPDIR doesn't existGravatar Laszlo Csomor2018-01-05
| | | | | | | | | | | | | | | | | | | | | | | | | | | | As part of setting up a sandbox, Bazel creates the list of writable paths. If the action's environment defines $TEST_TMPDIR, then it's a test action and the sandbox must allow writing to that path, therefore Bazel must add $TEST_TMPDIR to the writable paths. Bazel must resolve symlinks in that path though, at least on the last path segment, because in case the path points to a symlink, the action would be allowed to modify the symlink itself, and not access what the link points to. However the path must exist for Bazel to successfully resolve symlinks, therefore this commit adds a check for that. Given that the code was there since at least July 2017, and I'm not aware of bugs caused by it, I conclude that this code path either never runs or nobody ever triggered it. Either way, adding the check is the right thing to do. Change-Id: I87a5d3fc3fe7878a918ed318c71e8d135f10f1b8 PiperOrigin-RevId: 180931382
* Extract LocalEnvProvider members to classes.Gravatar Laszlo Csomor2018-01-04
| | | | | | | | | | | | | | | | | | Create a PosixLocalEnvProvider and WindowsLocalEnvProvider class, with singleton instances for now. This refactoring should not change functionality, it's just a requirement for an upcoming change. That upcoming change is for these classes to respect the client environment's TMPDIR or TMP/TEMP envvars. See https://github.com/bazelbuild/bazel/issues/4376 Change-Id: I032bb6f18adf8af9e43e6bc543c09c58adae3863 PiperOrigin-RevId: 180799936
* Enable local action execution statistics collection for sandboxed actions ↵Gravatar ruperts2017-12-22
| | | | | | | | | | | that use the DarwinSandboxedSpawnRunner. In particular, record metrics for user and system CPU execution time, block I/O and involuntary context switches. This feature is guarded behind a new option, --experimental_collect_local_sandbox_action_metrics. RELNOTES: None. PiperOrigin-RevId: 179980734
* Enable local action execution statistics collection for sandboxed actions ↵Gravatar ruperts2017-12-22
| | | | | | | | | | | | | that use either the LinuxSandboxedSpawnRunner or the ProcessWrapperSandboxedSpawnRunner. In particular, record metrics for user and system CPU execution time, block I/O and involuntary context switches. This feature is guarded behind a new option, --experimental_collect_local_sandbox_action_metrics. Note: We still need to enable execution statistics for the DarwinSandboxedSpawnRunner in a later change. RELNOTES: None. PiperOrigin-RevId: 179976217
* Remove cached version of FileSystemUtils#createDirectoryAndParents.Gravatar tomlu2017-12-21
| | | | | | Only the sandbox uses the cached version. We can let it do its own caching of the top-level directory. It won't help with higher-level directories, but if there are a lot of inputs in the same directory we'll avoid I/O there. PiperOrigin-RevId: 179830651
* Move compulsory ProcessWrapperUtil parameters into its builder() method, to ↵Gravatar ruperts2017-12-20
| | | | | | | be consistent with LinuxSandboxUtil. RELNOTES: None. PiperOrigin-RevId: 179758847
* Use linux-sandbox via the (new) LinuxSandboxUtil. (second attempt).Gravatar ruperts2017-12-20
| | | | | RELNOTES: None. PiperOrigin-RevId: 179705357
* Automated rollback of commit 52b62164af031c50b7a0584303caad67af5e1d4d.Gravatar aehlig2017-12-20
| | | | | | | | | | | | | *** Reason for rollback *** Breaks //src/test/shell/bazel:bazel_sandboxing_test *** Original change description *** Use linux-sandbox via the (new) LinuxSandboxUtil. RELNOTES: None. PiperOrigin-RevId: 179676894
* Use linux-sandbox via the (new) LinuxSandboxUtil.Gravatar ruperts2017-12-20
| | | | | RELNOTES: None. PiperOrigin-RevId: 179646155
* Fix: uploading artifacts of failed actions to remote cache stopped working.Gravatar olaola2017-12-11
| | | | | | | | | | | | | | | | To reproduce: run a failing test with --experimental_remote_spawn_cache or with --spawn_strategy=remote and no executor. Expected: test log is uploaded. Desired behavior: - regardless of whether a spawn is cacheable or not, its artifacts should be uploaded to the remote cache. - the spawn result should only be set if the spawn is cacheable *and* the action succeeded. - when executing remotely, the do_not_cache field should be set for non-cacheable spawns, and the remote execution engine should respect it. This CL contains multiple fixes to ensure the above behaviors, and adds a few tests, both end to end and unit tests. Important behavior change: it is no longer assumed that non-cacheable spawns should use a NO_CACHE SpawnCache! The appropriate test case was removed. Instead, an assumption was added that all implementations of SpawnCache should respect the Spawns.mayBeCached(spawn) property. Currently, only NO_CACHE and RemoteSpawnCache exist, and they (now) support it. TESTED=remote build execution backend. WANT_LGTM: philwo,buchgr RELNOTES: None PiperOrigin-RevId: 178617937
* Add a CommandLineBuilder for the process-wrapper embedded tool, and use it ↵Gravatar ruperts2017-11-30
| | | | | | | | | everywhere instead of duplicating process-wrapper --shell_arguments in Blaze. To avoid a cyclic dependency, I broke up exec/local:local into exec/local:local and exec/local:options. RELNOTES: None. PiperOrigin-RevId: 177419268
* Add user and system time to CommandResults, and plumb them into SpawnResults.Gravatar ruperts2017-11-29
| | | | | RELNOTES: None PiperOrigin-RevId: 177290508
* Clean up ExecutionRequirementsGravatar ulfjack2017-11-29
| | | | | | | | | | | | | | | | | | | - remove BaseSpawn.Local; instead, all callers pass in the full set of execution requirements they want to set - disable caching and sandboxing for the symlink tree action - it does not declare outputs, so it can't be cached or sandboxed (fixes #4041) - centralize the existing execution requirements in the ExecutionRequirements class - centralize checking for execution requirements in the Spawn class (it's possible that we may need a more decentralized, extensible design in the future, but for now having them in a single place is simple and effective) - update the documentation - forward the relevant tags to execution requirements in TargetUtils (progress on #3960) - this also contributes to #4153 PiperOrigin-RevId: 177288598
* Remove process-wrapper duplication from RunCommand.Gravatar ruperts2017-11-28
| | | | | | | Also rename ProcessWrapperRunner to ProcessWrapperUtil, since it isn't really a Runner (it doesn't run anything). RELNOTES: None. PiperOrigin-RevId: 177260951
* Simplify SpawnRunner interfaceGravatar ulfjack2017-11-28
| | | | | | | | | | | | | | It turns out that the SUCCESS status is often misunderstood to mean "zero exit", even though this is clearly documented. I've decided to add another status for non-zero exit, and use success only for zero exit to avoid this pitfall. Also, many of the status codes are set, but never used. I decided to reduce the number of status codes to only those that are actually relevant, which simplifies further processing. Instead, we should add a string message for the error case when we need one - we're not using it right now, so I decided not to add that yet. PiperOrigin-RevId: 177129441
* Replace all usages of Blaze's Preconditions class with guava.Gravatar tomlu2017-11-09
| | | | | | | | Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own. This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually. PiperOrigin-RevId: 175033526
* Use Durations to store wall, user and system execution time in SpawnResults, ↵Gravatar ruperts2017-11-06
| | | | | | | and make cumulative execution times available in ActionResults. RELNOTES: None PiperOrigin-RevId: 174553272
* Thread FileSystem through to a single Path#delete call site.Gravatar tomlu2017-10-30
| | | | | | This requires a fairly large amount of changes to fundamental objects like BlazeRuntime, Executor, and so on, as well as changing a lot of test code to thread the file system through. I expect future CLs to be much smaller. PiperOrigin-RevId: 173678144
* Actions now have a temp envvar.Gravatar László Csomor2017-10-16
| | | | | | | | | | | | | | | | | | | | | Every build and test action that creates a Spawn will now have platform-specific environment variables for temp directories: - on Windows: TMP and TEMP - on Linux/Darwin: TMPDIR This is particularly important on Windows where e.g. Java programs cannot create temp directories unless there's a valid TMP or TEMP environment variable set. Fixes: - https://github.com/bazelbuild/bazel/issues/1590 - https://github.com/bazelbuild/bazel/issues/2349 - https://github.com/bazelbuild/bazel/issues/2870 Change-Id: Ib758307daf6b3a51b0f71ae5e65e5bb564dad643 PiperOrigin-RevId: 172326371
* Simplify the SpawnExecException constructorGravatar ulfjack2017-10-16
| | | | | | | Whether or not there was a catastrophic error is stored in the SpawnResult, so we can just use that instead of passing in an additional boolean. PiperOrigin-RevId: 172083752
* More SpawnResult-related plumbing changes to Actions, Strategies, ↵Gravatar ruperts2017-10-07
| | | | | | | | | ActionContexts, etc., so that SpawnResult metadata is returned upwards. Note that the TODOs mostly refer to changes that will appear in a subsequent CL (a CL to return SpawnResults, contained in ActionResults, from Actions/AbstractActions). I split off the remaining SpawnResult-related changes into this CL and kept the ActionResult-related changes separate. RELNOTES: None. PiperOrigin-RevId: 171355611
* Move SpawnResult from build.lib.exec into build.lib.actions so that e.g. ↵Gravatar ruperts2017-09-22
| | | | | | | build.lib.actions.SpawnActionContext can import SpawnResult without creating a cyclic dependency. RELNOTES: None. PiperOrigin-RevId: 169642267
* More BUILD file refactorings.Gravatar philwo2017-09-06
| | | | | | | | | Split collect, concurrent, vfs, windows into package-level BUILD files. Move clock classes out of "util", into their own Java package. Move CompactHashSet into its own Java package to break a dependency cycle. Give nestedset and inmemoryfs their own package-level BUILD files. PiperOrigin-RevId: 167702127
* Split "shell", "graph" into their own packages.Gravatar philwo2017-08-29
| | | | PiperOrigin-RevId: 166849610
* Support executing as root in the linux sandbox.Gravatar Benjamin Peterson2017-08-11
| | | | | | | | | | | | | | | linux-sandbox has a useful option -R, that runs the spawn as fake root. However, it's not exposed to Bazel rules. Here, we do that via the "requires-fakeroot" tag. One possible usecase: In combination with "block-network", "requires-fakeroot" makes it possible to integration test services that insist on listening on privileged ports. Unsurprisingly, this is incompatible with --sandbox_fake_username. Change-Id: I9e8ab4d4abf0e45626e005ff21f73e6c17de0788 PiperOrigin-RevId: 164961019
* AbstractSpawnStrategy: use ActionExecutionContext.getVerboseFailuresGravatar ulfjack2017-08-11
| | | | | | | ...instead of injecting it through the constructor. Simplify all the callers accordingly. PiperOrigin-RevId: 164955391
* Move core test classes to lib.analysis.testGravatar ulfjack2017-08-11
| | | | | | | | These are depended upon by analysis code, so need to live in the same library as lib.analysis. Moving them here makes it possible to split the build-base library into separate libraries for analysis, execution, and rules. PiperOrigin-RevId: 164847161
* Unify input prefetchingGravatar ulfjack2017-08-10
| | | | | | | All prefetching now goes through AbstractSpawnStrategy's implementation of SpawnExecutionPolicy. Make sure the sandbox runners also do this consistently. PiperOrigin-RevId: 164836877
* Fix --verbose_failures w/ sandboxing to print the full command lineGravatar ulfjack2017-08-10
| | | | | | Fixes #3521. PiperOrigin-RevId: 164728178
* Rewrite the Command APIGravatar ulfjack2017-08-10
| | | | | | | | | | | | | | | | | | | | | | | | Important: the simplified API now defaults to forwarding interrupts to subprocesses. I did audit all the call sites, and I think this is a safe change to make. - Properly support timeouts with all implementations - Simplify the API - only provide two flavours of blocking calls, which require no input and forward interrupts; this is the most common usage - provide a number of async calls, which optionally takes input, and a flag whether to forward interrupts - only support input streams, no byte arrays or other 'convenience features' that are rarely needed and unnecessarily increase the surface area - use java.time.Duration to specify timeout; for consistency, interpret a timeout of <= 0 as no timeout (i.e., including rather than excluding 0) - KillableObserver and subclasses are no longer part of the public API, but still used to implement timeouts if the Subprocess.Factory does not support them - Update the documentation for Command - Update all callers; most callers now use the simplified API PiperOrigin-RevId: 164716782
* Use java.time.Duration for timeoutsGravatar ulfjack2017-08-09
| | | | PiperOrigin-RevId: 164577062
* Move most test options from BuildConfiguration to TestConfiguration.Gravatar tomlu2017-08-07
| | | | | | --test_env isn't moved in this CL since it's exposed to Skylark via BuildConfiguration, making it a somewhat riskier refactor. PiperOrigin-RevId: 164266168
* Refactor BuildRequest out of sandboxed spawn runner initialization.Gravatar Benjamin Peterson2017-08-04
| | | | | | | | | | | | | Remove BuildRequest as parameter to sandboxed spawn runner constructors. Previously, the build request was used to obtain some options, but those can be extricated from a CommandEnvironment, which is passed in, too. Also, remove LinuxSandboxedSpawnRunner's aliased sandboxOptions member variable. It can just use the superclass's. Change-Id: I1ef1a45cbf7e800d0809f05673f097a148289740 PiperOrigin-RevId: 164257471
* sandbox: Allow UNIX sockets on macOS even when block-network is used.Gravatar Philipp Wollermann2017-07-25
| | | | | | Closes #3444. PiperOrigin-RevId: 163047183
* Extract a common AbstractSpawnStrategy parent classGravatar ulfjack2017-07-24
| | | | | | This removes a bunch of code duplication that I previously introduced. PiperOrigin-RevId: 162909430
* Move ActionInputPrefetcher to the actions packageGravatar ulfjack2017-07-21
| | | | | | The plan is to add it to ActionExecutionContext, which is also there. PiperOrigin-RevId: 162656835
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-04 13:46:08 +0000