aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/test/java/com/google/devtools/build/lib/sandbox
Commit message (Collapse)AuthorAge
* Automated rollback of commit 7e87730de985b7099b9b683571d58efdaab70890.Gravatar ccalvarin2018-07-27
| | | | | | | | | | | | | | | | | *** Reason for rollback *** Go back to the default constructor - instead of requiring everywhere to know the correct hash function, we'll have the default rely on global state. It will make transition easier, even if it makes the origin of the hash less obvious. *** Original change description *** Remove default MD5 in most of Bazel's virtual filesystems. This forces the ex-default to be explicit in a lot of tests, but I'd rather that than have the risk of implicit md5-use in production code. To keep this CL smaller, do not remove the default from UnixFS quite yet. RELNOTES: None. PiperOrigin-RevId: 206358838
* Remove default MD5 in most of Bazel's virtual filesystems.Gravatar ccalvarin2018-07-26
| | | | | | | | | This forces the ex-default to be explicit in a lot of tests, but I'd rather that than have the risk of implicit md5-use in production code. To keep this CL smaller, do not remove the default from UnixFS quite yet. RELNOTES: None. PiperOrigin-RevId: 206223521
* Bazel server: ensure OutputStreams are closedGravatar laszlocsomor2018-07-05
| | | | | | | | | | | | | | | | | | | Use try-with-resources to ensure OutputStreams that we open via FileSystem.OutputStream(path) are closed. Eagerly closing OutputStreams avoids hanging on to file handles until the garbage collector finalizes the OutputStream, meaning Bazel on Windows (and other processes) can delete or mutate these files. Hopefully this avoids intermittent file deletion errors that sometimes occur on Windows. See https://github.com/bazelbuild/bazel/issues/5512 RELNOTES: none PiperOrigin-RevId: 203342889
* Fix build and run of RealSandboxfsProcessTest.Gravatar jmmv2018-05-25
| | | | | | | | | | | | | | https://github.com/bazelbuild/bazel/commit/56d1b1c3122a3d1ec111baab339631b8c42c2c31 changed the signature of SandboxfsProcess#mount() to take a PathFragment instead of a Path, and this broke the build of the test above. Given that the test is marked as manual, this was not caught by CI. Further, the test was failing to link in the test runner class, so it would not execute properly. The missing dependency was incorrectly removed by unknown commit because it had been specified as a deps instead of a runtime_deps. RELNOTES: None. PiperOrigin-RevId: 198048748
* Roll forward of ↵Gravatar philwo2018-04-16
| | | | | | | | | | | | | | | | | | | | https://github.com/bazelbuild/bazel/commit/656a0bab1e025ff3c27d595284a4bf1c5a8d8028 with test (unknown commit) and fix. Big round of sandbox fixes / performance improvements. - The number of stat() syscalls in the SymlinkedSandboxedSpawn was way too high. Do less, feel better. - When using --experimental_sandbox_base, ensure that symlinks in the path are resolved. Before this, you had to check whether on your system /dev/shm is a symlink to /run/shm and then use that instead. Now it no longer matters, as symlinks are resolved. - Remove an unnecessary directory creation from each sandboxed invocation. Turns out that the "tmpdir" that we created was no longer used after some changes to Bazel's TMPDIR handling. - Use simpler sandbox paths, by using the unique ID for each Spawn provided by SpawnExecutionPolicy instead of a randomly generated temp folder name. This also saves a round-trip from our VFS to NIO and back. Clean up the sandbox base before each build to ensure that the unique IDs are actually unique. ;) - Use Java 8's Process#isAlive to check whether a process is alive instead of trying to get the exitcode and catching an exception. Closes #4913. PiperOrigin-RevId: 193031017
* Fix handling of relative symlinks within sandboxfs.Gravatar jmmv2018-04-10
| | | | | | | | | | | | | | | | | | If an action expresses a symlink as an input, the target of the symlink does not necessarily appear as a file to map within the sandbox. This is a problem when the target of the symlink is relative because sandboxfs would expose the link verbatim and the target would be missing later on during resolution. To fix this, special-case the handling of symlinks: when trying to expose them via a sandboxfs mount point, resolve their final target instead of respecting the original contents. This loses the fact that the file was a symlink when running within the sandboxfs sandbox, but is easier to implement and slightly faster at runtime. We can reconsider this choice if this causes problems. RELNOTES: None. PiperOrigin-RevId: 192325932
* Internal changeGravatar dannark2018-04-04
| | | | PiperOrigin-RevId: 191642942
* Big round of sandbox fixes / performance improvements.Gravatar Philipp Wollermann2018-03-26
| | | | | | | | | | | | | | | | - The number of stat() syscalls in the SymlinkedSandboxedSpawn was way too high. Do less, feel better. - When using --experimental_sandbox_base, ensure that symlinks in the path are resolved. Before this, you had to check whether on your system /dev/shm is a symlink to /run/shm and then use that instead. Now it no longer matters, as symlinks are resolved. - Remove an unnecessary directory creation from each sandboxed invocation. Turns out that the "tmpdir" that we created was no longer used after some changes to Bazel's TMPDIR handling. - Use simpler sandbox paths, by using the unique ID for each Spawn provided by SpawnExecutionPolicy instead of a randomly generated temp folder name. This also saves a round-trip from our VFS to NIO and back. Clean up the sandbox base before each build to ensure that the unique IDs are actually unique. ;) - Use Java 8's Process#isAlive to check whether a process is alive instead of trying to get the exitcode and catching an exception. Closes #4913. PiperOrigin-RevId: 190472170
* Refactor and cleanup the sandboxing code.Gravatar Philipp Wollermann2018-03-23
| | | | | | | | | | | | | - Remove Optional<> where it's not needed. It's nice for return values, but IMHO it was overused in this code (e.g. Optional<List<X>> is an anti-pattern, as the list itself can already signal that it is empty). - Use Bazel's own Path class when dealing with paths, not String or java.io.File. - Move LinuxSandboxUtil into the "sandbox" package. - Remove dead code and unused fields. - Migrate deprecated VFS method calls to their replacements. - Fix a bug in ExecutionStatistics where a FileInputStream was not closed. Closes #4868. PiperOrigin-RevId: 190217476
* Add a new SandboxfsSandboxedSpawn to spawn actions using sandboxfs.Gravatar jmmv2018-03-22
| | | | | RELNOTES: None. PiperOrigin-RevId: 190062172
* Add an interface to interact with sandboxfs.Gravatar jmmv2018-03-08
| | | | | | | | | | | | | | | | The new SandboxfsProcess interface allows interacting with sandboxfs. There are two implementations: RealSandboxfsProcess, which spawns the sandboxfs binary, and FakeSandboxfsProcess, which mimics what sandboxfs does but using symlinks and is intended for testing purposes only. The RealSandboxfsProcess implementation works but still carries many TODOs. The most "painful" one may be that the test requires manual invocation because we do not yet have an easy way to integrate with sandboxfs. That will be solved later on; for now this is sufficient for initial testing. RELNOTES: None. PiperOrigin-RevId: 188347393
* Fix sandboxing when spawn outputs differ from action outputs.Gravatar Benjamin Peterson2017-07-19
| | | | | | | | | | | | TestRunner actions are special because their action outputs are different from their spawn outputs. If there's a spawn output that's not an action output, SymlinkExecroot can't rely on the parent directories for that output existing in the real execroot. Thus, copyOutputs() must ensure the real execroot has the relevant ancestral directories before copying the output over. Change-Id: I84fd69cd51628c51de9c8993b6a4407bbff038a0 PiperOrigin-RevId: 162470058
* Rewrite all the sandbox strategy implementationsGravatar ulfjack2017-07-12
| | | | | | | | | | | | | | - Make use of existing abstractions like SpawnRunner and SpawnExecutionPolicy. - Instead of having the *Strategy create a *Runner, and then call back into SandboxStrategy, create a single SandboxContainer which contains the full command line, environment, and everything needed to create and delete the sandbox directory. - Do all the work in SandboxStrategy, including creation and deletion of the sandbox directory. - Use SpawnResult instead of throwing, catching, and rethrowing. - Simplify the control flow a bit. PiperOrigin-RevId: 161644979
* Migrate Java tests to Truth.Gravatar lberki2017-05-30
| | | | | | RELNOTES: None. PiperOrigin-RevId: 157446717
* Fixes incorrectly-ordered arguments to calls to assertEqualsGravatar Googler2017-05-19
| | | | | | | | | | | | | | | [] This change has been automatically generated by an Error Prone check that detects incorrect argument ordering on calls to assertEquals-style methods. See [] Cleanup change automatically generated by javacflume/refactory Refactoring: third_party/java_src/error_prone/project/core/src/main/java/com/google/errorprone/bugpatterns/argumentselectiondefects:AssertEqualsArgumentOrderChecker_refactoring Tested: TAP --sample for global presubmit queue [] PiperOrigin-RevId: 156539781
* sandbox: Use the SpawnInputExpander everywhere and delete SpawnHelpers.Gravatar Philipp Wollermann2017-05-09
| | | | | | | | | This unifies our code to use just one standard implementation to get the entire expanded input files for a Spawn, including from Filesets and Runfiles. Change-Id: I1e286508adf0a9aeddf70934b010e6fcc144c4a7 PiperOrigin-RevId: 155497273
* Refactor all ctor callsites of PathFragment to instead call a static ↵Gravatar nharmata2017-04-05
| | | | | | | | | | | | 'create' method. This paves the way for changing PathFragment to e.g. an abstract class with multiple subclasses. This way we can split out the windows-specific stuff into one of these concrete classes, making the code more readable and also saving memory (since the shallow heap size of the NonWindowsPathFragment subclass will hopefully be smaller than that of the current PathFragment). This also lets us pursue gc churn optimizations. We can now do interning in PathFragment#create and can also get rid of unnecessary intermediate PathFragment allocations. RELNOTES: None PiperOrigin-RevId: 152145768
* sandbox should create regular empty files, not symlink to /dev/null.Gravatar Philipp Wollermann2017-02-28
| | | | | | | | | | | | | | Adds a test based on our Python rules that makes sure that this actually fixes the issue. Thanks to @duggelz for the suggestion. It seems like our Python rules are the only place that actually provides an EmptyFilesSupplier to Runfiles, so there's probably no simpler way to test this behavior in an integration test. Fix #1458. Fix #2394. -- PiperOrigin-RevId: 148656193 MOS_MIGRATED_REVID=148656193
* Add customized path mounting in Bazel sandbox.Gravatar Xin Gao2016-12-20
| | | | | | | | | | RELNOTES: New flag --sandbox_add_mount_pair to specify customized source:target path pairs to bind mount inside the sandbox. -- Change-Id: Ifbacfc0e16bbaedcf5b6d3937799710f2cfa3d58 Reviewed-on: https://cr.bazel.build/7150 PiperOrigin-RevId: 142542381 MOS_MIGRATED_REVID=142542381
* Add support for sandboxing persistent workers.Gravatar Philipp Wollermann2016-09-28
| | | | | -- MOS_MIGRATED_REVID=134523222
* sandbox: Extract the common methods that get the input files from a Spawn ↵Gravatar Philipp Wollermann2016-09-08
| | | | | | | into a shared helper class. -- MOS_MIGRATED_REVID=132546638
* sandbox: Allow network access for builds by default.Gravatar Philipp Wollermann2016-08-26
| | | | | | | | | | | | | This solves a performance issue that slowed down builds by about 40% at least on Linux, due to clone() with CLONE_NEWNET becoming extremely slow (>1 second) for highly parallel builds. See this thread for a discussion: https://lkml.org/lkml/2014/8/20/40 For the sake of consistency, we apply the same policy on OS X, too. If we find a better way to block network access for processes on Linux that doesn't have this performance hit, we will revisit this. RELNOTES: Sandboxed builds allow network access for builds by default. Tests will still be run without networking, unless "requires-network" is specified as a tag. -- MOS_MIGRATED_REVID=131393514
* Add client environment to ActionExecutionContextGravatar Klaus Aehlig2016-08-26
| | | | | | | | | | | As the execution of an action now also depends on the client environment, make the latter part of the ActionExecutionContext, so that enough context is provided to actually execute an action. -- Change-Id: Ida7bf407ef0c0375728faba92494bfd47dcbaeb8 Reviewed-on: https://bazel-review.googlesource.com/#/c/5391 MOS_MIGRATED_REVID=131377490
* Implement the first stage of Bazel's "Sandbox 2.0" for Linux.Gravatar Philipp Wollermann2016-08-18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This has the following improvements upon the older one: - Uses PID namespaces, PR_SET_PDEATHSIG and a number of other tricks for further process isolation and 100% reliable killing of child processes. - Uses clone() instead of unshare() to work around a Linux kernel bug that made creating a sandbox unreliable. - Instead of mounting a hardcoded list of paths + whatever you add with --sandbox_add_path, this sandbox instead mounts all of /, except for what you make inaccessible via --sandbox_block_path. This should solve the majority of "Sandboxing breaks my build, because my compiler is installed in /opt or /usr/local" issues that users have seen. - Instead of doing magic with bind mounts, we create a separate execroot for each process containing symlinks to the input files. This is simpler and gives more predictable performance. - Actually makes everything except the working directory read-only (fixes #1364). This means that a running process can no longer accidentally modify your source code (yay!). - Prevents a number of additional "attacks" or leaks, like accidentally inheriting file handles from the parent. - Simpler command-line interface. - We can provide the same semantics in a Mac OS X sandbox, which will come in a separate code review from yueg@. It has the following caveats / known issues: - The "fallback to /bin/bash on error" feature is gone, but now that the sandbox mounts everything by default, the main use-case for this is no longer needed. The following improvements are planned: - Use a FUSE filesystem if possible for the new execroot, instead of creating symlinks. - Mount a base image instead of "/". FAQ: Q: Why is mounting all of "/" okay, doesn't this make the whole sandbox useless? A: This is still a reasonable behavior, because the sandbox never tried to isolate your build from the operating system it runs in. Instead it is supposed to protect your data from a test running "rm -rf $HOME" and to make it difficult / impossible for actions to use input files that are not declared dependencies. For even more isolation the sandbox will support mounting a base image as its root in a future version (similar to Docker images). Q: Let's say my process-specific execroot contains a symlink to an input file "good.h", can't the process just resolve the symlink, strip off the file name and then look around in the workspace? A: Yes. Unfortunately we could not find any way on Linux to make a file appear in a different directory with *all* of the semantics we would like. The options investigated were: 1) Copying input files, which is much too slow. 2) Hard linking input files, which is fast, but doesn't work cross- filesystems and it's also not possible to make them read-only. 3) Bind mounts, which don't scale once you're up in the thousands of input files (across all actions) - it seems like the kernel has some non-linear performance behavior when the mount table grows too much, resulting in the mount syscall taking more time the more mounts you have. 4) FUSE filesystem, good in theory, but wasn't ready for the first iteration. RELNOTES: New sandboxing implementation for Linux in which all actions run in a separate execroot that contains input files as symlinks back to the originals in the workspace. The running action now has read-write access to its execroot and /tmp only and can no longer write in arbitrary other places in the file system. -- Change-Id: Ic91386fc92f8eef727ed6d22e6bd0f357d145063 Reviewed-on: https://bazel-review.googlesource.com/#/c/4053 MOS_MIGRATED_REVID=130638204
* Rename namespace-sandbox/-runner to linux-sandbox.Gravatar Philipp Wollermann2016-07-18
| | | | | -- MOS_MIGRATED_REVID=127538990
* Remove uses of TestConstants.Gravatar Ulf Adams2016-07-14
| | | | | -- MOS_MIGRATED_REVID=127331186
* Rollback of commit 1e37a5375f918376c132fa537e25695f673f41b8.Gravatar Dmitry Lomov2016-07-07
| | | | | | | | | | | | | *** Reason for rollback *** Apparently we now try to open output files for the process twice: once when we are constructing the output streams, and the second time when we tell the process to redirect its outputs. This causes the outputs to be empty on Windows *** Original change description *** Do redirection of stdout / stderr in Java instead of reimplementing it in every process wrapper again. -- MOS_MIGRATED_REVID=126801016
* Do redirection of stdout / stderr in Java instead of reimplementing it in ↵Gravatar Philipp Wollermann2016-06-30
| | | | | | | every process wrapper again. -- MOS_MIGRATED_REVID=126279021
* Fixed two Bazel java tests on Windows by using the right native file systemGravatar Yun Peng2016-06-21
| | | | | | | | | | | | | | Newly passing: //src/test/java/com/google/devtools/build/... lib/skyframe:SkyframeTests lib:actions_test Also refactored FileSystems.java -- Change-Id: I03ab9db5c1ab5e5be4ff1efbc5cf2d280084254a Reviewed-on: https://bazel-review.googlesource.com/#/c/3843 MOS_MIGRATED_REVID=125449456
* sandbox: Remove debug flags from LinuxSandboxedStrategyTestCase as they were ↵Gravatar Philipp Wollermann2016-06-21
| | | | | | | causing LocalLinuxSandboxedStrategyTest to fail due to unexpected lines in the output. -- MOS_MIGRATED_REVID=125439352
* sandbox: So refactoring, much no-op change, wow.Gravatar Philipp Wollermann2016-06-21
| | | | | | | (But this will make it easier to try out the overlayfs-based sandbox later.) -- MOS_MIGRATED_REVID=125320914
* Replace the occurrences of Constants.PRODUCT_NAME for a call toGravatar Luis Fernando Pino Duque2016-05-23
| | | | | | | | | BlazeRuntime#getProductName() or a reference to TestConstants.PRODUCT_NAME for tests. This CL prepares the codebase in order to delete the constant. -- MOS_MIGRATED_REVID=122993568
* Allows network in sandbox, when --java_debug. Fixes #393.Gravatar Yue Gan2016-04-28
| | | | | -- MOS_MIGRATED_REVID=121002371
* Split ActionMetadata into ActionAnalysisMetadata and ActionExecutionMetadata.Gravatar Rumou Duan2016-04-26
| | | | | | | Except in action execution logic (ActionExecutionFunction, SkyframeActionExecutor, etc.), switch Action interface references to either ActionAnalysisMetadata if possible or ActionExecutionMetadata. -- MOS_MIGRATED_REVID=120723431
* Automated [] rollback of commit 525fa71b0d6f096e9bfb180f688a4418c4974eb4.Gravatar Yue Gan2016-04-14
| | | | | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Contributor finds some bugs and after fixing some bugs there are more bugs to fix now. *** Original change description *** Mount whole directories into the sandbox when possible This halves the overhead with sandboxing enabled vs disabled for a test that basically only mounts a bunch of files out of a directory, and slows that same test with a single extra file added to the directory (but not mounted) by only ~4%. The test is <https://gist.github.com/bsilver8192/10527a862ce16bb7f79a> with 30000 inputs moved to a subdirectory and on... *** ROLLBACK_OF=119138157 -- MOS_MIGRATED_REVID=119828267
* Mount whole directories into the sandbox when possibleGravatar Brian Silverman2016-04-07
| | | | | | | | | | | | | | | | | | | | This halves the overhead with sandboxing enabled vs disabled for a test that basically only mounts a bunch of files out of a directory, and slows that same test with a single extra file added to the directory (but not mounted) by only ~4%. The test is <https://gist.github.com/bsilver8192/10527a862ce16bb7f79a> with 30000 inputs moved to a subdirectory and only 10 genrules. This change means symlinks will be mounted directly as their target rather than as a symlink, but this solves some weird behavior with multi-level symlinks and will only break things which don't declare all of their dependencies. -- Change-Id: I1aa39dccb2e5fca2893bdab9065ee043d34019b2 Reviewed-on: https://bazel-review.googlesource.com/#/c/3220/ MOS_MIGRATED_REVID=119138157
* sandbox:Gravatar Yue Gan2016-03-17
| | | | | | | | | | | | - add flag --sandbox_add_path, which takes a list of additional paths as argument and mount these paths to sandbox. Fixes #884. - mount target of /etc/resolv.conf if it is a symlink. Fixes #738. RELNOTES: - add flag --sandbox_add_path, which takes a list of additional paths as argument and mount these paths to sandbox. - mount target of /etc/resolv.conf if it is a symlink. -- MOS_MIGRATED_REVID=117364211
* Removes "startsWith" check in testGravatar Yue Gan2016-02-19
| | | | | | | | | | | | com.google.devtools.build.lib.sandbox.LocalLinuxSandboxedStrategyTest.testExecutionFailurePrintsCorrectMessage. We modified the error message of sandbox failure, and new error message (commandLineArgs) is not able to get in test. Fixes #939. -- MOS_MIGRATED_REVID=115058938
* Reduce the number of stat calls when setting up a sandboxGravatar Brian Silverman2016-02-16
| | | | | | | | | | | | This improved performance for a (somewhat artificial) test which runs 100 genrules each with 3000 inputs by 25% on my laptop (2x hyperthreaded cores, SSD, ext4). Test code at <https://gist.github.com/bsilver8192/10527a862ce16bb7f79a>. -- Change-Id: I7a7aaccdfbe2925c7e962c0192924ef1cf80b33a Reviewed-on: https://bazel-review.git.corp.google.com/#/c/2840/1..2 MOS_MIGRATED_REVID=114694334
* Fix up exception declarations; use EnvironmentalExecException.Gravatar Ulf Adams2015-12-04
| | | | | -- MOS_MIGRATED_REVID=109404922
* Migrated remaining tests in devtools/build/lib to JUnit 4.Gravatar Florian Weikert2015-12-03
| | | | | -- MOS_MIGRATED_REVID=109287267
* Add support for filesets in LinuxSandboxedStrategy.Gravatar Philipp Wollermann2015-10-09
| | | | | -- MOS_MIGRATED_REVID=105052078
* sandbox: Improve MountMap to always return entries sorted by path depth and ↵Gravatar Philipp Wollermann2015-10-06
| | | | | | | lexicographical order. This prevents certain edge cases in the sandbox, where a mounted child directory could be hidden by a later mount of a parent directory over its parent. -- MOS_MIGRATED_REVID=104749937
* Ensure that the LinuxSandboxedStrategy tests actually test the correct strategy.Gravatar Philipp Wollermann2015-10-06
| | | | | -- MOS_MIGRATED_REVID=104749361
* Add --sandbox_debug.Gravatar Han-Wen Nienhuys2015-09-30
| | | | | -- MOS_MIGRATED_REVID=104108613
* Rationalize copyright headersGravatar Damien Martin-Guillerez2015-09-25
| | | | | | | | | | | The headers were modified with `find . -type f -exec 'sed' '-Ei' 's|Copyright 201([45]) Google|Copyright 201\1 The Bazel Authors|' '{}' ';'` And manual edit for not Google owned copyright. Because of the nature of ijar, I did not modified the header of file owned by Alan Donovan. The list of authors were extracted from the git log. It is missing older Google contributors that can be added on-demand. -- MOS_MIGRATED_REVID=103938715
* Extract MountMap to a top-level class.Gravatar Ulf Adams2015-09-21
| | | | | | | -- Change-Id: I26cf10accaa6f62014c65f41637a36fbeab42b0a Reviewed-on: https://github.com/bazelbuild/bazel/pull/465 MOS_MIGRATED_REVID=103529462
* sandbox: Show user-friendly mount paths in the namespace-sandbox debug log.Gravatar Philipp Wollermann2015-09-08
| | | | | | | Improvement for #424. -- MOS_MIGRATED_REVID=102566748
* sandbox: Better error messages and the noisy debug logs of the ↵Gravatar Philipp Wollermann2015-09-08
| | | | | | | | | namespace-runner now have to be explicitly activated via --sandbox_debug. Fixes #424. -- MOS_MIGRATED_REVID=102566625
* sandbox: Use a better data structure for the mounts.Gravatar Philipp Wollermann2015-09-04
| | | | | -- MOS_MIGRATED_REVID=102330179
generated by cgit v1.2.3 (git 2.39.1) at 2024-06-26 23:22:46 +0000