aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/vfs/FileSystemUtils.java
Commit message (Collapse)AuthorAge
* Use a structured Exception type for the "unexpected short read" error ↵Gravatar nharmata2018-07-25
| | | | | | | detected by FileSystemUtils#readWithKnownFileSize. RELNOTES: None PiperOrigin-RevId: 205995852
* vfs: do not close streams twiceGravatar Laszlo Csomor2018-07-20
| | | | | | | | | | | | | | ByteStream's functions already close the underlying stream. See https://github.com/bazelbuild/bazel/commit/09d20311d982606093ed881d779bb05a5ee70ed3 Change-Id: Id389ef594946bfebb90ca66d97ea96f271b20331 Closes #5641. Change-Id: Id389ef594946bfebb90ca66d97ea96f271b20331 PiperOrigin-RevId: 205395805
* Bazel server, VFS: revert to using asByteSourceGravatar laszlocsomor2018-07-10
| | | | | | | | | | | | | | | | | | commit 59f17d6e0550bf63a0b6ef182e2d63474e058ede updated files to use try-with-resources when dealing with streams. The change also got rid of asByteSource, replacing it with throw-away ByteSource instances wrapping a try-with-resources-guarded InputStream. Doing so was unnecessary though, because all ByteSource methods (except for open[Buffered]InputStream) close the stream. Thanks to @jbduncan to point that out and explain in detail [1]. [1] see comment thread on https://github.com/bazelbuild/bazel/commit/59f17d6e0550bf63a0b6ef182e2d63474e058ede under `FilesetManifest.java` RELNOTES: none PiperOrigin-RevId: 203934830
* Bazel server: ensure OutputStreams are closedGravatar laszlocsomor2018-07-05
| | | | | | | | | | | | | | | | | | | Use try-with-resources to ensure OutputStreams that we open via FileSystem.OutputStream(path) are closed. Eagerly closing OutputStreams avoids hanging on to file handles until the garbage collector finalizes the OutputStream, meaning Bazel on Windows (and other processes) can delete or mutate these files. Hopefully this avoids intermittent file deletion errors that sometimes occur on Windows. See https://github.com/bazelbuild/bazel/issues/5512 RELNOTES: none PiperOrigin-RevId: 203342889
* Bazel server: ensure InputStreams are closedGravatar laszlocsomor2018-07-05
| | | | | | | | | | | | | | | | | | | Use try-with-resources to ensure InputStreams that we open via FileSystem.InputStream(path) are closed. Eagerly closing InputStreams avoids hanging on to file handles until the garbage collector finalizes the InputStream, meaning Bazel on Windows (and other processes) can delete or mutate these files. Hopefully this avoids intermittent file deletion errors that sometimes occur on Windows. See https://github.com/bazelbuild/bazel/issues/5512 RELNOTES: none PiperOrigin-RevId: 203338148
* Make FileSystemUtils.moveFile always preserve symlinks and use it in ↵Gravatar Benjamin Peterson2018-04-12
| | | | | | | | | | | | | | | | | | | | | | | SandboxedSpawn.copyOutputs. Previously, if moveFile fell back to copying from a true rename (e.g., if the move is across file systems), it would fully dereference the source and produce a regular file at the output location. This CL fixes moveFile to properly copy symlinks. Note we don't preserve metadata in the symlink case mostly because the Bazel VFS has no API to write metadata without dereferencing symlinks. (But, also, it's not important for the current use cases.) This breaks the backward-compatibility of FileSystemUtils.moveFile and FileSystemUtils.moveTreeBelow. This seems okay because the new behavior makes more sense, and sandbox is the only consumer of these APIs. Switching SandboxedSpawn.copyOutputs to use FileSystemUtils.moveFile rather than Guava's Files.move fixes https://github.com/bazelbuild/bazel/issues/4987. Closes #4989. Change-Id: I0283e8aa11fadff5b0afd7bdfd0490aca12a1f6b PiperOrigin-RevId: 192611227
* Replace path implementation.Gravatar tomlu2018-02-08
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Path and PathFragment have been replaced with String-based implementations. They are pretty similar, but each method is dissimilar enough that I did not feel sharing code was appropriate. A summary of changes: PATH ==== * Subsumes LocalPath (deleted, its tests repurposed) * Use a simple string to back Path * Path instances are no longer interned; Reference equality will no longer work * Always normalized (same as before) * Some operations will now be slower, like instance compares (which were previously just a reference check) * Multiple identical paths will now consume more memory since they are not interned PATH FRAGMENT ============= * Use a simple string to back PathFragment * No more segment arrays with interned strings * Always normalized * Remove isNormalized * Replace some isNormalizied uses with containsUpLevelReferences() to check if path fragments try to escape their scope * To check if user input is normalized, supply static methods on PathFragment to validate the string before constructing a PathFragment * Because PathFragments are always normalized, we have to replace checks for literal "." from PathFragment#getPathString to PathFragment#getSafePathString. The latter returns "." for the empty string. * The previous implementation supported efficient segment semantics (segment count, iterating over segments). This is now expensive since we do longer have a segment array. ARTIFACT ======== * Remove Path instance. It is instead dynamically constructed on request. This is necessary to avoid this CL becoming a memory regression. RELNOTES: None PiperOrigin-RevId: 185062932
* Automated rollback of commit cb7689404ef9a69488d64bfc2e0bfb9326e664d6.Gravatar tomlu2018-01-09
| | | | | | | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Rolling forward after the underlying issue has been fixed. *** Original change description *** Automated rollback of commit 6a54339bb943702bd7dffc43f85267dac98dc355. *** Reason for rollback *** b/71442447 *** Original change description *** Call through to Path#createDirectoryAndParents from FileUtils. This CL removes a method that due to its implementation causes threading difficulties for Path#createDirectory. The tests for the method are brought across to FileSystemTests since the methods are now implemented natively by the FileSystem classes. The tests were also cleaned up. The test revealed an edge c... *** PiperOrigin-RevId: 181367850
* Automated rollback of commit 6a54339bb943702bd7dffc43f85267dac98dc355.Gravatar tomlu2018-01-05
| | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** b/71442447 *** Original change description *** Call through to Path#createDirectoryAndParents from FileUtils. This CL removes a method that due to its implementation causes threading difficulties for Path#createDirectory. The tests for the method are brought across to FileSystemTests since the methods are now implemented natively by the FileSystem classes. The tests were also cleaned up. The test revealed an edge case bug in JavaIoFileSystem, so fix this. In two cases some code was using the return value from the old method. Returning "f... *** ROLLBACK_OF=179864042 PiperOrigin-RevId: 180946251
* Call through to Path#createDirectoryAndParents from FileUtils.Gravatar tomlu2017-12-21
| | | | | | | | | | | | This CL removes a method that due to its implementation causes threading difficulties for Path#createDirectory. The tests for the method are brought across to FileSystemTests since the methods are now implemented natively by the FileSystem classes. The tests were also cleaned up. The test revealed an edge case bug in JavaIoFileSystem, so fix this. In two cases some code was using the return value from the old method. Returning "false" essentially means that the directory already existed, and the code doesn't look racy, so we can replace it with an existence check. PiperOrigin-RevId: 179864042
* Remove cached version of FileSystemUtils#createDirectoryAndParents.Gravatar tomlu2017-12-21
| | | | | | Only the sandbox uses the cached version. We can let it do its own caching of the top-level directory. It won't help with higher-level directories, but if there are a lot of inputs in the same directory we'll avoid I/O there. PiperOrigin-RevId: 179830651
* Automated rollback of commit 82e68b75304438c96ff878a0c2b8d18b42002486.Gravatar aehlig2017-12-19
| | | | | | | | | | | | | | Fixes #4322, #4306. *** Reason for rollback *** Introduces a deadlock (see https://github.com/bazelbuild/bazel/issues/4322) *** Original change description *** Make FileSystem operate on LocalPath instead of Path. PiperOrigin-RevId: 179549866
* Make FileSystem operate on LocalPath instead of Path.Gravatar tomlu2017-12-14
| | | | PiperOrigin-RevId: 179082062
* Remove some unused extension points.Gravatar Googler2017-11-29
| | | | | RELNOTES: None. PiperOrigin-RevId: 177326265
* Replace all usages of Blaze's Preconditions class with guava.Gravatar tomlu2017-11-09
| | | | | | | | Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own. This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually. PiperOrigin-RevId: 175033526
* Add follow-symlinks parameter to FileSystemUtils#copyTreesBelow, so that all ↵Gravatar janakr2017-10-12
| | | | | | the tool setup we do in tests doesn't necessarily have to be copied if we copy a workspace over. PiperOrigin-RevId: 171864170
* Automated conversion to Java 8Gravatar laurentlb2017-06-30
| | | | | | | With a few manual fixes for readability. RELNOTES: None. PiperOrigin-RevId: 160582556
* Refactor all ctor callsites of PathFragment to instead call a static ↵Gravatar nharmata2017-04-05
| | | | | | | | | | | | 'create' method. This paves the way for changing PathFragment to e.g. an abstract class with multiple subclasses. This way we can split out the windows-specific stuff into one of these concrete classes, making the code more readable and also saving memory (since the shallow heap size of the NonWindowsPathFragment subclass will hopefully be smaller than that of the current PathFragment). This also lets us pursue gc churn optimizations. We can now do interning in PathFragment#create and can also get rid of unnecessary intermediate PathFragment allocations. RELNOTES: None PiperOrigin-RevId: 152145768
* Move UnixFileSystem to lib.unix, WindowsFileSystem to lib.windowsGravatar Ulf Adams2017-02-28
| | | | | | -- PiperOrigin-RevId: 148749485 MOS_MIGRATED_REVID=148749485
* Do not tickle TimestampGranularityMonitor for stable-status.txt no-op updates.Gravatar Julio Merino2016-11-03
| | | | | | | | | | | | | | | | | When rewriting stable-status.txt, which happens on each build, avoid updating the file's ctime and mtime if the new contents match what is already in the file. This prevents tickling the TimestampGranularityMonitor for what should be a no-op update, which in turn could cause null/incremental builds to stall for up to a second. The problem was magnified on macOS where the default HFS+ file system only has second-level granularity. (This also affects Linux, but because current Linux file systems have milli/nanosecond-level granularity, the wait imposed by TimestampGranularityMonitor is minimal and thus not generally noticeable.) -- MOS_MIGRATED_REVID=137983794
* Fixed symbolic link and hard link path not stripped when "strip_prefix" is set.Gravatar Xin Gao2016-09-23
| | | | | -- MOS_MIGRATED_REVID=134005484
* Rollback of commit 4e2f75d8e60020f13a7df452488ec4cc7a719f03.Gravatar Kristina Chodorow2016-09-22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** This is causing external_integration_test to fail See, for example, http://ci.bazel.io/job/bazel-tests/BAZEL_VERSION=HEAD,PLATFORM_NAME=linux-x86_64/236/console: ** test_http_archive_tar_xz **************************************************** GET /fox.tar.xz HTTP/1.1 User-Agent: Java/1.8.0_101 Host: localhost:36541 Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2 Connection: keep-alive cat: bazel-out/../external/endangered/fox/male_absolute: No such file or directory -- Test log: ----------------------------------------------------------- INFO: Reading 'startup' options from /home/ci/.cache/bazel/_bazel_ci/a761298a0949227106f79c62c3bebb6e/bazel-sandbox/81f51af6-eb50-417e-8f8d-b7bba207ee83-661/execroot/linux-x86_64/_tmp/external_integration_test_2/bazelrc: --output_user_root=/home/ci/.cache/bazel/_bazel_ci/a761298a0949227106f79c62c3bebb6e/bazel-sandbox/81f51af6-eb50-417e-8f8d-b7bba207ee83-661/execroot/linux-x86_64/_tmp/external_integration_test_2/root --host_javabase=/home/ci/.cache/bazel/_bazel_ci/a761298a0949227106f79c62c3bebb6e/bazel-sandbox/81f51af6-eb50-417e-8f8d-b7bba207ee83-661/execroot/linux-x86_64/bazel-out/local-fastbuild/bin/src/test/shell/bazel/external_integration_test.runfiles/local_jdk INFO: $TEST_TMPDIR defined: output root default is '/home/ci/.cache/bazel/_bazel_ci/a761298a0949227106f79c62c3bebb6e/bazel-sandbox/81f51af6-eb50-417e-8f8d-b7bba207ee83-661/execroot/linux-x86_64/_tmp/external_integration_test_2'. .............................................. ____Loading package: zoo ____Loading... ____Loading package: @bazel_tools//tools/cpp ____Loading package: @bazel_tools//tools/jdk ____Loading package: @local_config_xcode// ____Loading package: @local_jdk// ____Loading package: @local_config_cc// ____Loading complete. Analyzing... ____Downloading from [] 0B ____Downloading from [] 1KB ____Loading package: @endangered//fox ____Found 1 target... ____Building... ____[0 / 1] BazelWorkspaceStatusAction stable-status.txt ____[0 / 4] Creating source manifest for //zoo:breeding-program ____[0 / 4] Symlinking //zoo:breeding-program ____[3 / 4] Creating runfiles tree bazel-out/local-fastbuild/bin/zoo/breeding-program.runfiles ____Building complete. Target //zoo:breeding-program up-to-date: bazel-bin/zoo/breeding-program ____Elapsed time: 6.903s, Critical Path: 0.04s ____Running command line: bazel-bin/zoo/breeding-program Fraka-kaka-kaka-kaka-kow ------------------------------------------------------------------------ test_http_archive_tar_xz FAILED: Expected regexp #!/bin/bash echo Fraka-kaka-kaka-kaka-kow not found in bazel-out/../external/endangered/fox/male_absolute . /home/ci/.cache/bazel/_bazel_ci/a761298a0949227106f79c62c3bebb6e/bazel-sandbox/81f51af6-eb50-417e-8f8d-b7bba207ee83-661/execroot/linux-x86_64/bazel-out/local-fastbuild/bin/src/test/shell/bazel/external_integration_test.runfiles/io_bazel/src/test/shell/bazel/external_integration_test:154: in call to assert_files_same /home/ci/.cache/bazel/_bazel_ci/a761298a0949227106f79c62c3bebb6e/bazel-sandbox/81f51af6-eb50-417e-8f8d-b7bba207ee83-661/execroot/linux-x86_64/bazel-out/local-fastbuild/bin/src/test/shell/bazel/external_integration_test.runfiles/io_bazel/src/test/shell/bazel/external_integration_test:150: in call to http_archive_helper /home/ci/.cache/bazel/_bazel_ci/a761298a0949227106f79c62c3bebb6e/bazel-sandbox/81f51af6-eb50-417e-8f8d-b7bba207ee83-661/execroot/linux-x86_64/bazel-out/local-fastbuild/bin/src/test/shell/bazel/external_integration_test.runfiles/io_bazel/src/test/shell/bazel/external_integration_test:190: in call to test_http_archive_tar_xz FAILED: test_http_archive_tar_xz *** Original change description *** Fixed symbolic link and hard link path not stripped when "strip_prefix" is set. -- MOS_MIGRATED_REVID=133970692
* Fixed symbolic link and hard link path not stripped when "strip_prefix" is set.Gravatar Xin Gao2016-09-20
| | | | | -- MOS_MIGRATED_REVID=133628392
* Also copy directories as output for sandbox.Gravatar Yue Gan2016-09-19
| | | | | -- MOS_MIGRATED_REVID=133564429
* Fixed the issue that hard links are handled improperly when bazel ↵Gravatar Googler2016-09-08
| | | | | | | | | decompresses tarballs. Issue link: https://github.com/bazelbuild/bazel/issues/574 -- MOS_MIGRATED_REVID=132434278
* Implement the first stage of Bazel's "Sandbox 2.0" for Linux.Gravatar Philipp Wollermann2016-08-18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This has the following improvements upon the older one: - Uses PID namespaces, PR_SET_PDEATHSIG and a number of other tricks for further process isolation and 100% reliable killing of child processes. - Uses clone() instead of unshare() to work around a Linux kernel bug that made creating a sandbox unreliable. - Instead of mounting a hardcoded list of paths + whatever you add with --sandbox_add_path, this sandbox instead mounts all of /, except for what you make inaccessible via --sandbox_block_path. This should solve the majority of "Sandboxing breaks my build, because my compiler is installed in /opt or /usr/local" issues that users have seen. - Instead of doing magic with bind mounts, we create a separate execroot for each process containing symlinks to the input files. This is simpler and gives more predictable performance. - Actually makes everything except the working directory read-only (fixes #1364). This means that a running process can no longer accidentally modify your source code (yay!). - Prevents a number of additional "attacks" or leaks, like accidentally inheriting file handles from the parent. - Simpler command-line interface. - We can provide the same semantics in a Mac OS X sandbox, which will come in a separate code review from yueg@. It has the following caveats / known issues: - The "fallback to /bin/bash on error" feature is gone, but now that the sandbox mounts everything by default, the main use-case for this is no longer needed. The following improvements are planned: - Use a FUSE filesystem if possible for the new execroot, instead of creating symlinks. - Mount a base image instead of "/". FAQ: Q: Why is mounting all of "/" okay, doesn't this make the whole sandbox useless? A: This is still a reasonable behavior, because the sandbox never tried to isolate your build from the operating system it runs in. Instead it is supposed to protect your data from a test running "rm -rf $HOME" and to make it difficult / impossible for actions to use input files that are not declared dependencies. For even more isolation the sandbox will support mounting a base image as its root in a future version (similar to Docker images). Q: Let's say my process-specific execroot contains a symlink to an input file "good.h", can't the process just resolve the symlink, strip off the file name and then look around in the workspace? A: Yes. Unfortunately we could not find any way on Linux to make a file appear in a different directory with *all* of the semantics we would like. The options investigated were: 1) Copying input files, which is much too slow. 2) Hard linking input files, which is fast, but doesn't work cross- filesystems and it's also not possible to make them read-only. 3) Bind mounts, which don't scale once you're up in the thousands of input files (across all actions) - it seems like the kernel has some non-linear performance behavior when the mount table grows too much, resulting in the mount syscall taking more time the more mounts you have. 4) FUSE filesystem, good in theory, but wasn't ready for the first iteration. RELNOTES: New sandboxing implementation for Linux in which all actions run in a separate execroot that contains input files as symlinks back to the originals in the workspace. The running action now has read-write access to its execroot and /tmp only and can no longer write in arbitrary other places in the file system. -- Change-Id: Ic91386fc92f8eef727ed6d22e6bd0f357d145063 Reviewed-on: https://bazel-review.googlesource.com/#/c/4053 MOS_MIGRATED_REVID=130638204
* Move execroot tree creation to its own classGravatar Kristina Chodorow2016-06-21
| | | | | | | Part 1 of many for #1262, rolling forward. -- MOS_MIGRATED_REVID=125334954
* Rollback of commit bdfd58a8ca2ed5735d6aaa5b238fb0f689515724.Gravatar Laurent Le Brun2016-06-20
| | | | | -- MOS_MIGRATED_REVID=125160288
* Make the execution root match the runfiles tree structure for external ↵Gravatar Kristina Chodorow2016-06-17
| | | | | | | | | | | | | | | | | | | | | repositories One interesting side effect of how this is implemented is that for external repositories, bin/ and genfiles/ are combined. External repo output is under bazel-out/local-fastbuild/repo_name for each repo. Fixes #1262. RELNOTES[INC]: Previously, an external repository would be symlinked into the execution root at execroot/local_repo/external/remote_repo. This changes it to be at execroot/remote_repo. This may break genrules/Skylark actions that hardcode execution root paths. If this causes breakages for you, ensure that genrules are using $(location :target) to access files and Skylark rules are using http://bazel.io/docs/skylark/lib/File.html's path, dirname, etc. functions. -- MOS_MIGRATED_REVID=125095799
* Replace the occurrences of Constants.PRODUCT_NAME for a call toGravatar Luis Fernando Pino Duque2016-05-23
| | | | | | | | | BlazeRuntime#getProductName() or a reference to TestConstants.PRODUCT_NAME for tests. This CL prepares the codebase in order to delete the constant. -- MOS_MIGRATED_REVID=122993568
* When creating the exec root symlink forest, take account of the empty ↵Gravatar Janak Ramakrishnan2016-05-03
| | | | | | | | | package's package root when creating top-level directories. Otherwise, if the empty package references files in those top-level directories, they would be hidden. Fixes #1221 -- MOS_MIGRATED_REVID=121392128
* Publicize some ByteSink/ByteSource creators in FileSystemUtils.Gravatar Eric Fellheimer2016-02-11
| | | | | -- MOS_MIGRATED_REVID=114434668
* Make local_repository and new_local_repository work on Windows.Gravatar Lukacs Berki2016-02-11
| | | | | | | This makes it possible to compile //third_party/ijar with a bootstrapped Bazel on Windows in dslomov's tree. -- MOS_MIGRATED_REVID=114428109
* cache SDKROOT value for local execution of actions using apple toolkitGravatar Chris Parsons2015-12-15
| | | | | -- MOS_MIGRATED_REVID=110174447
* Use Bazel Preconditions variant which avoids varargs array creationGravatar Mark Schaller2015-12-10
| | | | | | | Reduces garbage. -- MOS_MIGRATED_REVID=109914243
* Avoid copying file contents when reading the expected number of bytes ↵Gravatar Eric Fellheimer2015-11-04
| | | | | | | in#readContentWithLimit(). -- MOS_MIGRATED_REVID=106995917
* Change the preprocessor interface to take the byte[] contents of the BUILD ↵Gravatar Nathan Harmata2015-10-21
| | | | | | | | | file rather than a ParserInputSource. This is part of a series of changes with the net result being that we open, read, and parse each BUILD file exactly once. -- MOS_MIGRATED_REVID=105911557
* Rationalize copyright headersGravatar Damien Martin-Guillerez2015-09-25
| | | | | | | | | | | The headers were modified with `find . -type f -exec 'sed' '-Ei' 's|Copyright 201([45]) Google|Copyright 201\1 The Bazel Authors|' '{}' ';'` And manual edit for not Google owned copyright. Because of the nature of ijar, I did not modified the header of file owned by Alan Donovan. The list of authors were extracted from the git log. It is missing older Google contributors that can be added on-demand. -- MOS_MIGRATED_REVID=103938715
* Introduce Path#readSymbolicLinkUnchecked, intended to only be used when the ↵Gravatar Nathan Harmata2015-09-17
| | | | | | | caller already knows the path is a symlink, and use this new method throughout the codebase. -- MOS_MIGRATED_REVID=103229983
* Make FileSystemUtils#asByteSource() private againGravatar Michajlo Matijkiw2015-09-01
| | | | | -- MOS_MIGRATED_REVID=101976891
* Expose the #asByteSource() method in FileSystemUtils.Gravatar Eric Fellheimer2015-08-27
| | | | | -- MOS_MIGRATED_REVID=101628787
* TemplateExpansionAction now consistently uses UTF-8 instead of mixing UTF-8 ↵Gravatar Florian Weikert2015-08-04
| | | | | | | with Latin-1 -- MOS_MIGRATED_REVID=99651466
* Tolerate "..." as the base filename for the fdo profile.Gravatar Janak Ramakrishnan2015-07-10
| | | | | | | This is fragile, and there might be other places in the codebase this issue occurs. -- MOS_MIGRATED_REVID=97784977
* Allow BUILD files directly under the build rootGravatar Kristina Chodorow2015-03-10
| | | | | | | | | This makes the empty package name legal (//:foo). If the empty package is used, this symlinks everything under the build root to the exec root. This includes directories. -- MOS_MIGRATED_REVID=87960882
* Some cleanup changes.Gravatar Ulf Adams2015-03-05
| | | | | -- MOS_MIGRATED_REVID=87821306
* Update from Google.Gravatar Han-Wen Nienhuys2015-02-25
-- MOE_MIGRATED_REVID=85702957