aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/vfs
Commit message (Collapse)AuthorAge
* For objc rules, generated headers are mandatory inputs to CppCompileAction.Gravatar cpeyser2017-05-26
| | | | PiperOrigin-RevId: 157218175
* Remove stale comment and add back a removed TODO.Gravatar nharmata2017-04-28
| | | | | RELNOTES: None PiperOrigin-RevId: 154543320
* Make PathFragment an abstract class.Gravatar nharmata2017-04-24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There are now four concrete implementations: RelativeUnixPathFragment, AbsoluteUnixPathFragment, RelativeWindowsPathFragment, AbsoluteWindowsPathFragment. Goals: -Reduce memory usage of PathFragment on non-Windows platforms while maintaining existing semantics. -Make a few simple performance improvements along the way. -Add TODOs for a few more simple performance improvements. -Open the way for reducing code complexity of PathFragment. All of the Windows-specific stuff ought not complicate the code so much. Non goals: -Make the entire codebase as pretty as possible wrt PathFragment & Windows. -Make PathFragment usage more sane in general (e.g. change semantics to ban coexistence of Windows and Unix PathFragments). -Optimize PathFragment as much as possible wrt memory or even in any other dimensions (e.g. gc churn, cpu). To elaborate, the primary motivation is per-instance memory usage of PathFragment on Unix platforms: Before this change ------------------ +UseCompressedOops --> 32 bytes per instance -UseCompressedOops --> 40 bytes per instance After this change ------------------ +UseCompressedOops --> 24 bytes per instance -UseCompressedOops --> 32 bytes per instance Since Bazel can retain lots of PathFragments, the memory savings of this CL are fairly large. RELNOTES: None PiperOrigin-RevId: 154052905
* Migrate UnixGlob to Path#statIfFound() instead of #statNullable(). The ↵Gravatar felly2017-04-18
| | | | | | | | | latter swallows all filesystem failures, and does not disambiguate missing files from filesystem problems. The syscall cache now tracks IOExceptions if they are present, just as it does with readdir(). RELNOTES: None PiperOrigin-RevId: 153185433
* fix: Allow parentheses in glob patternsGravatar Harmandeep Kaur2017-04-11
| | | | | | | More info here #2583 Closes #2679. PiperOrigin-RevId: 152685327
* Update stale Windows path comment.Gravatar ccalvarin2017-04-11
| | | | PiperOrigin-RevId: 152684266
* Remove the last remaining public ctor from PathFragment.Gravatar nharmata2017-04-07
| | | | | RELNOTES: None PiperOrigin-RevId: 152400979
* Refactor all ctor callsites of PathFragment to instead call a static ↵Gravatar nharmata2017-04-05
| | | | | | | | | | | | 'create' method. This paves the way for changing PathFragment to e.g. an abstract class with multiple subclasses. This way we can split out the windows-specific stuff into one of these concrete classes, making the code more readable and also saving memory (since the shallow heap size of the NonWindowsPathFragment subclass will hopefully be smaller than that of the current PathFragment). This also lets us pursue gc churn optimizations. We can now do interning in PathFragment#create and can also get rid of unnecessary intermediate PathFragment allocations. RELNOTES: None PiperOrigin-RevId: 152145768
* Restructure Path#getCachedChildPath to make direct method calls without ↵Gravatar nharmata2017-04-04
| | | | | | | | allocating a garbage TranslatedPath object. RELNOTES: None PiperOrigin-RevId: 152130170
* Make InMemoryFileSystem case-insensitive on Windows. Gravatar Dmitry Lomov2017-03-09
| | | | | | | | | | This fixes PackagePathLocatorTest. -- Change-Id: I3d14a80993f6b256acfc732adf2d97b1d2dcb804 Reviewed-on: https://cr.bazel.build/9310 PiperOrigin-RevId: 149634730 MOS_MIGRATED_REVID=149634730
* PathFragment comparisons are now platform-aware Gravatar László Csomor2017-03-06
| | | | | | | | | | | | | | | | | | | | | | | | | PathFragment's `equals`, `hashCode`, `compareTo`, `startsWith`, `endsWith`, and `relativeTo` are now aware of case-insensitivity when running on Windows. This approach is better than https://bazel-review.googlesource.com/c/9124/ because it preserves path casing, which is important when computing action output paths. This change contains two additional bugfixes: - `compareTo` now takes `driveLetter` into account - the `InMemoryFileSystem` in `PathWindowsTest` is not case-insensitive Fixes https://github.com/bazelbuild/bazel/issues/2613 -- Change-Id: I1a4250a373fff03fa02a6d8360457450b47a42a8 Reviewed-on: https://cr.bazel.build/9126 PiperOrigin-RevId: 149106930 MOS_MIGRATED_REVID=149106930
* Move UnixFileSystem to lib.unix, WindowsFileSystem to lib.windowsGravatar Ulf Adams2017-02-28
| | | | | | -- PiperOrigin-RevId: 148749485 MOS_MIGRATED_REVID=148749485
* Remove the deprecated com.google.devtools.build.lib.vfs.FileSystems classGravatar Ulf Adams2017-02-27
| | | | | | -- PiperOrigin-RevId: 148625618 MOS_MIGRATED_REVID=148625618
* Move FileAccessException to the vfs packageGravatar Ulf Adams2017-02-17
| | | | | | | | | | | | | | It was previously in unix, but also used from non-unix file systems, which means it's not actually unix-specific. This is in preparation for splitting compilation of the unix and windows file systems into separate libraries. That improves layering and reduces compile times - note that Bazel already injects the vfs into its lower layers, which should only rely on the normal vfs APIs, not on anything platform-specific. -- PiperOrigin-RevId: 147829659 MOS_MIGRATED_REVID=147829659
* Windows: use JNI CreateJunction in Java code Gravatar Laszlo Csomor2017-02-15
| | | | | | | | | | | | | | | | Use the new CreateJunction in the Windows JNI code every time we need to create junctions. This means updating WindowsFileOperations and related tests. Add test for WindowsFileSystem.createSymbolicLink. See https://github.com/bazelbuild/bazel/issues/2238 -- Change-Id: I5827e2e70e8e147f5f102fabf95fa9a148b3bcdc Reviewed-on: https://cr.bazel.build/8896 PiperOrigin-RevId: 147598107 MOS_MIGRATED_REVID=147598107
* WindowsFileSystem.java: Fix createSymbolicLink when target is a relative path Gravatar Yun Peng2017-02-07
| | | | | | | | | | Fixed https://github.com/bazelbuild/bazel/issues/2494 -- Change-Id: I2ca335fa5b3a7759f57085717128912f09363789 Reviewed-on: https://cr.bazel.build/8650 PiperOrigin-RevId: 146761747 MOS_MIGRATED_REVID=146761747
* The double-checked locking seems unnecessary.Gravatar Googler2017-02-06
| | | | | | | | | | | | | | | Every double-checked lock will be followed up by a regular lock. Either the children don't exist, in which case we take the lock twice, or they do exist and we take the lock once. This makes it so we only ever take the lock once. I don't know if this makes it faster, but it makes the code simpler, and shouldn't make it slower. -- PiperOrigin-RevId: 146553235 MOS_MIGRATED_REVID=146553235
* Windows: resolve shortened pathsGravatar Laszlo Csomor2017-01-03
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | TL;DR: resolve "C:/progra~1" style paths, don't cache failed resolutions in the parent's Path.children When creating WindowsPath objects, resolve the "C:/progra~1" style paths to "C:/program files". This enables us to correctly compare paths on Windows. Without this canonicalization we incorrectly determine "C:/progra~1" and "C:/program files" to be different when they are actually the same. We only attempt to resolve such paths if the name looks like it's an abbreviated path. If resolution fails, probably due to the path not existing, then we don't cache this Path object in the parent's `children` list. This avoids stale cache entries in case the path springs into existence later in the server's lifetime. Fixes https://github.com/bazelbuild/bazel/issues/2145 We also need to rectify https://github.com/bazelbuild/bazel/issues/2173 -- PiperOrigin-RevId: 143442134 MOS_MIGRATED_REVID=143442134
* WindowsFileSystem: use JNI impl. of isJunctionGravatar Laszlo Csomor2016-12-20
| | | | | | | | | | | | | | | | | | | Also build the JNI library while bootstraping. This was once submitted in commit 4a249b6962d32ed4cfd4165dfdae4a555b00bb69 but got rolled back due to some test breakage that's long since fixed. In this change I'm slightly modifying the original code in compile.sh. Using JNI methods however is necessary because we can't implement WindowsFileOperations.GetLongPath in native Java, and having that code is a prerequisite for the fix of https://github.com/bazelbuild/bazel/issues/2145 See also https://github.com/bazelbuild/bazel/issues/2238 -- PiperOrigin-RevId: 142535019 MOS_MIGRATED_REVID=142535019
* Bazel client, Windows: implement AsWindowsPathGravatar Laszlo Csomor2016-12-20
| | | | | | | | | | | | | | | | | | | | This method converts MSYS paths to Windows path. It uses the BAZEL_SH envvar to obtain the MSYS root directory, to which all Unix paths (except for mounts) are relative. We cannot handle mounts because we don't want to read /etc/mtab every time there's a file operation so we simply apply a heuristic similar to https://github.com/bazelbuild/bazel/blob/cd4cc09fa6ef96380a3d0888f825dfd1dbada651/src/main/java/com/google/devtools/build/lib/vfs/WindowsFileSystem.java#L52-L63 Also clean up the #ifdefs surrounding SyncFile. See https://github.com/bazelbuild/bazel/issues/2107 -- PiperOrigin-RevId: 142531986 MOS_MIGRATED_REVID=142531986
* Fix PathFragment to not use Java8-only static hashCode methods.Gravatar John Cater2016-12-15
| | | | | | | | Fixes #2247. -- PiperOrigin-RevId: 142143520 MOS_MIGRATED_REVID=142143520
* Replace UnixGlob's package glob prefetching functionality with anGravatar Googler2016-12-02
| | | | | | | | optional method in FileSystem. Custom FileSystem implementations can use this to provide their own implementation of glob prefetching. -- MOS_MIGRATED_REVID=140736304
* Introduce BlazeInterners, a Blaze-specific wrapper around Guava's Interners ↵Gravatar Nathan Harmata2016-11-24
| | | | | | | | | | | that makes an appropriate call to Interners.InternerBuilder#concurrencyLevel. For current readers of this CL, I used this class everywhere in the Blaze codebase. For future readers of this CL, this class should be used to create an Interner in the Blaze codebase. -- MOS_MIGRATED_REVID=140063271
* Adding an option to set the digest function that everything uses. Minor ↵Gravatar Ola Rozenfeld2016-11-18
| | | | | | | | | | | | | refactoring: enabling potential fast digest computation of more than one digest function type. Usage: bazel --host_jvm_args="-Dbazel.DigestFunction=SHA1" build ... Ugliness: using a system property (a static non-final variable), because the better way to do it (a flag) would result in a much, much larger refactoring. More ugliness: I have updated the minimal amount of tests. A lot of tests are still relying on the default value of MD5. Ideally, they need to be updated as well. -- MOS_MIGRATED_REVID=139490836
* Windows: can now remove bazel-* symlinksGravatar Laszlo Csomor2016-11-04
| | | | | | | | | | | | | | | | | This change updates WindowsFileSystem so it: - retrieves the DosFileAttributes instead of the BasicFileAttributes, because the latter does not report junctions as directories - uses just isJunction to decide if a file is a symlink, doesn't look at whether it's a directory (again because java.nio.File also incorrectly reports junctions as non-directories) Fixes https://github.com/bazelbuild/bazel/issues/1850 -- MOS_MIGRATED_REVID=138187220
* Do not tickle TimestampGranularityMonitor for stable-status.txt no-op updates.Gravatar Julio Merino2016-11-03
| | | | | | | | | | | | | | | | | When rewriting stable-status.txt, which happens on each build, avoid updating the file's ctime and mtime if the new contents match what is already in the file. This prevents tickling the TimestampGranularityMonitor for what should be a no-op update, which in turn could cause null/incremental builds to stall for up to a second. The problem was magnified on macOS where the default HFS+ file system only has second-level granularity. (This also affects Linux, but because current Linux file systems have milli/nanosecond-level granularity, the wait imposed by TimestampGranularityMonitor is minimal and thus not generally noticeable.) -- MOS_MIGRATED_REVID=137983794
* VFS: PathFragment is no longer aware of MSYS pathsGravatar Laszlo Csomor2016-10-27
| | | | | | | | | | | | | PathFragment no longer parses "/c/foo" as "C:/foo" on Windows, but as a driveletter-less absolute path. If such a PathFragment is used in creating a Path object, the WindowsPath.translatePath method will translate it correctly. Fixes https://github.com/bazelbuild/bazel/issues/1994 -- MOS_MIGRATED_REVID=137283176
* Refactor: extract anonymous classesGravatar Laszlo Csomor2016-10-26
| | | | | | | | | | | | | | | | | | | AbstractFileSystem.getFile{Input,Output}Stream created anonymous File{Input,Output}Stream objects. These hold a reference to the outer class instance (AbstractFileSystem). This may prevent memory release in case the returned objects are kept around even if the AbstractFileSystem instance could otherwise be released. This particular refactoring is unlikely to have caught any memory leaks like this, so it's not really necessary, but I came across it and thought it won't hurt and will future-prove the class against such leaks. -- MOS_MIGRATED_REVID=137254192
* VFS: implement a Windows-specific Path subclassGravatar Laszlo Csomor2016-10-25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This change rolls forward commit e0d7a540e3c615c628f63fcaaaba0c47fca2cb25 and commit 8bb4299b28de14eed9d3b57bcaeb9350c81c7db3, and adds a bugfix: - FileSystem.PathFactory got a new translatePath method that WindowsFileSystem.PathFactory overrides to translate absolute Unix paths to MSYS-relative paths - Path.getCachedChildPath calls this translatePath method so the child path is registered with the correct (translated) parent and under the correct name (e.g. "C:" instead of say "c") Below is the rest of the original change description: The new subclass WindowsFileSystem.WindowsPath is aware of Windows drives. This change: - introduces a new factory for Path objects so FileSystems can return a custom implementation that instantiates filesystem-specific Paths - implements the WindowsPath subclass of Path that is aware of Windows drives - introduces the bazel.windows_unix_root JVM argument that defines the MSYS root, which defines the absolute Windows path that is the root of all Unix paths that Bazel creates (e.g. "/usr/lib" -> "C:/tools/msys64/usr/lib") except if the path is of the form "/c/foo" which is treated as "C:/foo" - removes all Windows-specific logic from Path PathFragment is still aware of drive letters and it has to remain so because it is unaware of file systems. WindowsPath restricts the allowed path strings to absolute Unix paths where the first segment, if any, is a volume specifier. From now on if Bazel attempts to create a WindowsPath from an absolute Unix path, Bazel will make it relative to WindowsPath.UNIX_ROOT, unless the first component is a single-letter name (e.g. "/c/foo" which is "C:/foo"). Subclassing Path is necessary because a Unix-style absolute path doesn't sufficiently define a full Windows path, as it may be relative to any drive. Fixes https://github.com/bazelbuild/bazel/issues/1463 -- MOS_MIGRATED_REVID=137149483
* Rollback of commit e0d7a540e3c615c628f63fcaaaba0c47fca2cb25.Gravatar Philipp Wollermann2016-10-19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Suspected root cause for Windows bootstrap on Bazel CI breakage: java.lang.NullPointerException at com.google.devtools.build.lib.vfs.Path$1.run(Path.java:123) http://ci.bazel.io/view/Bazel%20bootstrap%20and%20maintenance/job/Bazel/922/JAVA_VERSION=1.8,PLATFORM_NAME=windows-x86_64/console *** Original change description *** VFS: implement a Windows-specific Path subclass The new subclass WindowsFileSystem.WindowsPath is aware of Windows drives. This change: - introduces a new factory for Path objects so FileSystems can return a custom implementation that instantiates filesystem-specific Paths - implements the WindowsPath subclass of Path that is aware of Windows drives - introduces the bazel.windows_unix_root JVM argument that defines the MSYS root, which defines the absolute Windows path that is the... *** -- MOS_MIGRATED_REVID=136583352
* Rollback of commit 8bb4299b28de14eed9d3b57bcaeb9350c81c7db3.Gravatar Philipp Wollermann2016-10-19
| | | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Suspected root cause for Windows bootstrap on Bazel CI breakage: java.lang.NullPointerException at com.google.devtools.build.lib.vfs.Path$1.run(Path.java:123) http://ci.bazel.io/view/Bazel%20bootstrap%20and%20maintenance/job/Bazel/922/JAVA_VERSION=1.8,PLATFORM_NAME=windows-x86_64/console *** Original change description *** VFS, WindowsFileSystem: fix UNIX_ROOT retrieval Executing bash.exe directly instead of through cmd.exe doesn't seem to work. This change fixes that problem. Fixes https://github.com/bazelbuild/bazel/issues/1463 (again) -- MOS_MIGRATED_REVID=136581532
* Add file extension property to SkylarkGravatar Paul Roberts2016-10-18
| | | | | | | RELNOTES[NEW]: Files now have an "extension" property in Skylark. -- MOS_MIGRATED_REVID=136425934
* VFS, WindowsFileSystem: fix UNIX_ROOT retrievalGravatar Laszlo Csomor2016-10-18
| | | | | | | | | | | Executing bash.exe directly instead of through cmd.exe doesn't seem to work. This change fixes that problem. Fixes https://github.com/bazelbuild/bazel/issues/1463 (again) -- MOS_MIGRATED_REVID=136364606
* VFS: implement a Windows-specific Path subclassGravatar Laszlo Csomor2016-10-18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The new subclass WindowsFileSystem.WindowsPath is aware of Windows drives. This change: - introduces a new factory for Path objects so FileSystems can return a custom implementation that instantiates filesystem-specific Paths - implements the WindowsPath subclass of Path that is aware of Windows drives - introduces the bazel.windows_unix_root JVM argument that defines the MSYS root, which defines the absolute Windows path that is the root of all Unix paths that Bazel creates (e.g. "/usr/lib" -> "C:/tools/msys64/usr/lib") except if the path is of the form "/c/foo" which is treated as "C:/foo" - removes all Windows-specific logic from Path PathFragment is still aware of drive letters and it has to remain so because it is unaware of file systems. WindowsPath restricts the allowed path strings to absolute Unix paths where the first segment, if any, is a volume specifier. From now on if Bazel attempts to create a WindowsPath from an absolute Unix path, Bazel will make it relative to WindowsPath.UNIX_ROOT, unless the first component is a single-letter name (e.g. "/c/foo" which is "C:/foo"). Subclassing Path is necessary because a Unix-style absolute path doesn't sufficiently define a full Windows path, as it may be relative to any drive. Fixes https://github.com/bazelbuild/bazel/issues/1463 -- MOS_MIGRATED_REVID=136350304
* Expand $(location :label) to string unambiguously a pathGravatar Klaus Aehlig2016-10-14
| | | | | | | | | | | | | | | | | | | In genrules, cmd strings of the form "$(location :label) ..." are used with the assumption that the executable named by the label will be called. This holds true as long as $(location :label) expands to a string that is recognizable as a path, i.e., as long as :label does not refer to a file in the top-level directory. In the latter case, however, that string will be the plain file name and the shell will search for that name in the search path. This will fail, if '.' is not in the search path; even worse, if a file with that name is in the search path before '.', then that one will be called which is not what the user intended to do. Fix this unintended behavior by expanding $(location :label) to a string that unambiguously is a path. -- Change-Id: If8681039a8befae6234fbe0cbe3a0f75eedba7aa Reviewed-on: https://bazel-review.googlesource.com/#/c/6691 MOS_MIGRATED_REVID=136151500
* Add the option --experimental_max_directories_to_eagerly_visit_in_globbing. ↵Gravatar Janak Ramakrishnan2016-10-05
| | | | | | | The first legacy glob that a package requires will, if this option is enabled, cause up to that many directories to be eagerly visited by a glob(['**']). The results are thrown away for memory reasons. -- MOS_MIGRATED_REVID=135148361
* Fixed symbolic link and hard link path not stripped when "strip_prefix" is set.Gravatar Xin Gao2016-09-23
| | | | | -- MOS_MIGRATED_REVID=134005484
* Rollback of commit 4e2f75d8e60020f13a7df452488ec4cc7a719f03.Gravatar Kristina Chodorow2016-09-22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** This is causing external_integration_test to fail See, for example, http://ci.bazel.io/job/bazel-tests/BAZEL_VERSION=HEAD,PLATFORM_NAME=linux-x86_64/236/console: ** test_http_archive_tar_xz **************************************************** GET /fox.tar.xz HTTP/1.1 User-Agent: Java/1.8.0_101 Host: localhost:36541 Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2 Connection: keep-alive cat: bazel-out/../external/endangered/fox/male_absolute: No such file or directory -- Test log: ----------------------------------------------------------- INFO: Reading 'startup' options from /home/ci/.cache/bazel/_bazel_ci/a761298a0949227106f79c62c3bebb6e/bazel-sandbox/81f51af6-eb50-417e-8f8d-b7bba207ee83-661/execroot/linux-x86_64/_tmp/external_integration_test_2/bazelrc: --output_user_root=/home/ci/.cache/bazel/_bazel_ci/a761298a0949227106f79c62c3bebb6e/bazel-sandbox/81f51af6-eb50-417e-8f8d-b7bba207ee83-661/execroot/linux-x86_64/_tmp/external_integration_test_2/root --host_javabase=/home/ci/.cache/bazel/_bazel_ci/a761298a0949227106f79c62c3bebb6e/bazel-sandbox/81f51af6-eb50-417e-8f8d-b7bba207ee83-661/execroot/linux-x86_64/bazel-out/local-fastbuild/bin/src/test/shell/bazel/external_integration_test.runfiles/local_jdk INFO: $TEST_TMPDIR defined: output root default is '/home/ci/.cache/bazel/_bazel_ci/a761298a0949227106f79c62c3bebb6e/bazel-sandbox/81f51af6-eb50-417e-8f8d-b7bba207ee83-661/execroot/linux-x86_64/_tmp/external_integration_test_2'. .............................................. ____Loading package: zoo ____Loading... ____Loading package: @bazel_tools//tools/cpp ____Loading package: @bazel_tools//tools/jdk ____Loading package: @local_config_xcode// ____Loading package: @local_jdk// ____Loading package: @local_config_cc// ____Loading complete. Analyzing... ____Downloading from [] 0B ____Downloading from [] 1KB ____Loading package: @endangered//fox ____Found 1 target... ____Building... ____[0 / 1] BazelWorkspaceStatusAction stable-status.txt ____[0 / 4] Creating source manifest for //zoo:breeding-program ____[0 / 4] Symlinking //zoo:breeding-program ____[3 / 4] Creating runfiles tree bazel-out/local-fastbuild/bin/zoo/breeding-program.runfiles ____Building complete. Target //zoo:breeding-program up-to-date: bazel-bin/zoo/breeding-program ____Elapsed time: 6.903s, Critical Path: 0.04s ____Running command line: bazel-bin/zoo/breeding-program Fraka-kaka-kaka-kaka-kow ------------------------------------------------------------------------ test_http_archive_tar_xz FAILED: Expected regexp #!/bin/bash echo Fraka-kaka-kaka-kaka-kow not found in bazel-out/../external/endangered/fox/male_absolute . /home/ci/.cache/bazel/_bazel_ci/a761298a0949227106f79c62c3bebb6e/bazel-sandbox/81f51af6-eb50-417e-8f8d-b7bba207ee83-661/execroot/linux-x86_64/bazel-out/local-fastbuild/bin/src/test/shell/bazel/external_integration_test.runfiles/io_bazel/src/test/shell/bazel/external_integration_test:154: in call to assert_files_same /home/ci/.cache/bazel/_bazel_ci/a761298a0949227106f79c62c3bebb6e/bazel-sandbox/81f51af6-eb50-417e-8f8d-b7bba207ee83-661/execroot/linux-x86_64/bazel-out/local-fastbuild/bin/src/test/shell/bazel/external_integration_test.runfiles/io_bazel/src/test/shell/bazel/external_integration_test:150: in call to http_archive_helper /home/ci/.cache/bazel/_bazel_ci/a761298a0949227106f79c62c3bebb6e/bazel-sandbox/81f51af6-eb50-417e-8f8d-b7bba207ee83-661/execroot/linux-x86_64/bazel-out/local-fastbuild/bin/src/test/shell/bazel/external_integration_test.runfiles/io_bazel/src/test/shell/bazel/external_integration_test:190: in call to test_http_archive_tar_xz FAILED: test_http_archive_tar_xz *** Original change description *** Fixed symbolic link and hard link path not stripped when "strip_prefix" is set. -- MOS_MIGRATED_REVID=133970692
* Fixed symbolic link and hard link path not stripped when "strip_prefix" is set.Gravatar Xin Gao2016-09-20
| | | | | -- MOS_MIGRATED_REVID=133628392
* Also copy directories as output for sandbox.Gravatar Yue Gan2016-09-19
| | | | | -- MOS_MIGRATED_REVID=133564429
* Exporting functions to compute SHA1 digests of files.Gravatar Ola Rozenfeld2016-09-12
| | | | | | | | TODO: need to stich them all the way through various ActionInputFileCache implementations! -- MOS_MIGRATED_REVID=132685408
* Windows: add a Java-native isJunction methodGravatar Laszlo Csomor2016-09-09
| | | | | | | | | | | | | | | | | Also add tests for WindowsFileSystem and some for WindowsFileOperations, so we test both the JNI-based and Java-native isJunction function, as well as handling of dangling symlinks/junctions. Having a Java-native version of this method means we don't need to use windows_jni.dll for any tests or for bootstrapping. This change could help with https://github.com/bazelbuild/bazel/issues/1735 -- MOS_MIGRATED_REVID=132556440
* Windows: go back to using Jury-rigged isJunctionGravatar Laszlo Csomor2016-09-08
| | | | | | | | | | | This should unbreak the tests, see https://github.com/bazelbuild/bazel/issues/1735 The tests still break but now due to https://github.com/bazelbuild/bazel/issues/1736 -- MOS_MIGRATED_REVID=132530222
* Fixed the issue that hard links are handled improperly when bazel ↵Gravatar Googler2016-09-08
| | | | | | | | | decompresses tarballs. Issue link: https://github.com/bazelbuild/bazel/issues/574 -- MOS_MIGRATED_REVID=132434278
* Making the PathFragment interned segments list public, to be used later inGravatar Googler2016-09-08
| | | | | | | generating the directory tree nodes efficiently. -- MOS_MIGRATED_REVID=132433991
* Windows, tests: move code into WindowsTestUtilGravatar Laszlo Csomor2016-09-07
| | | | | | | | | | | | Additionally: - clean up the corresponding BUILD file a bit - add a comment to Path A subsequent change will add tests for WindowsFileSystem, then fix a bug there. -- MOS_MIGRATED_REVID=132408212
* Windows, JNI: use native isJunction method.Gravatar Laszlo Csomor2016-09-06
| | | | | | | Fixes https://github.com/bazelbuild/bazel/issues/1680 -- MOS_MIGRATED_REVID=132051176
* Automated [] rollback of commit 846a5ab98fc26d72024890fdb79a5d3bc6a5a1ba + ↵Gravatar Nathan Harmata2016-08-23
| | | | | | | | | | | | | | | manual rollback of [] *** Reason for rollback *** Depot has been fixed / is in the process of being fixed. See the work tracked on [] *** Original change description *** Automated [] rollback of commit bb5d5efb4b50710241b5b374eb67084f4bf08278. -- MOS_MIGRATED_REVID=131095905
* Rollback of commit bb5d5efb4b50710241b5b374eb67084f4bf08278.Gravatar Tobias Werth2016-08-23
| | | | | -- MOS_MIGRATED_REVID=130941264
* Implement the first stage of Bazel's "Sandbox 2.0" for Linux.Gravatar Philipp Wollermann2016-08-18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This has the following improvements upon the older one: - Uses PID namespaces, PR_SET_PDEATHSIG and a number of other tricks for further process isolation and 100% reliable killing of child processes. - Uses clone() instead of unshare() to work around a Linux kernel bug that made creating a sandbox unreliable. - Instead of mounting a hardcoded list of paths + whatever you add with --sandbox_add_path, this sandbox instead mounts all of /, except for what you make inaccessible via --sandbox_block_path. This should solve the majority of "Sandboxing breaks my build, because my compiler is installed in /opt or /usr/local" issues that users have seen. - Instead of doing magic with bind mounts, we create a separate execroot for each process containing symlinks to the input files. This is simpler and gives more predictable performance. - Actually makes everything except the working directory read-only (fixes #1364). This means that a running process can no longer accidentally modify your source code (yay!). - Prevents a number of additional "attacks" or leaks, like accidentally inheriting file handles from the parent. - Simpler command-line interface. - We can provide the same semantics in a Mac OS X sandbox, which will come in a separate code review from yueg@. It has the following caveats / known issues: - The "fallback to /bin/bash on error" feature is gone, but now that the sandbox mounts everything by default, the main use-case for this is no longer needed. The following improvements are planned: - Use a FUSE filesystem if possible for the new execroot, instead of creating symlinks. - Mount a base image instead of "/". FAQ: Q: Why is mounting all of "/" okay, doesn't this make the whole sandbox useless? A: This is still a reasonable behavior, because the sandbox never tried to isolate your build from the operating system it runs in. Instead it is supposed to protect your data from a test running "rm -rf $HOME" and to make it difficult / impossible for actions to use input files that are not declared dependencies. For even more isolation the sandbox will support mounting a base image as its root in a future version (similar to Docker images). Q: Let's say my process-specific execroot contains a symlink to an input file "good.h", can't the process just resolve the symlink, strip off the file name and then look around in the workspace? A: Yes. Unfortunately we could not find any way on Linux to make a file appear in a different directory with *all* of the semantics we would like. The options investigated were: 1) Copying input files, which is much too slow. 2) Hard linking input files, which is fast, but doesn't work cross- filesystems and it's also not possible to make them read-only. 3) Bind mounts, which don't scale once you're up in the thousands of input files (across all actions) - it seems like the kernel has some non-linear performance behavior when the mount table grows too much, resulting in the mount syscall taking more time the more mounts you have. 4) FUSE filesystem, good in theory, but wasn't ready for the first iteration. RELNOTES: New sandboxing implementation for Linux in which all actions run in a separate execroot that contains input files as symlinks back to the originals in the workspace. The running action now has read-write access to its execroot and /tmp only and can no longer write in arbitrary other places in the file system. -- Change-Id: Ic91386fc92f8eef727ed6d22e6bd0f357d145063 Reviewed-on: https://bazel-review.googlesource.com/#/c/4053 MOS_MIGRATED_REVID=130638204
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-05 00:44:04 +0000