| Commit message (Collapse) | Author | Age |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
SandboxedSpawn.copyOutputs.
Previously, if moveFile fell back to copying from a true rename (e.g., if the
move is across file systems), it would fully dereference the source and produce
a regular file at the output location. This CL fixes moveFile to properly copy
symlinks. Note we don't preserve metadata in the symlink case mostly because the
Bazel VFS has no API to write metadata without dereferencing symlinks. (But,
also, it's not important for the current use cases.)
This breaks the backward-compatibility of FileSystemUtils.moveFile and
FileSystemUtils.moveTreeBelow. This seems okay because the new behavior makes
more sense, and sandbox is the only consumer of these APIs.
Switching SandboxedSpawn.copyOutputs to use FileSystemUtils.moveFile rather than
Guava's Files.move fixes https://github.com/bazelbuild/bazel/issues/4987.
Closes #4989.
Change-Id: I0283e8aa11fadff5b0afd7bdfd0490aca12a1f6b
PiperOrigin-RevId: 192611227
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Path and PathFragment have been replaced with String-based implementations. They are pretty similar, but each method is dissimilar enough that I did not feel sharing code was appropriate.
A summary of changes:
PATH
====
* Subsumes LocalPath (deleted, its tests repurposed)
* Use a simple string to back Path
* Path instances are no longer interned; Reference equality will no longer work
* Always normalized (same as before)
* Some operations will now be slower, like instance compares (which were previously just a reference check)
* Multiple identical paths will now consume more memory since they are not interned
PATH FRAGMENT
=============
* Use a simple string to back PathFragment
* No more segment arrays with interned strings
* Always normalized
* Remove isNormalized
* Replace some isNormalizied uses with containsUpLevelReferences() to check if path fragments try to escape their scope
* To check if user input is normalized, supply static methods on PathFragment to validate the string before constructing a PathFragment
* Because PathFragments are always normalized, we have to replace checks for literal "." from PathFragment#getPathString to PathFragment#getSafePathString. The latter returns "." for the empty string.
* The previous implementation supported efficient segment semantics (segment count, iterating over segments). This is now expensive since we do longer have a segment array.
ARTIFACT
========
* Remove Path instance. It is instead dynamically constructed on request. This is necessary to avoid this CL becoming a memory regression.
RELNOTES: None
PiperOrigin-RevId: 185062932
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Rolling forward after the underlying issue has been fixed.
*** Original change description ***
Automated rollback of commit 6a54339bb943702bd7dffc43f85267dac98dc355.
*** Reason for rollback ***
b/71442447
*** Original change description ***
Call through to Path#createDirectoryAndParents from FileUtils.
This CL removes a method that due to its implementation causes threading difficulties for Path#createDirectory.
The tests for the method are brought across to FileSystemTests since the methods are now implemented natively by the FileSystem classes. The tests were also cleaned up.
The test revealed an edge c...
***
PiperOrigin-RevId: 181367850
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
b/71442447
*** Original change description ***
Call through to Path#createDirectoryAndParents from FileUtils.
This CL removes a method that due to its implementation causes threading difficulties for Path#createDirectory.
The tests for the method are brought across to FileSystemTests since the methods are now implemented natively by the FileSystem classes. The tests were also cleaned up.
The test revealed an edge case bug in JavaIoFileSystem, so fix this.
In two cases some code was using the return value from the old method. Returning "f...
***
ROLLBACK_OF=179864042
PiperOrigin-RevId: 180946251
|
|
|
|
|
|
|
|
|
|
|
|
| |
This CL removes a method that due to its implementation causes threading difficulties for Path#createDirectory.
The tests for the method are brought across to FileSystemTests since the methods are now implemented natively by the FileSystem classes. The tests were also cleaned up.
The test revealed an edge case bug in JavaIoFileSystem, so fix this.
In two cases some code was using the return value from the old method. Returning "false" essentially means that the directory already existed, and the code doesn't look racy, so we can replace it with an existence check.
PiperOrigin-RevId: 179864042
|
|
|
|
|
|
| |
Only the sandbox uses the cached version. We can let it do its own caching of the top-level directory. It won't help with higher-level directories, but if there are a lot of inputs in the same directory we'll avoid I/O there.
PiperOrigin-RevId: 179830651
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fixes #4322, #4306.
*** Reason for rollback ***
Introduces a deadlock (see https://github.com/bazelbuild/bazel/issues/4322)
*** Original change description ***
Make FileSystem operate on LocalPath instead of Path.
PiperOrigin-RevId: 179549866
|
|
|
|
| |
PiperOrigin-RevId: 179082062
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 177326265
|
|
|
|
|
|
|
|
| |
Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own.
This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually.
PiperOrigin-RevId: 175033526
|
|
|
|
|
|
| |
the tool setup we do in tests doesn't necessarily have to be copied if we copy a workspace over.
PiperOrigin-RevId: 171864170
|
|
|
|
|
|
|
| |
With a few manual fixes for readability.
RELNOTES: None.
PiperOrigin-RevId: 160582556
|
|
|
|
|
|
|
|
|
|
|
|
| |
'create' method.
This paves the way for changing PathFragment to e.g. an abstract class with multiple subclasses. This way we can split out the windows-specific stuff into one of these concrete classes, making the code more readable and also saving memory (since the shallow heap size of the NonWindowsPathFragment subclass will hopefully be smaller than that of the current PathFragment).
This also lets us pursue gc churn optimizations. We can now do interning in PathFragment#create and can also get rid of unnecessary intermediate PathFragment allocations.
RELNOTES: None
PiperOrigin-RevId: 152145768
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 148749485
MOS_MIGRATED_REVID=148749485
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When rewriting stable-status.txt, which happens on each build, avoid updating
the file's ctime and mtime if the new contents match what is already in the
file.
This prevents tickling the TimestampGranularityMonitor for what should be a
no-op update, which in turn could cause null/incremental builds to stall for
up to a second. The problem was magnified on macOS where the default HFS+
file system only has second-level granularity. (This also affects Linux, but
because current Linux file systems have milli/nanosecond-level granularity,
the wait imposed by TimestampGranularityMonitor is minimal and thus not
generally noticeable.)
--
MOS_MIGRATED_REVID=137983794
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=134005484
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
This is causing external_integration_test to fail
See, for example, http://ci.bazel.io/job/bazel-tests/BAZEL_VERSION=HEAD,PLATFORM_NAME=linux-x86_64/236/console:
** test_http_archive_tar_xz ****************************************************
GET /fox.tar.xz HTTP/1.1
User-Agent: Java/1.8.0_101
Host: localhost:36541
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
cat: bazel-out/../external/endangered/fox/male_absolute: No such file or directory
-- Test log: -----------------------------------------------------------
INFO: Reading 'startup' options from /home/ci/.cache/bazel/_bazel_ci/a761298a0949227106f79c62c3bebb6e/bazel-sandbox/81f51af6-eb50-417e-8f8d-b7bba207ee83-661/execroot/linux-x86_64/_tmp/external_integration_test_2/bazelrc: --output_user_root=/home/ci/.cache/bazel/_bazel_ci/a761298a0949227106f79c62c3bebb6e/bazel-sandbox/81f51af6-eb50-417e-8f8d-b7bba207ee83-661/execroot/linux-x86_64/_tmp/external_integration_test_2/root --host_javabase=/home/ci/.cache/bazel/_bazel_ci/a761298a0949227106f79c62c3bebb6e/bazel-sandbox/81f51af6-eb50-417e-8f8d-b7bba207ee83-661/execroot/linux-x86_64/bazel-out/local-fastbuild/bin/src/test/shell/bazel/external_integration_test.runfiles/local_jdk
INFO: $TEST_TMPDIR defined: output root default is '/home/ci/.cache/bazel/_bazel_ci/a761298a0949227106f79c62c3bebb6e/bazel-sandbox/81f51af6-eb50-417e-8f8d-b7bba207ee83-661/execroot/linux-x86_64/_tmp/external_integration_test_2'.
..............................................
____Loading package: zoo
____Loading...
____Loading package: @bazel_tools//tools/cpp
____Loading package: @bazel_tools//tools/jdk
____Loading package: @local_config_xcode//
____Loading package: @local_jdk//
____Loading package: @local_config_cc//
____Loading complete. Analyzing...
____Downloading from [] 0B
____Downloading from [] 1KB
____Loading package: @endangered//fox
____Found 1 target...
____Building...
____[0 / 1] BazelWorkspaceStatusAction stable-status.txt
____[0 / 4] Creating source manifest for //zoo:breeding-program
____[0 / 4] Symlinking //zoo:breeding-program
____[3 / 4] Creating runfiles tree bazel-out/local-fastbuild/bin/zoo/breeding-program.runfiles
____Building complete.
Target //zoo:breeding-program up-to-date:
bazel-bin/zoo/breeding-program
____Elapsed time: 6.903s, Critical Path: 0.04s
____Running command line: bazel-bin/zoo/breeding-program
Fraka-kaka-kaka-kaka-kow
------------------------------------------------------------------------
test_http_archive_tar_xz FAILED: Expected regexp #!/bin/bash
echo Fraka-kaka-kaka-kaka-kow not found in bazel-out/../external/endangered/fox/male_absolute .
/home/ci/.cache/bazel/_bazel_ci/a761298a0949227106f79c62c3bebb6e/bazel-sandbox/81f51af6-eb50-417e-8f8d-b7bba207ee83-661/execroot/linux-x86_64/bazel-out/local-fastbuild/bin/src/test/shell/bazel/external_integration_test.runfiles/io_bazel/src/test/shell/bazel/external_integration_test:154: in call to assert_files_same
/home/ci/.cache/bazel/_bazel_ci/a761298a0949227106f79c62c3bebb6e/bazel-sandbox/81f51af6-eb50-417e-8f8d-b7bba207ee83-661/execroot/linux-x86_64/bazel-out/local-fastbuild/bin/src/test/shell/bazel/external_integration_test.runfiles/io_bazel/src/test/shell/bazel/external_integration_test:150: in call to http_archive_helper
/home/ci/.cache/bazel/_bazel_ci/a761298a0949227106f79c62c3bebb6e/bazel-sandbox/81f51af6-eb50-417e-8f8d-b7bba207ee83-661/execroot/linux-x86_64/bazel-out/local-fastbuild/bin/src/test/shell/bazel/external_integration_test.runfiles/io_bazel/src/test/shell/bazel/external_integration_test:190: in call to test_http_archive_tar_xz
FAILED: test_http_archive_tar_xz
*** Original change description ***
Fixed symbolic link and hard link path not stripped when "strip_prefix" is set.
--
MOS_MIGRATED_REVID=133970692
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=133628392
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=133564429
|
|
|
|
|
|
|
|
|
| |
decompresses tarballs.
Issue link: https://github.com/bazelbuild/bazel/issues/574
--
MOS_MIGRATED_REVID=132434278
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This has the following improvements upon the older one:
- Uses PID namespaces, PR_SET_PDEATHSIG and a number of other tricks for
further process isolation and 100% reliable killing of child processes.
- Uses clone() instead of unshare() to work around a Linux kernel bug that
made creating a sandbox unreliable.
- Instead of mounting a hardcoded list of paths + whatever you add with
--sandbox_add_path, this sandbox instead mounts all of /, except for what
you make inaccessible via --sandbox_block_path. This should solve the
majority of "Sandboxing breaks my build, because my compiler is installed
in /opt or /usr/local" issues that users have seen.
- Instead of doing magic with bind mounts, we create a separate execroot for
each process containing symlinks to the input files. This is simpler and
gives more predictable performance.
- Actually makes everything except the working directory read-only
(fixes #1364). This means that a running process can no longer accidentally
modify your source code (yay!).
- Prevents a number of additional "attacks" or leaks, like accidentally
inheriting file handles from the parent.
- Simpler command-line interface.
- We can provide the same semantics in a Mac OS X sandbox, which will come in
a separate code review from yueg@.
It has the following caveats / known issues:
- The "fallback to /bin/bash on error" feature is gone, but now that the
sandbox mounts everything by default, the main use-case for this is no
longer needed.
The following improvements are planned:
- Use a FUSE filesystem if possible for the new execroot, instead of creating
symlinks.
- Mount a base image instead of "/".
FAQ:
Q: Why is mounting all of "/" okay, doesn't this make the whole sandbox
useless?
A: This is still a reasonable behavior, because the sandbox never tried to
isolate your build from the operating system it runs in. Instead it is
supposed to protect your data from a test running "rm -rf $HOME" and to
make it difficult / impossible for actions to use input files that are not
declared dependencies. For even more isolation the sandbox will support
mounting a base image as its root in a future version (similar to Docker
images).
Q: Let's say my process-specific execroot contains a symlink to an input file
"good.h", can't the process just resolve the symlink, strip off the file
name and then look around in the workspace?
A: Yes. Unfortunately we could not find any way on Linux to make a file appear
in a different directory with *all* of the semantics we would like. The
options investigated were:
1) Copying input files, which is much too slow.
2) Hard linking input files, which is fast, but doesn't work cross-
filesystems and it's also not possible to make them read-only.
3) Bind mounts, which don't scale once you're up in the thousands of input
files (across all actions) - it seems like the kernel has some
non-linear performance behavior when the mount table grows too much,
resulting in the mount syscall taking more time the more mounts you
have.
4) FUSE filesystem, good in theory, but wasn't ready for the first
iteration.
RELNOTES: New sandboxing implementation for Linux in which all actions run in a separate execroot that contains input files as symlinks back to the originals in the workspace. The running action now has read-write access to its execroot and /tmp only and can no longer write in arbitrary other places in the file system.
--
Change-Id: Ic91386fc92f8eef727ed6d22e6bd0f357d145063
Reviewed-on: https://bazel-review.googlesource.com/#/c/4053
MOS_MIGRATED_REVID=130638204
|
|
|
|
|
|
|
| |
Part 1 of many for #1262, rolling forward.
--
MOS_MIGRATED_REVID=125334954
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=125160288
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
repositories
One interesting side effect of how this is implemented is that for external
repositories, bin/ and genfiles/ are combined. External repo output is under
bazel-out/local-fastbuild/repo_name for each repo.
Fixes #1262.
RELNOTES[INC]: Previously, an external repository would be symlinked into the
execution root at execroot/local_repo/external/remote_repo. This changes it to
be at execroot/remote_repo. This may break genrules/Skylark actions that
hardcode execution root paths. If this causes breakages for you, ensure that
genrules are using $(location :target) to access files and Skylark rules are
using http://bazel.io/docs/skylark/lib/File.html's path, dirname, etc.
functions.
--
MOS_MIGRATED_REVID=125095799
|
|
|
|
|
|
|
|
|
| |
BlazeRuntime#getProductName() or a reference to TestConstants.PRODUCT_NAME for tests.
This CL prepares the codebase in order to delete the constant.
--
MOS_MIGRATED_REVID=122993568
|
|
|
|
|
|
|
|
|
| |
package's package root when creating top-level directories. Otherwise, if the empty package references files in those top-level directories, they would be hidden.
Fixes #1221
--
MOS_MIGRATED_REVID=121392128
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=114434668
|
|
|
|
|
|
|
| |
This makes it possible to compile //third_party/ijar with a bootstrapped Bazel on Windows in dslomov's tree.
--
MOS_MIGRATED_REVID=114428109
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=110174447
|
|
|
|
|
|
|
| |
Reduces garbage.
--
MOS_MIGRATED_REVID=109914243
|
|
|
|
|
|
|
| |
in#readContentWithLimit().
--
MOS_MIGRATED_REVID=106995917
|
|
|
|
|
|
|
|
|
| |
file rather than a ParserInputSource.
This is part of a series of changes with the net result being that we open, read, and parse each BUILD file exactly once.
--
MOS_MIGRATED_REVID=105911557
|
|
|
|
|
|
|
|
|
|
|
| |
The headers were modified with
`find . -type f -exec 'sed' '-Ei' 's|Copyright 201([45]) Google|Copyright 201\1 The Bazel Authors|' '{}' ';'`
And manual edit for not Google owned copyright. Because of the nature of ijar, I did not modified the header of file owned by Alan Donovan.
The list of authors were extracted from the git log. It is missing older Google contributors that can be added on-demand.
--
MOS_MIGRATED_REVID=103938715
|
|
|
|
|
|
|
| |
caller already knows the path is a symlink, and use this new method throughout the codebase.
--
MOS_MIGRATED_REVID=103229983
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=101976891
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=101628787
|
|
|
|
|
|
|
| |
with Latin-1
--
MOS_MIGRATED_REVID=99651466
|
|
|
|
|
|
|
| |
This is fragile, and there might be other places in the codebase this issue occurs.
--
MOS_MIGRATED_REVID=97784977
|
|
|
|
|
|
|
|
|
| |
This makes the empty package name legal (//:foo). If the empty package is used,
this symlinks everything under the build root to the exec root. This includes
directories.
--
MOS_MIGRATED_REVID=87960882
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=87821306
|
|
--
MOE_MIGRATED_REVID=85702957
|