aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/vfs
Commit message (Collapse)AuthorAge
* Exporting functions to compute SHA1 digests of files.Gravatar Ola Rozenfeld2016-09-12
| | | | | | | | TODO: need to stich them all the way through various ActionInputFileCache implementations! -- MOS_MIGRATED_REVID=132685408
* Windows: add a Java-native isJunction methodGravatar Laszlo Csomor2016-09-09
| | | | | | | | | | | | | | | | | Also add tests for WindowsFileSystem and some for WindowsFileOperations, so we test both the JNI-based and Java-native isJunction function, as well as handling of dangling symlinks/junctions. Having a Java-native version of this method means we don't need to use windows_jni.dll for any tests or for bootstrapping. This change could help with https://github.com/bazelbuild/bazel/issues/1735 -- MOS_MIGRATED_REVID=132556440
* Windows: go back to using Jury-rigged isJunctionGravatar Laszlo Csomor2016-09-08
| | | | | | | | | | | This should unbreak the tests, see https://github.com/bazelbuild/bazel/issues/1735 The tests still break but now due to https://github.com/bazelbuild/bazel/issues/1736 -- MOS_MIGRATED_REVID=132530222
* Fixed the issue that hard links are handled improperly when bazel ↵Gravatar Googler2016-09-08
| | | | | | | | | decompresses tarballs. Issue link: https://github.com/bazelbuild/bazel/issues/574 -- MOS_MIGRATED_REVID=132434278
* Making the PathFragment interned segments list public, to be used later inGravatar Googler2016-09-08
| | | | | | | generating the directory tree nodes efficiently. -- MOS_MIGRATED_REVID=132433991
* Windows, tests: move code into WindowsTestUtilGravatar Laszlo Csomor2016-09-07
| | | | | | | | | | | | Additionally: - clean up the corresponding BUILD file a bit - add a comment to Path A subsequent change will add tests for WindowsFileSystem, then fix a bug there. -- MOS_MIGRATED_REVID=132408212
* Windows, JNI: use native isJunction method.Gravatar Laszlo Csomor2016-09-06
| | | | | | | Fixes https://github.com/bazelbuild/bazel/issues/1680 -- MOS_MIGRATED_REVID=132051176
* Automated [] rollback of commit 846a5ab98fc26d72024890fdb79a5d3bc6a5a1ba + ↵Gravatar Nathan Harmata2016-08-23
| | | | | | | | | | | | | | | manual rollback of [] *** Reason for rollback *** Depot has been fixed / is in the process of being fixed. See the work tracked on [] *** Original change description *** Automated [] rollback of commit bb5d5efb4b50710241b5b374eb67084f4bf08278. -- MOS_MIGRATED_REVID=131095905
* Rollback of commit bb5d5efb4b50710241b5b374eb67084f4bf08278.Gravatar Tobias Werth2016-08-23
| | | | | -- MOS_MIGRATED_REVID=130941264
* Implement the first stage of Bazel's "Sandbox 2.0" for Linux.Gravatar Philipp Wollermann2016-08-18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This has the following improvements upon the older one: - Uses PID namespaces, PR_SET_PDEATHSIG and a number of other tricks for further process isolation and 100% reliable killing of child processes. - Uses clone() instead of unshare() to work around a Linux kernel bug that made creating a sandbox unreliable. - Instead of mounting a hardcoded list of paths + whatever you add with --sandbox_add_path, this sandbox instead mounts all of /, except for what you make inaccessible via --sandbox_block_path. This should solve the majority of "Sandboxing breaks my build, because my compiler is installed in /opt or /usr/local" issues that users have seen. - Instead of doing magic with bind mounts, we create a separate execroot for each process containing symlinks to the input files. This is simpler and gives more predictable performance. - Actually makes everything except the working directory read-only (fixes #1364). This means that a running process can no longer accidentally modify your source code (yay!). - Prevents a number of additional "attacks" or leaks, like accidentally inheriting file handles from the parent. - Simpler command-line interface. - We can provide the same semantics in a Mac OS X sandbox, which will come in a separate code review from yueg@. It has the following caveats / known issues: - The "fallback to /bin/bash on error" feature is gone, but now that the sandbox mounts everything by default, the main use-case for this is no longer needed. The following improvements are planned: - Use a FUSE filesystem if possible for the new execroot, instead of creating symlinks. - Mount a base image instead of "/". FAQ: Q: Why is mounting all of "/" okay, doesn't this make the whole sandbox useless? A: This is still a reasonable behavior, because the sandbox never tried to isolate your build from the operating system it runs in. Instead it is supposed to protect your data from a test running "rm -rf $HOME" and to make it difficult / impossible for actions to use input files that are not declared dependencies. For even more isolation the sandbox will support mounting a base image as its root in a future version (similar to Docker images). Q: Let's say my process-specific execroot contains a symlink to an input file "good.h", can't the process just resolve the symlink, strip off the file name and then look around in the workspace? A: Yes. Unfortunately we could not find any way on Linux to make a file appear in a different directory with *all* of the semantics we would like. The options investigated were: 1) Copying input files, which is much too slow. 2) Hard linking input files, which is fast, but doesn't work cross- filesystems and it's also not possible to make them read-only. 3) Bind mounts, which don't scale once you're up in the thousands of input files (across all actions) - it seems like the kernel has some non-linear performance behavior when the mount table grows too much, resulting in the mount syscall taking more time the more mounts you have. 4) FUSE filesystem, good in theory, but wasn't ready for the first iteration. RELNOTES: New sandboxing implementation for Linux in which all actions run in a separate execroot that contains input files as symlinks back to the originals in the workspace. The running action now has read-write access to its execroot and /tmp only and can no longer write in arbitrary other places in the file system. -- Change-Id: Ic91386fc92f8eef727ed6d22e6bd0f357d145063 Reviewed-on: https://bazel-review.googlesource.com/#/c/4053 MOS_MIGRATED_REVID=130638204
* RELNOTES: The string list returned by the skylark 'glob' function is now ↵Gravatar Nathan Harmata2016-08-18
| | | | | | | | | | | | | | sorted. Previously, it would return a list formed by concatenating the sorted results of each pattern in the 'includes' list. A bunch of cleanups and one bug fix: -Remove the unused-except-for tests GlobCache#globsUpToDate. This code has been dead for a very very long time, ever since we switched to using Skyframe. -Change the semantics of the 'glob' function as described above. -Change UnixGlob to return unsorted results. Document this in UnixGlob and GlobCache. -Change LegacyGlobber to conditionally return sorted results. Have users other than PackageFunction get sorted results (as described above). Have PackageFunction's use case get completely unsorted results, and have PackageFunction do the sorting itself. -Have PackageFunction's HybridGlobber unconditionally sort the glob result list. This ensure deterministic glob results, fixing a bug where the order of the elements of the result depended on the contents of the Skyframe graph, which of course depends on the sequence of incremental Blaze commands. -- MOS_MIGRATED_REVID=130540152
* Remove the experimental hardlink / junction code from UnixFileSystem.Gravatar Ulf Adams2016-08-11
| | | | | -- MOS_MIGRATED_REVID=129863453
* Fixed JavaIoFileSystem.setLastModifiedTime to actually match its documented ↵Gravatar Chris Parsons2016-07-27
| | | | | | | behavior (passing -1 modified time should use system time). -- MOS_MIGRATED_REVID=128489592
* Typo fixes in markdown and javadoc as suggested by intellij typo inspection.Gravatar Googler2016-07-27
| | | | | -- MOS_MIGRATED_REVID=128476121
* Remove not-quite necessary serialization bitsGravatar Michajlo Matijkiw2016-07-15
| | | | | | | What we really are doing here is formatting. -- MOS_MIGRATED_REVID=127481183
* Fix interning in PathFragment.create(char, bool, String[])Gravatar Michajlo Matijkiw2016-06-24
| | | | | -- MOS_MIGRATED_REVID=125721556
* Move execroot tree creation to its own classGravatar Kristina Chodorow2016-06-21
| | | | | | | Part 1 of many for #1262, rolling forward. -- MOS_MIGRATED_REVID=125334954
* Rollback of commit bdfd58a8ca2ed5735d6aaa5b238fb0f689515724.Gravatar Laurent Le Brun2016-06-20
| | | | | -- MOS_MIGRATED_REVID=125160288
* Make the execution root match the runfiles tree structure for external ↵Gravatar Kristina Chodorow2016-06-17
| | | | | | | | | | | | | | | | | | | | | repositories One interesting side effect of how this is implemented is that for external repositories, bin/ and genfiles/ are combined. External repo output is under bazel-out/local-fastbuild/repo_name for each repo. Fixes #1262. RELNOTES[INC]: Previously, an external repository would be symlinked into the execution root at execroot/local_repo/external/remote_repo. This changes it to be at execroot/remote_repo. This may break genrules/Skylark actions that hardcode execution root paths. If this causes breakages for you, ensure that genrules are using $(location :target) to access files and Skylark rules are using http://bazel.io/docs/skylark/lib/File.html's path, dirname, etc. functions. -- MOS_MIGRATED_REVID=125095799
* Refactor UnixGlob by consolidating the context of a glob subtask into a ↵Gravatar Nathan Harmata2016-06-07
| | | | | | | GlobTaskContext object. Also dedupe identical recursive calls that arise from our naive implementation of the glob algorithm. -- MOS_MIGRATED_REVID=124159729
* Remove the dead UnixGlob#Builder#addExcludes and move some existing tests ↵Gravatar Nathan Harmata2016-06-07
| | | | | | | around so that we still have good test coverage for 'excludes' in globs. -- MOS_MIGRATED_REVID=124152499
* Replace the occurrences of Constants.PRODUCT_NAME for a call toGravatar Luis Fernando Pino Duque2016-05-23
| | | | | | | | | BlazeRuntime#getProductName() or a reference to TestConstants.PRODUCT_NAME for tests. This CL prepares the codebase in order to delete the constant. -- MOS_MIGRATED_REVID=122993568
* Fix Windows issue in Path.getRelative, which wasn't handling backslashes.Gravatar Ulf Adams2016-05-19
| | | | | -- MOS_MIGRATED_REVID=122627792
* Remove extraneous space added by commit 5e34a3d4ea30525971e320ce283ba8cd67c74e3cGravatar Nathan Harmata2016-05-17
| | | | | -- MOS_MIGRATED_REVID=122542339
* Support case-insensitive comparision in Path.java with WindowsFileSystemGravatar Yun Peng2016-05-09
| | | | | | | | Since file path is case-insensitive on Windows, we need to support this. Also fixed .d file inclusions check in CppCompileAction.java on Windows -- MOS_MIGRATED_REVID=121823250
* When creating the exec root symlink forest, take account of the empty ↵Gravatar Janak Ramakrishnan2016-05-03
| | | | | | | | | package's package root when creating top-level directories. Otherwise, if the empty package references files in those top-level directories, they would be hidden. Fixes #1221 -- MOS_MIGRATED_REVID=121392128
* Short-circuit Path.equals() with hashcode.Gravatar Googler2016-04-21
| | | | | | | Optimize .equals() by using the hashcode calculated in the constructor to determine if we can short-circuit equals and avoid traversing the parents if the two paths can't be equal. -- MOS_MIGRATED_REVID=120363376
* Move FDO support to the analysis phase by wrapping FdoSupport in its own ↵Gravatar Lukacs Berki2016-04-07
| | | | | | | | | | | SkyFunction. This removes one of the two reasons for the existence of BuildConfiguration#prepareToBuild() which makes implementing dynamic configurations impossible and also makes FDO support halfway sane; now FDO is exactly as ugly as remote repositories, that is to say, reasonably okay. Ideally, we'd implement the zip extraction as an Action and make it a TreeArtifact, but support for TreeArtifacts is not mature yet enough, so it's not possible at the moment. -- MOS_MIGRATED_REVID=119150223
* Roll-forward of commit 4bf0018ed1cf8616297b951dc03dbde3f0db2503 with code to ↵Gravatar Janak Ramakrishnan2016-03-15
| | | | | | | preserve order of glob matches: Parallelize fetches of symlink file values, subdirectory globs, and subdirectory package lookup values. This should improve change pruning speed when we have to check a glob. It also keeps GlobFunction closer to the contract of Skyframe, because in order to avoid quadratic restarts, it wasn't checking for missing deps between getValue calls. -- MOS_MIGRATED_REVID=117139471
* Global cleanup change.Gravatar Googler2016-03-02
| | | | | -- MOS_MIGRATED_REVID=116138214
* Remove stray e.printStackTrace()Gravatar Dmitry Lomov2016-02-16
| | | | | -- MOS_MIGRATED_REVID=114697873
* Publicize some ByteSink/ByteSource creators in FileSystemUtils.Gravatar Eric Fellheimer2016-02-11
| | | | | -- MOS_MIGRATED_REVID=114434668
* Make local_repository and new_local_repository work on Windows.Gravatar Lukacs Berki2016-02-11
| | | | | | | This makes it possible to compile //third_party/ijar with a bootstrapped Bazel on Windows in dslomov's tree. -- MOS_MIGRATED_REVID=114428109
* Merge Dmitry's jury-rigged WindowsFileSystem to decrease the number of ↵Gravatar Lukacs Berki2016-01-29
| | | | | | | | | changes between his branch and HEAD. This code will need some JNI to make it work properly, but for now, it will do. -- MOS_MIGRATED_REVID=113349143
* Add documentation about our plans to handle symlinks on Windows.Gravatar Lukacs Berki2016-01-27
| | | | | -- MOS_MIGRATED_REVID=113154309
* Rename FilesystemUtils to NativePosixFiles.Gravatar Lukacs Berki2016-01-27
| | | | | | | This helps avoid confusion with File*S*ystemUtils, which differs in only the case of a character but is a completely different class. -- MOS_MIGRATED_REVID=113054116
* Make local_repository and new_local_repository work with the limitations of ↵Gravatar Lukacs Berki2016-01-26
| | | | | | | our plans for symlink support on Windows. -- MOS_MIGRATED_REVID=113043269
* Add a (working) windows compatibility mode that makes Bazel create hardlinks ↵Gravatar Lukacs Berki2016-01-26
| | | | | | | | | | | for links to writable files. Curiously enough, the native Unix JNI wrapper already had a function for link(), but it wasn't on the Java interface. build-runfiles is also updated accordingly. -- MOS_MIGRATED_REVID=113029168
* Create the convenience symlinks only after the build, and only those that ↵Gravatar Lukacs Berki2016-01-21
| | | | | | | | | point to an existing directory. This required fixing the worker strategy so that it reads params files not through said convenience symlinks, but from the execroot. -- MOS_MIGRATED_REVID=112682485
* Scaffolding for implementing symlinks on Windows. Gravatar Lukacs Berki2016-01-21
| | | | | | | | | | | | | | | At first, this will only be used for emulating the planned implementation on Linux to validate it a little more before starting with the big work of porting everything to Windows in case it is doomed to failure. In logging mode, the only places where we create symbolic links that we can't emulate with the plan (pointing to a non-existent file or to a file outside the output base and the source root, which are assumed to be writable): - ExecutionTool.createOutputDirectoryLinks(). If we won't have the convenience symlinks on Windows, I won't shed a tear (I'm wondering why, though, because they are between the output base and the source tree) - In the implementation of new_local_repository (Would need to be special-cased for Windows. No big deal.) - In the implementation of the .tar.gz decompressor (doesn't seem to be serious, either.) So this seems to be alright. Note, however, that we didn't check build-runfiles.cc, which might cause trouble. I don't remember any place where we create a link there that is illegal according to the above rules, though. -- MOS_MIGRATED_REVID=112659070
* Add a Java property to influence symlinking strategy so that we can test ↵Gravatar Lukacs Berki2016-01-20
| | | | | | | what kind of performance we could get from how we imagine it would work under Windows. -- MOS_MIGRATED_REVID=112572621
* cache SDKROOT value for local execution of actions using apple toolkitGravatar Chris Parsons2015-12-15
| | | | | -- MOS_MIGRATED_REVID=110174447
* Use Bazel Preconditions variant which avoids varargs array creationGravatar Mark Schaller2015-12-10
| | | | | | | Reduces garbage. -- MOS_MIGRATED_REVID=109914243
* ASwB aspect: parse java packages during executionGravatar Googler2015-12-03
| | | | | -- MOS_MIGRATED_REVID=109305952
* Make AbstractFileSystem correct in isolation. Previously there was ↵Gravatar Nathan Harmata2015-11-19
| | | | | | | | | (accidentally) the implicit requirement that the deriving class override FileSystem#stat. Even though this wasn't a problem in practice in the Bazel codebase (since all of our transitive subclasses had custom 'stat' implementation), it's good hygiene to have things correct (e.g. if we added a new subclass). -- MOS_MIGRATED_REVID=108158039
* C++ libraries in remote repos don't need to set include pathsGravatar Kristina Chodorow2015-11-12
| | | | | | | | | Fixes #445, based on https://github.com/bazelbuild/bazel/compare/master...ulfjack:cpp-include-path. RELNOTES: C++ libraries no longer need includes = ["."] (or similar copts) to include paths relative to a remote repository's root. -- MOS_MIGRATED_REVID=107593486
* Close some streams that we didn'tGravatar Miguel Alcon Pinto2015-11-05
| | | | | -- MOS_MIGRATED_REVID=107048547
* Avoid copying file contents when reading the expected number of bytes ↵Gravatar Eric Fellheimer2015-11-04
| | | | | | | in#readContentWithLimit(). -- MOS_MIGRATED_REVID=106995917
* Augment JNI utime() method.Gravatar Eric Fellheimer2015-10-30
| | | | | -- MOS_MIGRATED_REVID=106634616
* Reduce memory churn while creating PathFragments.Gravatar Eric Fellheimer2015-10-27
| | | | | -- MOS_MIGRATED_REVID=106329484
generated by cgit v1.2.3 (git 2.39.1) at 2024-10-21 00:34:57 +0000