aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/vfs/FileSystem.java
Commit message (Collapse)AuthorAge
* Add easy access to MessageDigests in DigestHashFunctionGravatar ccalvarin2018-08-13
| | | | | | | Move the message-digest cloning to DigestHashFunction and out of Fingerprint, to make it possible to configure Fingerprint to use different hash functions. We keep the default MD5 for now, we'd like it to use the global default but want to isolate the configuration change from any change adding potential contention. RELNOTES: None. PiperOrigin-RevId: 208502993
* Add a normal startup-option for setting the digest function.Gravatar ccalvarin2018-08-01
| | | | | | | | | We continue to support the jvm property -Dbazel.DigestFunction, for backwards compatibility, but this will go away. The startup-option is marked experimental for now as we iron out issues. (note: leaving this out of release notes until the experimental tag is removed) As part of this refactor, the default constructor calls for FileSystem and derived classes will now use this default. This should remove the need for constructors that accept custom hash functions. RELNOTES: None. PiperOrigin-RevId: 207035217
* Automated rollback of commit 7e87730de985b7099b9b683571d58efdaab70890.Gravatar ccalvarin2018-07-27
| | | | | | | | | | | | | | | | | *** Reason for rollback *** Go back to the default constructor - instead of requiring everywhere to know the correct hash function, we'll have the default rely on global state. It will make transition easier, even if it makes the origin of the hash less obvious. *** Original change description *** Remove default MD5 in most of Bazel's virtual filesystems. This forces the ex-default to be explicit in a lot of tests, but I'd rather that than have the risk of implicit md5-use in production code. To keep this CL smaller, do not remove the default from UnixFS quite yet. RELNOTES: None. PiperOrigin-RevId: 206358838
* Remove default MD5 in most of Bazel's virtual filesystems.Gravatar ccalvarin2018-07-26
| | | | | | | | | This forces the ex-default to be explicit in a lot of tests, but I'd rather that than have the risk of implicit md5-use in production code. To keep this CL smaller, do not remove the default from UnixFS quite yet. RELNOTES: None. PiperOrigin-RevId: 206223521
* Remove redundancy in DigestHashFunction use in FileSystem.Gravatar ccalvarin2018-07-23
| | | | | | | Each FileSystem instance has a digest function, but the getDigest and getHashDigest functions also accepted their own custom parameter functions. We only support 1 hash per filesystem instance, these parameters are redundant. RELNOTES: None. PiperOrigin-RevId: 205758571
* Bazel server, VFS: revert to using asByteSourceGravatar laszlocsomor2018-07-10
| | | | | | | | | | | | | | | | | | commit 59f17d6e0550bf63a0b6ef182e2d63474e058ede updated files to use try-with-resources when dealing with streams. The change also got rid of asByteSource, replacing it with throw-away ByteSource instances wrapping a try-with-resources-guarded InputStream. Doing so was unnecessary though, because all ByteSource methods (except for open[Buffered]InputStream) close the stream. Thanks to @jbduncan to point that out and explain in detail [1]. [1] see comment thread on https://github.com/bazelbuild/bazel/commit/59f17d6e0550bf63a0b6ef182e2d63474e058ede under `FilesetManifest.java` RELNOTES: none PiperOrigin-RevId: 203934830
* Bazel server: ensure InputStreams are closedGravatar laszlocsomor2018-07-05
| | | | | | | | | | | | | | | | | | | Use try-with-resources to ensure InputStreams that we open via FileSystem.InputStream(path) are closed. Eagerly closing InputStreams avoids hanging on to file handles until the garbage collector finalizes the InputStream, meaning Bazel on Windows (and other processes) can delete or mutate these files. Hopefully this avoids intermittent file deletion errors that sometimes occur on Windows. See https://github.com/bazelbuild/bazel/issues/5512 RELNOTES: none PiperOrigin-RevId: 203338148
* Move HashFunction out of FileSystem, and turn it into a class, instead of an ↵Gravatar ccalvarin2018-06-21
| | | | | | | | | enum. Now that we aren't using enum names for the hash functions, we also accept the standard names, such as SHA-256. RELNOTES: None. PiperOrigin-RevId: 201624286
* Clean up code that directly imports nested classes like Builder, Entry, etc.Gravatar jcater2018-05-01
| | | | PiperOrigin-RevId: 195040539
* Replace path implementation.Gravatar tomlu2018-02-08
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Path and PathFragment have been replaced with String-based implementations. They are pretty similar, but each method is dissimilar enough that I did not feel sharing code was appropriate. A summary of changes: PATH ==== * Subsumes LocalPath (deleted, its tests repurposed) * Use a simple string to back Path * Path instances are no longer interned; Reference equality will no longer work * Always normalized (same as before) * Some operations will now be slower, like instance compares (which were previously just a reference check) * Multiple identical paths will now consume more memory since they are not interned PATH FRAGMENT ============= * Use a simple string to back PathFragment * No more segment arrays with interned strings * Always normalized * Remove isNormalized * Replace some isNormalizied uses with containsUpLevelReferences() to check if path fragments try to escape their scope * To check if user input is normalized, supply static methods on PathFragment to validate the string before constructing a PathFragment * Because PathFragments are always normalized, we have to replace checks for literal "." from PathFragment#getPathString to PathFragment#getSafePathString. The latter returns "." for the empty string. * The previous implementation supported efficient segment semantics (segment count, iterating over segments). This is now expensive since we do longer have a segment array. ARTIFACT ======== * Remove Path instance. It is instead dynamically constructed on request. This is necessary to avoid this CL becoming a memory regression. RELNOTES: None PiperOrigin-RevId: 185062932
* Add absolute root concept.Gravatar tomlu2018-01-18
| | | | | | | | An absolute root accepts any absolute path fragments and simply returns it. This concept replaces the FileSystem root directory concept. PiperOrigin-RevId: 182400471
* Introduce Root class.Gravatar tomlu2018-01-17
| | | | | | | | | | | This class represents a root (such as a package path or an output root) used for file lookups and artifacts. It is meant to be as opaque as possible in order to hide the user's environment from sky keys and sky functions. Roots are used by RootedPaths and ArtifactRoots. This CL attempts to make the minimum number of modifications necessary to change RootedPath and ArtifactRoot to use these fields. Deprecated methods and invasive accessors are permitted to minimise the risk of any observable changes. RELNOTES: None PiperOrigin-RevId: 182271759
* Add FileSystem#createDirectoryAndParents.Gravatar tomlu2017-12-21
| | | | | | | | A native implementation of this (instead of using FileSystemUtils, which can only use public interfaces) should be more efficient and more easy to make correct. In particular, it should allow removing FileSystemUtils#createDirectoriesAndParents, which has poor thread safety characteristics. The latter method has a lot of logic that forces certain unnatural atomicity guarantees on createDirectory, and it also has logic that is conditional on sub-string content of exception messages. PiperOrigin-RevId: 179819623
* Automated rollback of commit 82e68b75304438c96ff878a0c2b8d18b42002486.Gravatar aehlig2017-12-19
| | | | | | | | | | | | | | Fixes #4322, #4306. *** Reason for rollback *** Introduces a deadlock (see https://github.com/bazelbuild/bazel/issues/4322) *** Original change description *** Make FileSystem operate on LocalPath instead of Path. PiperOrigin-RevId: 179549866
* Make FileSystem operate on LocalPath instead of Path.Gravatar tomlu2017-12-14
| | | | PiperOrigin-RevId: 179082062
* Thread FileSystem instance through a few call sites.Gravatar tomlu2017-12-11
| | | | PiperOrigin-RevId: 178704585
* Thread filesystem through codebase.Gravatar tomlu2017-12-08
| | | | | RELNOTES: None PiperOrigin-RevId: 178426166
* Refactor the FileSystem API to allow for different hash functions.Gravatar buchgr2017-11-30
| | | | | | | | | | | | | Refactor the FileSystem class to include the hash function as an instance field. This allows us to have a different hash function per FileSystem and removes technical debt, as currently that's somewhat accomplished by a horrible hack that has a static method to set the hash function for all FileSystem instances. The FileSystem's default hash function remains MD5. RELNOTES: None PiperOrigin-RevId: 177479772
* Change FileSystem#getDirectoryEntries to return strings of the ↵Gravatar tomlu2017-10-23
| | | | | | | | file/directory names instead of paths. This is a small isolated change that can be done ahead of the big refactoring. PiperOrigin-RevId: 173124518
* Lazy-initialise rootPath in FileSystem.Gravatar tomlu2017-10-23
| | | | | | Path construction entails calling isFileSystemCaseSensitive at a time when subclasses have not been initialised. This can cause a crash. Solve by deferring init of the path. PiperOrigin-RevId: 172914501
* Add Path argument to supportsModifications, supports*LinksNatively.Gravatar tomlu2017-10-20
| | | | | | | | This works better with UnionFileSystem, as (eg.) different mapped file systems might not agree on whether they are read-only. No actual code changes have been made that actually uses this argument, so this should be a behaviour no-op. PiperOrigin-RevId: 172782759
* Adding support for SHA256 for remote execution. Switch remote execution to useGravatar olaola2017-06-20
| | | | | | | | | the currently defined hash function for blobs. Some refactoring. Adding an option to set the hash function in the remote worker, defaulting to the current behavior (unfortunately it is a build option, have not found a clean way to specify it at runtime). BUG=62622420 TESTED=remote worker RELNOTES: none PiperOrigin-RevId: 159473116
* Cleanup: addressing comments from commit ↵Gravatar olaola2017-06-16
| | | | | | | 9eea05d068a06ab642dd9d86d46ee5fa2e36b02e. RELNOTES: n/a PiperOrigin-RevId: 158988688
* Refactor all ctor callsites of PathFragment to instead call a static ↵Gravatar nharmata2017-04-05
| | | | | | | | | | | | 'create' method. This paves the way for changing PathFragment to e.g. an abstract class with multiple subclasses. This way we can split out the windows-specific stuff into one of these concrete classes, making the code more readable and also saving memory (since the shallow heap size of the NonWindowsPathFragment subclass will hopefully be smaller than that of the current PathFragment). This also lets us pursue gc churn optimizations. We can now do interning in PathFragment#create and can also get rid of unnecessary intermediate PathFragment allocations. RELNOTES: None PiperOrigin-RevId: 152145768
* Restructure Path#getCachedChildPath to make direct method calls without ↵Gravatar nharmata2017-04-04
| | | | | | | | allocating a garbage TranslatedPath object. RELNOTES: None PiperOrigin-RevId: 152130170
* Move FileAccessException to the vfs packageGravatar Ulf Adams2017-02-17
| | | | | | | | | | | | | | It was previously in unix, but also used from non-unix file systems, which means it's not actually unix-specific. This is in preparation for splitting compilation of the unix and windows file systems into separate libraries. That improves layering and reduces compile times - note that Bazel already injects the vfs into its lower layers, which should only rely on the normal vfs APIs, not on anything platform-specific. -- PiperOrigin-RevId: 147829659 MOS_MIGRATED_REVID=147829659
* Replace UnixGlob's package glob prefetching functionality with anGravatar Googler2016-12-02
| | | | | | | | optional method in FileSystem. Custom FileSystem implementations can use this to provide their own implementation of glob prefetching. -- MOS_MIGRATED_REVID=140736304
* Adding an option to set the digest function that everything uses. Minor ↵Gravatar Ola Rozenfeld2016-11-18
| | | | | | | | | | | | | refactoring: enabling potential fast digest computation of more than one digest function type. Usage: bazel --host_jvm_args="-Dbazel.DigestFunction=SHA1" build ... Ugliness: using a system property (a static non-final variable), because the better way to do it (a flag) would result in a much, much larger refactoring. More ugliness: I have updated the minimal amount of tests. A lot of tests are still relying on the default value of MD5. Ideally, they need to be updated as well. -- MOS_MIGRATED_REVID=139490836
* VFS: implement a Windows-specific Path subclassGravatar Laszlo Csomor2016-10-25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This change rolls forward commit e0d7a540e3c615c628f63fcaaaba0c47fca2cb25 and commit 8bb4299b28de14eed9d3b57bcaeb9350c81c7db3, and adds a bugfix: - FileSystem.PathFactory got a new translatePath method that WindowsFileSystem.PathFactory overrides to translate absolute Unix paths to MSYS-relative paths - Path.getCachedChildPath calls this translatePath method so the child path is registered with the correct (translated) parent and under the correct name (e.g. "C:" instead of say "c") Below is the rest of the original change description: The new subclass WindowsFileSystem.WindowsPath is aware of Windows drives. This change: - introduces a new factory for Path objects so FileSystems can return a custom implementation that instantiates filesystem-specific Paths - implements the WindowsPath subclass of Path that is aware of Windows drives - introduces the bazel.windows_unix_root JVM argument that defines the MSYS root, which defines the absolute Windows path that is the root of all Unix paths that Bazel creates (e.g. "/usr/lib" -> "C:/tools/msys64/usr/lib") except if the path is of the form "/c/foo" which is treated as "C:/foo" - removes all Windows-specific logic from Path PathFragment is still aware of drive letters and it has to remain so because it is unaware of file systems. WindowsPath restricts the allowed path strings to absolute Unix paths where the first segment, if any, is a volume specifier. From now on if Bazel attempts to create a WindowsPath from an absolute Unix path, Bazel will make it relative to WindowsPath.UNIX_ROOT, unless the first component is a single-letter name (e.g. "/c/foo" which is "C:/foo"). Subclassing Path is necessary because a Unix-style absolute path doesn't sufficiently define a full Windows path, as it may be relative to any drive. Fixes https://github.com/bazelbuild/bazel/issues/1463 -- MOS_MIGRATED_REVID=137149483
* Rollback of commit e0d7a540e3c615c628f63fcaaaba0c47fca2cb25.Gravatar Philipp Wollermann2016-10-19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Suspected root cause for Windows bootstrap on Bazel CI breakage: java.lang.NullPointerException at com.google.devtools.build.lib.vfs.Path$1.run(Path.java:123) http://ci.bazel.io/view/Bazel%20bootstrap%20and%20maintenance/job/Bazel/922/JAVA_VERSION=1.8,PLATFORM_NAME=windows-x86_64/console *** Original change description *** VFS: implement a Windows-specific Path subclass The new subclass WindowsFileSystem.WindowsPath is aware of Windows drives. This change: - introduces a new factory for Path objects so FileSystems can return a custom implementation that instantiates filesystem-specific Paths - implements the WindowsPath subclass of Path that is aware of Windows drives - introduces the bazel.windows_unix_root JVM argument that defines the MSYS root, which defines the absolute Windows path that is the... *** -- MOS_MIGRATED_REVID=136583352
* VFS: implement a Windows-specific Path subclassGravatar Laszlo Csomor2016-10-18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The new subclass WindowsFileSystem.WindowsPath is aware of Windows drives. This change: - introduces a new factory for Path objects so FileSystems can return a custom implementation that instantiates filesystem-specific Paths - implements the WindowsPath subclass of Path that is aware of Windows drives - introduces the bazel.windows_unix_root JVM argument that defines the MSYS root, which defines the absolute Windows path that is the root of all Unix paths that Bazel creates (e.g. "/usr/lib" -> "C:/tools/msys64/usr/lib") except if the path is of the form "/c/foo" which is treated as "C:/foo" - removes all Windows-specific logic from Path PathFragment is still aware of drive letters and it has to remain so because it is unaware of file systems. WindowsPath restricts the allowed path strings to absolute Unix paths where the first segment, if any, is a volume specifier. From now on if Bazel attempts to create a WindowsPath from an absolute Unix path, Bazel will make it relative to WindowsPath.UNIX_ROOT, unless the first component is a single-letter name (e.g. "/c/foo" which is "C:/foo"). Subclassing Path is necessary because a Unix-style absolute path doesn't sufficiently define a full Windows path, as it may be relative to any drive. Fixes https://github.com/bazelbuild/bazel/issues/1463 -- MOS_MIGRATED_REVID=136350304
* Exporting functions to compute SHA1 digests of files.Gravatar Ola Rozenfeld2016-09-12
| | | | | | | | TODO: need to stich them all the way through various ActionInputFileCache implementations! -- MOS_MIGRATED_REVID=132685408
* Fixed the issue that hard links are handled improperly when bazel ↵Gravatar Googler2016-09-08
| | | | | | | | | decompresses tarballs. Issue link: https://github.com/bazelbuild/bazel/issues/574 -- MOS_MIGRATED_REVID=132434278
* Support case-insensitive comparision in Path.java with WindowsFileSystemGravatar Yun Peng2016-05-09
| | | | | | | | Since file path is case-insensitive on Windows, we need to support this. Also fixed .d file inclusions check in CppCompileAction.java on Windows -- MOS_MIGRATED_REVID=121823250
* Make local_repository and new_local_repository work with the limitations of ↵Gravatar Lukacs Berki2016-01-26
| | | | | | | our plans for symlink support on Windows. -- MOS_MIGRATED_REVID=113043269
* Close some streams that we didn'tGravatar Miguel Alcon Pinto2015-11-05
| | | | | -- MOS_MIGRATED_REVID=107048547
* Introduce Path#isSpecialFile, FileSystem#isSpecialFile, and ↵Gravatar Nathan Harmata2015-10-21
| | | | | | | FileStatus#isSpecialFile to help disambiguate between a regular file and a special file, since the file size of a special file cannot be trusted. -- MOS_MIGRATED_REVID=105903622
* Rationalize copyright headersGravatar Damien Martin-Guillerez2015-09-25
| | | | | | | | | | | The headers were modified with `find . -type f -exec 'sed' '-Ei' 's|Copyright 201([45]) Google|Copyright 201\1 The Bazel Authors|' '{}' ';'` And manual edit for not Google owned copyright. Because of the nature of ijar, I did not modified the header of file owned by Alan Donovan. The list of authors were extracted from the git log. It is missing older Google contributors that can be added on-demand. -- MOS_MIGRATED_REVID=103938715
* Introduce Path#readSymbolicLinkUnchecked, intended to only be used when the ↵Gravatar Nathan Harmata2015-09-17
| | | | | | | caller already knows the path is a symlink, and use this new method throughout the codebase. -- MOS_MIGRATED_REVID=103229983
* Remove unused 'followSymlinks' parameter to FileSystem#getxattr. It is ↵Gravatar Nathan Harmata2015-09-16
| | | | | | | always true in practice. -- MOS_MIGRATED_REVID=103221081
* Minor FileSystem refactoring.Gravatar Eric Fellheimer2015-06-03
| | | | | -- MOS_MIGRATED_REVID=95054907
* Slight refactoring of readdir().Gravatar Eric Fellheimer2015-06-03
| | | | | -- MOS_MIGRATED_REVID=95036998
* Some cleanup changes.Gravatar Ulf Adams2015-03-05
| | | | | -- MOS_MIGRATED_REVID=87821306
* Update from Google.Gravatar Han-Wen Nienhuys2015-02-25
-- MOE_MIGRATED_REVID=85702957
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-18 20:22:26 +0000