aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/vfs
Commit message (Collapse)AuthorAge
* Support tree artifacts in target complete.Gravatar tomlu2018-08-15
| | | | | RELNOTES: None PiperOrigin-RevId: 208861642
* Have DigestMap support multiple hash functions.Gravatar tomlu2018-08-15
| | | | | RELNOTES: None PiperOrigin-RevId: 208837641
* Add easy access to MessageDigests in DigestHashFunctionGravatar ccalvarin2018-08-13
| | | | | | | Move the message-digest cloning to DigestHashFunction and out of Fingerprint, to make it possible to configure Fingerprint to use different hash functions. We keep the default MD5 for now, we'd like it to use the global default but want to isolate the configuration change from any change adding potential contention. RELNOTES: None. PiperOrigin-RevId: 208502993
* Pass Fileset information to the Action filesystem.Gravatar felly2018-08-09
| | | | PiperOrigin-RevId: 208096548
* Add support to completion function to create a path resolver from its ↵Gravatar tomlu2018-08-03
| | | | | | | looked-up artifact values. RELNOTES: None PiperOrigin-RevId: 207295716
* Disables ActionFS sandboxing of discovered derived inputs forGravatar shahan2018-08-02
| | | | | | performance reasons. PiperOrigin-RevId: 207171305
* Add a normal startup-option for setting the digest function.Gravatar ccalvarin2018-08-01
| | | | | | | | | We continue to support the jvm property -Dbazel.DigestFunction, for backwards compatibility, but this will go away. The startup-option is marked experimental for now as we iron out issues. (note: leaving this out of release notes until the experimental tag is removed) As part of this refactor, the default constructor calls for FileSystem and derived classes will now use this default. This should remove the need for constructors that accept custom hash functions. RELNOTES: None. PiperOrigin-RevId: 207035217
* Slightly improve UnixGlob implementation. For recursive patterns, the currentGravatar Googler2018-07-31
| | | | | | | | | | | | | | | implementation leads to two identical recursive calls. This isn't harmful as there is a cache that will make the second call very cheap, but does not seem right. As a consequence, a cache is only needed if there are two recursive patterns in a single glob-path, e.g. /a/b/**/c/**/d/... as either "**" can expand to 0 or more directories (so /a/b/c/c/d could be reached in two different ways). If there is only a single recursive pattern, even with "*" placeholders, there is always a unique expansion (a "*" always represents exactly one path element). RELNOTES: None. PiperOrigin-RevId: 206753919
* Automated rollback of commit 7e87730de985b7099b9b683571d58efdaab70890.Gravatar ccalvarin2018-07-27
| | | | | | | | | | | | | | | | | *** Reason for rollback *** Go back to the default constructor - instead of requiring everywhere to know the correct hash function, we'll have the default rely on global state. It will make transition easier, even if it makes the origin of the hash less obvious. *** Original change description *** Remove default MD5 in most of Bazel's virtual filesystems. This forces the ex-default to be explicit in a lot of tests, but I'd rather that than have the risk of implicit md5-use in production code. To keep this CL smaller, do not remove the default from UnixFS quite yet. RELNOTES: None. PiperOrigin-RevId: 206358838
* Remove default MD5 in most of Bazel's virtual filesystems.Gravatar ccalvarin2018-07-26
| | | | | | | | | This forces the ex-default to be explicit in a lot of tests, but I'd rather that than have the risk of implicit md5-use in production code. To keep this CL smaller, do not remove the default from UnixFS quite yet. RELNOTES: None. PiperOrigin-RevId: 206223521
* Use the path resolver when dealing with tree artifacts.Gravatar felly2018-07-26
| | | | | RELNOTES: None PiperOrigin-RevId: 206157591
* Use a structured Exception type for the "unexpected short read" error ↵Gravatar nharmata2018-07-25
| | | | | | | detected by FileSystemUtils#readWithKnownFileSize. RELNOTES: None PiperOrigin-RevId: 205995852
* Remove redundancy in DigestHashFunction use in FileSystem.Gravatar ccalvarin2018-07-23
| | | | | | | Each FileSystem instance has a digest function, but the getDigest and getHashDigest functions also accepted their own custom parameter functions. We only support 1 hash per filesystem instance, these parameters are redundant. RELNOTES: None. PiperOrigin-RevId: 205758571
* vfs: do not close streams twiceGravatar Laszlo Csomor2018-07-20
| | | | | | | | | | | | | | ByteStream's functions already close the underlying stream. See https://github.com/bazelbuild/bazel/commit/09d20311d982606093ed881d779bb05a5ee70ed3 Change-Id: Id389ef594946bfebb90ca66d97ea96f271b20331 Closes #5641. Change-Id: Id389ef594946bfebb90ca66d97ea96f271b20331 PiperOrigin-RevId: 205395805
* Add a utility function to hash a directoryGravatar Klaus Aehlig2018-07-20
| | | | | | | | | | Return a hash of a directory that is suitable to verify whether a repository rule contained a good snapshot of source code. So certain aspects of the directory, in particular ownership of the files, are deliberately not included in the hash. Change-Id: I1b35f7af47b376808acad3b6e54daaaec4f9ebfd PiperOrigin-RevId: 205382020
* Bazel server, VFS: revert to using asByteSourceGravatar laszlocsomor2018-07-10
| | | | | | | | | | | | | | | | | | commit 59f17d6e0550bf63a0b6ef182e2d63474e058ede updated files to use try-with-resources when dealing with streams. The change also got rid of asByteSource, replacing it with throw-away ByteSource instances wrapping a try-with-resources-guarded InputStream. Doing so was unnecessary though, because all ByteSource methods (except for open[Buffered]InputStream) close the stream. Thanks to @jbduncan to point that out and explain in detail [1]. [1] see comment thread on https://github.com/bazelbuild/bazel/commit/59f17d6e0550bf63a0b6ef182e2d63474e058ede under `FilesetManifest.java` RELNOTES: none PiperOrigin-RevId: 203934830
* Delete UnionFileSystem.Gravatar lberki2018-07-09
| | | | | | | It was used in a single, Google-internal test which I cleaned up as pre-work for this change. RELNOTES: None. PiperOrigin-RevId: 203727293
* vfs: fix race condition in JavaIoFileSystem.deleteGravatar Laszlo Csomor2018-07-09
| | | | | | | | | | | | | | | | | | | | | | | | | | | JavaIoFileSystem.delete() now uses Files.deleteIfExists() to delete files and empty directories. The old code used to have a race condition: a file could have been deleted or changed to a non-directory between checking it's a directory and listing its contents. WindowsFileSystem.delete() now uses the DeletePath JNI method (wrapped by WindowsFileOperations.deletePath) which is more robust than JavaIoFileSystem.delete(), because it can delete readonly files and can tolerate some concurrent filesystem modifications. Fixes https://github.com/bazelbuild/bazel/issues/5513 Change-Id: I5d2720afff8dfd3725880a6d7408d22de5935c3d Closes #5527. Change-Id: I5d2720afff8dfd3725880a6d7408d22de5935c3d PiperOrigin-RevId: 203720575
* Bazel server: ensure OutputStreams are closedGravatar laszlocsomor2018-07-05
| | | | | | | | | | | | | | | | | | | Use try-with-resources to ensure OutputStreams that we open via FileSystem.OutputStream(path) are closed. Eagerly closing OutputStreams avoids hanging on to file handles until the garbage collector finalizes the OutputStream, meaning Bazel on Windows (and other processes) can delete or mutate these files. Hopefully this avoids intermittent file deletion errors that sometimes occur on Windows. See https://github.com/bazelbuild/bazel/issues/5512 RELNOTES: none PiperOrigin-RevId: 203342889
* Bazel server: ensure InputStreams are closedGravatar laszlocsomor2018-07-05
| | | | | | | | | | | | | | | | | | | Use try-with-resources to ensure InputStreams that we open via FileSystem.InputStream(path) are closed. Eagerly closing InputStreams avoids hanging on to file handles until the garbage collector finalizes the InputStream, meaning Bazel on Windows (and other processes) can delete or mutate these files. Hopefully this avoids intermittent file deletion errors that sometimes occur on Windows. See https://github.com/bazelbuild/bazel/issues/5512 RELNOTES: none PiperOrigin-RevId: 203338148
* Move HashFunction out of FileSystem, and turn it into a class, instead of an ↵Gravatar ccalvarin2018-06-21
| | | | | | | | | enum. Now that we aren't using enum names for the hash functions, we also accept the standard names, such as SHA-256. RELNOTES: None. PiperOrigin-RevId: 201624286
* Refactoring: uses OutputService for ActionFileSystem injectionGravatar shahan2018-06-18
| | | | | | Moves more Action-oriented from skyframe package, which has become very unwieldy, to action package. This is needed to avoid circular dependencies caused when build-base is needed for skyframe. PiperOrigin-RevId: 200996982
* Add AutoProfiler-like API to ProfilerGravatar ulfjack2018-06-11
| | | | | | - migrate all startTask/completeTask pairs to the new API PiperOrigin-RevId: 200038703
* Allow for Path and Root resolution when extracting action metadata.Gravatar felly2018-06-06
| | | | | RELNOTES: None PiperOrigin-RevId: 199572597
* Hand-roll PathFragment$Codec, to squeeze a bit of performance out of it.Gravatar janakr2018-05-25
| | | | PiperOrigin-RevId: 198124705
* Adds an action-scoped filesystem.Gravatar shahan2018-05-09
| | | | PiperOrigin-RevId: 195973862
* Clean up code that directly imports nested classes like Builder, Entry, etc.Gravatar jcater2018-05-01
| | | | PiperOrigin-RevId: 195040539
* Normalize parameter name commentsGravatar cushon2018-04-27
| | | | PiperOrigin-RevId: 194512971
* Make FileSystemUtils.moveFile always preserve symlinks and use it in ↵Gravatar Benjamin Peterson2018-04-12
| | | | | | | | | | | | | | | | | | | | | | | SandboxedSpawn.copyOutputs. Previously, if moveFile fell back to copying from a true rename (e.g., if the move is across file systems), it would fully dereference the source and produce a regular file at the output location. This CL fixes moveFile to properly copy symlinks. Note we don't preserve metadata in the symlink case mostly because the Bazel VFS has no API to write metadata without dereferencing symlinks. (But, also, it's not important for the current use cases.) This breaks the backward-compatibility of FileSystemUtils.moveFile and FileSystemUtils.moveTreeBelow. This seems okay because the new behavior makes more sense, and sandbox is the only consumer of these APIs. Switching SandboxedSpawn.copyOutputs to use FileSystemUtils.moveFile rather than Guava's Files.move fixes https://github.com/bazelbuild/bazel/issues/4987. Closes #4989. Change-Id: I0283e8aa11fadff5b0afd7bdfd0490aca12a1f6b PiperOrigin-RevId: 192611227
* Make some objects frequently encountered during serialization into ↵Gravatar janakr2018-04-10
| | | | | | | | constants. A lot of care is needed here because we're using reference equality. I plan to add value-equality constants in a follow-up. Add ImmutableSortedSet marshaller because I think it might have been needed, and hey, why not. PiperOrigin-RevId: 192326359
* Change profiling to only accept strings for its "description" argument. ↵Gravatar janakr2018-04-01
| | | | | | Profiling can hold onto objects for the duration of the build, and some of those objects may be temporary that should not be persisted. In particular, UnixGlob and its inner classes should not outlive loading and analysis. For the most part, care was taken in this CL to only use strings that required no additional construction, mainly to minimize garbage (retaining references to newly created strings is not as great a concern since only the strings corresponding to the slowest K tasks are retained, for some relatively small values of K). Action descriptions for actually executing actions are eagerly expanded because that work is minimal compared to the work of actually executing an action. PiperOrigin-RevId: 191251488
* Refactor and cleanup the sandboxing code.Gravatar Philipp Wollermann2018-03-23
| | | | | | | | | | | | | - Remove Optional<> where it's not needed. It's nice for return values, but IMHO it was overused in this code (e.g. Optional<List<X>> is an anti-pattern, as the list itself can already signal that it is empty). - Use Bazel's own Path class when dealing with paths, not String or java.io.File. - Move LinuxSandboxUtil into the "sandbox" package. - Remove dead code and unused fields. - Migrate deprecated VFS method calls to their replacements. - Fix a bug in ExecutionStatistics where a FileInputStream was not closed. Closes #4868. PiperOrigin-RevId: 190217476
* Optimize GC churn due to PackageFunction#getContainingDirectory. While I'm ↵Gravatar nharmata2018-03-15
| | | | | | | | | here, also slightly restructure the code in #handleLabelsCrossingSubpackagesAndPropagateInconsistentFilesystemExceptions to make it more readable (and defer PackageIdentifier allocations). Alternatives considered: Add a PathFragment#getParentDirectoryOfRelative(String other) instance method, and use it in PackageFunction#getContainingDirectory. I thought the approach in this CL would be preferable to adding a specialized method like that to PathFragment. RELNOTES: None PiperOrigin-RevId: 189197855
* Add mostly-ide-generated hashCode for RootedPathGravatar michajlo2018-03-12
| | | | | | | Removed null-checks since not expecting null. RootedPath's hashCode is called enough that it's worth optimizing. PiperOrigin-RevId: 188801280
* @AutoCodec TransitiveTraversalValue, DirectoryListingStateValue.Gravatar janakr2018-03-06
| | | | PiperOrigin-RevId: 188077900
* Serialize Path using AutoCodec.Gravatar janakr2018-03-05
| | | | PiperOrigin-RevId: 187945746
* Deletes CODEC fields now that they are no longer needed.Gravatar shahan2018-02-28
| | | | PiperOrigin-RevId: 187397314
* Optimize Path (and the underlying String) allocations in UnixGlob for the ↵Gravatar nharmata2018-02-27
| | | | | | | | | case where a pattern like '*.blah' doesn't match all the dirents. Before the recent Path.java rewrite, the unconditional allocation here was free/amortized (in the sense that if Blaze ever had to consider the path in question, then we'd be doing the Path allocation anyway and caching the object). But now it's potentially GC churny, especially in the case where a single BUILD file has multiple glob expressions. RELNOTES: None PiperOrigin-RevId: 187185573
* Adds ObjectCodecRegistry to {Des|S}erializationContext.Gravatar shahan2018-02-20
| | | | | | | | | | | | * AutoCodec now delegates to the registry. * Adds getSuperclass logic for resolving a codec. * Small cleanups for classes that break the registry. TODO after this change: * Explicit CODEC definitions are no longer needed and existing ones should be cleaned up. * POLYMORPHIC is no longer be needed and should be cleaned up. PiperOrigin-RevId: 186351845
* Move CommandLine, CommandLineItem, and ParamFileInfo from ↵Gravatar tomlu2018-02-15
| | | | | | | | | lib.analysis.actions -> lib.actions. These are fundamental types that want to sit alongside types like Spawn. RELNOTES: None PiperOrigin-RevId: 185887971
* Don't intern PathFragments when we create PackageIdentifiers. This should be ↵Gravatar shreyax2018-02-15
| | | | | | | | a strict win: There are only two places Canonicalizer did PathFragment interning: PackageIdentifier creation and Package creation. PackageIdentifiers are always interned by a separate interner, and so the underlying PathFragment will be in 1-1 correspondence with PackageIdentifiers (per repo). Moreover, Packages are created with a PackageIdentifier already existing, so it will use the same PathFragment of the unique PackageIdentifier. PiperOrigin-RevId: 185877942
* Replaces InjectingObjectCodec with dependencies threaded through ↵Gravatar shahan2018-02-13
| | | | | | (Des|S)erializationContext. PiperOrigin-RevId: 185547740
* Add context argument to ObjectCodec.{serialize,deserialize}Gravatar michajlo2018-02-11
| | | | | | | | | | Context implementations are currently empty, just doing the plumbing in this change. Once this is in we can start passing along the ObjectCodecRegistry, which will allow runtime codec resolution for classes not known at compile time. We'll also inevitably add some memoization helpers, allowing us to optimize the serialization process further. PiperOrigin-RevId: 185305674
* Do not store Path object in RootedPath.Gravatar tomlu2018-02-09
| | | | | | | This can be computed on the fly if we need it. RELNOTES: None PiperOrigin-RevId: 185180257
* Replace path implementation.Gravatar tomlu2018-02-08
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Path and PathFragment have been replaced with String-based implementations. They are pretty similar, but each method is dissimilar enough that I did not feel sharing code was appropriate. A summary of changes: PATH ==== * Subsumes LocalPath (deleted, its tests repurposed) * Use a simple string to back Path * Path instances are no longer interned; Reference equality will no longer work * Always normalized (same as before) * Some operations will now be slower, like instance compares (which were previously just a reference check) * Multiple identical paths will now consume more memory since they are not interned PATH FRAGMENT ============= * Use a simple string to back PathFragment * No more segment arrays with interned strings * Always normalized * Remove isNormalized * Replace some isNormalizied uses with containsUpLevelReferences() to check if path fragments try to escape their scope * To check if user input is normalized, supply static methods on PathFragment to validate the string before constructing a PathFragment * Because PathFragments are always normalized, we have to replace checks for literal "." from PathFragment#getPathString to PathFragment#getSafePathString. The latter returns "." for the empty string. * The previous implementation supported efficient segment semantics (segment count, iterating over segments). This is now expensive since we do longer have a segment array. ARTIFACT ======== * Remove Path instance. It is instead dynamically constructed on request. This is necessary to avoid this CL becoming a memory regression. RELNOTES: None PiperOrigin-RevId: 185062932
* Remove Root#getCodec, to bring Root in line with general style for codecGravatar cpeyser2018-02-02
| | | | | | declaration. PiperOrigin-RevId: 184304653
* Rephrase RootCodec as an InjectingObjectCodec.Gravatar cpeyser2018-01-29
| | | | PiperOrigin-RevId: 183677348
* Add CommandLineItem interface.Gravatar tomlu2018-01-25
| | | | | | | | This interface makes it clearer in the type system exactly how items that go into a CustomCommandLine are turned into strings. It is a preparatory change to allow command line fingerprints to be more cheaply calculated, but it is valuable in itself from a code quality standpoint. PiperOrigin-RevId: 183274022
* Fix usages of PathFragment segments that will become inefficient.Gravatar tomlu2018-01-19
| | | | | | | An upcoming replacement to PathFragment will not have efficient segment semantics, causing code to become unnecessarily inefficient. RELNOTES: None PiperOrigin-RevId: 182553098
* Rename relativePath -> rootRelativePath in Root and friends.Gravatar tomlu2018-01-19
| | | | | | This makes it clearer that the path fragments in question are relative *to the root*. Confusingly, when the root is absolute, the root relative fragment is also absolute. This makes it a tiny bit clearer that the path fragment may be absolute. PiperOrigin-RevId: 182544893
generated by cgit v1.2.3 (git 2.39.1) at 2024-06-13 22:36:12 +0000