aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/test/java/com/google/devtools/build/lib/util
Commit message (Collapse)AuthorAge
* Introduce a new file-based logging handler for Bazel.Gravatar arostovtsev2018-08-03
| | | | | | | | | | | | | | | | | | | | | | It provides a number of features that we want and whose combination cannot be accomplished using the standard FileHandler: * Using a different filename per server process, by putting a timestamp and process ID in the filename. This means Bazel will no longer overwrite its log when the server is restarted, making it easier for developers and maintainers to diagnose issues. * Putting the hostname and username in the filename (useful when running on a shared network filesystem). * Automatically setting a symlink to the latest log file, ensuring that the latest log can still be found under the usual Bazel server log path. * Providing an API for getting the filename of the current log file, for use by Bazel itself. * Cleaning up old log files when their total size exceeds a set limit. This commit only introduces the handler; its usage in Bazel will be enabled by a follow-up commit. RELNOTES: None. PiperOrigin-RevId: 207274587
* Automated rollback of commit 7e87730de985b7099b9b683571d58efdaab70890.Gravatar ccalvarin2018-07-27
| | | | | | | | | | | | | | | | | *** Reason for rollback *** Go back to the default constructor - instead of requiring everywhere to know the correct hash function, we'll have the default rely on global state. It will make transition easier, even if it makes the origin of the hash less obvious. *** Original change description *** Remove default MD5 in most of Bazel's virtual filesystems. This forces the ex-default to be explicit in a lot of tests, but I'd rather that than have the risk of implicit md5-use in production code. To keep this CL smaller, do not remove the default from UnixFS quite yet. RELNOTES: None. PiperOrigin-RevId: 206358838
* Remove default MD5 in most of Bazel's virtual filesystems.Gravatar ccalvarin2018-07-26
| | | | | | | | | This forces the ex-default to be explicit in a lot of tests, but I'd rather that than have the risk of implicit md5-use in production code. To keep this CL smaller, do not remove the default from UnixFS quite yet. RELNOTES: None. PiperOrigin-RevId: 206223521
* Modify --subcommands to allow pretty printing the arguments of a subcommandGravatar ahumesky2018-07-26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | as a list, rather than as a single line (i.e., newline delimited rather than space delimited). Before: SUBCOMMAND: # //src/main/java/com/google/devtools/build/lib:string_util [action 'Building src/main/java/com/google/devtools/build/lib/libstring_util.jar (2 source files) [for host]'] (cd /tmp/devbazel_output_base/execroot/io_bazel && \ exec env - \ LC_CTYPE=en_US.UTF-8 \ external/embedded_jdk/bin/java -XX:+UseParallelOldGC -XX:-CompactStrings '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--patch-module=java.compiler=external/bazel_tools/third_party/java/jdk/langtools/java_compiler.jar' '--patch-module=jdk.compiler=external/bazel_tools/third_party/java/jdk/langtools/jdk_compiler.jar' '--add-opens=java.base/java.nio=ALL-UNNAMED' -jar external/bazel_tools/tools/jdk/JavaBuilder_deploy.jar @bazel-out/host/bin/src/main/java/com/google/devtools/build/lib/libstring_util.jar-2.params) After: SUBCOMMAND: # //src/main/java/com/google/devtools/build/lib:string_util [action 'Building src/main/java/com/google/devtools/build/lib/libstring_util.jar (2 source files) [for host]'] (cd /tmp/devbazel_output_base/execroot/io_bazel && \ exec env - \ LC_CTYPE=en_US.UTF-8 \ external/embedded_jdk/bin/java \ -XX:+UseParallelOldGC \ -XX:-CompactStrings \ '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' \ '--add-opens=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' \ '--patch-module=java.compiler=external/bazel_tools/third_party/java/jdk/langtools/java_compiler.jar' \ '--patch-module=jdk.compiler=external/bazel_tools/third_party/java/jdk/langtools/jdk_compiler.jar' \ '--add-opens=java.base/java.nio=ALL-UNNAMED' \ -jar \ external/bazel_tools/tools/jdk/JavaBuilder_deploy.jar \ @bazel-out/host/bin/src/main/java/com/google/devtools/build/lib/libstring_util.jar-2.params) RELNOTES: --subcommands can now take a "pretty_print" value ("--subcommands=pretty_print") to print the arguments of subcommands as a list for easier reading. PiperOrigin-RevId: 206213009
* Reimplement AsynchronousFileOutputStream to use a writer threadGravatar ulfjack2018-06-15
| | | | | | | | | | | | | | | | | | | | | Background: the original code is implementing the OutputStream interface. Given a sequence of write calls, the code puts each of these writes into a queue. On the other side of the queue is an unbounded thread pool, which takes the writes off the queue one by one, and then does individual blocking writes with a fixed file offset. There are three problems with the original code: 1. Writes are sent to the Kernel one-by-one. Imagine if the incoming writes a single-byte writes, then we do one kernel call for every single byte. This is the worst case. 2. Due to multithreading, the writes can get reordered. In the worst case, the order is reversed, and the kernel flushes each write to disk individually. Since the writes are not aligned to disk block boundaries, each write has to first *read* the block from disk, overwrite a few bytes, and then flush the block back to disk. On a spinning platter, this is the worst possible sequence of operations: write a block, read the same block back, write the same block again, read the same block back, etc., with each operation having to wait for a full disk rotation. Note that this gets worse if there's high thread and / or disk contention, e.g., when running a build system in the background. 3. Due to the unbounded thread pool, it may end up creating a new thread for every single write (possibly as many as one per byte written). This is also the worst case, although it's probably negligible compared to 1+2. Compared to that, this change uses an in-memory buffer before sending writes to the kernel so non-block sized writes are batched, it writes sequentially, and it uses a single thread created at the start. A single thread should be more than able to fully saturate local disk I/O, so multi-threading should ~never be a improvement here. This might look different if we had perfectly aligned writes of an integer multiple of the disk block size to a distributed network file system or a local SSD raid. If you look at the clients of this class, that's definitely not the case: this code is primarily used for local file BEP transports - we wouldn't use local file BEP transports to write to a network file system, we'd use the BES instead. It's also used to write a couple of log files, also all local - otherwise we'd add the data to BEP. These are all unaligned, ~random-sized writes. If we created a lot of these files, then using a thread pool with fewer threads than files and using non-blocking I/O might be an improvement due to the reduction in thread count, but I think it's very unlikely that we'll ever need that complexity. PiperOrigin-RevId: 200694423
* Remove unused StringUtil method.Gravatar felly2018-05-21
| | | | PiperOrigin-RevId: 197461794
* Rollback https://github.com/bazelbuild/bazel/pull/5007Gravatar cparsons2018-04-16
| | | | | | | It breaks downstream rules_nodejs. See https://github.com/bazelbuild/bazel/issues/5028 for details. RELNOTES: None. PiperOrigin-RevId: 193074798
* CommandBuilder: remove useShell and setWorkingDirGravatar Laszlo Csomor2018-04-12
| | | | | | | | | | | | | | | | | | | | | Remove the .useShell method, expect callers to just pass the shell interpreter if they need it. This removes the argument vector transformation heuristic, and stops shelling out to cmd.exe on Windows. Also remove the .setWorkingDir method because callers always had to set the working directory. Instead, the CommandBuilder constructor takes the working directory. Closes #2190. (Test code still shells out to cmd.exe though.) Closes #5007. Change-Id: I545e01c811daaf34913cb585492923da81aa02ee PiperOrigin-RevId: 192611370
* Remove categories from Bazel options.Gravatar ccalvarin2018-03-28
| | | | | | | | | These have all had a chance to be categorized with the OptionDocumentationCategory enum, and the help output already uses the enum-grouped format. The "incompatible changes" category has meaning for --all_incompatible_changes and will be removed separately. RELNOTES: None. PiperOrigin-RevId: 190773778
* Allow path options to use user specific pathsGravatar David Ostrovsky2018-03-20
| | | | | | | | | | | | | | | | | | | | | Fixes #2054. Allow users to be able to specify user specific paths. With this option we can now commit bazel configuration file and force local action cache activation per default: $ cat tools/bazel.rc build --experimental_local_disk_cache_path=~/.gerrit/bazel-cache/cas build --experimental_local_disk_cache build --experimental_strict_action_env Test Plan: $ bazel test //src/test/java/com/google/devtools/build/lib:util_test Closes #4852. PiperOrigin-RevId: 189744599
* Adding async proto or text logging utility class.Gravatar olaola2018-03-06
| | | | | | | WANT_LGTM=buchgr TESTED=unit tests, 500 runs per test RELNOTES: None PiperOrigin-RevId: 188093043
* Optimize GC usage of iterating over all elements of GroupedLists when we ↵Gravatar shreyax2018-03-02
| | | | | | don't care about the group structure, and simplify the logic for prefetching old deps. PiperOrigin-RevId: 187681887
* Deletes CODEC fields now that they are no longer needed.Gravatar shahan2018-02-28
| | | | PiperOrigin-RevId: 187397314
* Fix bug where an was-inflight-and-is-about-to-be-done NodeEntry has ↵Gravatar nharmata2018-01-18
| | | | | | | | | | | | | | | | | | | | incomplete deps that need to be removed, and these deps are currently duplicated in the NodeEntry's newly requested deps GroupedList. Also add a fast-path to GroupedListHelper#remove(List<Object> elements, Set<E> toRemove) for the incredibly common case where toRemove is empty. This saves a wasteful O(elements.size()) scan over elements. This method is unconditionally called each time a SkyFunction restart causes us to add new direct deps (with elements=<the new direct deps> and toRemove=<unfinished direct deps>); in the case where there are a ton of new direct deps, this scan entails wasted cpu and gc churn. The bug only occurs in uncommon case that there are deps to remove. The bug has existed since GroupedList was first introduced into the codebase. In Skyframe-land, this is only observable in nokeep_going mode because in keep_going mode "we do not let SkyFunctions throw errors with missing deps" (quote from comment in AbstractParallelEvaluator). A Bazel-on-Skyframe-land example how this bug could occur in practice is PackageFunction's Skyframe hybrid globbing. If an io error is encountered during legacy globbing, the PackageFunction eagerly throws a SkyFunctionException but it has already requested the Skyframe GlobValue deps. RELNOTES: None PiperOrigin-RevId: 182403943
* Redo FileType to reduce generated garbage.Gravatar tomlu2017-12-22
| | | | | | | | | | | | | * Change FileType to no longer assume it operates on just the base name (it can now be given a full path). * Move the responsibility to specific classes (Artifact, Path, PathFragment) to decide how they want to offer up a string that includes the file name. * Flip the order in which users are expected to check Artifact type, from FileType#matches(Artifact) to Artifact#isFileType(FileType). This looks natural and should encourage developers to use efficient file type checking methods. * Change CppCompileAction to use the new API. RELNOTES: None PiperOrigin-RevId: 179903239
* Automated rollback of commit 82e68b75304438c96ff878a0c2b8d18b42002486.Gravatar aehlig2017-12-19
| | | | | | | | | | | | | | Fixes #4322, #4306. *** Reason for rollback *** Introduces a deadlock (see https://github.com/bazelbuild/bazel/issues/4322) *** Original change description *** Make FileSystem operate on LocalPath instead of Path. PiperOrigin-RevId: 179549866
* Make FileSystem operate on LocalPath instead of Path.Gravatar tomlu2017-12-14
| | | | PiperOrigin-RevId: 179082062
* Move msys path support into DependencySet, removing it from the general Path ↵Gravatar tomlu2017-12-14
| | | | | | | | system. This is the only place that should actually need it. PiperOrigin-RevId: 179054861
* Adds a codec for RegexFilter.Gravatar Googler2017-12-13
| | | | PiperOrigin-RevId: 178942449
* Add a static method GroupedList#numElements to allow to count the number of ↵Gravatar shreyax2017-11-29
| | | | | | deps from a compressed GroupedList without uncompressing it. Also some minor GC improvements. PiperOrigin-RevId: 177338852
* Track Option placement within a priority category.Gravatar ccalvarin2017-10-18
| | | | | | | | | An option has precedence over previous options at the same enum-valued priority. Track its placement in this ordering explicitly. This will allow after-the-fact expansion of expansion options such that they correctly take precedence or not compared to other mentions of the same flag. This is needed to fix --config's expansion. RELNOTES: None. PiperOrigin-RevId: 172367996
* Make UnionFileSystem accept all paths Bazel can throw at it.Gravatar ccalvarin2017-09-27
| | | | | | | | | Instead of relying on a character-by-character StringTrie, segment paths based on PathFragments. This means UnionFS can accept any path that Bazel stores internally, removing the ASCII limitations. This also means removing the ability to have a filesystem bound for sub-PathFragments, /foo/barbar, /foo/barqux could have the same filesystem bound at /foo/bar. This feature was tested for when a use case was envisioned, but it was never used, so removing it is safe. RELNOTES: None. PiperOrigin-RevId: 170054656
* More BUILD file refactorings.Gravatar philwo2017-09-06
| | | | | | | | | Split collect, concurrent, vfs, windows into package-level BUILD files. Move clock classes out of "util", into their own Java package. Move CompactHashSet into its own Java package to break a dependency cycle. Give nestedset and inmemoryfs their own package-level BUILD files. PiperOrigin-RevId: 167702127
* Rewrite all code to use the new Java 8 java.time classes.Gravatar Philipp Wollermann2017-09-05
| | | | | | | This removes our dependency on third_party/joda_time, which can be removed in the next commit. Change-Id: Ibda131d34d0abdc2d675db4bfbd2e99480c055ee PiperOrigin-RevId: 167515260
* CommandBuilder.java: Don't quote the last argument when passing Windows ↵Gravatar pcloudy2017-08-18
| | | | | | | | | command to CMD.EXE. Fix https://github.com/bazelbuild/bazel/issues/2040#issuecomment-322980147 RELNOTES: None PiperOrigin-RevId: 165562681
* Make the @Option annotation depend on the java version of the tagging enums.Gravatar ccalvarin2017-07-18
| | | | | | | The option filters proto dependency can be removed from the OptionsParser. This is in response to option parser users that want to avoid the bazel-internal proto file in their dependencies. RELNOTES: None. PiperOrigin-RevId: 162249778
* Fold OptionUsageRestrictions into OptionDocumentationCategory and ↵Gravatar ccalvarin2017-07-11
| | | | | | | | | | OptionMetadataTags. These are similar, no need to have both fields. Removing the "DOCUMENTED" default, the absence of UNDOCUMENTED will be used instead. Since requiring a documentation category for undocumented options doesn't make sense, list that as one of the OptionDocumentationCategories, but list HIDDEN and INTERNAL as part of OptionMetadata. These options should list UNDOCUMENTED as their category. PiperOrigin-RevId: 161515674
* Cleanup for Java 8Gravatar laurentlb2017-07-05
| | | | PiperOrigin-RevId: 160832316
* Explicitly state the default options categories for options used in blaze ↵Gravatar ccalvarin2017-06-29
| | | | | | | | testing. Unlike in the production flags, these flags are only used for internal testing. Tagged them as NO_OP instead of the default UNKNOWN to make it clear that we do not need these to become properly tagged. PiperOrigin-RevId: 160526472
* Migrate most of the assertions to Truth that the auto-migration tool did not ↵Gravatar lberki2017-05-31
| | | | | | | | | catch. IntelliJ's "replace structurally" command was surprisingly useful. RELNOTES: None. PiperOrigin-RevId: 157463734
* Migrate Java tests to Truth.Gravatar lberki2017-05-30
| | | | | | RELNOTES: None. PiperOrigin-RevId: 157446717
* Fixes incorrectly-ordered arguments to calls to assertEqualsGravatar Googler2017-05-19
| | | | | | | | | | | | | | | [] This change has been automatically generated by an Error Prone check that detects incorrect argument ordering on calls to assertEquals-style methods. See [] Cleanup change automatically generated by javacflume/refactory Refactoring: third_party/java_src/error_prone/project/core/src/main/java/com/google/errorprone/bugpatterns/argumentselectiondefects:AssertEqualsArgumentOrderChecker_refactoring Tested: TAP --sample for global presubmit queue [] PiperOrigin-RevId: 156539781
* Remove the overloading of option category to cover documentation level.Gravatar ccalvarin2017-04-18
| | | | | | | | This prevents having to parse the category, and allows the category to potentially be used in the future for information about all options, included undocumented, hidden, or internal options. Also rename DocumentationLevel to OptionUsageRestrictions, since INTERNAL was not really documentation related. PiperOrigin-RevId: 153367769
* Add a custom single-line formatter for java.log.Gravatar schmitt2017-04-13
| | | | | | | | This logger makes it easier to parse log statements and is now enabled for Bazel's java.log. RELNOTES[INC]: Bazel now prints logs in single lines to java.log PiperOrigin-RevId: 152954337
* Do not trample the PersistentMap journalGravatar Benjamin Peterson2017-04-07
| | | | | | | | | | | | | | | This fixes https://github.com/bazelbuild/bazel/issues/2660. Basically, if we elect to keep the journal during PersistentMap.save(), we shouldn't stomp over it the next time save() is called. In writeJournal(), we now check if the journal file exists, and open it in append mode if it does. Alternatively, we could simply not close (and thus forget about) the journal in save(), but that would leak the journal file handle if save() was never called with keepJournal() returning false. Change-Id: Id00732f161c8b5a082a6c109aee115591ace2ea7 PiperOrigin-RevId: 152480978
* Refactor all ctor callsites of PathFragment to instead call a static ↵Gravatar nharmata2017-04-05
| | | | | | | | | | | | 'create' method. This paves the way for changing PathFragment to e.g. an abstract class with multiple subclasses. This way we can split out the windows-specific stuff into one of these concrete classes, making the code more readable and also saving memory (since the shallow heap size of the NonWindowsPathFragment subclass will hopefully be smaller than that of the current PathFragment). This also lets us pursue gc churn optimizations. We can now do interning in PathFragment#create and can also get rid of unnecessary intermediate PathFragment allocations. RELNOTES: None PiperOrigin-RevId: 152145768
* Add new utility function to encode options as argument list Gravatar Klaus Aehlig2017-03-23
| | | | | | | | | | | ...in a way preserving the list structure and without escaping. In other words, return the options as needed for an execve(3) call. -- Change-Id: Ifb168d2e720392fb3a97d557960e0c1f1a83f543 Reviewed-on: https://cr.bazel.build/9458 PiperOrigin-RevId: 151019830 MOS_MIGRATED_REVID=151019830
* Make the spell checker a bit more conservative.Gravatar Laurent Le Brun2017-03-21
| | | | | | | | | Reduce the max spell-checking distance for matching words. "target" will not match "range" anymore. -- PiperOrigin-RevId: 150638073 MOS_MIGRATED_REVID=150638073
* Move UnixFileSystem to lib.unix, WindowsFileSystem to lib.windowsGravatar Ulf Adams2017-02-28
| | | | | | -- PiperOrigin-RevId: 148749485 MOS_MIGRATED_REVID=148749485
* Move ProcMeminfoParser to lib.unix, where it belongsGravatar Ulf Adams2017-01-09
| | | | | | -- PiperOrigin-RevId: 143939410 MOS_MIGRATED_REVID=143939410
* Streamline Fingerprint implementationGravatar Michajlo Matijkiw2016-12-15
| | | | | | | | | | | | Thread all updates through a CodedOutputStream. This has the benefit of potentially hashing less data, as many int values can be represented more compactly, and reducing the churn of hashing strings (generally), since CodedOutputStream is already heavily optimized for this. While the buffer size is a little generous, it winds up paying off. -- PiperOrigin-RevId: 142151062 MOS_MIGRATED_REVID=142151062
* Use MessageDigest instead of HashFunction for FingerprintGravatar Michajlo Matijkiw2016-12-01
| | | | | | | | | This allows us to reset and reuse the underlying digest implementation, which guava's HashFunction doesn't allow. We do take the clone-if-possible page out of guava's book. -- MOS_MIGRATED_REVID=140624836
* Implement build tag filtering.Gravatar Lukacs Berki2016-11-11
| | | | | | | If the --build_tag_filters option is specified, targets built will be filtered according to their tags (at least one included, none excluded) -- MOS_MIGRATED_REVID=138856195
* Stop consulting the environment for "originating user" information.Gravatar Googler2016-11-03
| | | | | -- MOS_MIGRATED_REVID=138039276
* If a group of deps has only one unique element, store it bare.Gravatar Janak Ramakrishnan2016-10-25
| | | | | | | | | | Not storing it bare both breaks equality comparison for GroupedList and uses more space. -- Change-Id: Iaf4f88908ecf4293cfc31dbd896f46e1da3b4184 Reviewed-on: https://bazel-review.googlesource.com/#/c/6850/ MOS_MIGRATED_REVID=137099785
* Fix unnecessary static imports of typesGravatar Liam Miller-Cushon2016-10-18
| | | | | -- MOS_MIGRATED_REVID=136431274
* Stop storing a set in GroupedListHelper to deduplicate SkyKey dep requests. ↵Gravatar Janak Ramakrishnan2016-10-14
| | | | | | | Instead, deduplicate when the helper is actually added to a GroupedList. -- MOS_MIGRATED_REVID=136145321
* Rollback of commit 0e9a1b2d51f9e03f4493e52d23cb3efc568be59f.Gravatar Googler2016-10-07
| | | | | | | | | | | | | | *** Reason for rollback *** [] hasn't released unknown commit yet. We need to keep the old behavior until then. *** Original change description *** Stop consulting the environment for "originating user" information. -- MOS_MIGRATED_REVID=135441115
* Stop consulting the environment for "originating user" information.Gravatar Googler2016-10-07
| | | | | -- MOS_MIGRATED_REVID=135422529
* Remove Fingerprint.toString() to avoid relying on MessageDigest.clone().Gravatar Googler2016-07-04
| | | | | -- MOS_MIGRATED_REVID=126566280
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-04 09:43:05 +0000