aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/util
Commit message (Collapse)AuthorAge
* Reimplement AsynchronousFileOutputStream to use a writer threadGravatar ulfjack2018-06-15
| | | | | | | | | | | | | | | | | | | | | Background: the original code is implementing the OutputStream interface. Given a sequence of write calls, the code puts each of these writes into a queue. On the other side of the queue is an unbounded thread pool, which takes the writes off the queue one by one, and then does individual blocking writes with a fixed file offset. There are three problems with the original code: 1. Writes are sent to the Kernel one-by-one. Imagine if the incoming writes a single-byte writes, then we do one kernel call for every single byte. This is the worst case. 2. Due to multithreading, the writes can get reordered. In the worst case, the order is reversed, and the kernel flushes each write to disk individually. Since the writes are not aligned to disk block boundaries, each write has to first *read* the block from disk, overwrite a few bytes, and then flush the block back to disk. On a spinning platter, this is the worst possible sequence of operations: write a block, read the same block back, write the same block again, read the same block back, etc., with each operation having to wait for a full disk rotation. Note that this gets worse if there's high thread and / or disk contention, e.g., when running a build system in the background. 3. Due to the unbounded thread pool, it may end up creating a new thread for every single write (possibly as many as one per byte written). This is also the worst case, although it's probably negligible compared to 1+2. Compared to that, this change uses an in-memory buffer before sending writes to the kernel so non-block sized writes are batched, it writes sequentially, and it uses a single thread created at the start. A single thread should be more than able to fully saturate local disk I/O, so multi-threading should ~never be a improvement here. This might look different if we had perfectly aligned writes of an integer multiple of the disk block size to a distributed network file system or a local SSD raid. If you look at the clients of this class, that's definitely not the case: this code is primarily used for local file BEP transports - we wouldn't use local file BEP transports to write to a network file system, we'd use the BES instead. It's also used to write a couple of log files, also all local - otherwise we'd add the data to BEP. These are all unaligned, ~random-sized writes. If we created a lot of these files, then using a thread pool with fewer threads than files and using non-blocking I/O might be an improvement due to the reduction in thread count, but I think it's very unlikely that we'll ever need that complexity. PiperOrigin-RevId: 200694423
* Remove unused StringUtil method.Gravatar felly2018-05-21
| | | | PiperOrigin-RevId: 197461794
* Remove AbruptException(ExitCode exitCode) constructor.Gravatar lpino2018-05-17
| | | | | | | | | This constructor was creating an Exception with a null message leading to possible NullPointerExceptions in a few places in our codebase. The call sites have been replaced with calls to AbruptException(String message, ExitCode exitCode) with a meaningful message. PiperOrigin-RevId: 196973540
* Augment comments about blaze exit code classifications.Gravatar Googler2018-05-14
| | | | | RELNOTES: None PiperOrigin-RevId: 196553226
* Document that getuid/getgid/getpid can throw exceptions.Gravatar philwo2018-04-25
| | | | | RELNOTES: None. PiperOrigin-RevId: 194236052
* Add BazelCrashUtils to make it easier to crash the JVM when uncaught ↵Gravatar janakr2018-04-20
| | | | | | exceptions are detected in async worker threads. PiperOrigin-RevId: 193725871
* Logging the spawn graph on demand. First step on #4891.Gravatar olaola2018-04-19
| | | | | | | | This is a very heavy and slow option by design. It will be enabled only when a user wants to debug their build, most frequently to compare the step-by-step results of two builds. TESTED: manually on various rules, including directories. RELNOTES: None PiperOrigin-RevId: 193539034
* Rollback https://github.com/bazelbuild/bazel/pull/5007Gravatar cparsons2018-04-16
| | | | | | | It breaks downstream rules_nodejs. See https://github.com/bazelbuild/bazel/issues/5028 for details. RELNOTES: None. PiperOrigin-RevId: 193074798
* CommandBuilder: remove useShell and setWorkingDirGravatar Laszlo Csomor2018-04-12
| | | | | | | | | | | | | | | | | | | | | Remove the .useShell method, expect callers to just pass the shell interpreter if they need it. This removes the argument vector transformation heuristic, and stops shelling out to cmd.exe on Windows. Also remove the .setWorkingDir method because callers always had to set the working directory. Instead, the CommandBuilder constructor takes the working directory. Closes #2190. (Test code still shells out to cmd.exe though.) Closes #5007. Change-Id: I545e01c811daaf34913cb585492923da81aa02ee PiperOrigin-RevId: 192611370
* Separate out single-extension matching in FileType to reduce garbage.Gravatar janakr2018-03-26
| | | | PiperOrigin-RevId: 190532114
* Redo FileType @AutoCodec: it is risky to assume that all subclasses are ↵Gravatar janakr2018-03-26
| | | | | | | | totally described by their extensions, since many are not. Instead, explicitly @AutoCodec those that are, and leave the rest to be individually handled. This also allows us to do equality checking. There is a slight memory penalty here, but there are ~100s of FileTypes in a large build, so it should be negligible. PiperOrigin-RevId: 190477427
* Make the script written blaze "blaze run --direct_run --script_path" not ↵Gravatar lberki2018-03-26
| | | | | | | | | overwrite the environment of their caller. Admittedly, doing that was a bad idea in the first place. RELNOTES: None. PiperOrigin-RevId: 190430189
* Allow path options to use user specific pathsGravatar David Ostrovsky2018-03-20
| | | | | | | | | | | | | | | | | | | | | Fixes #2054. Allow users to be able to specify user specific paths. With this option we can now commit bazel configuration file and force local action cache activation per default: $ cat tools/bazel.rc build --experimental_local_disk_cache_path=~/.gerrit/bazel-cache/cas build --experimental_local_disk_cache build --experimental_strict_action_env Test Plan: $ bazel test //src/test/java/com/google/devtools/build/lib:util_test Closes #4852. PiperOrigin-RevId: 189744599
* tests: add TimestampGranularityUtilsGravatar laszlocsomor2018-03-13
| | | | | | | | | | | | | | Add a new test utility class TimestampGranularityUtils which offers methods to wait for the filesystem timestamp granularity to pass. Such waiting is necessary if a test asserts changes in a file's mtime or ctime. This is a follow-up to comments on https://github.com/bazelbuild/bazel/commit/a31e035fb42aa8db9fd248ef2ebd665a411053b8. PiperOrigin-RevId: 188847729
* tests: fix flaky RecursiveFSTraversalFunctionTestGravatar Laszlo Csomor2018-03-09
| | | | | | | | | | | | | | | | | | Fix the testTraversalOfGeneratedDirectory method in RecursiveFilesystemTraversalFunctionTest that was flaky on Windows. The fix is to wait so that changes to files in the InMemoryFileSystem will have observable effects on the file ctimes. Depends on https://github.com/bazelbuild/bazel/pull/4787 Fixes https://github.com/bazelbuild/bazel/issues/4755 Closes #4788. PiperOrigin-RevId: 188470080
* tests: fix flaky SkyframeAwareActionTestGravatar Laszlo Csomor2018-03-09
| | | | | | | | | | | | | Fix testCacheBypassingActionWithMtimeChangingInput in SkyframeAwareActionTest by ensuring that enough time elapses between file updates so their effects are observable on the file's ctime. Fixes https://github.com/bazelbuild/bazel/issues/4755 Closes #4787. PiperOrigin-RevId: 188458542
* Adding async proto or text logging utility class.Gravatar olaola2018-03-06
| | | | | | | WANT_LGTM=buchgr TESTED=unit tests, 500 runs per test RELNOTES: None PiperOrigin-RevId: 188093043
* Automated rollback of commit 96145511b34a2d7be834e3eb05927674e875c813.Gravatar janakr2018-03-05
| | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Fixed issue that led to rollback. RELNOTES: None *** Original change description *** Automated rollback of commit eee53d3a33dde441f7e7adaecde81ef2d3db7c1b. *** Reason for rollback *** breaks Blaze_CorpTest *** Original change description *** @AutoCodec ConfiguredTargetValue. PiperOrigin-RevId: 187869560
* Automated rollback of commit eee53d3a33dde441f7e7adaecde81ef2d3db7c1b.Gravatar dannark2018-03-04
| | | | | | | | | | | | *** Reason for rollback *** breaks Blaze_CorpTest *** Original change description *** @AutoCodec ConfiguredTargetValue. PiperOrigin-RevId: 187783702
* Optimize GC usage of iterating over all elements of GroupedLists when we ↵Gravatar shreyax2018-03-02
| | | | | | don't care about the group structure, and simplify the logic for prefetching old deps. PiperOrigin-RevId: 187681887
* @AutoCodec ConfiguredTargetValue.Gravatar cpeyser2018-03-02
| | | | PiperOrigin-RevId: 187635570
* Deletes CODEC fields now that they are no longer needed.Gravatar shahan2018-02-28
| | | | PiperOrigin-RevId: 187397314
* Add @AutoCodec to FileType and FileTypeSet.Gravatar mjhalupka2018-02-28
| | | | | | More strictly type the arguments for creating FileTypes. PiperOrigin-RevId: 187391336
* A couple quality-of-life improvements for Bazel devs, in response to an ↵Gravatar nharmata2018-02-23
| | | | | | | | | | email from philwo@. (i) Only have TimestampGranularityMonitor log the first file of relevance. This reduces log spam, especially in tests, while still maintaining useful information in the logs. (ii) Don't have ExternalFilesHelper log the fact that it encountered an external file when we're in a unit test or an integration test. Tests, especially bazel tests that use external repositories, tend to involve lots of "external" files. RELNOTES: None PiperOrigin-RevId: 186799176
* Pass ctime to the TimestampGranularityMonitorGravatar ulfjack2018-02-05
| | | | | | | | We're now using ctime to detect file changes, so the timestamp granularity monitor should as well. Unfortunately, we currently get nanosecond ctime from Linux, but then only return millis from FileStatus, so this doesn't change the accuracy of the monitor at all. PiperOrigin-RevId: 184536539
* Allow CustomCommandLine's mapFn to expand each object into multiple items.Gravatar tomlu2018-02-01
| | | | | | This is needed to migrate JavaCompileAction away from CustomMultiArgv. PiperOrigin-RevId: 184136486
* Add memory-efficient map for storing nested set -> digest.Gravatar tomlu2018-01-31
| | | | | | | | | Instead of using ConcurrentHashMap, we use a dead-simple open addressed hash hable with a giant byte array with 16-byte slots. We then read or write fingerprints straight into and out of the array, obviating the need to generate intermediate garbage. Locking mechanism is a read-write lock. This should be faster than full synchronisation for read-heavy loads. RELNOTES: None PiperOrigin-RevId: 184019301
* Allows @AutoCodec to use factory methods.Gravatar shahan2018-01-29
| | | | | | Generalizes @AutoCodec.Constructor to @AutoCodec.Instantiator. PiperOrigin-RevId: 183702768
* Fix bug where an was-inflight-and-is-about-to-be-done NodeEntry has ↵Gravatar nharmata2018-01-18
| | | | | | | | | | | | | | | | | | | | incomplete deps that need to be removed, and these deps are currently duplicated in the NodeEntry's newly requested deps GroupedList. Also add a fast-path to GroupedListHelper#remove(List<Object> elements, Set<E> toRemove) for the incredibly common case where toRemove is empty. This saves a wasteful O(elements.size()) scan over elements. This method is unconditionally called each time a SkyFunction restart causes us to add new direct deps (with elements=<the new direct deps> and toRemove=<unfinished direct deps>); in the case where there are a ton of new direct deps, this scan entails wasted cpu and gc churn. The bug only occurs in uncommon case that there are deps to remove. The bug has existed since GroupedList was first introduced into the codebase. In Skyframe-land, this is only observable in nokeep_going mode because in keep_going mode "we do not let SkyFunctions throw errors with missing deps" (quote from comment in AbstractParallelEvaluator). A Bazel-on-Skyframe-land example how this bug could occur in practice is PackageFunction's Skyframe hybrid globbing. If an io error is encountered during legacy globbing, the PackageFunction eagerly throws a SkyFunctionException but it has already requested the Skyframe GlobValue deps. RELNOTES: None PiperOrigin-RevId: 182403943
* Adds the @AutoCodec.Constructor annotation for selecting constructors.Gravatar shahan2018-01-03
| | | | | | Uses the constructor having the @AutoCodec.Constructor annotation to generate a codec (instead of choosing the first in source code). This annotation is required when a class has more than one constructor. PiperOrigin-RevId: 180685902
* @AutoCodec: Uses sun.misc.Unsafe to retrieve fields for serialization.Gravatar shahan2018-01-02
| | | | | | | | This is fast (profiles indicate a 9% penalty vs getters) and does not require getters. * Also adds a Marshaller for java.util.regex.Pattern. * Minor fixes for codecs that were broken by this change. PiperOrigin-RevId: 180619680
* Replaces some existing codecs with @AutoCodec.Gravatar Googler2017-12-22
| | | | PiperOrigin-RevId: 179931575
* Redo FileType to reduce generated garbage.Gravatar tomlu2017-12-22
| | | | | | | | | | | | | * Change FileType to no longer assume it operates on just the base name (it can now be given a full path). * Move the responsibility to specific classes (Artifact, Path, PathFragment) to decide how they want to offer up a string that includes the file name. * Flip the order in which users are expected to check Artifact type, from FileType#matches(Artifact) to Artifact#isFileType(FileType). This looks natural and should encourage developers to use efficient file type checking methods. * Change CppCompileAction to use the new API. RELNOTES: None PiperOrigin-RevId: 179903239
* Move msys path support into DependencySet, removing it from the general Path ↵Gravatar tomlu2017-12-14
| | | | | | | | system. This is the only place that should actually need it. PiperOrigin-RevId: 179054861
* Adds a codec for RegexFilter.Gravatar Googler2017-12-13
| | | | PiperOrigin-RevId: 178942449
* Add a static method GroupedList#numElements to allow to count the number of ↵Gravatar shreyax2017-11-29
| | | | | | deps from a compressed GroupedList without uncompressing it. Also some minor GC improvements. PiperOrigin-RevId: 177338852
* Remove useless abstractions in C++ rules after internal cleanupGravatar hlopko2017-11-24
| | | | | RELNOTES: PiperOrigin-RevId: 176844836
* Delete unused preconditions library.Gravatar tomlu2017-11-09
| | | | PiperOrigin-RevId: 175092923
* Replace all usages of Blaze's Preconditions class with guava.Gravatar tomlu2017-11-09
| | | | | | | | Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own. This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually. PiperOrigin-RevId: 175033526
* Make --workspace_status_command accept relative pathGravatar Yun Peng2017-10-25
| | | | | | | Fixed https://github.com/bazelbuild/bazel/issues/3962 Change-Id: I772a409f03b9cc2c0e69e4f51d8a7b95688138a4 PiperOrigin-RevId: 173374845
* Prepare Classpath reading for Java 9Gravatar cushon2017-10-06
| | | | PiperOrigin-RevId: 171180273
* Move the canonicalization of an option value to the option value itself.Gravatar ccalvarin2017-09-27
| | | | | | | Ideally, the canonical form we output from OptionUtils would be the same as for the command canonicalize-flags, but that must wait for dependencies to be cleaned up. Still, in the meantime, keep the --foo=1 normalization of --foo, and apply it to all other boolean flag values (t,true,yes,y, and false equivalents), so that the canoncalize-flags output is more canonical, even if it isn't using the --[no]foo form yet. RELNOTES: Boolean flag values will now get normalized to 1 or 0 in canonicalize-flags output. PiperOrigin-RevId: 170084599
* Make UnionFileSystem accept all paths Bazel can throw at it.Gravatar ccalvarin2017-09-27
| | | | | | | | | Instead of relying on a character-by-character StringTrie, segment paths based on PathFragments. This means UnionFS can accept any path that Bazel stores internally, removing the ASCII limitations. This also means removing the ability to have a filesystem bound for sub-PathFragments, /foo/barbar, /foo/barqux could have the same filesystem bound at /foo/bar. This feature was tested for when a use case was envisioned, but it was never used, so removing it is safe. RELNOTES: None. PiperOrigin-RevId: 170054656
* Allow alternate GroupedList constructor to be public, but add warnings ↵Gravatar shreyax2017-09-25
| | | | | | regarding the integrity of the structure to various operations. PiperOrigin-RevId: 169715233
* Fix bug in NetUtil caching.Gravatar felly2017-09-20
| | | | | | | Fixes #3586. RELNOTES: None. PiperOrigin-RevId: 169303481
* Throw away the confusing UnparsedOptionValueDescription name.Gravatar ccalvarin2017-09-18
| | | | | | | | | | | | | | We get UnparsedValues after ... parsing the options. So that doesn't make sense. What was meant was that it wasn't converted to the final value. In an effort to make this distinction more clear, this change will make the terminology more consistent. The `--foo=bar` step is "parsing" and the `bar -> Object` step is "converting" (it is, in fact, done by Converters). RELNOTES: None. PiperOrigin-RevId: 168852847
* Automatic code cleanup.Gravatar cushon2017-09-15
| | | | PiperOrigin-RevId: 168802886
* Only compute hostname once per server lifetimeGravatar hlopko2017-09-14
| | | | | | | | | | | | NetUtil.getShortHostName can take seconds on mac and on windows (like, 20!), since it performs reverse dns lookup. We already cached hostname for the BazelWorkspaceStatusModule, let's cache it for entire bazel server. Also make sure that users of the method understand it's cached. Fixes #3586. RELNOTES: None. PiperOrigin-RevId: 168691615
* Consolidate the "unparsed" option value tracking.Gravatar ccalvarin2017-09-14
| | | | | | | In preparation for linking the parsed and unparsed values of options, consolidate and standardize our representation of the flag values as we received them (what is meant by "unparsed" values in this case). This was being done separately in ParseOptionResult, which, with extra context added, is being folded into UnparsedOptionValueDescription. We now track how an option was provided and where it came from for all option parsing. RELNOTES: None. PiperOrigin-RevId: 168682082
* Move option value tracking classes to their own file.Gravatar ccalvarin2017-09-13
| | | | | | | These classes are mostly used during the options parsing process itself, and are barely a part of the options parser interface, so they really don't belong in OptionsParser.java. They are also about to change significantly, so taking this opportunity to split them out. RELNOTES: None. PiperOrigin-RevId: 168400162
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-05 13:07:46 +0000