| Commit message (Collapse) | Author | Age |
|
|
|
|
|
|
| |
Move the message-digest cloning to DigestHashFunction and out of Fingerprint, to make it possible to configure Fingerprint to use different hash functions. We keep the default MD5 for now, we'd like it to use the global default but want to isolate the configuration change from any change adding potential contention.
RELNOTES: None.
PiperOrigin-RevId: 208502993
|
|
|
|
|
|
|
|
|
|
|
| |
We want a way for Bazel to find a logging handler's current log file without
direct dependencies on the exact handler class. We do this with an abstract
parent class whose concrete child class (to be used as a singleton) will be
given in startup_options, i.e. in the same place as the server logging
configuration.
RELNOTES: None.
PiperOrigin-RevId: 208171084
|
|
|
|
|
|
|
|
|
|
|
| |
As nobody uses bazel on a traditional teletyper terminal, we
can as well use the line ending of native files instead of
terminal/network line endings. While the console output is not
meant to be handled by anything but a terminal, this still seems
to be done.
Change-Id: Ied745eeedaec70572ef4b3a3c716a0cf57dfdfd1
PiperOrigin-RevId: 207511649
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It provides a number of features that we want and whose combination cannot be
accomplished using the standard FileHandler:
* Using a different filename per server process, by putting a timestamp and
process ID in the filename. This means Bazel will no longer overwrite its
log when the server is restarted, making it easier for developers and
maintainers to diagnose issues.
* Putting the hostname and username in the filename (useful when running on a
shared network filesystem).
* Automatically setting a symlink to the latest log file, ensuring that the
latest log can still be found under the usual Bazel server log path.
* Providing an API for getting the filename of the current log file, for use
by Bazel itself.
* Cleaning up old log files when their total size exceeds a set limit.
This commit only introduces the handler; its usage in Bazel will be enabled by
a follow-up commit.
RELNOTES: None.
PiperOrigin-RevId: 207274587
|
|
|
|
| |
PiperOrigin-RevId: 206573961
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
as a list, rather than as a single line (i.e., newline delimited rather than
space delimited).
Before:
SUBCOMMAND: # //src/main/java/com/google/devtools/build/lib:string_util [action 'Building src/main/java/com/google/devtools/build/lib/libstring_util.jar (2 source files) [for host]']
(cd /tmp/devbazel_output_base/execroot/io_bazel && \
exec env - \
LC_CTYPE=en_US.UTF-8 \
external/embedded_jdk/bin/java -XX:+UseParallelOldGC -XX:-CompactStrings '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--patch-module=java.compiler=external/bazel_tools/third_party/java/jdk/langtools/java_compiler.jar' '--patch-module=jdk.compiler=external/bazel_tools/third_party/java/jdk/langtools/jdk_compiler.jar' '--add-opens=java.base/java.nio=ALL-UNNAMED' -jar external/bazel_tools/tools/jdk/JavaBuilder_deploy.jar @bazel-out/host/bin/src/main/java/com/google/devtools/build/lib/libstring_util.jar-2.params)
After:
SUBCOMMAND: # //src/main/java/com/google/devtools/build/lib:string_util [action 'Building src/main/java/com/google/devtools/build/lib/libstring_util.jar (2 source files) [for host]']
(cd /tmp/devbazel_output_base/execroot/io_bazel && \
exec env - \
LC_CTYPE=en_US.UTF-8 \
external/embedded_jdk/bin/java \
-XX:+UseParallelOldGC \
-XX:-CompactStrings \
'--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' \
'--add-exports=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED' \
'--add-exports=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED' \
'--add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' \
'--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' \
'--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED' \
'--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' \
'--add-opens=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' \
'--patch-module=java.compiler=external/bazel_tools/third_party/java/jdk/langtools/java_compiler.jar' \
'--patch-module=jdk.compiler=external/bazel_tools/third_party/java/jdk/langtools/jdk_compiler.jar' \
'--add-opens=java.base/java.nio=ALL-UNNAMED' \
-jar \
external/bazel_tools/tools/jdk/JavaBuilder_deploy.jar \
@bazel-out/host/bin/src/main/java/com/google/devtools/build/lib/libstring_util.jar-2.params)
RELNOTES: --subcommands can now take a "pretty_print" value ("--subcommands=pretty_print") to print the
arguments of subcommands as a list for easier reading.
PiperOrigin-RevId: 206213009
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
On two fronts: First, it should follow standard command line semantics. Second, it should work as intended: --noblock_for_lock means the client will not wait for another command to finish, but will exit eagerly. It can be useful for preventing hanging in applications that are non-interactively calling bazel.
It should have standard startup-option semantics: the default value is accepted as a no-op or can be provided to override a previous value.
The next issue involves 2 different locks - the client lock, and the server-side command lock. This duality exists because we would like, one day, to be able to run certain commands, like info or help, at the same time, so multiple commands would need specialized locks that allow some duality but blocks others. This can only be done at the server level, so as soon as the client gets the "we're connected" grpc message from the server, it releases the client lock and lets the server manage multiple requests.
There are basically 3 possible states that are relevant to this option:
1) no other client is active, so no one holds the client lock or the command lock - the server can be used, shutdown or started as needed. - no blocking, but no need to block, either, so we're safe
2) another client (client1) holds the lock, but it is currently using a server that we want to reuse. If client1 still holds the client lock, we fail fast. Same thing if client1 is holding the server-side lock: we will exit gracefully when the BlazeCommandDispatcher responds with a failure.
3) client1 holds the lock but its server cannot be reused. (batch clients also fall into this category, as there is no server to reuse - but in this case, the client lock is still in play). However, for server mode, this is broken - the following happens:
- Server is occupied with client1's request, holds the command lock
- client2 wants to restart the server, so sends the old server a "shutdown" command
- the BlazeCommandDispatcher says - nuh-uh, this is busy, and you said you didn't want to wait for the lock
- client2 absorbs this response
- waits (blocks...)
- for a minute
- then force shuts-down the old server.
So we had 2 problems - we block, and we shutdown a server that we truly intended to keep going. Now, if the server responds saying another action is using it, the client will exit correctly, and leave the old server to do its thing.
RELNOTES: None.
PiperOrigin-RevId: 205671817
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Use try-with-resources to ensure OutputStreams
that we open via FileSystem.OutputStream(path)
are closed.
Eagerly closing OutputStreams avoids hanging on to
file handles until the garbage collector finalizes
the OutputStream, meaning Bazel on Windows (and
other processes) can delete or mutate these files.
Hopefully this avoids intermittent file deletion
errors that sometimes occur on Windows.
See https://github.com/bazelbuild/bazel/issues/5512
RELNOTES: none
PiperOrigin-RevId: 203342889
|
|
|
|
|
|
|
|
| |
Based on work of Sun Zhiyi <zhiyisun@gmail.com>.
Closes #5360.
PiperOrigin-RevId: 201339882
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Background: the original code is implementing the OutputStream interface. Given a sequence of write calls, the code puts each of these writes into a queue. On the other side of the queue is an unbounded thread pool, which takes the writes off the queue one by one, and then does individual blocking writes with a fixed file offset.
There are three problems with the original code:
1. Writes are sent to the Kernel one-by-one. Imagine if the incoming writes a single-byte writes, then we do one kernel call for every single byte. This is the worst case.
2. Due to multithreading, the writes can get reordered. In the worst case, the order is reversed, and the kernel flushes each write to disk individually. Since the writes are not aligned to disk block boundaries, each write has to first *read* the block from disk, overwrite a few bytes, and then flush the block back to disk. On a spinning platter, this is the worst possible sequence of operations: write a block, read the same block back, write the same block again, read the same block back, etc., with each operation having to wait for a full disk rotation.
Note that this gets worse if there's high thread and / or disk contention, e.g., when running a build system in the background.
3. Due to the unbounded thread pool, it may end up creating a new thread for every single write (possibly as many as one per byte written). This is also the worst case, although it's probably negligible compared to 1+2.
Compared to that, this change uses an in-memory buffer before sending writes to the kernel so non-block sized writes are batched, it writes sequentially, and it uses a single thread created at the start. A single thread should be more than able to fully saturate local disk I/O, so multi-threading should ~never be a improvement here.
This might look different if we had perfectly aligned writes of an integer multiple of the disk block size to a distributed network file system or a local SSD raid. If you look at the clients of this class, that's definitely not the case: this code is primarily used for local file BEP transports - we wouldn't use local file BEP transports to write to a network file system, we'd use the BES instead. It's also used to write a couple of log files, also all local - otherwise we'd add the data to BEP. These are all unaligned, ~random-sized writes.
If we created a lot of these files, then using a thread pool with fewer threads than files and using non-blocking I/O might be an improvement due to the reduction in thread count, but I think it's very unlikely that we'll ever need that complexity.
PiperOrigin-RevId: 200694423
|
|
|
|
| |
PiperOrigin-RevId: 197461794
|
|
|
|
|
|
|
|
|
| |
This constructor was creating an Exception with a null message leading to possible
NullPointerExceptions in a few places in our codebase. The call sites have
been replaced with calls to AbruptException(String message, ExitCode exitCode) with
a meaningful message.
PiperOrigin-RevId: 196973540
|
|
|
|
|
| |
RELNOTES: None
PiperOrigin-RevId: 196553226
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 194236052
|
|
|
|
|
|
| |
exceptions are detected in async worker threads.
PiperOrigin-RevId: 193725871
|
|
|
|
|
|
|
|
| |
This is a very heavy and slow option by design. It will be enabled only when a user wants to debug their build, most frequently to compare the step-by-step results of two builds.
TESTED: manually on various rules, including directories.
RELNOTES: None
PiperOrigin-RevId: 193539034
|
|
|
|
|
|
|
| |
It breaks downstream rules_nodejs. See https://github.com/bazelbuild/bazel/issues/5028 for details.
RELNOTES: None.
PiperOrigin-RevId: 193074798
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Remove the .useShell method, expect callers to
just pass the shell interpreter if they need it.
This removes the argument vector transformation
heuristic, and stops shelling out to cmd.exe on
Windows.
Also remove the .setWorkingDir method because
callers always had to set the working directory.
Instead, the CommandBuilder constructor takes the
working directory.
Closes #2190. (Test code still shells out to
cmd.exe though.)
Closes #5007.
Change-Id: I545e01c811daaf34913cb585492923da81aa02ee
PiperOrigin-RevId: 192611370
|
|
|
|
| |
PiperOrigin-RevId: 190532114
|
|
|
|
|
|
|
|
| |
totally described by their extensions, since many are not. Instead, explicitly @AutoCodec those that are, and leave the rest to be individually handled. This also allows us to do equality checking.
There is a slight memory penalty here, but there are ~100s of FileTypes in a large build, so it should be negligible.
PiperOrigin-RevId: 190477427
|
|
|
|
|
|
|
|
|
| |
overwrite the environment of their caller.
Admittedly, doing that was a bad idea in the first place.
RELNOTES: None.
PiperOrigin-RevId: 190430189
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fixes #2054.
Allow users to be able to specify user specific paths. With this option
we can now commit bazel configuration file and force local action cache
activation per default:
$ cat tools/bazel.rc
build --experimental_local_disk_cache_path=~/.gerrit/bazel-cache/cas
build --experimental_local_disk_cache
build --experimental_strict_action_env
Test Plan:
$ bazel test //src/test/java/com/google/devtools/build/lib:util_test
Closes #4852.
PiperOrigin-RevId: 189744599
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add a new test utility class
TimestampGranularityUtils which offers methods to
wait for the filesystem timestamp granularity to
pass.
Such waiting is necessary if a test asserts
changes in a file's mtime or ctime.
This is a follow-up to comments on https://github.com/bazelbuild/bazel/commit/a31e035fb42aa8db9fd248ef2ebd665a411053b8.
PiperOrigin-RevId: 188847729
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fix the testTraversalOfGeneratedDirectory method
in RecursiveFilesystemTraversalFunctionTest that
was flaky on Windows.
The fix is to wait so that changes to files in the
InMemoryFileSystem will have observable effects on
the file ctimes.
Depends on https://github.com/bazelbuild/bazel/pull/4787
Fixes https://github.com/bazelbuild/bazel/issues/4755
Closes #4788.
PiperOrigin-RevId: 188470080
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fix testCacheBypassingActionWithMtimeChangingInput
in SkyframeAwareActionTest by ensuring that enough
time elapses between file updates so their effects
are observable on the file's ctime.
Fixes https://github.com/bazelbuild/bazel/issues/4755
Closes #4787.
PiperOrigin-RevId: 188458542
|
|
|
|
|
|
|
| |
WANT_LGTM=buchgr
TESTED=unit tests, 500 runs per test
RELNOTES: None
PiperOrigin-RevId: 188093043
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Fixed issue that led to rollback.
RELNOTES: None
*** Original change description ***
Automated rollback of commit eee53d3a33dde441f7e7adaecde81ef2d3db7c1b.
*** Reason for rollback ***
breaks Blaze_CorpTest
*** Original change description ***
@AutoCodec ConfiguredTargetValue.
PiperOrigin-RevId: 187869560
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
breaks Blaze_CorpTest
*** Original change description ***
@AutoCodec ConfiguredTargetValue.
PiperOrigin-RevId: 187783702
|
|
|
|
|
|
| |
don't care about the group structure, and simplify the logic for prefetching old deps.
PiperOrigin-RevId: 187681887
|
|
|
|
| |
PiperOrigin-RevId: 187635570
|
|
|
|
| |
PiperOrigin-RevId: 187397314
|
|
|
|
|
|
| |
More strictly type the arguments for creating FileTypes.
PiperOrigin-RevId: 187391336
|
|
|
|
|
|
|
|
|
|
| |
email from philwo@.
(i) Only have TimestampGranularityMonitor log the first file of relevance. This reduces log spam, especially in tests, while still maintaining useful information in the logs.
(ii) Don't have ExternalFilesHelper log the fact that it encountered an external file when we're in a unit test or an integration test. Tests, especially bazel tests that use external repositories, tend to involve lots of "external" files.
RELNOTES: None
PiperOrigin-RevId: 186799176
|
|
|
|
|
|
|
|
| |
We're now using ctime to detect file changes, so the timestamp granularity monitor should as well.
Unfortunately, we currently get nanosecond ctime from Linux, but then only return millis from FileStatus, so this doesn't change the accuracy of the monitor at all.
PiperOrigin-RevId: 184536539
|
|
|
|
|
|
| |
This is needed to migrate JavaCompileAction away from CustomMultiArgv.
PiperOrigin-RevId: 184136486
|
|
|
|
|
|
|
|
|
| |
Instead of using ConcurrentHashMap, we use a dead-simple open addressed hash hable with a giant byte array with 16-byte slots. We then read or write fingerprints straight into and out of the array, obviating the need to generate intermediate garbage.
Locking mechanism is a read-write lock. This should be faster than full synchronisation for read-heavy loads.
RELNOTES: None
PiperOrigin-RevId: 184019301
|
|
|
|
|
|
| |
Generalizes @AutoCodec.Constructor to @AutoCodec.Instantiator.
PiperOrigin-RevId: 183702768
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
incomplete deps that need to be removed, and
these deps are currently duplicated in the NodeEntry's newly requested deps
GroupedList.
Also add a fast-path to GroupedListHelper#remove(List<Object> elements, Set<E> toRemove) for the incredibly common case where toRemove is empty. This saves a wasteful O(elements.size()) scan over elements. This method is unconditionally called each time a SkyFunction restart causes us to add new direct deps (with elements=<the new direct deps> and toRemove=<unfinished direct deps>); in the case where there are a ton of new direct deps, this scan entails wasted cpu and gc churn. The bug only occurs in uncommon case that there are deps to remove.
The bug has existed since GroupedList was first introduced into the codebase.
In Skyframe-land, this is only observable in nokeep_going mode because in keep_going mode "we
do not let SkyFunctions throw errors with missing deps" (quote from
comment in AbstractParallelEvaluator).
A Bazel-on-Skyframe-land example how this bug could occur in practice is PackageFunction's Skyframe hybrid globbing. If an io error is encountered during legacy globbing, the PackageFunction eagerly throws a SkyFunctionException but it has already requested the Skyframe GlobValue deps.
RELNOTES: None
PiperOrigin-RevId: 182403943
|
|
|
|
|
|
| |
Uses the constructor having the @AutoCodec.Constructor annotation to generate a codec (instead of choosing the first in source code). This annotation is required when a class has more than one constructor.
PiperOrigin-RevId: 180685902
|
|
|
|
|
|
|
|
| |
This is fast (profiles indicate a 9% penalty vs getters) and does not require getters.
* Also adds a Marshaller for java.util.regex.Pattern.
* Minor fixes for codecs that were broken by this change.
PiperOrigin-RevId: 180619680
|
|
|
|
| |
PiperOrigin-RevId: 179931575
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Change FileType to no longer assume it operates on just the base name (it can now be given a full path).
* Move the responsibility to specific classes (Artifact, Path, PathFragment) to decide how they want to offer up a string that includes the file name.
* Flip the order in which users are expected to check Artifact type, from FileType#matches(Artifact) to Artifact#isFileType(FileType). This looks natural and should encourage developers to use efficient file type checking methods.
* Change CppCompileAction to use the new API.
RELNOTES: None
PiperOrigin-RevId: 179903239
|
|
|
|
|
|
|
|
| |
system.
This is the only place that should actually need it.
PiperOrigin-RevId: 179054861
|
|
|
|
| |
PiperOrigin-RevId: 178942449
|
|
|
|
|
|
| |
deps from a compressed GroupedList without uncompressing it. Also some minor GC improvements.
PiperOrigin-RevId: 177338852
|
|
|
|
|
| |
RELNOTES:
PiperOrigin-RevId: 176844836
|
|
|
|
| |
PiperOrigin-RevId: 175092923
|
|
|
|
|
|
|
|
| |
Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own.
This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually.
PiperOrigin-RevId: 175033526
|
|
|
|
|
|
|
| |
Fixed https://github.com/bazelbuild/bazel/issues/3962
Change-Id: I772a409f03b9cc2c0e69e4f51d8a7b95688138a4
PiperOrigin-RevId: 173374845
|
|
|
|
| |
PiperOrigin-RevId: 171180273
|