| Commit message (Collapse) | Author | Age |
|
|
|
|
| |
RELNOTES: None
PiperOrigin-RevId: 208646319
|
|
|
|
|
|
|
|
| |
removed "sychronized" from Reporter.handle() as we want to be able to let
EventHandlers care about their own synchronization.
RELNOTES: None.
PiperOrigin-RevId: 208628638
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Using a thread-safe container to store handlers.
- Using a ThreadLocal NumberFormat to produce readable action counts.
NumberFormat (or rather the default implementation DecimalFormat) use
internal state and locking.
- Adding a lock-free fast path for START and FINISH events in
ExperimentalEventHandler.
RELNOTES: None.
PiperOrigin-RevId: 208479896
|
|
|
|
|
|
|
|
|
| |
...in a single event instead of relying on the synchronisation on the
reporter. In this way, it will be possible to focus on output of errors,
and also to deduplicate warnings.
Change-Id: I9669e8497ca6e22a01f3e285ec9ee42c1ecf3fb5
PiperOrigin-RevId: 208223882
|
|
|
|
|
| |
RELNOTES: None
PiperOrigin-RevId: 208010973
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
If a Bazel server is idle for 10 seconds, it unconditionally triggers a full-scale Java GC via System.gc(). This behavior doesn't have clear benefits and causes Bazel to steal resources from whatever the user does after invoking Bazel. This CL adds a startup option, --idle_server_tasks, to toggle the idle GC behavior.
Also, add some logging for when idle GC is enabled, so it's easier to evaluate its effects. Example of logging:
```
180718 17:43:04.609:I 247 [com.google.devtools.build.lib.server.IdleServerTasks.lambda$idle$0] [Idle GC] used: 157MB -> 15MB, committed: 421MB -> 422MB
```
Fixes https://github.com/bazelbuild/bazel/issues/5589.
Closes #5628.
PiperOrigin-RevId: 207869996
|
|
|
|
|
|
|
|
|
|
|
| |
As nobody uses bazel on a traditional teletyper terminal, we
can as well use the line ending of native files instead of
terminal/network line endings. While the console output is not
meant to be handled by anything but a terminal, this still seems
to be done.
Change-Id: Ied745eeedaec70572ef4b3a3c716a0cf57dfdfd1
PiperOrigin-RevId: 207511649
|
|
|
|
|
|
|
| |
At the moment, an identity path resolver is passed. This will later be replaced by a contextual path resolver.
RELNOTES: None
PiperOrigin-RevId: 207138772
|
|
|
|
|
| |
RELNOTES: None
PiperOrigin-RevId: 207133488
|
|
|
|
|
|
|
|
| |
In some cases, we may be reporting a lot of actions, and this has shown up as a hotspot in our profiling. LinkedHashSet should be just as fast for insertion, keep order as Deque, and be faster for removal.
There are a couple of more aggressive steps we can take if this doesn't help.
PiperOrigin-RevId: 207064758
|
|
|
|
|
|
|
|
|
| |
We continue to support the jvm property -Dbazel.DigestFunction, for backwards compatibility, but this will go away. The startup-option is marked experimental for now as we iron out issues. (note: leaving this out of release notes until the experimental tag is removed)
As part of this refactor, the default constructor calls for FileSystem and derived classes will now use this default. This should remove the need for constructors that accept custom hash functions.
RELNOTES: None.
PiperOrigin-RevId: 207035217
|
|
|
|
|
| |
RELNOTES: None
PiperOrigin-RevId: 206741115
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 206310700
|
|
|
|
|
|
|
| |
chrome://tracing is able to load gzipped profiles out of the box.
RELNOTES: None
PiperOrigin-RevId: 206308018
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When the experimental UI is asked to limit its overall output, reserve a certain
fraction for output to be generated after the build is complete. In this way,
at least a final status, or a bit of the test summary, is shown.
Also, to help staying within the limit also for small limits, from a certain
threshold onwards, more severely restrict the output we allow for each individual
action.
Change-Id: I912aec0dd17639d9864133a10359f93537b473ad
PiperOrigin-RevId: 206288651
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
as a list, rather than as a single line (i.e., newline delimited rather than
space delimited).
Before:
SUBCOMMAND: # //src/main/java/com/google/devtools/build/lib:string_util [action 'Building src/main/java/com/google/devtools/build/lib/libstring_util.jar (2 source files) [for host]']
(cd /tmp/devbazel_output_base/execroot/io_bazel && \
exec env - \
LC_CTYPE=en_US.UTF-8 \
external/embedded_jdk/bin/java -XX:+UseParallelOldGC -XX:-CompactStrings '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--patch-module=java.compiler=external/bazel_tools/third_party/java/jdk/langtools/java_compiler.jar' '--patch-module=jdk.compiler=external/bazel_tools/third_party/java/jdk/langtools/jdk_compiler.jar' '--add-opens=java.base/java.nio=ALL-UNNAMED' -jar external/bazel_tools/tools/jdk/JavaBuilder_deploy.jar @bazel-out/host/bin/src/main/java/com/google/devtools/build/lib/libstring_util.jar-2.params)
After:
SUBCOMMAND: # //src/main/java/com/google/devtools/build/lib:string_util [action 'Building src/main/java/com/google/devtools/build/lib/libstring_util.jar (2 source files) [for host]']
(cd /tmp/devbazel_output_base/execroot/io_bazel && \
exec env - \
LC_CTYPE=en_US.UTF-8 \
external/embedded_jdk/bin/java \
-XX:+UseParallelOldGC \
-XX:-CompactStrings \
'--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' \
'--add-exports=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED' \
'--add-exports=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED' \
'--add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' \
'--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' \
'--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED' \
'--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' \
'--add-opens=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' \
'--patch-module=java.compiler=external/bazel_tools/third_party/java/jdk/langtools/java_compiler.jar' \
'--patch-module=jdk.compiler=external/bazel_tools/third_party/java/jdk/langtools/jdk_compiler.jar' \
'--add-opens=java.base/java.nio=ALL-UNNAMED' \
-jar \
external/bazel_tools/tools/jdk/JavaBuilder_deploy.jar \
@bazel-out/host/bin/src/main/java/com/google/devtools/build/lib/libstring_util.jar-2.params)
RELNOTES: --subcommands can now take a "pretty_print" value ("--subcommands=pretty_print") to print the
arguments of subcommands as a list for easier reading.
PiperOrigin-RevId: 206213009
|
|
|
|
|
|
|
|
|
|
| |
to avoid confusion between the LHS and RHS host_javabases.
The LHS --host_javabase option should be considered deprecated and will
eventually be removed.
RELNOTES: Rename the startup flag --host_javabase to --server_javabase to avoid confusion with the build flag --host_javabase
PiperOrigin-RevId: 206015757
|
|
|
|
|
|
|
|
|
|
| |
When there is a hard limit on the output of the experimental UI, change
the behavior to only write to the underlying stream on flush. In this
way, we can avoid semantically incomplete writes at the moment we
run out of characters.
Change-Id: I024c776ae2139d76d380bb89d13b8fe61d6cfe51
PiperOrigin-RevId: 206000817
|
|
|
|
|
|
|
|
|
|
|
| |
Also simplify LoadingPhaseCompleteEvent, and SkyframeExecutor, and remove
LoadingCallback, which is unnecessary now that we only have a single
implementation (previously LoadingPhaseRunner).
This also removes some of the excessive Skyframe calls introduced by
https://github.com/bazelbuild/bazel/commit/1067310e18cb9ac203110726de0be53bdc403cea, and prepares for interleaving target pattern eval and loading.
PiperOrigin-RevId: 205813197
|
|
|
|
|
|
|
|
|
|
|
| |
The conversion approach we were previously using is not stable - the resulting
offsets can differ based on what other things are going on on the same machine
at the same time.
By using an interface and passing it to the relevant places (and only computing
the offset once), we ensure that all conversions are consistent with each other.
PiperOrigin-RevId: 205787309
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
On two fronts: First, it should follow standard command line semantics. Second, it should work as intended: --noblock_for_lock means the client will not wait for another command to finish, but will exit eagerly. It can be useful for preventing hanging in applications that are non-interactively calling bazel.
It should have standard startup-option semantics: the default value is accepted as a no-op or can be provided to override a previous value.
The next issue involves 2 different locks - the client lock, and the server-side command lock. This duality exists because we would like, one day, to be able to run certain commands, like info or help, at the same time, so multiple commands would need specialized locks that allow some duality but blocks others. This can only be done at the server level, so as soon as the client gets the "we're connected" grpc message from the server, it releases the client lock and lets the server manage multiple requests.
There are basically 3 possible states that are relevant to this option:
1) no other client is active, so no one holds the client lock or the command lock - the server can be used, shutdown or started as needed. - no blocking, but no need to block, either, so we're safe
2) another client (client1) holds the lock, but it is currently using a server that we want to reuse. If client1 still holds the client lock, we fail fast. Same thing if client1 is holding the server-side lock: we will exit gracefully when the BlazeCommandDispatcher responds with a failure.
3) client1 holds the lock but its server cannot be reused. (batch clients also fall into this category, as there is no server to reuse - but in this case, the client lock is still in play). However, for server mode, this is broken - the following happens:
- Server is occupied with client1's request, holds the command lock
- client2 wants to restart the server, so sends the old server a "shutdown" command
- the BlazeCommandDispatcher says - nuh-uh, this is busy, and you said you didn't want to wait for the lock
- client2 absorbs this response
- waits (blocks...)
- for a minute
- then force shuts-down the old server.
So we had 2 problems - we block, and we shutdown a server that we truly intended to keep going. Now, if the server responds saying another action is using it, the client will exit correctly, and leave the old server to do its thing.
RELNOTES: None.
PiperOrigin-RevId: 205671817
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 205635805
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 205447838
|
|
|
|
|
| |
RELNOTES: Add aquery command to get analysis time information about the action graph.
PiperOrigin-RevId: 205064145
|
|
|
|
|
|
|
| |
It was not opensourced, and even internally was not used. And we hate having internal-only code.
RELNOTES: None.
PiperOrigin-RevId: 204441702
|
|
|
|
|
|
|
| |
uploader cannot upload a particular file.
RELNOTES: None
PiperOrigin-RevId: 204167372
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There can be multiple BuildEventTransports active at
the same time and we need to ensure that each transport
gets its own BuildEventArtifactUploader as these transports
might have different lifecycles.
We do that by introducing another level of indirection via
the BuildEventArtifactUploaderFactory. BlazeModules now
register a factory object instead of an uploader.
In addition, the BuildEventArtifactUploader gets a shutdown()
method that allows to free any resources associated with it.
PiperOrigin-RevId: 203752092
|
|
|
|
|
|
|
| |
PiperOrigin-RevId: 203300374
Change-Id: Iaa47f870ab2e0cad40a202aad2c7f9430f73c856
PiperOrigin-RevId: 203407392
|
|
|
|
|
|
|
| |
Instead of just a path, events now include information about the type of file (output, source file, stdout/stderr, test logs, etc.). This information can be used by the uploaders to determine a) whether to upload, b) what kind of lease to give the files.
RELNOTES: None
PiperOrigin-RevId: 203285549
|
|
|
|
|
| |
RELNOTES: --[no]allow_undefined_configs no longer exists, passing undefined configs is an error.
PiperOrigin-RevId: 202518987
|
|
|
|
|
|
|
|
|
| |
enum.
Now that we aren't using enum names for the hash functions, we also accept the standard names, such as SHA-256.
RELNOTES: None.
PiperOrigin-RevId: 201624286
|
|
|
|
|
|
|
| |
RELNOTES[INC]: The mobile-install --skylark_incremental_res flag is no longer
available, use the --skylark flag instead.
PiperOrigin-RevId: 201574876
|
|
|
|
|
|
|
|
|
|
| |
This gives us more coverage for command startup, especially wrt. modules. We
have some modules that are known to be slow, and this immediately shows any
such issues. I've already tracked down two issues with this change, and it
would also have shown a recently fixed regression.
RELNOTES: None.
PiperOrigin-RevId: 201337059
|
|
|
|
|
|
|
| |
The memory savings from this flag are not worth the complexity, and it interferes with action restarting.
RELNOTES: Remove support for --discard_actions_after_execution.
PiperOrigin-RevId: 201077905
|
|
|
|
|
|
| |
Moves more Action-oriented from skyframe package, which has become very unwieldy, to action package. This is needed to avoid circular dependencies caused when build-base is needed for skyframe.
PiperOrigin-RevId: 200996982
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The intent is that the main thread has ~zero gaps in the profile (though there
may still be small gaps due to the time between one try block and an
immediately subsequent try block).
We need to be careful not to wrap markPhase calls (or methods that call
markPhase) in try blocks for the profiler - the Profiler requires that all
markPhase calls happen at top level, and throws an exception if not.
This should not have any performance impact - all of these are once per build,
or at most once per module per build, and we don't expect a very large number
of modules (and if we see an increasing number, we need to change the module
API to not have to call every single module, but only those that are actually
interested in certain events, maybe with an EventBus-based setup).
PiperOrigin-RevId: 200712677
|
|
|
|
|
|
|
|
|
|
| |
The intend is to move the profiler setup earlier in the startup sequence, so we
can get more profiler coverage. Ideally, we'd setup the profiler immediately
after getting the client call, but this is not currently possible due to the
requirement that initialization and shutdown happen in pairs (otherwise the
next command invocation crashes with an exception from the Profiler).
PiperOrigin-RevId: 200707029
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
They weren't actually doing anything. With default flags, createHandler never
returns a FancyTerminalEventHandler, because that always gets wrapped in a
RateLimitingEventHandler. Even so, under normal circumstances, the fancy
terminal event handler always leaves the terminal in a valid state at the end
of each printed line. We can only end up in a bad state if the blaze server is
killed, but even then, this code wasn't doing anything because it didn't get to
run.
The only way to guarantee that the terminal is always in a valid state is to
make sure that every stdout/stderr buffer we send to the client ends with a
reset code sequence. The experimental ui handler is better at sending only
complete buffers (we were previously using an auto-line flushing output stream,
which didn't allow the event handler to reason about how much stuff got
actually printed in a single write call).
Anyway, I think we're in a better place now, and this code wasn't run, and if
we want to fix it properly (if the switch to experimental_ui isn't sufficient),
then we need to fix it elsewhere.
PiperOrigin-RevId: 200699609
|
|
|
|
|
|
|
| |
- reduce the parameters to the minimum required
- change to void, move additional setup into the method
PiperOrigin-RevId: 200692559
|
|
|
|
|
|
|
|
| |
This should be a no-op, mostly replacing PathConverter with
BuildEventArtifactUploader, since none of the implementations perform any
upload yet.
PiperOrigin-RevId: 200685325
|
|
|
|
|
|
|
| |
Deprecates the flag, though it continues to exist as a no-op so that users get a warning before errors.
RELNOTES: --noexpand_configs_in_place is deprecated.
PiperOrigin-RevId: 200572053
|
|
|
|
|
|
|
| |
These are no longer necessary since we don't pipe the subprocess stdout/stderr
through Bazel anymore.
PiperOrigin-RevId: 200534552
|
|
|
|
| |
PiperOrigin-RevId: 200518973
|
|
|
|
| |
PiperOrigin-RevId: 200516863
|
|
|
|
|
|
|
| |
of bazel info crosstool-configuration
RELNOTES: None.
PiperOrigin-RevId: 200437121
|
|
|
|
|
|
|
| |
This is in preparation for dismantling BuildView and merging the relevant
parts into AnalysisPhaseRunner.
PiperOrigin-RevId: 200391088
|
|
|
|
|
|
| |
Do the work in beforeCommand instead.
PiperOrigin-RevId: 200391086
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When this flag is turned on, and the user's working directory is beneath the
workspace (but is not the workspace itself), the workspace's absolute path is
printed as a prefix to the convenience symlink.
With this flag off, the displayed convenience symlink path is always relative
to the user's workspace, even if that isn't the current working directory.
(This is the current behavior.)
The new (flag-on) behavior will become default soon, and then this flag will
be removed.
RELNOTES: None.
PiperOrigin-RevId: 200278355
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add a --experimental_generate_json_trace_profile option that puts a file into
the output base (or uses --profile if set).
There are still a lot of problems with this.
- unexplained holes
- way too many threads
- nonsensical event titles
- too many detail events, too little overview
- it may also cause unnecessary load
- it silently overwrites the existing file on subsequent invocations
The format is documented here: goo.gl/oMZPLh
PiperOrigin-RevId: 200259431
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- move the save method to an inner class
- don't use a timer, use a blocking queue instead
- add a format enum (in anticipation of adding a json output format)
- update the test to use an in memory buffer, and avoid FoundationTestCase
Compared to the original https://github.com/bazelbuild/bazel/commit/15b8c259db111012b4642287172cb4d1d82151f3, it contains these changes:
- Make it so we don't create a queue if we are not going to write any
data! The queue is now owned by the writer, and if there is no writer, there
is no queue.
This was causing a memory regression because slowest task profiling is
enabled by default, in which case the profiler is started with no output
file. In that case, there's no thread that is emptying the queue, but the
queue was still created by default.
- add additional tests for slowest task and histogram handling; these also
provide coverage for the case where the profiler is started without an output
stream
- move all the writer thread handling into the inner class
- make writer access thread-safe
- add a bunch of documentation
PiperOrigin-RevId: 200212978
|