aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/runtime
Commit message (Collapse)AuthorAge
* Fix lock contention in Reporter.startTask/.finishTask by:Gravatar Googler2018-08-13
| | | | | | | | | | | | - Using a thread-safe container to store handlers. - Using a ThreadLocal NumberFormat to produce readable action counts. NumberFormat (or rather the default implementation DecimalFormat) use internal state and locking. - Adding a lock-free fast path for START and FINISH events in ExperimentalEventHandler. RELNOTES: None. PiperOrigin-RevId: 208479896
* Combine prefix event and payload stdout/stderrGravatar Klaus Aehlig2018-08-10
| | | | | | | | | ...in a single event instead of relying on the synchronisation on the reporter. In this way, it will be possible to focus on output of errors, and also to deduplicate warnings. Change-Id: I9669e8497ca6e22a01f3e285ec9ee42c1ecf3fb5 PiperOrigin-RevId: 208223882
* Remove unused workspace variable.Gravatar twerth2018-08-09
| | | | | RELNOTES: None PiperOrigin-RevId: 208010973
* Add a option to disable idle gc.Gravatar Benjamin Peterson2018-08-08
| | | | | | | | | | | | | | | If a Bazel server is idle for 10 seconds, it unconditionally triggers a full-scale Java GC via System.gc(). This behavior doesn't have clear benefits and causes Bazel to steal resources from whatever the user does after invoking Bazel. This CL adds a startup option, --idle_server_tasks, to toggle the idle GC behavior. Also, add some logging for when idle GC is enabled, so it's easier to evaluate its effects. Example of logging: ``` 180718 17:43:04.609:I 247 [com.google.devtools.build.lib.server.IdleServerTasks.lambda$idle$0] [Idle GC] used: 157MB -> 15MB, committed: 421MB -> 422MB ``` Fixes https://github.com/bazelbuild/bazel/issues/5589. Closes #5628. PiperOrigin-RevId: 207869996
* Console output: switch to file line endingGravatar Klaus Aehlig2018-08-06
| | | | | | | | | | | As nobody uses bazel on a traditional teletyper terminal, we can as well use the line ending of native files instead of terminal/network line endings. While the console output is not meant to be handled by anything but a terminal, this still seems to be done. Change-Id: Ied745eeedaec70572ef4b3a3c716a0cf57dfdfd1 PiperOrigin-RevId: 207511649
* Thread a path resolver through target/aspect complete event.Gravatar tomlu2018-08-02
| | | | | | | At the moment, an identity path resolver is passed. This will later be replaced by a contextual path resolver. RELNOTES: None PiperOrigin-RevId: 207138772
* Make TestSummary event paths use the file system from ActionExecutionContext.Gravatar tomlu2018-08-02
| | | | | RELNOTES: None PiperOrigin-RevId: 207133488
* Change Deque -> LinkedHashSetGravatar ulfjack2018-08-02
| | | | | | | | In some cases, we may be reporting a lot of actions, and this has shown up as a hotspot in our profiling. LinkedHashSet should be just as fast for insertion, keep order as Deque, and be faster for removal. There are a couple of more aggressive steps we can take if this doesn't help. PiperOrigin-RevId: 207064758
* Add a normal startup-option for setting the digest function.Gravatar ccalvarin2018-08-01
| | | | | | | | | We continue to support the jvm property -Dbazel.DigestFunction, for backwards compatibility, but this will go away. The startup-option is marked experimental for now as we iron out issues. (note: leaving this out of release notes until the experimental tag is removed) As part of this refactor, the default constructor calls for FileSystem and derived classes will now use this default. This should remove the need for constructors that accept custom hash functions. RELNOTES: None. PiperOrigin-RevId: 207035217
* Add option to post ProfileStartedEvent containing the profile's path.Gravatar twerth2018-07-31
| | | | | RELNOTES: None PiperOrigin-RevId: 206741115
* Update ProjectFileSupport to take an OptionsParser.Gravatar Googler2018-07-27
| | | | | RELNOTES: None. PiperOrigin-RevId: 206310700
* Add an option to compress the JSON trace profile.Gravatar twerth2018-07-27
| | | | | | | chrome://tracing is able to load gzipped profiles out of the box. RELNOTES: None PiperOrigin-RevId: 206308018
* Experimental UI: on limited output, reserve capacity for post-build statusGravatar Klaus Aehlig2018-07-27
| | | | | | | | | | | | | When the experimental UI is asked to limit its overall output, reserve a certain fraction for output to be generated after the build is complete. In this way, at least a final status, or a bit of the test summary, is shown. Also, to help staying within the limit also for small limits, from a certain threshold onwards, more severely restrict the output we allow for each individual action. Change-Id: I912aec0dd17639d9864133a10359f93537b473ad PiperOrigin-RevId: 206288651
* Modify --subcommands to allow pretty printing the arguments of a subcommandGravatar ahumesky2018-07-26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | as a list, rather than as a single line (i.e., newline delimited rather than space delimited). Before: SUBCOMMAND: # //src/main/java/com/google/devtools/build/lib:string_util [action 'Building src/main/java/com/google/devtools/build/lib/libstring_util.jar (2 source files) [for host]'] (cd /tmp/devbazel_output_base/execroot/io_bazel && \ exec env - \ LC_CTYPE=en_US.UTF-8 \ external/embedded_jdk/bin/java -XX:+UseParallelOldGC -XX:-CompactStrings '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--patch-module=java.compiler=external/bazel_tools/third_party/java/jdk/langtools/java_compiler.jar' '--patch-module=jdk.compiler=external/bazel_tools/third_party/java/jdk/langtools/jdk_compiler.jar' '--add-opens=java.base/java.nio=ALL-UNNAMED' -jar external/bazel_tools/tools/jdk/JavaBuilder_deploy.jar @bazel-out/host/bin/src/main/java/com/google/devtools/build/lib/libstring_util.jar-2.params) After: SUBCOMMAND: # //src/main/java/com/google/devtools/build/lib:string_util [action 'Building src/main/java/com/google/devtools/build/lib/libstring_util.jar (2 source files) [for host]'] (cd /tmp/devbazel_output_base/execroot/io_bazel && \ exec env - \ LC_CTYPE=en_US.UTF-8 \ external/embedded_jdk/bin/java \ -XX:+UseParallelOldGC \ -XX:-CompactStrings \ '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' \ '--add-opens=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' \ '--patch-module=java.compiler=external/bazel_tools/third_party/java/jdk/langtools/java_compiler.jar' \ '--patch-module=jdk.compiler=external/bazel_tools/third_party/java/jdk/langtools/jdk_compiler.jar' \ '--add-opens=java.base/java.nio=ALL-UNNAMED' \ -jar \ external/bazel_tools/tools/jdk/JavaBuilder_deploy.jar \ @bazel-out/host/bin/src/main/java/com/google/devtools/build/lib/libstring_util.jar-2.params) RELNOTES: --subcommands can now take a "pretty_print" value ("--subcommands=pretty_print") to print the arguments of subcommands as a list for easier reading. PiperOrigin-RevId: 206213009
* Add a --server_javabase startup option as an alias for the LHS --host_javabaseGravatar cushon2018-07-25
| | | | | | | | | | to avoid confusion between the LHS and RHS host_javabases. The LHS --host_javabase option should be considered deprecated and will eventually be removed. RELNOTES: Rename the startup flag --host_javabase to --server_javabase to avoid confusion with the build flag --host_javabase PiperOrigin-RevId: 206015757
* Experimental UI: only write on flush for limited outputGravatar Klaus Aehlig2018-07-25
| | | | | | | | | | When there is a hard limit on the output of the experimental UI, change the behavior to only write to the underlying stream on flush. In this way, we can avoid semantically incomplete writes at the moment we run out of characters. Change-Id: I024c776ae2139d76d380bb89d13b8fe61d6cfe51 PiperOrigin-RevId: 206000817
* Move LoadingPhaseCompleteEvent posting to TargetPatternPhaseFunctionGravatar ulfjack2018-07-24
| | | | | | | | | | | Also simplify LoadingPhaseCompleteEvent, and SkyframeExecutor, and remove LoadingCallback, which is unnecessary now that we only have a single implementation (previously LoadingPhaseRunner). This also removes some of the excessive Skyframe calls introduced by https://github.com/bazelbuild/bazel/commit/1067310e18cb9ac203110726de0be53bdc403cea, and prepares for interleaving target pattern eval and loading. PiperOrigin-RevId: 205813197
* Use an interface to convert from nanos to millis since epochGravatar ulfjack2018-07-24
| | | | | | | | | | | The conversion approach we were previously using is not stable - the resulting offsets can differ based on what other things are going on on the same machine at the same time. By using an interface and passing it to the relevant places (and only computing the offset once), we ensure that all conversions are consistent with each other. PiperOrigin-RevId: 205787309
* Fix block_for_lock.Gravatar ccalvarin2018-07-23
| | | | | | | | | | | | | | | | | | | | | | | | | | On two fronts: First, it should follow standard command line semantics. Second, it should work as intended: --noblock_for_lock means the client will not wait for another command to finish, but will exit eagerly. It can be useful for preventing hanging in applications that are non-interactively calling bazel. It should have standard startup-option semantics: the default value is accepted as a no-op or can be provided to override a previous value. The next issue involves 2 different locks - the client lock, and the server-side command lock. This duality exists because we would like, one day, to be able to run certain commands, like info or help, at the same time, so multiple commands would need specialized locks that allow some duality but blocks others. This can only be done at the server level, so as soon as the client gets the "we're connected" grpc message from the server, it releases the client lock and lets the server manage multiple requests. There are basically 3 possible states that are relevant to this option: 1) no other client is active, so no one holds the client lock or the command lock - the server can be used, shutdown or started as needed. - no blocking, but no need to block, either, so we're safe 2) another client (client1) holds the lock, but it is currently using a server that we want to reuse. If client1 still holds the client lock, we fail fast. Same thing if client1 is holding the server-side lock: we will exit gracefully when the BlazeCommandDispatcher responds with a failure. 3) client1 holds the lock but its server cannot be reused. (batch clients also fall into this category, as there is no server to reuse - but in this case, the client lock is still in play). However, for server mode, this is broken - the following happens: - Server is occupied with client1's request, holds the command lock - client2 wants to restart the server, so sends the old server a "shutdown" command - the BlazeCommandDispatcher says - nuh-uh, this is busy, and you said you didn't want to wait for the lock - client2 absorbs this response - waits (blocks...) - for a minute - then force shuts-down the old server. So we had 2 problems - we block, and we shutdown a server that we truly intended to keep going. Now, if the server responds saying another action is using it, the client will exit correctly, and leave the old server to do its thing. RELNOTES: None. PiperOrigin-RevId: 205671817
* Remove gender specific prononuns from Bazel codebaseGravatar hlopko2018-07-23
| | | | | RELNOTES: None. PiperOrigin-RevId: 205635805
* Close the query environment after running a query.Gravatar shreyax2018-07-20
| | | | | RELNOTES: None. PiperOrigin-RevId: 205447838
* Add aquery as proper command to Bazel.Gravatar twerth2018-07-18
| | | | | RELNOTES: Add aquery command to get analysis time information about the action graph. PiperOrigin-RevId: 205064145
* Remove bazel info crosstool-configurationGravatar hlopko2018-07-13
| | | | | | | It was not opensourced, and even internally was not used. And we hate having internal-only code. RELNOTES: None. PiperOrigin-RevId: 204441702
* Omit parts of BEP events (like stdout, stderr) when the associated artifact ↵Gravatar tomlu2018-07-11
| | | | | | | uploader cannot upload a particular file. RELNOTES: None PiperOrigin-RevId: 204167372
* bep: introduce BuildEventArtifactUploaderFactoryGravatar buchgr2018-07-09
| | | | | | | | | | | | | | | | There can be multiple BuildEventTransports active at the same time and we need to ensure that each transport gets its own BuildEventArtifactUploader as these transports might have different lifecycles. We do that by introducing another level of indirection via the BuildEventArtifactUploaderFactory. BlazeModules now register a factory object instead of an uploader. In addition, the BuildEventArtifactUploader gets a shutdown() method that allows to free any resources associated with it. PiperOrigin-RevId: 203752092
* Replace instances of Blaze with Bazel.Gravatar Dan Fabulich2018-07-05
| | | | | | | PiperOrigin-RevId: 203300374 Change-Id: Iaa47f870ab2e0cad40a202aad2c7f9430f73c856 PiperOrigin-RevId: 203407392
* Include more information about build events' referenced local files.Gravatar tomlu2018-07-04
| | | | | | | Instead of just a path, events now include information about the type of file (output, source file, stdout/stderr, test logs, etc.). This information can be used by the uploaders to determine a) whether to upload, b) what kind of lease to give the files. RELNOTES: None PiperOrigin-RevId: 203285549
* Remove --allow_undefined_configsGravatar ccalvarin2018-06-28
| | | | | RELNOTES: --[no]allow_undefined_configs no longer exists, passing undefined configs is an error. PiperOrigin-RevId: 202518987
* Move HashFunction out of FileSystem, and turn it into a class, instead of an ↵Gravatar ccalvarin2018-06-21
| | | | | | | | | enum. Now that we aren't using enum names for the hash functions, we also accept the standard names, such as SHA-256. RELNOTES: None. PiperOrigin-RevId: 201624286
* Remove deprecated flag "skylark_incremental_res" from mobile-install command.Gravatar Googler2018-06-21
| | | | | | | RELNOTES[INC]: The mobile-install --skylark_incremental_res flag is no longer available, use the --skylark flag instead. PiperOrigin-RevId: 201574876
* Move the profiler setup earlier in the startup.Gravatar twerth2018-06-20
| | | | | | | | | | This gives us more coverage for command startup, especially wrt. modules. We have some modules that are known to be slow, and this immediately shows any such issues. I've already tracked down two issues with this change, and it would also have shown a recently fixed regression. RELNOTES: None. PiperOrigin-RevId: 201337059
* Remove support for --discard_actions_after_execution.Gravatar tomlu2018-06-18
| | | | | | | The memory savings from this flag are not worth the complexity, and it interferes with action restarting. RELNOTES: Remove support for --discard_actions_after_execution. PiperOrigin-RevId: 201077905
* Refactoring: uses OutputService for ActionFileSystem injectionGravatar shahan2018-06-18
| | | | | | Moves more Action-oriented from skyframe package, which has become very unwieldy, to action package. This is needed to avoid circular dependencies caused when build-base is needed for skyframe. PiperOrigin-RevId: 200996982
* Add a bit more profiler coverageGravatar ulfjack2018-06-15
| | | | | | | | | | | | | | | | | | The intent is that the main thread has ~zero gaps in the profile (though there may still be small gaps due to the time between one try block and an immediately subsequent try block). We need to be careful not to wrap markPhase calls (or methods that call markPhase) in try blocks for the profiler - the Profiler requires that all markPhase calls happen at top level, and throws an exception if not. This should not have any performance impact - all of these are once per build, or at most once per module per build, and we don't expect a very large number of modules (and if we see an increasing number, we need to change the module API to not have to call every single module, but only those that are actually interested in certain events, maybe with an EventBus-based setup). PiperOrigin-RevId: 200712677
* Move the BlazeRuntime.initProfiler call to BlazeCommandDispatcherGravatar ulfjack2018-06-15
| | | | | | | | | | The intend is to move the profiler setup earlier in the startup sequence, so we can get more profiler coverage. Ideally, we'd setup the profiler immediately after getting the client call, but this is not currently possible due to the requirement that initialization and shutdown happen in pairs (otherwise the next command invocation crashes with an exception from the Profiler). PiperOrigin-RevId: 200707029
* Remove the removeHandler/releaseHandler pairsGravatar ulfjack2018-06-15
| | | | | | | | | | | | | | | | | | | | | | | They weren't actually doing anything. With default flags, createHandler never returns a FancyTerminalEventHandler, because that always gets wrapped in a RateLimitingEventHandler. Even so, under normal circumstances, the fancy terminal event handler always leaves the terminal in a valid state at the end of each printed line. We can only end up in a bad state if the blaze server is killed, but even then, this code wasn't doing anything because it didn't get to run. The only way to guarantee that the terminal is always in a valid state is to make sure that every stdout/stderr buffer we send to the client ends with a reset code sequence. The experimental ui handler is better at sending only complete buffers (we were previously using an auto-line flushing output stream, which didn't allow the event handler to reason about how much stuff got actually printed in a single write call). Anyway, I think we're in a better place now, and this code wasn't run, and if we want to fix it properly (if the switch to experimental_ui isn't sufficient), then we need to fix it elsewhere. PiperOrigin-RevId: 200699609
* Refactor BlazeRuntime.initProfilerGravatar ulfjack2018-06-15
| | | | | | | - reduce the parameters to the minimum required - change to void, move additional setup into the method PiperOrigin-RevId: 200692559
* Add a mechanism for build event protocol events to upload filesGravatar ulfjack2018-06-15
| | | | | | | | This should be a no-op, mostly replacing PathConverter with BuildEventArtifactUploader, since none of the implementations perform any upload yet. PiperOrigin-RevId: 200685325
* Remove fixed point config expansion.Gravatar ccalvarin2018-06-14
| | | | | | | Deprecates the flag, though it continues to exist as a no-op so that users get a warning before errors. RELNOTES: --noexpand_configs_in_place is deprecated. PiperOrigin-RevId: 200572053
* Remove unnecessary flags from RunCommand command annotationGravatar ulfjack2018-06-14
| | | | | | | These are no longer necessary since we don't pipe the subprocess stdout/stderr through Bazel anymore. PiperOrigin-RevId: 200534552
* Fix Profiler setup to actually enable the Json formatGravatar ulfjack2018-06-14
| | | | PiperOrigin-RevId: 200518973
* Simplify CommandEnvironment constructor, make fields finalGravatar ulfjack2018-06-14
| | | | PiperOrigin-RevId: 200516863
* Add --experimental_supports_info_crosstool_configuration to support removal ↵Gravatar hlopko2018-06-13
| | | | | | | of bazel info crosstool-configuration RELNOTES: None. PiperOrigin-RevId: 200437121
* Move BuildView.Options to a new top-level classGravatar ulfjack2018-06-13
| | | | | | | This is in preparation for dismantling BuildView and merging the relevant parts into AnalysisPhaseRunner. PiperOrigin-RevId: 200391088
* Remove BlazeModule.checkEnvironmentGravatar ulfjack2018-06-13
| | | | | | Do the work in beforeCommand instead. PiperOrigin-RevId: 200391086
* Add --print_workspace_in_output_paths_if_needed flag.Gravatar mstaib2018-06-12
| | | | | | | | | | | | | | | | When this flag is turned on, and the user's working directory is beneath the workspace (but is not the workspace itself), the workspace's absolute path is printed as a prefix to the convenience symlink. With this flag off, the displayed convenience symlink path is always relative to the user's workspace, even if that isn't the current working directory. (This is the current behavior.) The new (flag-on) behavior will become default soon, and then this flag will be removed. RELNOTES: None. PiperOrigin-RevId: 200278355
* Add support for the Json trace file formatGravatar ulfjack2018-06-12
| | | | | | | | | | | | | | | | | Add a --experimental_generate_json_trace_profile option that puts a file into the output base (or uses --profile if set). There are still a lot of problems with this. - unexplained holes - way too many threads - nonsensical event titles - too many detail events, too little overview - it may also cause unnecessary load - it silently overwrites the existing file on subsequent invocations The format is documented here: goo.gl/oMZPLh PiperOrigin-RevId: 200259431
* Refactor profilerGravatar ulfjack2018-06-12
| | | | | | | | | | | | | | | | | | | | | | | | | | - move the save method to an inner class - don't use a timer, use a blocking queue instead - add a format enum (in anticipation of adding a json output format) - update the test to use an in memory buffer, and avoid FoundationTestCase Compared to the original https://github.com/bazelbuild/bazel/commit/15b8c259db111012b4642287172cb4d1d82151f3, it contains these changes: - Make it so we don't create a queue if we are not going to write any data! The queue is now owned by the writer, and if there is no writer, there is no queue. This was causing a memory regression because slowest task profiling is enabled by default, in which case the profiler is started with no output file. In that case, there's no thread that is emptying the queue, but the queue was still created by default. - add additional tests for slowest task and histogram handling; these also provide coverage for the case where the profiler is started without an output stream - move all the writer thread handling into the inner class - make writer access thread-safe - add a bunch of documentation PiperOrigin-RevId: 200212978
* Split TargetPatternEvaluator into two interfacesGravatar ulfjack2018-06-12
| | | | | | | | | | | | | | | | | - the TargetPatternPreloader is still used for query in all its forms - the remaining TargetPatternEvaluator part is no longer used except in tests - also make both implementations stateless and pass the offset to the methods instead; note that they both modify the underlying skyframe graph, so there are side effects to the calls even if there's no direct state anymore The intent is to migrate the relevant tests to LoadingPhaseRunnerTest (which could also now be renamed since it's not doing a loading phase), and then delete the TargetPatternEvaluator interface. This depends on the previous commit that removed the last direct use of TPE from an internal command. PiperOrigin-RevId: 200198067
* Automated rollback of commit 15b8c259db111012b4642287172cb4d1d82151f3.Gravatar jmmv2018-06-11
| | | | | | | | *** Reason for rollback *** Breaks internal performance tests. PiperOrigin-RevId: 200103033
generated by cgit v1.2.3 (git 2.39.1) at 2024-06-26 01:25:05 +0000