aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/worker
Commit message (Collapse)AuthorAge
* Modify --subcommands to allow pretty printing the arguments of a subcommandGravatar ahumesky2018-07-26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | as a list, rather than as a single line (i.e., newline delimited rather than space delimited). Before: SUBCOMMAND: # //src/main/java/com/google/devtools/build/lib:string_util [action 'Building src/main/java/com/google/devtools/build/lib/libstring_util.jar (2 source files) [for host]'] (cd /tmp/devbazel_output_base/execroot/io_bazel && \ exec env - \ LC_CTYPE=en_US.UTF-8 \ external/embedded_jdk/bin/java -XX:+UseParallelOldGC -XX:-CompactStrings '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--patch-module=java.compiler=external/bazel_tools/third_party/java/jdk/langtools/java_compiler.jar' '--patch-module=jdk.compiler=external/bazel_tools/third_party/java/jdk/langtools/jdk_compiler.jar' '--add-opens=java.base/java.nio=ALL-UNNAMED' -jar external/bazel_tools/tools/jdk/JavaBuilder_deploy.jar @bazel-out/host/bin/src/main/java/com/google/devtools/build/lib/libstring_util.jar-2.params) After: SUBCOMMAND: # //src/main/java/com/google/devtools/build/lib:string_util [action 'Building src/main/java/com/google/devtools/build/lib/libstring_util.jar (2 source files) [for host]'] (cd /tmp/devbazel_output_base/execroot/io_bazel && \ exec env - \ LC_CTYPE=en_US.UTF-8 \ external/embedded_jdk/bin/java \ -XX:+UseParallelOldGC \ -XX:-CompactStrings \ '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' \ '--add-opens=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' \ '--patch-module=java.compiler=external/bazel_tools/third_party/java/jdk/langtools/java_compiler.jar' \ '--patch-module=jdk.compiler=external/bazel_tools/third_party/java/jdk/langtools/jdk_compiler.jar' \ '--add-opens=java.base/java.nio=ALL-UNNAMED' \ -jar \ external/bazel_tools/tools/jdk/JavaBuilder_deploy.jar \ @bazel-out/host/bin/src/main/java/com/google/devtools/build/lib/libstring_util.jar-2.params) RELNOTES: --subcommands can now take a "pretty_print" value ("--subcommands=pretty_print") to print the arguments of subcommands as a list for easier reading. PiperOrigin-RevId: 206213009
* Prefetch inputs in bazel workers.Gravatar Googler2018-06-26
| | | | PiperOrigin-RevId: 202092962
* Log "Worker configuration has changed, restarting worker pool..." regardless ↵Gravatar Googler2018-06-22
| | | | | | | | of the --worker_verbose setting. This message is important because it can catch cases where people are unknowingly running consecutive builds with different options, leading to much slower builds. In this regard it is a lot like "Build options have changed, discarding analysis cache". PiperOrigin-RevId: 201648554
* Inline some ActionContextProvider classes into their modulesGravatar ulfjack2018-06-11
| | | | | | | | | | | | | | | This also gets rid of some boilerplate. The ExecutionTool.addActionContext method has been around for a while, but is underused. There are still a few ActionContextProvider implementations left, which are implementing other functionality besides adding action contexts. As a side effect, this change reduces null build time with a hot server on linux by about a quarter. We were running the linux sandbox twice on every build, which takes about 70ms each (on my machine), with the total null build time around 300ms. PiperOrigin-RevId: 200045145
* Remove ActionContextConsumerGravatar ulfjack2018-06-11
| | | | | | | | Instead, add some simple APIs to ExecutorBuilder and inline all the previous subclasses into their corresponding modules. This removes a bunch of boilerplate. PiperOrigin-RevId: 200017162
* Deletes ActionInputFileCache.Gravatar shahan2018-06-07
| | | | PiperOrigin-RevId: 199732415
* Refactoring: makes the code simpler by deleting Metadata and InputArtifactData.Gravatar shahan2018-06-06
| | | | | | (minor) ActionFS now implements MetadataProvider.getInput PiperOrigin-RevId: 199575194
* Allow --worker_max_instances to take MnemonicName=value to specify max for ↵Gravatar Googler2018-05-03
| | | | | | | each named worker. RELNOTES: Allow --worker_max_instances to take MnemonicName=value to specify max for each worker. PiperOrigin-RevId: 195244295
* Clean up code that directly imports nested classes like Builder, Entry, etc.Gravatar jcater2018-05-02
| | | | PiperOrigin-RevId: 195100670
* Add param file support to sandboxed spawn runners.Gravatar tomlu2018-05-01
| | | | | | | Spawns can contain param files in the form of virtual action inputs. Sandboxed runners must write these prior to execution of the action. RELNOTES: None PiperOrigin-RevId: 194951208
* Remove the on-disk caching of Xcode locations.Gravatar jmmv2018-04-28
| | | | | | | | | | | | | | | | | | | | | | | There is no need for the cache to be on disk. Originally, there was a desire to share this cache with other tools... but this never happened. And, actually, because Bazel is in control of what it runs, it can just inject the "cached" values into those tools via flags. Instead, just store the cache in-memory. This avoids having to open and read the cache on every single action that is locally executed on a Mac. Results when building a large iOS app from a clean slate show up to a 1% wall time improvement on my Mac Pro 2013 and a reduction in the variance of the measurements. This change also gets rid of the OS check from the action execution's critical path. There is not much use in checking this: if we instantiate this by mistake, the actual calls will fail. But sometimes we want to actually run this code on non-macOS systems (e.g. for unit-testing with mocked tools), so we should allow that. And this change also ensures that XcodeLocalEnvProviderTest builds and runs... RELNOTES: None. PiperOrigin-RevId: 194681802
* Blaze: add --high_priority_workers flag.Gravatar Googler2018-04-20
| | | | | | | | | | | When two or more instances of high priority workers are running, no other worker execution may start. Tweak WorkerSpawnRunner. Only reserve resources for a worker after it has been obtained from the pool. This allows us to block waiting for high priority workers to have finished before resources are reserved. RELNOTES: Add --high_priority_workers flag. PiperOrigin-RevId: 193672343
* Rename SpawnExecutionPolicy -> SpawnExecutionContext.Gravatar tomlu2018-04-19
| | | | | | | | This class will be used to tie a Spawn to a SpawnRunner, and isn't really a policy object. It will carry state such as the expanded inputs and expanded command line. Currently a context can be passed between different SpawnRunners. This will be addressed independently, so a context is tied to a particular spawn runner. PiperOrigin-RevId: 193501918
* Roll forward of ↵Gravatar philwo2018-04-16
| | | | | | | | | | | | | | | | | | | | https://github.com/bazelbuild/bazel/commit/656a0bab1e025ff3c27d595284a4bf1c5a8d8028 with test (unknown commit) and fix. Big round of sandbox fixes / performance improvements. - The number of stat() syscalls in the SymlinkedSandboxedSpawn was way too high. Do less, feel better. - When using --experimental_sandbox_base, ensure that symlinks in the path are resolved. Before this, you had to check whether on your system /dev/shm is a symlink to /run/shm and then use that instead. Now it no longer matters, as symlinks are resolved. - Remove an unnecessary directory creation from each sandboxed invocation. Turns out that the "tmpdir" that we created was no longer used after some changes to Bazel's TMPDIR handling. - Use simpler sandbox paths, by using the unique ID for each Spawn provided by SpawnExecutionPolicy instead of a randomly generated temp folder name. This also saves a round-trip from our VFS to NIO and back. Clean up the sandbox base before each build to ensure that the unique IDs are actually unique. ;) - Use Java 8's Process#isAlive to check whether a process is alive instead of trying to get the exitcode and catching an exception. Closes #4913. PiperOrigin-RevId: 193031017
* Remove WorkerTestStrategy and with it, ↵Gravatar lberki2018-04-12
| | | | | | | | | BuildConfiguration.Fragment#compatibleWithStrategy(). The experiment did not work out. RELNOTES: None. PiperOrigin-RevId: 192567832
* Internal changeGravatar dannark2018-04-04
| | | | PiperOrigin-RevId: 191642942
* Remove categories from Bazel options.Gravatar ccalvarin2018-03-28
| | | | | | | | | These have all had a chance to be categorized with the OptionDocumentationCategory enum, and the help output already uses the enum-grouped format. The "incompatible changes" category has meaning for --all_incompatible_changes and will be removed separately. RELNOTES: None. PiperOrigin-RevId: 190773778
* Simplified ActionContextConsumer by having it operate on a new class which ↵Gravatar Googler2018-03-26
| | | | | | holds a variety of strategy/context maps. PiperOrigin-RevId: 190491357
* Big round of sandbox fixes / performance improvements.Gravatar Philipp Wollermann2018-03-26
| | | | | | | | | | | | | | | | - The number of stat() syscalls in the SymlinkedSandboxedSpawn was way too high. Do less, feel better. - When using --experimental_sandbox_base, ensure that symlinks in the path are resolved. Before this, you had to check whether on your system /dev/shm is a symlink to /run/shm and then use that instead. Now it no longer matters, as symlinks are resolved. - Remove an unnecessary directory creation from each sandboxed invocation. Turns out that the "tmpdir" that we created was no longer used after some changes to Bazel's TMPDIR handling. - Use simpler sandbox paths, by using the unique ID for each Spawn provided by SpawnExecutionPolicy instead of a randomly generated temp folder name. This also saves a round-trip from our VFS to NIO and back. Clean up the sandbox base before each build to ensure that the unique IDs are actually unique. ;) - Use Java 8's Process#isAlive to check whether a process is alive instead of trying to get the exitcode and catching an exception. Closes #4913. PiperOrigin-RevId: 190472170
* Improve error message when a persistent worker sends an unparseable response.Gravatar philwo2018-03-22
| | | | | | Fixes #4897. PiperOrigin-RevId: 190045930
* Begins cleanup to allow ActionFS to be injected into all action executions.Gravatar shahan2018-03-15
| | | | PiperOrigin-RevId: 189244665
* Replace instances of XCode with Xcode which is the proper casing for the name.Gravatar Sergio Campama2018-03-06
| | | | | | Closes #4640. PiperOrigin-RevId: 188022228
* Adding a property name to the SpawnRunner. Most runners already had it, I ↵Gravatar olaola2018-02-22
| | | | | | | | | | just add it to the interface, and include it in the SpawnResult. This will be used to categorize/aggregate spawns executed by various runners. Also, minor refinement to the cacheHit property of the SpawnResult with remote execution. RELNOTES: None TESTED=presubmit, next cl PiperOrigin-RevId: 186637978
* Inject the product name in the constructor where neededGravatar ulfjack2018-02-20
| | | | | | ...instead of passing it everywhere so we can rewrite the local env. PiperOrigin-RevId: 186316859
* Remove hard-wired basenames for the test setup and the coverage collection ↵Gravatar lberki2018-02-09
| | | | | | | | | script. Accomplished by creating an explicit attribute for both of them on test actions. RELNOTES: None. PiperOrigin-RevId: 185132460
* actions,temp: respect TMPDIR envvarGravatar Laszlo Csomor2018-01-15
| | | | | | | Fixes https://github.com/bazelbuild/bazel/issues/4376 Change-Id: Id78bb0930044626304e54f07735db4d4b2c84720 PiperOrigin-RevId: 181959528
* Plumb exec root through to all spawn runners.Gravatar tomlu2018-01-11
| | | | | | They need this to parse input manifests. Previously we would grab the exec root from the Root, but wish to unsupport this. PiperOrigin-RevId: 181669143
* Extract LocalEnvProvider members to classes.Gravatar Laszlo Csomor2018-01-04
| | | | | | | | | | | | | | | | | | Create a PosixLocalEnvProvider and WindowsLocalEnvProvider class, with singleton instances for now. This refactoring should not change functionality, it's just a requirement for an upcoming change. That upcoming change is for these classes to respect the client environment's TMPDIR or TMP/TEMP envvars. See https://github.com/bazelbuild/bazel/issues/4376 Change-Id: I032bb6f18adf8af9e43e6bc543c09c58adae3863 PiperOrigin-RevId: 180799936
* Move FileStateValue.Type to Metadata; add Metadata.getType()Gravatar ulfjack2017-12-21
| | | | | | | | This is in preparation for merging FileArtifactValue and FileStateValue. Progress on #3360. PiperOrigin-RevId: 179832948
* Add a CommandLineBuilder for the process-wrapper embedded tool, and use it ↵Gravatar ruperts2017-11-30
| | | | | | | | | everywhere instead of duplicating process-wrapper --shell_arguments in Blaze. To avoid a cyclic dependency, I broke up exec/local:local into exec/local:local and exec/local:options. RELNOTES: None. PiperOrigin-RevId: 177419268
* Simplify SpawnRunner interfaceGravatar ulfjack2017-11-28
| | | | | | | | | | | | | | It turns out that the SUCCESS status is often misunderstood to mean "zero exit", even though this is clearly documented. I've decided to add another status for non-zero exit, and use success only for zero exit to avoid this pitfall. Also, many of the status codes are set, but never used. I decided to reduce the number of status codes to only those that are actually relevant, which simplifies further processing. Instead, we should add a string message for the error case when we need one - we're not using it right now, so I decided not to add that yet. PiperOrigin-RevId: 177129441
* Return SpawnResults using a List instead of a Set.Gravatar ruperts2017-11-21
| | | | | | | | | Currently we don't care about the list order of SpawnResults. However, we may care about the list order later. Also, if the equals() method for SpawnResults is ever changed then it may be problematic to return SpawnResults in a Set. Aside: ActionResults use SpawnResults to calculate cumulative execution times for Actions, and may provide other metrics in future. RELNOTES: None. PiperOrigin-RevId: 176579460
* Replace all usages of Blaze's Preconditions class with guava.Gravatar tomlu2017-11-09
| | | | | | | | Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own. This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually. PiperOrigin-RevId: 175033526
* Use Durations to store wall, user and system execution time in SpawnResults, ↵Gravatar ruperts2017-11-06
| | | | | | | and make cumulative execution times available in ActionResults. RELNOTES: None PiperOrigin-RevId: 174553272
* Windows, jni: Don't close stdout/stderr in nativeWaitFor functionGravatar Yun Peng2017-10-24
| | | | | | | | | | | | | | | | | | | | | | | | These two close operations were added to work around #1708, but caused #2675. We found the root cause of the hanging problem in #1708 is a race condition when creating Windows processes: When Bazel trys to create two processes, one for a local command execution, one for starting the worker process. The worker process might accidentally inherits handles opened when creating the local command process, and it holds those handles as long as it lives. Therefore, ReadFile function hangs when handles for the write end of stdout/stderr pipes are released by the worker. The solution is to make Bazel native createProcess JNI function explicitly inheirts handles as needed, and use this function to start worker process. Related: http://support.microsoft.com/kb/315939 Fixed https://github.com/bazelbuild/bazel/issues/2675 Change-Id: I1c9b1ac3c9383ed2fd28ea92f528f19649693275 PiperOrigin-RevId: 173244832
* Actions now have a temp envvar.Gravatar László Csomor2017-10-16
| | | | | | | | | | | | | | | | | | | | | Every build and test action that creates a Spawn will now have platform-specific environment variables for temp directories: - on Windows: TMP and TEMP - on Linux/Darwin: TMPDIR This is particularly important on Windows where e.g. Java programs cannot create temp directories unless there's a valid TMP or TEMP environment variable set. Fixes: - https://github.com/bazelbuild/bazel/issues/1590 - https://github.com/bazelbuild/bazel/issues/2349 - https://github.com/bazelbuild/bazel/issues/2870 Change-Id: Ib758307daf6b3a51b0f71ae5e65e5bb564dad643 PiperOrigin-RevId: 172326371
* More SpawnResult-related plumbing changes to Actions, Strategies, ↵Gravatar ruperts2017-10-07
| | | | | | | | | ActionContexts, etc., so that SpawnResult metadata is returned upwards. Note that the TODOs mostly refer to changes that will appear in a subsequent CL (a CL to return SpawnResults, contained in ActionResults, from Actions/AbstractActions). I split off the remaining SpawnResult-related changes into this CL and kept the ActionResult-related changes separate. RELNOTES: None. PiperOrigin-RevId: 171355611
* Move SpawnResult from build.lib.exec into build.lib.actions so that e.g. ↵Gravatar ruperts2017-09-22
| | | | | | | build.lib.actions.SpawnActionContext can import SpawnResult without creating a cyclic dependency. RELNOTES: None. PiperOrigin-RevId: 169642267
* More BUILD file refactorings.Gravatar philwo2017-09-06
| | | | | | | | | Split collect, concurrent, vfs, windows into package-level BUILD files. Move clock classes out of "util", into their own Java package. Move CompactHashSet into its own Java package to break a dependency cycle. Give nestedset and inmemoryfs their own package-level BUILD files. PiperOrigin-RevId: 167702127
* Fix NPE in WorkerFilesHash.Gravatar philwo2017-08-29
| | | | | | | | We must check that a given artifact is a file, before calling getDigest, according to the interface contract specified in actions/Metadata.java. Add a regression test to bazel_worker_test.sh, too. It's enough to simply add a directory entry to the worker's data to trigger the original bug. PiperOrigin-RevId: 166856512
* Add the workspace name to the sandbox path for sandboxed workers.Gravatar philwo2017-08-18
| | | | PiperOrigin-RevId: 165571541
* workers: Don't skip empty lines when expanding flagfiles.Gravatar philwo2017-08-17
| | | | | | Fixes #3329. PiperOrigin-RevId: 165443367
* When a worker can no longer be used, because one or more of its files on ↵Gravatar philwo2017-08-17
| | | | | | | disk have changed since it was launched, print *which* files have changed. RELNOTES: Improved logging when workers have to be restarted due to its files having changed. PiperOrigin-RevId: 165419664
* Refactor persistent workers to use SpawnRunner.Gravatar Benjamin Peterson2017-08-11
| | | | | | | | | | | | | | | | | | | | | | | | | | Change the persistent worker spawn strategy to extend AbstractSpawnStrategy and put the actual logic into WorkerSpawnRunner. WorkerTestStrategy is unaffected. I had to extend SpawnPolicy with a speculating() method. Persistent workers need to know if speculation is happening in order to require sandboxing. Additionally, I added java_test rules for the local runner tests and worker tests. See https://github.com/bazelbuild/bazel/issues/3481. NOTE: ulfjack@ made some changes to this change before merging: - changed Reporter to EventHandler; added TODO about its usage - reverted non-semantic indentation change in AbstractSpawnStrategy - reverted a non-semantic indentation change in WorkerSpawnRunner - updated some internal classes to match - removed catch IOException in WorkerSpawnRunner in some cases, removed verboseFailures flag from WorkerSpawnRunner, updated callers - disable some tests on Windows; we were previously not running them, now that we do, they fail :-( Change-Id: I207b3938f0dc84d374ab052d5030020886451d47 PiperOrigin-RevId: 164965398
* Move core test classes to lib.analysis.testGravatar ulfjack2017-08-11
| | | | | | | | These are depended upon by analysis code, so need to live in the same library as lib.analysis. Moving them here makes it possible to split the build-base library into separate libraries for analysis, execution, and rules. PiperOrigin-RevId: 164847161
* Make the @Option annotation depend on the java version of the tagging enums.Gravatar ccalvarin2017-07-18
| | | | | | | The option filters proto dependency can be removed from the OptionsParser. This is in response to option parser users that want to avoid the bazel-internal proto file in their dependencies. RELNOTES: None. PiperOrigin-RevId: 162249778
* Simplify exception handling in spawn strategiesGravatar ulfjack2017-07-17
| | | | | | | | | | | | | | | | | | | | | | | | | The main change here is to only catch SpawnExecException in StandaloneTestStrategy, so all other exceptions simplify propagate up. As a result, Bazel no longer retries tests that fail with an exception, we only retry tests that actually ran, had a spawn result, and resulted in a UserExecException. That is probably what we want. Also do some cleanup: - Remove ExecException.timedOut; nobody was calling it (but there's still SpawnExecException.timedOut) - Remove SpawnActionContext.shouldPropagateExecException; all exceptions (except SpawnExecException) are now propagated by default - Remote the SandboxOptions from the SandboxStrategies; all sandboxing options are now handled by the underlying SpawnRunner implementations I'll send a followup CL to remove the UserExecException and EnvironmentalExecException types; the types don't do anything special, and there are no catch blocks in production code that catch one of these more specific types. This should fix #3322 by removing a bunch of special handling. PiperOrigin-RevId: 161960919
* Rewrite all the sandbox strategy implementationsGravatar ulfjack2017-07-12
| | | | | | | | | | | | | | - Make use of existing abstractions like SpawnRunner and SpawnExecutionPolicy. - Instead of having the *Strategy create a *Runner, and then call back into SandboxStrategy, create a single SandboxContainer which contains the full command line, environment, and everything needed to create and delete the sandbox directory. - Do all the work in SandboxStrategy, including creation and deletion of the sandbox directory. - Use SpawnResult instead of throwing, catching, and rethrowing. - Simplify the control flow a bit. PiperOrigin-RevId: 161644979
* Cache flaky tests (remove the cachable flag from test result data)Gravatar ulfjack2017-07-07
| | | | | | | | | | | | | | | | | | | The cachable flag was only set to false for flaky tests. Before this CL, we did not cache flaky tests on subsequent runs, instead rerunning them, even though this is both very expensive and inconsistent with our normal handling, which is to cache passes but not errors. If cache_test_results is enabled, we now also cached timed out test results. If cache_test_results=auto (the default), we now also cache flaky tests. We still do not cache failed tests; the TestRunnerAction checks if the previous test result was marked as passed (which also applies to flaky tests, but not to failed or timed-out tests). Also add some unit tests. PiperOrigin-RevId: 161187950
* Simplify the ActionInputFileCacheGravatar ulfjack2017-07-06
| | | | | | | | | Add a single getMetadata method (matching MetadataHandler), and rewrite everything in those terms. This is in preparation for merging ActionInputFileCache and MetadataHandler. PiperOrigin-RevId: 161053535
generated by cgit v1.2.3 (git 2.39.1) at 2024-06-14 01:17:28 +0000