aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/exec/AbstractSpawnStrategy.java
Commit message (Collapse)AuthorAge
* Modify --subcommands to allow pretty printing the arguments of a subcommandGravatar ahumesky2018-07-26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | as a list, rather than as a single line (i.e., newline delimited rather than space delimited). Before: SUBCOMMAND: # //src/main/java/com/google/devtools/build/lib:string_util [action 'Building src/main/java/com/google/devtools/build/lib/libstring_util.jar (2 source files) [for host]'] (cd /tmp/devbazel_output_base/execroot/io_bazel && \ exec env - \ LC_CTYPE=en_US.UTF-8 \ external/embedded_jdk/bin/java -XX:+UseParallelOldGC -XX:-CompactStrings '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--patch-module=java.compiler=external/bazel_tools/third_party/java/jdk/langtools/java_compiler.jar' '--patch-module=jdk.compiler=external/bazel_tools/third_party/java/jdk/langtools/jdk_compiler.jar' '--add-opens=java.base/java.nio=ALL-UNNAMED' -jar external/bazel_tools/tools/jdk/JavaBuilder_deploy.jar @bazel-out/host/bin/src/main/java/com/google/devtools/build/lib/libstring_util.jar-2.params) After: SUBCOMMAND: # //src/main/java/com/google/devtools/build/lib:string_util [action 'Building src/main/java/com/google/devtools/build/lib/libstring_util.jar (2 source files) [for host]'] (cd /tmp/devbazel_output_base/execroot/io_bazel && \ exec env - \ LC_CTYPE=en_US.UTF-8 \ external/embedded_jdk/bin/java \ -XX:+UseParallelOldGC \ -XX:-CompactStrings \ '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' \ '--add-opens=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' \ '--patch-module=java.compiler=external/bazel_tools/third_party/java/jdk/langtools/java_compiler.jar' \ '--patch-module=jdk.compiler=external/bazel_tools/third_party/java/jdk/langtools/jdk_compiler.jar' \ '--add-opens=java.base/java.nio=ALL-UNNAMED' \ -jar \ external/bazel_tools/tools/jdk/JavaBuilder_deploy.jar \ @bazel-out/host/bin/src/main/java/com/google/devtools/build/lib/libstring_util.jar-2.params) RELNOTES: --subcommands can now take a "pretty_print" value ("--subcommands=pretty_print") to print the arguments of subcommands as a list for easier reading. PiperOrigin-RevId: 206213009
* Weave the artifact path resolver through the Spawn execution context.Gravatar felly2018-07-25
| | | | | RELNOTES: None PiperOrigin-RevId: 206063489
* Deletes ActionInputFileCache.Gravatar shahan2018-06-07
| | | | PiperOrigin-RevId: 199732415
* Allow banning symlink action outputs from being uploaded to a remote cache.Gravatar Benjamin Peterson2018-05-03
| | | | | | | | | | | | | | | | | | | This is mostly a roll-forward of 4465dae23de989f1452e93d0a88ac2a289103dd9, which was reverted by fa36d2f48965b127e8fd397348d16e991135bfb6. The main difference is that the new behavior is now gated behind the --noremote_allow_symlink_upload flag. https://docs.google.com/document/d/1gnOYszitgrLVet3sQk-TKGqIcpkkDsc6aw-izoo-d64 is a design proposal to support symlinks in the remote cache, which would render this change moot. I'd like to be able to prevent incorrect cache behavior until that change is implemented, though. This fixes https://github.com/bazelbuild/bazel/issues/4840 (again). Closes #5122. Change-Id: I2136cfe82c2e1a8a9f5856e12a37d42cabd0e299 PiperOrigin-RevId: 195261827
* Report what RemoteSpawnCache is doing.Gravatar Benjamin Peterson2018-05-03
| | | | | | | | | | | | | | | | Post ProgressStatus.CHECKING_CACHE if RemoteSpawnCache is checking the cache. The UI sees CHECKING_CACHE exactly the same as EXECUTING because no UIs currently have any special behavior for actions in cache-lookup state. This is still a UX improvement with --experimental_spawn_cache because EXECUTING is generally more correct than the old action state, which varies from harmless but unhelpful (no known state) to just wrong (C++ compile actions claimed they were doing include scanning during cache lookups). Closes #5130. Change-Id: I77421c3667c180875216f937fe0713f0e9415a7a PiperOrigin-RevId: 195233123
* Use the in-memory metadata in blaze as the source of truth for Fileset mappingsGravatar kush2018-05-02
| | | | | | | instead of the manifest files. RELNOTES: None PiperOrigin-RevId: 195149880
* Logging the spawn graph on demand. First step on #4891.Gravatar olaola2018-04-19
| | | | | | | | This is a very heavy and slow option by design. It will be enabled only when a user wants to debug their build, most frequently to compare the step-by-step results of two builds. TESTED: manually on various rules, including directories. RELNOTES: None PiperOrigin-RevId: 193539034
* Rename SpawnExecutionPolicy -> SpawnExecutionContext.Gravatar tomlu2018-04-19
| | | | | | | | This class will be used to tie a Spawn to a SpawnRunner, and isn't really a policy object. It will carry state such as the expanded inputs and expanded command line. Currently a context can be passed between different SpawnRunners. This will be addressed independently, so a context is tied to a particular spawn runner. PiperOrigin-RevId: 193501918
* Automated rollback of commit 4465dae23de989f1452e93d0a88ac2a289103dd9.Gravatar buchgr2018-04-18
| | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** The no-cache tag is not respected (see b/77857812) and thus this breaks remote caching for all projects with symlink outputs. *** Original change description *** Only allow regular files and directories spawn outputs to be uploaded to a remote cache. The remote cache protocol only knows about regular files and directories. Currently, during action output upload, symlinks are resolved into regular files. This means cached "executions" of an action may have different output file types than the original execution, which can be a footgun. This CL bans symlinks from cachable spawn outputs and fixes http... *** PiperOrigin-RevId: 193338629
* Only allow regular files and directories spawn outputs to be uploaded to a ↵Gravatar Benjamin Peterson2018-03-27
| | | | | | | | | | | | | | | | | | | | remote cache. The remote cache protocol only knows about regular files and directories. Currently, during action output upload, symlinks are resolved into regular files. This means cached "executions" of an action may have different output file types than the original execution, which can be a footgun. This CL bans symlinks from cachable spawn outputs and fixes https://github.com/bazelbuild/bazel/issues/4840. The interface of SpawnCache.CacheHandle.store is refactored: 1. The outputs parameter is removed, since that can be retrieved from the underlying Spawn. 2. It can now throw ExecException in order to fail actions. Closes #4902. Change-Id: I0d1d94d48779b970bb5d0840c66a14c189ab0091 PiperOrigin-RevId: 190608852
* AbstractSpawnStrategy should not crash when executing actions with a null owner.Gravatar tomlu2018-02-01
| | | | | RELNOTES: None PiperOrigin-RevId: 184127422
* Plumb exec root through to all spawn runners.Gravatar tomlu2018-01-11
| | | | | | They need this to parse input manifests. Previously we would grab the exec root from the Root, but wish to unsupport this. PiperOrigin-RevId: 181669143
* Use EmptyActionInput instead of null in SpawnInputExpanderGravatar ulfjack2018-01-08
| | | | | | | | This simplifies some spawn runners, which no longer have to specially handle null; unfortunately, the sandbox runners do not support VirtualActionInput, so they still have to special-case it. PiperOrigin-RevId: 181175408
* Fix: uploading artifacts of failed actions to remote cache stopped working.Gravatar olaola2017-12-11
| | | | | | | | | | | | | | | | To reproduce: run a failing test with --experimental_remote_spawn_cache or with --spawn_strategy=remote and no executor. Expected: test log is uploaded. Desired behavior: - regardless of whether a spawn is cacheable or not, its artifacts should be uploaded to the remote cache. - the spawn result should only be set if the spawn is cacheable *and* the action succeeded. - when executing remotely, the do_not_cache field should be set for non-cacheable spawns, and the remote execution engine should respect it. This CL contains multiple fixes to ensure the above behaviors, and adds a few tests, both end to end and unit tests. Important behavior change: it is no longer assumed that non-cacheable spawns should use a NO_CACHE SpawnCache! The appropriate test case was removed. Instead, an assumption was added that all implementations of SpawnCache should respect the Spawns.mayBeCached(spawn) property. Currently, only NO_CACHE and RemoteSpawnCache exist, and they (now) support it. TESTED=remote build execution backend. WANT_LGTM: philwo,buchgr RELNOTES: None PiperOrigin-RevId: 178617937
* Simplify SpawnRunner interfaceGravatar ulfjack2017-11-28
| | | | | | | | | | | | | | It turns out that the SUCCESS status is often misunderstood to mean "zero exit", even though this is clearly documented. I've decided to add another status for non-zero exit, and use success only for zero exit to avoid this pitfall. Also, many of the status codes are set, but never used. I decided to reduce the number of status codes to only those that are actually relevant, which simplifies further processing. Instead, we should add a string message for the error case when we need one - we're not using it right now, so I decided not to add that yet. PiperOrigin-RevId: 177129441
* Return SpawnResults using a List instead of a Set.Gravatar ruperts2017-11-21
| | | | | | | | | Currently we don't care about the list order of SpawnResults. However, we may care about the list order later. Also, if the equals() method for SpawnResults is ever changed then it may be problematic to return SpawnResults in a Set. Aside: ActionResults use SpawnResults to calculate cumulative execution times for Actions, and may provide other metrics in future. RELNOTES: None. PiperOrigin-RevId: 176579460
* Simplify the SpawnExecException constructorGravatar ulfjack2017-10-16
| | | | | | | Whether or not there was a catastrophic error is stored in the SpawnResult, so we can just use that instead of passing in an additional boolean. PiperOrigin-RevId: 172083752
* Ensure that SpawnResults make their way back to the SpawnActions that caused ↵Gravatar ruperts2017-09-27
| | | | | | | them to be created. RELNOTES: None. PiperOrigin-RevId: 170058295
* Move SpawnResult from build.lib.exec into build.lib.actions so that e.g. ↵Gravatar ruperts2017-09-22
| | | | | | | build.lib.actions.SpawnActionContext can import SpawnResult without creating a cyclic dependency. RELNOTES: None. PiperOrigin-RevId: 169642267
* Introduce a new SpawnCache API; add a RemoteSpawnCache implementationGravatar ulfjack2017-08-14
| | | | | | | | | | AbstractSpawnRunner now uses a SpawnCache if one is registered, this allows adding caching to any spawn runner without having to be aware of the implementations. I will delete the old CachedLocalSpawnRunner in a follow-up CL. PiperOrigin-RevId: 165024382
* Refactor persistent workers to use SpawnRunner.Gravatar Benjamin Peterson2017-08-11
| | | | | | | | | | | | | | | | | | | | | | | | | | Change the persistent worker spawn strategy to extend AbstractSpawnStrategy and put the actual logic into WorkerSpawnRunner. WorkerTestStrategy is unaffected. I had to extend SpawnPolicy with a speculating() method. Persistent workers need to know if speculation is happening in order to require sandboxing. Additionally, I added java_test rules for the local runner tests and worker tests. See https://github.com/bazelbuild/bazel/issues/3481. NOTE: ulfjack@ made some changes to this change before merging: - changed Reporter to EventHandler; added TODO about its usage - reverted non-semantic indentation change in AbstractSpawnStrategy - reverted a non-semantic indentation change in WorkerSpawnRunner - updated some internal classes to match - removed catch IOException in WorkerSpawnRunner in some cases, removed verboseFailures flag from WorkerSpawnRunner, updated callers - disable some tests on Windows; we were previously not running them, now that we do, they fail :-( Change-Id: I207b3938f0dc84d374ab052d5030020886451d47 PiperOrigin-RevId: 164965398
* AbstractSpawnStrategy: use ActionExecutionContext.getVerboseFailuresGravatar ulfjack2017-08-11
| | | | | | | ...instead of injecting it through the constructor. Simplify all the callers accordingly. PiperOrigin-RevId: 164955391
* Move anonymous inner class to nested classGravatar ulfjack2017-08-11
| | | | PiperOrigin-RevId: 164844520
* Unify input prefetchingGravatar ulfjack2017-08-10
| | | | | | | All prefetching now goes through AbstractSpawnStrategy's implementation of SpawnExecutionPolicy. Make sure the sandbox runners also do this consistently. PiperOrigin-RevId: 164836877
* Use java.time.Duration for timeoutsGravatar ulfjack2017-08-09
| | | | PiperOrigin-RevId: 164577062
* Fixing #3380: output stack traces and informative messages.Gravatar olaola2017-07-27
| | | | | | | | Also added tests specifically for the output, to ensure we don't break it again. TESTED=remote worker, unit tests RELNOTES: fixes #3380 PiperOrigin-RevId: 163283558
* Extract a common AbstractSpawnStrategy parent classGravatar ulfjack2017-07-24
This removes a bunch of code duplication that I previously introduced. PiperOrigin-RevId: 162909430
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-10 02:10:23 +0000