aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/worker/WorkerSpawnStrategy.java
Commit message (Collapse)AuthorAge
* Plumb exec root through to all spawn runners.Gravatar tomlu2018-01-11
| | | | | | They need this to parse input manifests. Previously we would grab the exec root from the Root, but wish to unsupport this. PiperOrigin-RevId: 181669143
* Refactor persistent workers to use SpawnRunner.Gravatar Benjamin Peterson2017-08-11
| | | | | | | | | | | | | | | | | | | | | | | | | | Change the persistent worker spawn strategy to extend AbstractSpawnStrategy and put the actual logic into WorkerSpawnRunner. WorkerTestStrategy is unaffected. I had to extend SpawnPolicy with a speculating() method. Persistent workers need to know if speculation is happening in order to require sandboxing. Additionally, I added java_test rules for the local runner tests and worker tests. See https://github.com/bazelbuild/bazel/issues/3481. NOTE: ulfjack@ made some changes to this change before merging: - changed Reporter to EventHandler; added TODO about its usage - reverted non-semantic indentation change in AbstractSpawnStrategy - reverted a non-semantic indentation change in WorkerSpawnRunner - updated some internal classes to match - removed catch IOException in WorkerSpawnRunner in some cases, removed verboseFailures flag from WorkerSpawnRunner, updated callers - disable some tests on Windows; we were previously not running them, now that we do, they fail :-( Change-Id: I207b3938f0dc84d374ab052d5030020886451d47 PiperOrigin-RevId: 164965398
* Simplify exception handling in spawn strategiesGravatar ulfjack2017-07-17
| | | | | | | | | | | | | | | | | | | | | | | | | The main change here is to only catch SpawnExecException in StandaloneTestStrategy, so all other exceptions simplify propagate up. As a result, Bazel no longer retries tests that fail with an exception, we only retry tests that actually ran, had a spawn result, and resulted in a UserExecException. That is probably what we want. Also do some cleanup: - Remove ExecException.timedOut; nobody was calling it (but there's still SpawnExecException.timedOut) - Remove SpawnActionContext.shouldPropagateExecException; all exceptions (except SpawnExecException) are now propagated by default - Remote the SandboxOptions from the SandboxStrategies; all sandboxing options are now handled by the underlying SpawnRunner implementations I'll send a followup CL to remove the UserExecException and EnvironmentalExecException types; the types don't do anything special, and there are no catch blocks in production code that catch one of these more specific types. This should fix #3322 by removing a bunch of special handling. PiperOrigin-RevId: 161960919
* Simplify the ActionInputFileCacheGravatar ulfjack2017-07-06
| | | | | | | | | Add a single getMetadata method (matching MetadataHandler), and rewrite everything in those terms. This is in preparation for merging ActionInputFileCache and MetadataHandler. PiperOrigin-RevId: 161053535
* Rewrite the Executor/ActionExecutionContext splitGravatar ulfjack2017-06-19
| | | | | | | Move everything to ActionExecutionContext, and drop Executor whereever possible. This clarifies the API, makes it simpler to test, and simplifies the code. PiperOrigin-RevId: 159414816
* Don't create the same String/ImmutableMap over and over again.Gravatar twerth2017-06-09
| | | | | RELNOTES: None PiperOrigin-RevId: 158498863
* Automated g4 rollback of commit c4134802dd15d6ef5cca6521f6bf6aac395ee2ad.Gravatar kchodorow2017-05-23
| | | | | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Roll forward of directory name change *** Original change description *** Automated g4 rollback of commit 1d9e1ac90197b1d3d7b137ba3c1ada67bb9ba31b. *** Reason for rollback *** Breaks //src/test/shell/integration:force_delete_output_test *** Original change description *** Symlink output directories to the correct directory name If the workspace directory is /path/to/my/proj and the name in the WORKSPACE file is "floop", this will symlink the output directories to output_base/execroot/floop instead of output_base/execroot/proj. More prep for #1262, fixes #1681. PiperOrigin-RevId: 156892980
* sandbox: Use the SpawnInputExpander everywhere and delete SpawnHelpers.Gravatar Philipp Wollermann2017-05-09
| | | | | | | | | This unifies our code to use just one standard implementation to get the entire expanded input files for a Spawn, including from Filesets and Runfiles. Change-Id: I1e286508adf0a9aeddf70934b010e6fcc144c4a7 PiperOrigin-RevId: 155497273
* Add support for logging exception details via ErrorMessage.Gravatar philwo2017-04-28
| | | | | | Part of #2855. PiperOrigin-RevId: 154477854
* worker: Do not check if the process is still alive prior to using it.Gravatar philwo2017-04-28
| | | | | | | | | | | | | | | This might sound strange at first, but the reasoning is this: A worker should never simply exit. Bazel controls the lifetime of its subprocesses, so a worker quitting is considered a failure (also because it cannot be distinguished from a crash). With that set, we can improve two things: - Bazel will now only notice that a worker crashed / quit when it tries to use one during a build. This is also the only time when we can print error messages to the user. Earlier we might have noticed that a worker crashed during validation, but had no mechanism to alert the user to this, because this wasn't necessarily during a build. - This also fixes a race condition where a worker is still alive during validation, then quits, then the WorkerSpawnStrategy tries to send a WorkRequest, which fails, triggering an IOException. This fixes the flaky test test_worker_restarts_after_exit (which is now called test_build_fails_when_worker_exits). Part of #2855. RELNOTES: Bazel will no longer gracefully restart workers that crashed / quit, instead this triggers a build failure. PiperOrigin-RevId: 154470257
* worker: Extract two methods from actuallyExec for better readability.Gravatar philwo2017-04-28
| | | | | | Part of #2855. PiperOrigin-RevId: 154461639
* worker: Small refactoring of RecordingInputStream.Gravatar philwo2017-04-28
| | | | | | | | Prevents it from potentially throwing an UnsupportedEncodingException. Part of #2855. PiperOrigin-RevId: 154446897
* worker: Remove the "retry on failure" feature.Gravatar philwo2017-04-28
| | | | | | | | | | | | | It wasn't a good idea to have in the first place: - It only ever worked in certain random circumstances (e.g. worker returning an unparseable protobuf, but not when it crashed). - No other strategy implements a behavior like this. - Confusing to users and hard to describe what use case this is good for. - Complex and error prone code. Part of #2855. RELNOTES: The flag --worker_max_retries was removed. The WorkerSpawnStrategy no longer retries execution of failed Spawns, the reason being that this just masks compiler bugs and isn't done for any other execution strategy either. PiperOrigin-RevId: 154422561
* Use the new ErrorMessage class in WorkerTestStrategy. Also adds a nullGravatar philwo2017-04-28
| | | | | | | | check for the response and shows a helpful error message in that case. Part of #2855. PiperOrigin-RevId: 154416882
* Dump the stderr log of a worker when it crashes during a build.Gravatar philwo2017-04-25
| | | | | | | | This should help users and developers a lot figuring out *why* the worker crashed as the log file was very undiscoverable for them. Part of the "make error messages great" effort (#2855). PiperOrigin-RevId: 154165665
* Use the RecordingInputStream from the WorkerSpawnStrategy in the ↵Gravatar kush2017-04-25
| | | | | | | WorkerTestStrategy as well, to better log errors. RELNOTES: None PiperOrigin-RevId: 154080821
* Automated g4 rollback of commit 1d9e1ac90197b1d3d7b137ba3c1ada67bb9ba31b.Gravatar hlopko2017-04-04
| | | | | | | | | | | | | | | | | | *** Reason for rollback *** Breaks //src/test/shell/integration:force_delete_output_test *** Original change description *** Symlink output directories to the correct directory name If the workspace directory is /path/to/my/proj and the name in the WORKSPACE file is "floop", this will symlink the output directories to output_base/execroot/floop instead of output_base/execroot/proj. More prep for #1262, fixes #1681. PiperOrigin-RevId: 152126545
* Symlink output directories to the correct directory nameGravatar kchodorow2017-03-31
| | | | | | | | | | If the workspace directory is /path/to/my/proj and the name in the WORKSPACE file is "floop", this will symlink the output directories to output_base/execroot/floop instead of output_base/execroot/proj. More prep for #1262, fixes #1681. PiperOrigin-RevId: 151712384
* Add support for --flagfile=flagfile.txt args in addition to the usual ↵Gravatar Philipp Wollermann2017-03-06
| | | | | | | | | | @flagfile.txt style. Let the worker strategy correctly handle multiple flagfiles, instead of just assuming that the last argument will be the one and only @flagfile. -- PiperOrigin-RevId: 149291230 MOS_MIGRATED_REVID=149291230
* Remove all the action resource estimation codeGravatar Ulf Adams2017-03-06
| | | | | | -- PiperOrigin-RevId: 149110466 MOS_MIGRATED_REVID=149110466
* Rationalize local resource acquisitionGravatar Ulf Adams2017-03-03
| | | | | | | | | | | | | | | | | | | | | | Move all local resource acquisition to where local execution actually happens. Don't attempt to acquire resources per action, but only for individual spawns. This significantly simplifies the code. The downside is that we don't account for action-level work anymore. In general, actions should not perform any process execution themselves, but always delegate such work to a SpawnStrategy implementation. This change makes sure that every Spawn has local resources set in a way that is consistent with the previous state. However, there are two actions - Fileset and FileWrite -, which are not spawns, and so we now don't limit their concurrent execution anymore. For Fileset, all work is done in a custom Fileset-specific thread pool, so this shouldn't be a problem. I'm not sure about FileWriteAction. -- PiperOrigin-RevId: 149012600 MOS_MIGRATED_REVID=149012600
* Simplify the Spawn interface, update Executor interfaceGravatar Ulf Adams2017-02-01
| | | | | | | | | | | | | | | | | | | | Drop the Spawn.getOwner() method, which was duplicating functionality already available through Spawn.getResourceOwner(), in favor of the latter. In order to do that, change Executor.reportSubcommand to take a Spawn instance instead, which in turn requires updating all call sites. This is part of establishing Spawn as an abstraction for lower-level local and remote execution in all cases instead of passing args, env, input/output files and possibly other metadata as individual method parameters to the underlying implementations. In order for that to be maintainable, Spawn should be light-weight and easy to construct correctly, and not contain too much unnecessary data. -- PiperOrigin-RevId: 146224914 MOS_MIGRATED_REVID=146224914
* Simplify the Spawn interfaceGravatar Ulf Adams2017-02-01
| | | | | | | | | | | | | | | | | | | | | | | | Remove getSpawnInfo and asShellCommand, which are never really overridden in a useful way. asShellCommand moves to the Spawns class, and getSpawnInfo is only ever called by SpawnAction, and the implementation moves there. I'm considering using Spawn as the general lower-level abstraction for both local and remote execution. It sort of is that already, except it's not used consistently - we often pass a tuple of (args, env) plus possibly input and output files through parameter-heavy method call hierarchies instead of using this existing abstraction. However, I'm concerned about the amount of baggage it's carrying as well as the number of implementations for what is supposed to be a simple interface (or possibly even a simple value class), and this is an attempt to slim it down a bit. This should have no visible effects on builds. -- PiperOrigin-RevId: 146109838 MOS_MIGRATED_REVID=146109838
* Initial code for Persistent Java Test Runner.Gravatar Kush Chakraborty2016-12-09
| | | | | | | | | | | | At this point this does nothing more than re-run the exact same test without having to re-start the test runner. In future iterations the aim is to be able to re-run tests with modified code, without having to re-start the test runner. To test out the WorkerTestStrategy simply use --test_strategy=experimental_worker for a test with bazel. -- PiperOrigin-RevId: 141465929 MOS_MIGRATED_REVID=141465929
* If a worker returns unparseable output, print it to the terminal.Gravatar Philipp Wollermann2016-12-06
| | | | | | | | When a persistent worker returns a WorkResponse that cannot be parsed by protobuf, it probably means that the worker has encountered an error and mistakenly printed a stack trace or error message to stdout. For users who want to know why their compiler crashes and for developers of tools that support the worker mode, it is extremely helpful to actually see these error messages, so let's print them. -- PiperOrigin-RevId: 141176835 MOS_MIGRATED_REVID=141176835
* Add support for sandboxing persistent workers.Gravatar Philipp Wollermann2016-09-28
| | | | | -- MOS_MIGRATED_REVID=134523222
* Use byte[] rather than ByteString for file digests.Gravatar Googler2016-07-20
| | | | | | | | | | | | | | | | | ActionInputFileCache: Change getDigest() to return the underlying byte[16] owned by each FileArtifactValue. Remove throws clause from getInputFromDigest(); this should be an in-memory operation, and no implementation actually throws. PerActionFileCache: Invert mapping from artifact to digest only if needed. Remove interner, as it was used only for the reverse map keys, not the returned values. This should be a significant cpu savings as eagerly constructing the reverse maps was a noticeable hotspot. -- MOS_MIGRATED_REVID=127972359
* workers: WorkerSpawnStrategy will now only execute SpawnActions if their ↵Gravatar Philipp Wollermann2016-07-19
| | | | | | | | | | | ExecutionInfo contains the tag "support-workers" set to "1" and fallback to non-worker execution if it is not present. This will eventually allow us to safely automatically decide whether to use workers to execute an action or not. RELNOTES[INC]: If you maintain a rule that uses persistent workers, you'll have to specify execution_requirements={"supports-workers": 1} in the ctx.action that intends to run a tool with workers. The WorkerSpawnStrategy will alert you with a warning message if you forget to make this change and fallback to non-worker based execution. -- MOS_MIGRATED_REVID=127822788
* More detailed error message for sandbox failures in test, when ↵Gravatar Yue Gan2016-04-13
| | | | | | | | | --sandbox_debug and --verbose_failures are on. See discussion in #1049. RELNOTES: -- MOS_MIGRATED_REVID=119635080
* Remove incremental heuristic from workers, as it was not really useful.Gravatar Philipp Wollermann2016-02-29
| | | | | -- MOS_MIGRATED_REVID=115800229
* Rename SpawnActionContext#isRemotable() to more appropriate ↵Gravatar Philipp Wollermann2016-02-28
| | | | | | | willExecuteRemotely(). Remove unused parameter "mnemonic". -- MOS_MIGRATED_REVID=115666410
* Rename MiddlemanExpander to ArtifactExpander, and refactor it to yield ↵Gravatar Michael Thvedt2016-02-09
| | | | | | | ArtifactFiles. -- MOS_MIGRATED_REVID=114166208
* Make The Build Faster: More explicit exception handling in the worker strategy.Gravatar Philipp Wollermann2016-02-09
| | | | | -- MOS_MIGRATED_REVID=114102899
* Make The Build Faster: Drop the describeStrategy() and strategyLocality() ↵Gravatar Philipp Wollermann2016-02-03
| | | | | | | | | methods, as we can simply pass an ActionStatusMessage to the EventBus instead. All SpawnActionContexts now send an appropriate message when they execute a Spawn. This also gets rid of the idiom that an Action knows which strategy will be used to execute it - this decision and knowledge belongs to the executor, not the action. -- MOS_MIGRATED_REVID=113731846
* Create the convenience symlinks only after the build, and only those that ↵Gravatar Lukacs Berki2016-01-21
| | | | | | | | | point to an existing directory. This required fixing the worker strategy so that it reads params files not through said convenience symlinks, but from the execroot. -- MOS_MIGRATED_REVID=112682485
* Use Bazel Preconditions variant which avoids varargs array creationGravatar Mark Schaller2015-12-10
| | | | | | | Reduces garbage. -- MOS_MIGRATED_REVID=109914243
* workers: Pass a map of input filenames -> digest of file contents to workers ↵Gravatar Philipp Wollermann2015-11-05
| | | | | | | so that they can cache and reuse state for unchanged inputs over multiple builds. -- MOS_MIGRATED_REVID=107066408
* workers: Put command-line arguments into the WorkRequest instead of passing ↵Gravatar Philipp Wollermann2015-11-05
| | | | | | | | | the @flagfile. Note that this does not resolve recursive @flagfile inclusion in the args if they're present. -- MOS_MIGRATED_REVID=107052447
* workers: Restart worker processes when their binary has changed since they ↵Gravatar Philipp Wollermann2015-11-05
| | | | | | | were launched. -- MOS_MIGRATED_REVID=107050157
* Rationalize copyright headersGravatar Damien Martin-Guillerez2015-09-25
| | | | | | | | | | | The headers were modified with `find . -type f -exec 'sed' '-Ei' 's|Copyright 201([45]) Google|Copyright 201\1 The Bazel Authors|' '{}' ';'` And manual edit for not Google owned copyright. Because of the nature of ijar, I did not modified the header of file owned by Alan Donovan. The list of authors were extracted from the git log. It is missing older Google contributors that can be added on-demand. -- MOS_MIGRATED_REVID=103938715
* workers: Make sure to wait for worker processes to exit so that they don't ↵Gravatar Philipp Wollermann2015-09-21
| | | | | | | become zombies. -- MOS_MIGRATED_REVID=103541217
* Move Label from the lib.syntax to the lib.cmdline package so that:Gravatar Lukacs Berki2015-09-21
| | | | | | | | | | - Label parsing can be simplified - lib.syntax is only contains the code for Skylark and is reasonably independent from the problem domain of building things This change is mostly only changes to imports declarations. The rest is reversing the dependency between :cmdline and :syntax and moving a tiny amount of code between Printer and FilesetEntry and the addition of SkylarkPrintableValue that I couldn't be bothered to separate out into its own change. -- MOS_MIGRATED_REVID=103527877
* workers: A multitude of bug fixes and improved logging.Gravatar Philipp Wollermann2015-09-14
| | | | | | | | | | | | | | | | | | | | | | | I know this should have been split up, but I was a bit on fire today and did it all in one go ^^; Fixes spurious "Stream closed: Stream closed" errors, by noticing dead workers and retrying with a fresh one. (Configurable with the --worker_max_retries flag.) Fixes an "IllegalArgumentException" when a non-worker compatible Spawn is given to the strategy. We fall back to StandaloneSpawnStrategy now. Redirect the stderr of worker processes to separate log files in a common sub-directory and print a message that tells you about the location on worker start-up for easier debugging. The log can be found in <output_base>/worker-logs/*.log. Adds the mnemonic of the Spawn to log messages and the log filename. Adds verbose messages on worker start-up and shutdown. (Enable it with --worker_verbose!) Shuts down the worker pool after a build finished by default, until we sort out one last remaining correctness issue. This also conserves resources, though makes incremental builds a bit slower. Want the maximum performance anyway? Try --experimental_workers_keep_running. Adds stack traces to errors that are caused by buggy workers to aid development. Fixes weird dupli..tripli..quadruple error messages ("Compiling failed: Stream closed: Stream closed: Stream closed: Stream closed."). -- MOS_MIGRATED_REVID=102983853
* Code cleanupGravatar Laurent Le Brun2015-09-03
| | | | | -- MOS_MIGRATED_REVID=102239051
* Send the output of spawns ran in worker processes to the outErr of the ↵Gravatar Philipp Wollermann2015-08-31
| | | | | | | execution context instead of printing it to System.err. -- MOS_MIGRATED_REVID=101941516
* Print some diagnostics if Precondition check fails.Gravatar Han-Wen Nienhuys2015-07-02
| | | | | -- MOS_MIGRATED_REVID=97407601
* Only run the worker if the build is incremental. Gravatar Han-Wen Nienhuys2015-06-26
| | | | | | | This feature is tunable through --worker_max_changed_files flag. If unspecified, all builds are considered incremental. -- MOS_MIGRATED_REVID=96968367
* Implement persistent worker processes and a spawn strategy that uses them.Gravatar Philipp Wollermann2015-06-10
This will be used by the persistent JavaBuilder, which improves performance of Java compilation by 4x due to profiting from JVM JIT optimizations and not having to relaunch the JVM for every spawn. It is completely generic though, so as long as a tool (ProGuard? Dexer? Whatever.) conforms to the Worker process protocol, it can use the new spawn strategy. -- MOS_MIGRATED_REVID=95527132
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-01 15:53:20 +0000