aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/worker/WorkerSpawnStrategy.java
Commit message (Collapse)AuthorAge
* Simplify the Spawn interface, update Executor interfaceGravatar Ulf Adams2017-02-01
| | | | | | | | | | | | | | | | | | | | Drop the Spawn.getOwner() method, which was duplicating functionality already available through Spawn.getResourceOwner(), in favor of the latter. In order to do that, change Executor.reportSubcommand to take a Spawn instance instead, which in turn requires updating all call sites. This is part of establishing Spawn as an abstraction for lower-level local and remote execution in all cases instead of passing args, env, input/output files and possibly other metadata as individual method parameters to the underlying implementations. In order for that to be maintainable, Spawn should be light-weight and easy to construct correctly, and not contain too much unnecessary data. -- PiperOrigin-RevId: 146224914 MOS_MIGRATED_REVID=146224914
* Simplify the Spawn interfaceGravatar Ulf Adams2017-02-01
| | | | | | | | | | | | | | | | | | | | | | | | Remove getSpawnInfo and asShellCommand, which are never really overridden in a useful way. asShellCommand moves to the Spawns class, and getSpawnInfo is only ever called by SpawnAction, and the implementation moves there. I'm considering using Spawn as the general lower-level abstraction for both local and remote execution. It sort of is that already, except it's not used consistently - we often pass a tuple of (args, env) plus possibly input and output files through parameter-heavy method call hierarchies instead of using this existing abstraction. However, I'm concerned about the amount of baggage it's carrying as well as the number of implementations for what is supposed to be a simple interface (or possibly even a simple value class), and this is an attempt to slim it down a bit. This should have no visible effects on builds. -- PiperOrigin-RevId: 146109838 MOS_MIGRATED_REVID=146109838
* Initial code for Persistent Java Test Runner.Gravatar Kush Chakraborty2016-12-09
| | | | | | | | | | | | At this point this does nothing more than re-run the exact same test without having to re-start the test runner. In future iterations the aim is to be able to re-run tests with modified code, without having to re-start the test runner. To test out the WorkerTestStrategy simply use --test_strategy=experimental_worker for a test with bazel. -- PiperOrigin-RevId: 141465929 MOS_MIGRATED_REVID=141465929
* If a worker returns unparseable output, print it to the terminal.Gravatar Philipp Wollermann2016-12-06
| | | | | | | | When a persistent worker returns a WorkResponse that cannot be parsed by protobuf, it probably means that the worker has encountered an error and mistakenly printed a stack trace or error message to stdout. For users who want to know why their compiler crashes and for developers of tools that support the worker mode, it is extremely helpful to actually see these error messages, so let's print them. -- PiperOrigin-RevId: 141176835 MOS_MIGRATED_REVID=141176835
* Add support for sandboxing persistent workers.Gravatar Philipp Wollermann2016-09-28
| | | | | -- MOS_MIGRATED_REVID=134523222
* Use byte[] rather than ByteString for file digests.Gravatar Googler2016-07-20
| | | | | | | | | | | | | | | | | ActionInputFileCache: Change getDigest() to return the underlying byte[16] owned by each FileArtifactValue. Remove throws clause from getInputFromDigest(); this should be an in-memory operation, and no implementation actually throws. PerActionFileCache: Invert mapping from artifact to digest only if needed. Remove interner, as it was used only for the reverse map keys, not the returned values. This should be a significant cpu savings as eagerly constructing the reverse maps was a noticeable hotspot. -- MOS_MIGRATED_REVID=127972359
* workers: WorkerSpawnStrategy will now only execute SpawnActions if their ↵Gravatar Philipp Wollermann2016-07-19
| | | | | | | | | | | ExecutionInfo contains the tag "support-workers" set to "1" and fallback to non-worker execution if it is not present. This will eventually allow us to safely automatically decide whether to use workers to execute an action or not. RELNOTES[INC]: If you maintain a rule that uses persistent workers, you'll have to specify execution_requirements={"supports-workers": 1} in the ctx.action that intends to run a tool with workers. The WorkerSpawnStrategy will alert you with a warning message if you forget to make this change and fallback to non-worker based execution. -- MOS_MIGRATED_REVID=127822788
* More detailed error message for sandbox failures in test, when ↵Gravatar Yue Gan2016-04-13
| | | | | | | | | --sandbox_debug and --verbose_failures are on. See discussion in #1049. RELNOTES: -- MOS_MIGRATED_REVID=119635080
* Remove incremental heuristic from workers, as it was not really useful.Gravatar Philipp Wollermann2016-02-29
| | | | | -- MOS_MIGRATED_REVID=115800229
* Rename SpawnActionContext#isRemotable() to more appropriate ↵Gravatar Philipp Wollermann2016-02-28
| | | | | | | willExecuteRemotely(). Remove unused parameter "mnemonic". -- MOS_MIGRATED_REVID=115666410
* Rename MiddlemanExpander to ArtifactExpander, and refactor it to yield ↵Gravatar Michael Thvedt2016-02-09
| | | | | | | ArtifactFiles. -- MOS_MIGRATED_REVID=114166208
* Make The Build Faster: More explicit exception handling in the worker strategy.Gravatar Philipp Wollermann2016-02-09
| | | | | -- MOS_MIGRATED_REVID=114102899
* Make The Build Faster: Drop the describeStrategy() and strategyLocality() ↵Gravatar Philipp Wollermann2016-02-03
| | | | | | | | | methods, as we can simply pass an ActionStatusMessage to the EventBus instead. All SpawnActionContexts now send an appropriate message when they execute a Spawn. This also gets rid of the idiom that an Action knows which strategy will be used to execute it - this decision and knowledge belongs to the executor, not the action. -- MOS_MIGRATED_REVID=113731846
* Create the convenience symlinks only after the build, and only those that ↵Gravatar Lukacs Berki2016-01-21
| | | | | | | | | point to an existing directory. This required fixing the worker strategy so that it reads params files not through said convenience symlinks, but from the execroot. -- MOS_MIGRATED_REVID=112682485
* Use Bazel Preconditions variant which avoids varargs array creationGravatar Mark Schaller2015-12-10
| | | | | | | Reduces garbage. -- MOS_MIGRATED_REVID=109914243
* workers: Pass a map of input filenames -> digest of file contents to workers ↵Gravatar Philipp Wollermann2015-11-05
| | | | | | | so that they can cache and reuse state for unchanged inputs over multiple builds. -- MOS_MIGRATED_REVID=107066408
* workers: Put command-line arguments into the WorkRequest instead of passing ↵Gravatar Philipp Wollermann2015-11-05
| | | | | | | | | the @flagfile. Note that this does not resolve recursive @flagfile inclusion in the args if they're present. -- MOS_MIGRATED_REVID=107052447
* workers: Restart worker processes when their binary has changed since they ↵Gravatar Philipp Wollermann2015-11-05
| | | | | | | were launched. -- MOS_MIGRATED_REVID=107050157
* Rationalize copyright headersGravatar Damien Martin-Guillerez2015-09-25
| | | | | | | | | | | The headers were modified with `find . -type f -exec 'sed' '-Ei' 's|Copyright 201([45]) Google|Copyright 201\1 The Bazel Authors|' '{}' ';'` And manual edit for not Google owned copyright. Because of the nature of ijar, I did not modified the header of file owned by Alan Donovan. The list of authors were extracted from the git log. It is missing older Google contributors that can be added on-demand. -- MOS_MIGRATED_REVID=103938715
* workers: Make sure to wait for worker processes to exit so that they don't ↵Gravatar Philipp Wollermann2015-09-21
| | | | | | | become zombies. -- MOS_MIGRATED_REVID=103541217
* Move Label from the lib.syntax to the lib.cmdline package so that:Gravatar Lukacs Berki2015-09-21
| | | | | | | | | | - Label parsing can be simplified - lib.syntax is only contains the code for Skylark and is reasonably independent from the problem domain of building things This change is mostly only changes to imports declarations. The rest is reversing the dependency between :cmdline and :syntax and moving a tiny amount of code between Printer and FilesetEntry and the addition of SkylarkPrintableValue that I couldn't be bothered to separate out into its own change. -- MOS_MIGRATED_REVID=103527877
* workers: A multitude of bug fixes and improved logging.Gravatar Philipp Wollermann2015-09-14
| | | | | | | | | | | | | | | | | | | | | | | I know this should have been split up, but I was a bit on fire today and did it all in one go ^^; Fixes spurious "Stream closed: Stream closed" errors, by noticing dead workers and retrying with a fresh one. (Configurable with the --worker_max_retries flag.) Fixes an "IllegalArgumentException" when a non-worker compatible Spawn is given to the strategy. We fall back to StandaloneSpawnStrategy now. Redirect the stderr of worker processes to separate log files in a common sub-directory and print a message that tells you about the location on worker start-up for easier debugging. The log can be found in <output_base>/worker-logs/*.log. Adds the mnemonic of the Spawn to log messages and the log filename. Adds verbose messages on worker start-up and shutdown. (Enable it with --worker_verbose!) Shuts down the worker pool after a build finished by default, until we sort out one last remaining correctness issue. This also conserves resources, though makes incremental builds a bit slower. Want the maximum performance anyway? Try --experimental_workers_keep_running. Adds stack traces to errors that are caused by buggy workers to aid development. Fixes weird dupli..tripli..quadruple error messages ("Compiling failed: Stream closed: Stream closed: Stream closed: Stream closed."). -- MOS_MIGRATED_REVID=102983853
* Code cleanupGravatar Laurent Le Brun2015-09-03
| | | | | -- MOS_MIGRATED_REVID=102239051
* Send the output of spawns ran in worker processes to the outErr of the ↵Gravatar Philipp Wollermann2015-08-31
| | | | | | | execution context instead of printing it to System.err. -- MOS_MIGRATED_REVID=101941516
* Print some diagnostics if Precondition check fails.Gravatar Han-Wen Nienhuys2015-07-02
| | | | | -- MOS_MIGRATED_REVID=97407601
* Only run the worker if the build is incremental. Gravatar Han-Wen Nienhuys2015-06-26
| | | | | | | This feature is tunable through --worker_max_changed_files flag. If unspecified, all builds are considered incremental. -- MOS_MIGRATED_REVID=96968367
* Implement persistent worker processes and a spawn strategy that uses them.Gravatar Philipp Wollermann2015-06-10
This will be used by the persistent JavaBuilder, which improves performance of Java compilation by 4x due to profiting from JVM JIT optimizations and not having to relaunch the JVM for every spawn. It is completely generic though, so as long as a tool (ProGuard? Dexer? Whatever.) conforms to the Worker process protocol, it can use the new spawn strategy. -- MOS_MIGRATED_REVID=95527132
generated by cgit v1.2.3 (git 2.39.1) at 2024-10-20 03:41:55 +0000