aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/worker
Commit message (Collapse)AuthorAge
...
* Add the default category and tag to all options.Gravatar ccalvarin2017-06-28
| | | | | | | | Move the default from the annotation to every mention. This makes the incompleteness explicit. Will add the defaults to test targets in a separate change. Once all dependencies are cleaned up, the Option annotation will no longer allow options without the documentationCategory or effectTag, to prevent new options being added without categories while we migrate to the new option categorization. PiperOrigin-RevId: 160281252
* Extract ActionContext to a top-level classGravatar ulfjack2017-06-19
| | | | PiperOrigin-RevId: 159423459
* Rewrite the Executor/ActionExecutionContext splitGravatar ulfjack2017-06-19
| | | | | | | Move everything to ActionExecutionContext, and drop Executor whereever possible. This clarifies the API, makes it simpler to test, and simplifies the code. PiperOrigin-RevId: 159414816
* Rewrite StandaloneSpawnStrategy to use LocalSpawnRunnerGravatar ulfjack2017-06-19
| | | | PiperOrigin-RevId: 159221067
* Simplify BlazeModule.beforeCommandGravatar ulfjack2017-06-14
| | | | | | | | | | | Don't pass the Command annotation explicitly, but add it to CommandEnvironment instead; most modules don't need it in the first place, so it was a lot of boilerplate for not much. Also change it so that the command is passed to the constructor. Add some documentation to the beforeCommand method. PiperOrigin-RevId: 158847128
* Don't create the same String/ImmutableMap over and over again.Gravatar twerth2017-06-09
| | | | | RELNOTES: None PiperOrigin-RevId: 158498863
* Track the test environment in Skyframe, like the action environmentGravatar ulfjack2017-06-09
| | | | | | | | Instead of passing a client env into the test strategies, use the same mechansim as --action_env, by depending on the right set of Skyframe nodes that correspond to client env entries. PiperOrigin-RevId: 158401670
* Automated g4 rollback of commit c4134802dd15d6ef5cca6521f6bf6aac395ee2ad.Gravatar kchodorow2017-05-23
| | | | | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Roll forward of directory name change *** Original change description *** Automated g4 rollback of commit 1d9e1ac90197b1d3d7b137ba3c1ada67bb9ba31b. *** Reason for rollback *** Breaks //src/test/shell/integration:force_delete_output_test *** Original change description *** Symlink output directories to the correct directory name If the workspace directory is /path/to/my/proj and the name in the WORKSPACE file is "floop", this will symlink the output directories to output_base/execroot/floop instead of output_base/execroot/proj. More prep for #1262, fixes #1681. PiperOrigin-RevId: 156892980
* sandbox: Use the SpawnInputExpander everywhere and delete SpawnHelpers.Gravatar Philipp Wollermann2017-05-09
| | | | | | | | | This unifies our code to use just one standard implementation to get the entire expanded input files for a Spawn, including from Filesets and Runfiles. Change-Id: I1e286508adf0a9aeddf70934b010e6fcc144c4a7 PiperOrigin-RevId: 155497273
* Add support for logging exception details via ErrorMessage.Gravatar philwo2017-04-28
| | | | | | Part of #2855. PiperOrigin-RevId: 154477854
* worker: Do not check if the process is still alive prior to using it.Gravatar philwo2017-04-28
| | | | | | | | | | | | | | | This might sound strange at first, but the reasoning is this: A worker should never simply exit. Bazel controls the lifetime of its subprocesses, so a worker quitting is considered a failure (also because it cannot be distinguished from a crash). With that set, we can improve two things: - Bazel will now only notice that a worker crashed / quit when it tries to use one during a build. This is also the only time when we can print error messages to the user. Earlier we might have noticed that a worker crashed during validation, but had no mechanism to alert the user to this, because this wasn't necessarily during a build. - This also fixes a race condition where a worker is still alive during validation, then quits, then the WorkerSpawnStrategy tries to send a WorkRequest, which fails, triggering an IOException. This fixes the flaky test test_worker_restarts_after_exit (which is now called test_build_fails_when_worker_exits). Part of #2855. RELNOTES: Bazel will no longer gracefully restart workers that crashed / quit, instead this triggers a build failure. PiperOrigin-RevId: 154470257
* worker: Extract two methods from actuallyExec for better readability.Gravatar philwo2017-04-28
| | | | | | Part of #2855. PiperOrigin-RevId: 154461639
* worker: Small refactoring of RecordingInputStream.Gravatar philwo2017-04-28
| | | | | | | | Prevents it from potentially throwing an UnsupportedEncodingException. Part of #2855. PiperOrigin-RevId: 154446897
* Slight tweaks to text formatting of ErrorMessages.Gravatar philwo2017-04-28
| | | | | | Part of #2855. PiperOrigin-RevId: 154427566
* worker: Remove the "retry on failure" feature.Gravatar philwo2017-04-28
| | | | | | | | | | | | | It wasn't a good idea to have in the first place: - It only ever worked in certain random circumstances (e.g. worker returning an unparseable protobuf, but not when it crashed). - No other strategy implements a behavior like this. - Confusing to users and hard to describe what use case this is good for. - Complex and error prone code. Part of #2855. RELNOTES: The flag --worker_max_retries was removed. The WorkerSpawnStrategy no longer retries execution of failed Spawns, the reason being that this just masks compiler bugs and isn't done for any other execution strategy either. PiperOrigin-RevId: 154422561
* Use the new ErrorMessage class in WorkerTestStrategy. Also adds a nullGravatar philwo2017-04-28
| | | | | | | | check for the response and shows a helpful error message in that case. Part of #2855. PiperOrigin-RevId: 154416882
* Dump the stderr log of a worker when it crashes during a build.Gravatar philwo2017-04-25
| | | | | | | | This should help users and developers a lot figuring out *why* the worker crashed as the log file was very undiscoverable for them. Part of the "make error messages great" effort (#2855). PiperOrigin-RevId: 154165665
* Use the RecordingInputStream from the WorkerSpawnStrategy in the ↵Gravatar kush2017-04-25
| | | | | | | WorkerTestStrategy as well, to better log errors. RELNOTES: None PiperOrigin-RevId: 154080821
* Fix #2769: Worker processes should be killed on 'bazel clean'.Gravatar philwo2017-04-12
| | | | PiperOrigin-RevId: 152799488
* Automated g4 rollback of commit 1d9e1ac90197b1d3d7b137ba3c1ada67bb9ba31b.Gravatar hlopko2017-04-04
| | | | | | | | | | | | | | | | | | *** Reason for rollback *** Breaks //src/test/shell/integration:force_delete_output_test *** Original change description *** Symlink output directories to the correct directory name If the workspace directory is /path/to/my/proj and the name in the WORKSPACE file is "floop", this will symlink the output directories to output_base/execroot/floop instead of output_base/execroot/proj. More prep for #1262, fixes #1681. PiperOrigin-RevId: 152126545
* Change the tag experimental_testrunner into a couple of flags.Gravatar kush2017-03-31
| | | | | | | | | | | | | | | Note 1) Explanation of the flags: --explicit_java_test_deps: This is a flag independent of the others, and forces users to specify the JUnit deps. We should try to make this flag default to true in a future release, irrespective of the work around persistent java tests. --experimental_testrunner: Bazel-only flag affecting switching from the BazelTestRunner to the ExperimentalTestRunner which will run the tests in a separate classloader. --explicit_java_test_deps is desired for this to ensure once we split the classpaths of the TestRunner and the TestTarget, the TestTarget does not hit a ClassNotFoundException, due to missing JUnit deps. --test_strategy=experimental_worker: This is the existing flag, which in turn depends on --experimental_testrunner flag, since only the ExperimentalTestRunner is capable to running java tests persistently. Note 2) There was no clean way to check for the flags defined in JavaOptions within TestActionBuilder (as I was checking the "tag=experimental_testrunner" before), without making TeasActionBuilder's build rules depend on java rules (yikes!). Hence, I created a new method compatibleWithStrategy() within each fragment, so I could check if WorkerTestStrategy could be compatible with the user specified flags. Thanks to Greg for suggesting this approach! RELNOTES: None PiperOrigin-RevId: 151729869
* Symlink output directories to the correct directory nameGravatar kchodorow2017-03-31
| | | | | | | | | | If the workspace directory is /path/to/my/proj and the name in the WORKSPACE file is "floop", this will symlink the output directories to output_base/execroot/floop instead of output_base/execroot/proj. More prep for #1262, fixes #1681. PiperOrigin-RevId: 151712384
* Prevent users from using the WorkerTestStrategy unless they're running with ↵Gravatar Kush Chakraborty2017-03-09
| | | | | | | | the ExperimentalTestRunner. -- PiperOrigin-RevId: 149636903 MOS_MIGRATED_REVID=149636903
* Bring back the persistent test runner functionality, on ExperimentalTestRunner.Gravatar Kush Chakraborty2017-03-09
| | | | | | | | This is essentially a rollforward of commit 7d0561b6ca92d72bd8767d4dca50e5437976812c, and changes triggering the perisitent runner using an environment variable instead of argument as suggested in commit 7d0561b6ca92d72bd8767d4dca50e5437976812c -- PiperOrigin-RevId: 149540564 MOS_MIGRATED_REVID=149540564
* Add support for --flagfile=flagfile.txt args in addition to the usual ↵Gravatar Philipp Wollermann2017-03-06
| | | | | | | | | | @flagfile.txt style. Let the worker strategy correctly handle multiple flagfiles, instead of just assuming that the last argument will be the one and only @flagfile. -- PiperOrigin-RevId: 149291230 MOS_MIGRATED_REVID=149291230
* Rewrite TestStrategy.getArgsGravatar Ulf Adams2017-03-06
| | | | | | | | | | | | | | | | | | | | | | This changes command-line computation for tests: - run the coverage collector before the run_under command - no shell escaping: if all tools just call "$@", then this should work Note that we still wrap the command in a sub-shell to support shell built-ins and PATH lookup if the command does not contain a slash character '/'. A side effect of this change is that the --run_under command now executes in the test's runfiles directory, rather than in the exec root, if coverage is enabled at the same time. Inside Google, it's very rare for --run_under to be used in combination with coverage, and it seems likely to be rare externally as well, so I don't think it warrants covering it in the release notes. Also set TEST_BINARY to the root-relative path of the test executable for all tests (in TestRunnerAction.java). -- PiperOrigin-RevId: 149275688 MOS_MIGRATED_REVID=149275688
* Remove all the action resource estimation codeGravatar Ulf Adams2017-03-06
| | | | | | -- PiperOrigin-RevId: 149110466 MOS_MIGRATED_REVID=149110466
* Rationalize local resource acquisitionGravatar Ulf Adams2017-03-03
| | | | | | | | | | | | | | | | | | | | | | Move all local resource acquisition to where local execution actually happens. Don't attempt to acquire resources per action, but only for individual spawns. This significantly simplifies the code. The downside is that we don't account for action-level work anymore. In general, actions should not perform any process execution themselves, but always delegate such work to a SpawnStrategy implementation. This change makes sure that every Spawn has local resources set in a way that is consistent with the previous state. However, there are two actions - Fileset and FileWrite -, which are not spawns, and so we now don't limit their concurrent execution anymore. For Fileset, all work is done in a custom Fileset-specific thread pool, so this shouldn't be a problem. I'm not sure about FileWriteAction. -- PiperOrigin-RevId: 149012600 MOS_MIGRATED_REVID=149012600
* Rollback of commit 786cfa2ed980e278c42ee474408844f7e3720385.Gravatar Kush Chakraborty2017-02-24
| | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Breaks dagger []: [] *** Original change description *** Separate the classpaths of the TestRunner with the test target, and use a separate Classloader to load the test target's classes. This enables a clean separation of the classes of the TestRunner with the target under test. This is achieved with the following steps: 1. Start the test runner with only the bare bones classpaths to the Test Runner's classes which are used by the system ClassLoader. 2. Have all the classpaths required to load the test target's classes in a TEST_TARGET_CLASSPATH envi... *** -- PiperOrigin-RevId: 148405598 MOS_MIGRATED_REVID=148405598
* Separate the classpaths of the TestRunner with the test target, and use a ↵Gravatar Kush Chakraborty2017-02-23
| | | | | | | | | | | | | | | | | | | | | | separate Classloader to load the test target's classes. This enables a clean separation of the classes of the TestRunner with the target under test. This is achieved with the following steps: 1. Start the test runner with only the bare bones classpaths to the Test Runner's classes which are used by the system ClassLoader. 2. Have all the classpaths required to load the test target's classes in a TEST_TARGET_CLASSPATH environment variable exported by the stub script. 3. Use a new classloader to load all the test target's classes using the paths in TEST_TARGET_CLASSPATH. This additionally enables the persistent test runner (currently experimental), to reload all the target's classes for every subsequent test run, so it can pick up any changes to the classes in between runs. The persistent test runner can be used by adding the argument --test_strategy=experimental_worker to the bazel test command. Tested this against: 1. gerrit/gerrit-common:client_tests: Dismal avg. improvement of 580ms to 557ms (just 23ms) 2. intellij/intellij/base:unit_tests: Somewhat modest avg. improvement 1661ms to 913ms (748 ms) RELNOTES: 1) Java tests and suites will now have to explicitly declare JUnit dependency 2) All non-legacy java_tests will now be run in a -- PiperOrigin-RevId: 148309979 MOS_MIGRATED_REVID=148309979
* Rollback of commit 1cb5059396e46d9ff2c317b8a0a046af32b3a1d8.Gravatar Ulf Adams2017-02-20
| | | | | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Breaks TEST_BINARY env setting if --run_under is used. *** Original change description *** Rewrite TestStrategy.getArgs This changes command-line computation for tests: - run the coverage collector before the run_under command - no shell escaping: if all tools just call "$@", then this should work Note that we still wrap the command in a sub-shell to support shell built-ins and PATH lookup if the command does not contain a slash character '/'. A side effect of this change is that the --run_under command now executes in the test's runfiles directory, rather than in the exec root, if... *** -- PiperOrigin-RevId: 148003525 MOS_MIGRATED_REVID=148003525
* Rewrite TestStrategy.getArgsGravatar Ulf Adams2017-02-17
| | | | | | | | | | | | | | | | | | | This changes command-line computation for tests: - run the coverage collector before the run_under command - no shell escaping: if all tools just call "$@", then this should work Note that we still wrap the command in a sub-shell to support shell built-ins and PATH lookup if the command does not contain a slash character '/'. A side effect of this change is that the --run_under command now executes in the test's runfiles directory, rather than in the exec root, if coverage is enabled at the same time. Inside Google, it's very rare for --run_under to be used in combination with coverage, and it seems likely to be rare externally as well, so I don't think it warrants covering it in the release notes. -- PiperOrigin-RevId: 147815017 MOS_MIGRATED_REVID=147815017
* Use //third_party/protobuf:protobuf_java instead of ↵Gravatar Carmi Grushko2017-02-09
| | | | | | | | | | | | //third_party/protobuf:protobuf to refer to the Java proto runtime. (second attempt) This is the name in the upstream protobuf repo. -- PiperOrigin-RevId: 147057949 MOS_MIGRATED_REVID=147057949
* Remove the first argument of TestStrategy.getArgs.Gravatar Ulf Adams2017-02-09
| | | | | | | | | All callers were performing identical steps before calling getArgs - move the functionality into getArgs instead. -- PiperOrigin-RevId: 147028753 MOS_MIGRATED_REVID=147028753
* Simplify the Spawn interface, update Executor interfaceGravatar Ulf Adams2017-02-01
| | | | | | | | | | | | | | | | | | | | Drop the Spawn.getOwner() method, which was duplicating functionality already available through Spawn.getResourceOwner(), in favor of the latter. In order to do that, change Executor.reportSubcommand to take a Spawn instance instead, which in turn requires updating all call sites. This is part of establishing Spawn as an abstraction for lower-level local and remote execution in all cases instead of passing args, env, input/output files and possibly other metadata as individual method parameters to the underlying implementations. In order for that to be maintainable, Spawn should be light-weight and easy to construct correctly, and not contain too much unnecessary data. -- PiperOrigin-RevId: 146224914 MOS_MIGRATED_REVID=146224914
* Throw an exception when persistent workers are used to run tests with ↵Gravatar Kush Chakraborty2017-02-01
| | | | | | | | use_testrunner=0 -- PiperOrigin-RevId: 146150454 MOS_MIGRATED_REVID=146150454
* Simplify the Spawn interfaceGravatar Ulf Adams2017-02-01
| | | | | | | | | | | | | | | | | | | | | | | | Remove getSpawnInfo and asShellCommand, which are never really overridden in a useful way. asShellCommand moves to the Spawns class, and getSpawnInfo is only ever called by SpawnAction, and the implementation moves there. I'm considering using Spawn as the general lower-level abstraction for both local and remote execution. It sort of is that already, except it's not used consistently - we often pass a tuple of (args, env) plus possibly input and output files through parameter-heavy method call hierarchies instead of using this existing abstraction. However, I'm concerned about the amount of baggage it's carrying as well as the number of implementations for what is supposed to be a simple interface (or possibly even a simple value class), and this is an attempt to slim it down a bit. This should have no visible effects on builds. -- PiperOrigin-RevId: 146109838 MOS_MIGRATED_REVID=146109838
* Refactor StandaloneTestStrategy; tiny step towards test strategy unification.Gravatar Ulf Adams2017-01-10
| | | | | | | | | | Create the test spawn upfront instead of lazily. In the common case, this doesn't cost anything, but the worker test strategy doesn't use the spawn right now. -- PiperOrigin-RevId: 144086321 MOS_MIGRATED_REVID=144086321
* Move some classes to lib.exec in preparation for layering enforcementGravatar Ulf Adams2016-12-21
| | | | | | | | | | | | This change is part of the mu-bazel effort, which aims to build a minimally useful Bazel binary with most extraneous functionality removed. As part of that, we want to enforce layering of packages. In particular, lib.actions must not depend on lib.rules or lib.exec. lib.rules must not depend on lib.exec. Moving these classes is a necessary step to enforce that layering. -- PiperOrigin-RevId: 142668172 MOS_MIGRATED_REVID=142668172
* Continued TestStrategy refactoring.Gravatar Ulf Adams2016-12-16
| | | | | | | | | More renaming and some reformatting to make StandaloneTestStrategy more closely resemble the internal implementation of TestStrategy. -- PiperOrigin-RevId: 142254302 MOS_MIGRATED_REVID=142254302
* Move TestStrategy to lib.exec package.Gravatar Ulf Adams2016-12-15
| | | | | | | | | | | | | | | | This is part of refactoring test strategy to unify its implementation between Bazel and Blaze (Google's internal version of Bazel), which should fix several issues in Bazel. It's also necessary to untangle lib.rules and lib.exec to enforce proper layering by separating compilation of these packages, and to provide a minimal Bazel binary. In particular, no core part of Bazel should depend on any of the rules, to facilitate moving them out of Bazel / reimplementing them in Skylark (except for some core rules like test_suite, alias, and genquery). -- PiperOrigin-RevId: 142151901 MOS_MIGRATED_REVID=142151901
* Initial code for Persistent Java Test Runner.Gravatar Kush Chakraborty2016-12-09
| | | | | | | | | | | | At this point this does nothing more than re-run the exact same test without having to re-start the test runner. In future iterations the aim is to be able to re-run tests with modified code, without having to re-start the test runner. To test out the WorkerTestStrategy simply use --test_strategy=experimental_worker for a test with bazel. -- PiperOrigin-RevId: 141465929 MOS_MIGRATED_REVID=141465929
* If a worker returns unparseable output, print it to the terminal.Gravatar Philipp Wollermann2016-12-06
| | | | | | | | When a persistent worker returns a WorkResponse that cannot be parsed by protobuf, it probably means that the worker has encountered an error and mistakenly printed a stack trace or error message to stdout. For users who want to know why their compiler crashes and for developers of tools that support the worker mode, it is extremely helpful to actually see these error messages, so let's print them. -- PiperOrigin-RevId: 141176835 MOS_MIGRATED_REVID=141176835
* Refactor the module API to use the builder pattern for executor creation.Gravatar Ulf Adams2016-11-02
| | | | | | | This significantly simplifies several of our modules. -- MOS_MIGRATED_REVID=137713119
* Rollback of commit e6191f7209ebf3f67480bc7aa9db5ac58beb2156.Gravatar Damien Martin-Guillerez2016-10-19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Roll-forward with fix. Tested with `bazel build src:srcs //src/test/...` using a bootstrapped bazel. Fixes #1923. RELNOTES[INC]: Non test-only targets can no longer depends on test-only targets. *** Original change description *** Automated [] rollback of commit a9f20b0d6459d395444c45cf5e94a899f3443633. *** Reason for rollback *** Broke Bazel CI: ERROR: /home/ci/workspace/Bazel/JAVA_VERSION/1.8/PLATFORM_NAME/linux-x86_64/src/BUILD:284:1: in filegroup rule //src:srcs: non-test target '//src:srcs' depends on testonly target '//src/java_tools/junitrunner/java/com/google/testing/junit/junit4:srcs' and doesn't have testonly attribute set. ERROR: Analysis of target '//scripts/packages:packages' failed; build aborted. http://ci.bazel.io/view/Bazel%20b... *** -- MOS_MIGRATED_REVID=136589937
* Add support for sandboxing persistent workers.Gravatar Philipp Wollermann2016-09-28
| | | | | -- MOS_MIGRATED_REVID=134523222
* Start worker processes with a clean environment, instead of accidentally ↵Gravatar Philipp Wollermann2016-07-28
| | | | | | | inheriting a copy of the server's environment. -- MOS_MIGRATED_REVID=128584538
* Use byte[] rather than ByteString for file digests.Gravatar Googler2016-07-20
| | | | | | | | | | | | | | | | | ActionInputFileCache: Change getDigest() to return the underlying byte[16] owned by each FileArtifactValue. Remove throws clause from getInputFromDigest(); this should be an in-memory operation, and no implementation actually throws. PerActionFileCache: Invert mapping from artifact to digest only if needed. Remove interner, as it was used only for the reverse map keys, not the returned values. This should be a significant cpu savings as eagerly constructing the reverse maps was a noticeable hotspot. -- MOS_MIGRATED_REVID=127972359
* --Gravatar Luis Fernando Pino Duque2016-07-20
| | | | MOS_MIGRATED_REVID=127927495
* workers: WorkerSpawnStrategy will now only execute SpawnActions if their ↵Gravatar Philipp Wollermann2016-07-19
| | | | | | | | | | | ExecutionInfo contains the tag "support-workers" set to "1" and fallback to non-worker execution if it is not present. This will eventually allow us to safely automatically decide whether to use workers to execute an action or not. RELNOTES[INC]: If you maintain a rule that uses persistent workers, you'll have to specify execution_requirements={"supports-workers": 1} in the ctx.action that intends to run a tool with workers. The WorkerSpawnStrategy will alert you with a warning message if you forget to make this change and fallback to non-worker based execution. -- MOS_MIGRATED_REVID=127822788
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-10 01:34:39 +0000