aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/exec
Commit message (Collapse)AuthorAge
* More SpawnResult-related plumbing changes to Actions, Strategies, ↵Gravatar ruperts2017-10-07
| | | | | | | | | ActionContexts, etc., so that SpawnResult metadata is returned upwards. Note that the TODOs mostly refer to changes that will appear in a subsequent CL (a CL to return SpawnResults, contained in ActionResults, from Actions/AbstractActions). I split off the remaining SpawnResult-related changes into this CL and kept the ActionResult-related changes separate. RELNOTES: None. PiperOrigin-RevId: 171355611
* Fix more linter errorsGravatar ruperts2017-10-06
| | | | | RELNOTES: None. PiperOrigin-RevId: 170955877
* Extract Fileset manifest parsing out of SpawnInputExpanderGravatar ulfjack2017-10-02
| | | | | | | | | | - add a mode to control how to handle relative symlinks - if set, attempt to resolve relative symlinks in manifests by looking for another entry that defines a file with the intended symlink target (we don't do recursive resolution for now) - add more test coverage; fix a bug in handling empty target locations PiperOrigin-RevId: 170691492
* Ensure that SpawnResults make their way back to the SpawnActions that caused ↵Gravatar ruperts2017-09-27
| | | | | | | them to be created. RELNOTES: None. PiperOrigin-RevId: 170058295
* BEP: Add coverage data to test_resultGravatar Klaus Aehlig2017-09-26
| | | | | | | | Include the coverage data file into the list of files repeorted in the test_result event in the BEP. Change-Id: Ia7b653addd5589268ce919b3e0349371dbc18bf2 PiperOrigin-RevId: 169956864
* Move SpawnResult from build.lib.exec into build.lib.actions so that e.g. ↵Gravatar ruperts2017-09-22
| | | | | | | build.lib.actions.SpawnActionContext can import SpawnResult without creating a cyclic dependency. RELNOTES: None. PiperOrigin-RevId: 169642267
* Include all test output files (e.g. undeclared outputs) for non-cached test ↵Gravatar Googler2017-09-22
| | | | | | | | | | | | | | executions and tests with multiple attempts. Previously, non-cached test execution would incorrectly only include test.log and test.xml. Implement this by unifying test output list building in a static TestResult method. For additional output files in multi-attempt tests, we follow the existing convention: Existing example: test.xml => test_attempts/attempt_N.xml New example (all new outputs done similarly): test.outputs/outputs.zip => test_attempts/attempt_N.outputs/outputs.zip RELNOTES: All test output files included for cached, uncached, and multiple attempt tests. PiperOrigin-RevId: 169556608
* Only compute hostname once per server lifetimeGravatar hlopko2017-09-14
| | | | | | | | | | | | NetUtil.getShortHostName can take seconds on mac and on windows (like, 20!), since it performs reverse dns lookup. We already cached hostname for the BazelWorkspaceStatusModule, let's cache it for entire bazel server. Also make sure that users of the method understand it's cached. Fixes #3586. RELNOTES: None. PiperOrigin-RevId: 168691615
* More BUILD file refactorings.Gravatar philwo2017-09-06
| | | | | | | | | Split collect, concurrent, vfs, windows into package-level BUILD files. Move clock classes out of "util", into their own Java package. Move CompactHashSet into its own Java package to break a dependency cycle. Give nestedset and inmemoryfs their own package-level BUILD files. PiperOrigin-RevId: 167702127
* Rename all logger instances to "logger" (instead "LOG" or "log").Gravatar lberki2017-09-05
| | | | | RELNOTES: None. PiperOrigin-RevId: 167505493
* remote: Return exit code 34 for remote caching/execution errors.Gravatar buchgr2017-09-01
| | | | | | | | | | For any errors that are due to failures in the remote caching / execution layers Bazel now returns exit code 34 (ExitCode.REMOTE_ERROR). This includes errors where the remote cache / executor is unreachable or crashes. It does not include errors if the test / build failure is due to user errors i.e. compilation or test failures. PiperOrigin-RevId: 167259236
* In terse test_summary, skip no-status testsGravatar Klaus Aehlig2017-08-30
| | | | | | | | | If there is a build failure, don't clobber the terse test summary by naming all the (usually many) tests that were skipped due to this failure. Change-Id: I6daae3efb1594c2b1018f87a50cf63949a34535b PiperOrigin-RevId: 166983264
* Split "shell", "graph" into their own packages.Gravatar philwo2017-08-29
| | | | PiperOrigin-RevId: 166849610
* Allow CommandLine expansion to throw an exception.Gravatar tomlu2017-08-24
| | | | | | | | | | | | | | | | | | | This paves the way for Skylark-side compact command lines that can fail during expansion. In general, expansion happens in these scenarios: * Action execution expands the command line to execute it. This is fine since we are well equipped already to handle failing actions. * In the analysis phase we expand command lines to investigate whether we need a params file. This could be moved to the execution phase, which would have the benefit of getting params files out of the action graph and saving memory. * Key computation expands the command line. This could be fixed by allowing command lines to compute their own keys (which wouldn't try to expand the command line). This could have the benefit of being more efficient. * Extra actions expand the command line to construct the extra action proto. This could maybe be deferred to the execution phase (writing the extra action), which would also be more memory efficient. For failed key computations, we return a singleton "error" key. This means multiple actions that will fail will map to the same key. These actions will necessarily fail if executed, so we should not need to worry about these ending up in the action cache. If we do manage to make the command line compute its own keys then this will become moot in the future. RELNOTES: None PiperOrigin-RevId: 166266385
* Introduce a new SpawnCache API; add a RemoteSpawnCache implementationGravatar ulfjack2017-08-14
| | | | | | | | | | AbstractSpawnRunner now uses a SpawnCache if one is registered, this allows adding caching to any spawn runner without having to be aware of the implementations. I will delete the old CachedLocalSpawnRunner in a follow-up CL. PiperOrigin-RevId: 165024382
* Refactor persistent workers to use SpawnRunner.Gravatar Benjamin Peterson2017-08-11
| | | | | | | | | | | | | | | | | | | | | | | | | | Change the persistent worker spawn strategy to extend AbstractSpawnStrategy and put the actual logic into WorkerSpawnRunner. WorkerTestStrategy is unaffected. I had to extend SpawnPolicy with a speculating() method. Persistent workers need to know if speculation is happening in order to require sandboxing. Additionally, I added java_test rules for the local runner tests and worker tests. See https://github.com/bazelbuild/bazel/issues/3481. NOTE: ulfjack@ made some changes to this change before merging: - changed Reporter to EventHandler; added TODO about its usage - reverted non-semantic indentation change in AbstractSpawnStrategy - reverted a non-semantic indentation change in WorkerSpawnRunner - updated some internal classes to match - removed catch IOException in WorkerSpawnRunner in some cases, removed verboseFailures flag from WorkerSpawnRunner, updated callers - disable some tests on Windows; we were previously not running them, now that we do, they fail :-( Change-Id: I207b3938f0dc84d374ab052d5030020886451d47 PiperOrigin-RevId: 164965398
* AbstractSpawnStrategy: use ActionExecutionContext.getVerboseFailuresGravatar ulfjack2017-08-11
| | | | | | | ...instead of injecting it through the constructor. Simplify all the callers accordingly. PiperOrigin-RevId: 164955391
* Move OutputService to lib.skyframeGravatar ulfjack2017-08-11
| | | | | | | | | | | This isn't ideal, but is the fastest way to split analysis, execution, and rules into separate java libraries, with lib.skyframe still being in the analysis library. Ideally, we'd split up the lib.skyframe package, move the analysis stuff to the analysis library, the execution stuff to the execution library, and so on. That wouldn't require us moving OutputService out of lib.exec. PiperOrigin-RevId: 164856998
* Move core test classes to lib.analysis.testGravatar ulfjack2017-08-11
| | | | | | | | These are depended upon by analysis code, so need to live in the same library as lib.analysis. Moving them here makes it possible to split the build-base library into separate libraries for analysis, execution, and rules. PiperOrigin-RevId: 164847161
* Move anonymous inner class to nested classGravatar ulfjack2017-08-11
| | | | PiperOrigin-RevId: 164844520
* Unify input prefetchingGravatar ulfjack2017-08-10
| | | | | | | All prefetching now goes through AbstractSpawnStrategy's implementation of SpawnExecutionPolicy. Make sure the sandbox runners also do this consistently. PiperOrigin-RevId: 164836877
* Rewrite the Command APIGravatar ulfjack2017-08-10
| | | | | | | | | | | | | | | | | | | | | | | | Important: the simplified API now defaults to forwarding interrupts to subprocesses. I did audit all the call sites, and I think this is a safe change to make. - Properly support timeouts with all implementations - Simplify the API - only provide two flavours of blocking calls, which require no input and forward interrupts; this is the most common usage - provide a number of async calls, which optionally takes input, and a flag whether to forward interrupts - only support input streams, no byte arrays or other 'convenience features' that are rarely needed and unnecessarily increase the surface area - use java.time.Duration to specify timeout; for consistency, interpret a timeout of <= 0 as no timeout (i.e., including rather than excluding 0) - KillableObserver and subclasses are no longer part of the public API, but still used to implement timeouts if the Subprocess.Factory does not support them - Update the documentation for Command - Update all callers; most callers now use the simplified API PiperOrigin-RevId: 164716782
* Use java.time.Duration for timeoutsGravatar ulfjack2017-08-09
| | | | PiperOrigin-RevId: 164577062
* Slight refactor of helper methods in TestStrategy.Gravatar nharmata2017-07-31
| | | | | RELNOTES: None PiperOrigin-RevId: 163471182
* Fixing #3380: output stack traces and informative messages.Gravatar olaola2017-07-27
| | | | | | | | Also added tests specifically for the output, to ensure we don't break it again. TESTED=remote worker, unit tests RELNOTES: fixes #3380 PiperOrigin-RevId: 163283558
* Extract a common AbstractSpawnStrategy parent classGravatar ulfjack2017-07-24
| | | | | | This removes a bunch of code duplication that I previously introduced. PiperOrigin-RevId: 162909430
* SpawnResult: allow signaling catastrophic errors, and compute detail messageGravatar ulfjack2017-07-24
| | | | PiperOrigin-RevId: 162906926
* Move ActionInputPrefetcher to the actions packageGravatar ulfjack2017-07-21
| | | | | | The plan is to add it to ActionExecutionContext, which is also there. PiperOrigin-RevId: 162656835
* Make the @Option annotation depend on the java version of the tagging enums.Gravatar ccalvarin2017-07-18
| | | | | | | The option filters proto dependency can be removed from the OptionsParser. This is in response to option parser users that want to avoid the bazel-internal proto file in their dependencies. RELNOTES: None. PiperOrigin-RevId: 162249778
* Extend the SpawnRunner APIGravatar ulfjack2017-07-17
| | | | | | | | | | | | | - add an id for logging; this allows us to correlate log entries for the same spawn from multiple spawn runner implementations in the future - add a prefetch method to the SpawnExecutionPolicy; better than relying on the ActionInputPrefetcher being injected in the constructor - add a name parameter to the report method; this is in preparation for a single unified SpawnStrategy implementation - it's basically the last bit of difference between SandboxStrategy and RemoteSpawnStrategy; they're otherwise equivalent (if not identical) PiperOrigin-RevId: 162194684
* Simplify exception handling in spawn strategiesGravatar ulfjack2017-07-17
| | | | | | | | | | | | | | | | | | | | | | | | | The main change here is to only catch SpawnExecException in StandaloneTestStrategy, so all other exceptions simplify propagate up. As a result, Bazel no longer retries tests that fail with an exception, we only retry tests that actually ran, had a spawn result, and resulted in a UserExecException. That is probably what we want. Also do some cleanup: - Remove ExecException.timedOut; nobody was calling it (but there's still SpawnExecException.timedOut) - Remove SpawnActionContext.shouldPropagateExecException; all exceptions (except SpawnExecException) are now propagated by default - Remote the SandboxOptions from the SandboxStrategies; all sandboxing options are now handled by the underlying SpawnRunner implementations I'll send a followup CL to remove the UserExecException and EnvironmentalExecException types; the types don't do anything special, and there are no catch blocks in production code that catch one of these more specific types. This should fix #3322 by removing a bunch of special handling. PiperOrigin-RevId: 161960919
* LocalSpawnRunner now always logs as itselfGravatar ulfjack2017-07-14
| | | | PiperOrigin-RevId: 161944209
* remote: Don't cache test if marked "external". Fixes #3362Gravatar buchgr2017-07-14
| | | | | RELNOTES: None. PiperOrigin-RevId: 161937673
* Rewrite all the sandbox strategy implementationsGravatar ulfjack2017-07-12
| | | | | | | | | | | | | | - Make use of existing abstractions like SpawnRunner and SpawnExecutionPolicy. - Instead of having the *Strategy create a *Runner, and then call back into SandboxStrategy, create a single SandboxContainer which contains the full command line, environment, and everything needed to create and delete the sandbox directory. - Do all the work in SandboxStrategy, including creation and deletion of the sandbox directory. - Use SpawnResult instead of throwing, catching, and rethrowing. - Simplify the control flow a bit. PiperOrigin-RevId: 161644979
* Fold OptionUsageRestrictions into OptionDocumentationCategory and ↵Gravatar ccalvarin2017-07-11
| | | | | | | | | | OptionMetadataTags. These are similar, no need to have both fields. Removing the "DOCUMENTED" default, the absence of UNDOCUMENTED will be used instead. Since requiring a documentation category for undocumented options doesn't make sense, list that as one of the OptionDocumentationCategories, but list HIDDEN and INTERNAL as part of OptionMetadata. These options should list UNDOCUMENTED as their category. PiperOrigin-RevId: 161515674
* Pass an EventHandler to OutputService.startBuildGravatar ulfjack2017-07-11
| | | | PiperOrigin-RevId: 161512574
* BEP: Report test warningsGravatar aehlig2017-07-10
| | | | | | | | Also report test warnings, if any, for completed test actions in the build event protocol. RELNOTES: None PiperOrigin-RevId: 161384061
* Cache flaky tests (remove the cachable flag from test result data)Gravatar ulfjack2017-07-07
| | | | | | | | | | | | | | | | | | | The cachable flag was only set to false for flaky tests. Before this CL, we did not cache flaky tests on subsequent runs, instead rerunning them, even though this is both very expensive and inconsistent with our normal handling, which is to cache passes but not errors. If cache_test_results is enabled, we now also cached timed out test results. If cache_test_results=auto (the default), we now also cache flaky tests. We still do not cache failed tests; the TestRunnerAction checks if the previous test result was marked as passed (which also applies to flaky tests, but not to failed or timed-out tests). Also add some unit tests. PiperOrigin-RevId: 161187950
* Simplify the ActionInputFileCacheGravatar ulfjack2017-07-06
| | | | | | | | | Add a single getMetadata method (matching MetadataHandler), and rewrite everything in those terms. This is in preparation for merging ActionInputFileCache and MetadataHandler. PiperOrigin-RevId: 161053535
* Add ActionEnvironment, which is a wrapper class for a fixed+inherited envGravatar ulfjack2017-06-28
| | | | | | | | | | | | | | | | We are currently tracking the action environment (computed from --action_env), and the test action environment (computed from --test_env, and not including the action env) as two pairs of fields, and a lot of callers haven't been updated to handle both parts (TestRunnerAction does, but many 'normal' actions don't). However, both parts have always both be handled, and moving them into a single abstraction makes it harder to accidentally miss one part. We'll subsequently need to update all the action implementations to use the new abstraction, and also update the Skylark API (or bypass it and always apply the action environment for all actions, except we don't know which configuration to use). PiperOrigin-RevId: 160386181
* Add the default category and tag to all options.Gravatar ccalvarin2017-06-28
| | | | | | | | Move the default from the annotation to every mention. This makes the incompleteness explicit. Will add the defaults to test targets in a separate change. Once all dependencies are cleaned up, the Option annotation will no longer allow options without the documentationCategory or effectTag, to prevent new options being added without categories while we migrate to the new option categorization. PiperOrigin-RevId: 160281252
* Use getopt to parse process-wrapper's command-line.Gravatar philwo2017-06-22
| | | | | | | | This will allow us to add new and optional flags like selecting a strategy used to spawn / wait for the child process. No one except Bazel should be calling "process-wrapper" and I couldn't find any references, so this breaking change should be fine. PiperOrigin-RevId: 159685867
* Extract ActionContext to a top-level classGravatar ulfjack2017-06-19
| | | | PiperOrigin-RevId: 159423459
* Rewrite the Executor/ActionExecutionContext splitGravatar ulfjack2017-06-19
| | | | | | | Move everything to ActionExecutionContext, and drop Executor whereever possible. This clarifies the API, makes it simpler to test, and simplifies the code. PiperOrigin-RevId: 159414816
* Rewrite StandaloneSpawnStrategy to use LocalSpawnRunnerGravatar ulfjack2017-06-19
| | | | PiperOrigin-RevId: 159221067
* Fix local execution to not throw an exceptionGravatar ulfjack2017-06-14
| | | | | | | | The SpawnInputExpander returns null for empty files, but the ActionInputPrefetcher does not expect null values, and implementations may throw NPE. PiperOrigin-RevId: 158960425
* Extract the MacOS/XCode env rewrite logic into lib.exec.appleGravatar ulfjack2017-06-13
| | | | | | | | | | | | | | | | | | | Also add an interface to allow injecting that logic into LocalSpawnRunner; this is in preparation for rewriting StandaloneSpawnStrategy to use LocalSpawnRunner. At the same time, this reduces the dependencies from exec / standalone to rules.apple, which is a prerequisite for micro-Bazel. There's a small semantic change hidden here - we now only set the new XCodeLocalEnvProvider if we're actually running on Darwin, so we no longer fail execution on non-Darwin platforms if XCODE_VERSION_OVERRIDE or APPLE_SDK_VERSION_OVERRIDE is set. As a result, I moved the corresponding test from StandaloneSpawnStrategyTest to the new XCodeLocalEnvProviderTest. While I'm at it, also open source DottedVersionTest and CacheManagerTest. PiperOrigin-RevId: 158829077
* Track the test environment in Skyframe, like the action environmentGravatar ulfjack2017-06-09
| | | | | | | | Instead of passing a client env into the test strategies, use the same mechansim as --action_env, by depending on the right set of Skyframe nodes that correspond to client env entries. PiperOrigin-RevId: 158401670
* Add some tests to the LocalSpawnRunner, and make some small tweaksGravatar ulfjack2017-06-08
| | | | | | | | | - Only manually check the timeout if the process wrapper is not used - Set the timeout correctly; the process API uses milliseconds - flush the error output stream after writing - return SIGALRM as exit code for timeout cases PiperOrigin-RevId: 158374246
* Switch to correct runfiles directory when running python test on WindowsGravatar Yun Peng2017-06-08
| | | | | | | | | | | | | | | | | py_test cannot find it's data file at runtime. The reason is it's running in a wrong directory. The test directory is set to runfiles directory by test-setup.sh, but on Windows, python binary unzip itself to another temp directory which test-setup.sh doesn't know. So let the python stub template switch to the correct runfiles directory if RUN_UNDER_RUNFILES = 1 Fixed https://github.com/bazelbuild/bazel/issues/3134 Change-Id: If5dbee811330372d86484ebd871ea55d84bc29a8 PiperOrigin-RevId: 158299041
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-08 15:00:38 +0000