aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/exec
Commit message (Collapse)AuthorAge
* Move BazelRuleClassProvider.TESTING_SUPPORT to its own class in rules/test.Gravatar lpino2018-01-29
| | | | PiperOrigin-RevId: 183672444
* Fix usages of PathFragment segments that will become inefficient.Gravatar tomlu2018-01-19
| | | | | | | An upcoming replacement to PathFragment will not have efficient segment semantics, causing code to become unnecessarily inefficient. RELNOTES: None PiperOrigin-RevId: 182553098
* Prevent broken cache entries on concurrent file changesGravatar ulfjack2018-01-19
| | | | | | | | | | | | | | | | Local execution has an inherent race condition: if a user modifies a file while an action is executed, then it is impossible for Bazel to tell which version of the file was actually read during action execution. The file may have been modified before or after the tool has read it, or, in the worst case, the tool may have read both the original and the modified version. In addition, the file may be changed back to the original state before Bazel can check the file, so computing the digest before / after may not be sufficient. This is a concern for both local and remote caches, although the cost of poisoning a shared remote cache is significantly higher, and is what has triggered this work. Fixes #3360. We solve this by keeping a reference to the FileContentsProxy, and using that to check for modificaitons before storing the cache entry. We output a warning if this check fails. This change does not increase memory consumption; Java objects are always allocated in multiples of 8 bytes, we use compressed oops, and the FileArtifactValue currently has 12 bytes worth of fields (excl. object overhead), so adding another pointer is effectively free. As a possible performance optimization on purely local builds, we could also consider not computing digests at all, and only use the FileContentsProxy for caching. PiperOrigin-RevId: 182510358
* actions,temp: respect TMPDIR envvarGravatar Laszlo Csomor2018-01-15
| | | | | | | Fixes https://github.com/bazelbuild/bazel/issues/4376 Change-Id: Id78bb0930044626304e54f07735db4d4b2c84720 PiperOrigin-RevId: 181959528
* Plumb exec root through to all spawn runners.Gravatar tomlu2018-01-11
| | | | | | They need this to parse input manifests. Previously we would grab the exec root from the Root, but wish to unsupport this. PiperOrigin-RevId: 181669143
* Adjust semantics of Metadata interface, remove isFile (use getType)Gravatar ulfjack2018-01-10
| | | | | | | | After some consideration, I think it makes sense to always allow a getDigest call, instead of specifying it as disallowed based on type. This is a follow-up CL for a previous CL introducing the getType method, which increased the complexity of the specification. I have a follow-up CL, which is related, namely unknown commit. After that CL, Metadata instances for directories (but not Filesets) also have digests rather than using mtime, which is compatible with the documentation changes made here. Said CL is solving a correctness issue with directory dependencies, which I think we want, and using the digest in the Metadata is a natural way to get correct action cache lookups. PiperOrigin-RevId: 181440548
* Use EmptyActionInput instead of null in SpawnInputExpanderGravatar ulfjack2018-01-08
| | | | | | | | This simplifies some spawn runners, which no longer have to specially handle null; unfortunately, the sandbox runners do not support VirtualActionInput, so they still have to special-case it. PiperOrigin-RevId: 181175408
* Extract LocalEnvProvider members to classes.Gravatar Laszlo Csomor2018-01-04
| | | | | | | | | | | | | | | | | | Create a PosixLocalEnvProvider and WindowsLocalEnvProvider class, with singleton instances for now. This refactoring should not change functionality, it's just a requirement for an upcoming change. That upcoming change is for these classes to respect the client environment's TMPDIR or TMP/TEMP envvars. See https://github.com/bazelbuild/bazel/issues/4376 Change-Id: I032bb6f18adf8af9e43e6bc543c09c58adae3863 PiperOrigin-RevId: 180799936
* Rework implementation of --nobuild_runfile_manifests.Gravatar Benjamin Peterson2018-01-03
| | | | | | | | | | | | | When --nobuild_runfile_manifests is passed, don't create runfiles input or output manifests at all. This seems better than creating fake manifest artifacts that are actually a middleman. Fail fast for local tests and the run command when --nobuild_runfiles_manifests is passed. (These cases were failing with obscure errors before under --nobuild_runfile_manifests-I just improved the messaging. See https://github.com/bazelbuild/bazel/issues/4177.) Change-Id: I351d26f746ecbe47016b58e4662768a5b6a72ff2 PiperOrigin-RevId: 180659571
* Enable local action execution statistics collection for sandboxed actions ↵Gravatar ruperts2017-12-22
| | | | | | | | | | | | | that use either the LinuxSandboxedSpawnRunner or the ProcessWrapperSandboxedSpawnRunner. In particular, record metrics for user and system CPU execution time, block I/O and involuntary context switches. This feature is guarded behind a new option, --experimental_collect_local_sandbox_action_metrics. Note: We still need to enable execution statistics for the DarwinSandboxedSpawnRunner in a later change. RELNOTES: None. PiperOrigin-RevId: 179976217
* Move compulsory ProcessWrapperUtil parameters into its builder() method, to ↵Gravatar ruperts2017-12-20
| | | | | | | be consistent with LinuxSandboxUtil. RELNOTES: None. PiperOrigin-RevId: 179758847
* Remove some unused codeGravatar ulfjack2017-12-19
| | | | PiperOrigin-RevId: 179542482
* Use java.nio.file.Files.createTempDirectory() to create temporary directory ↵Gravatar ruperts2017-12-13
| | | | | | | | | | | | | | | | | in LocalSpawnRunner, to avoid a race condition. Aside: The old, real temporary directory paths looked like this: TMPDIR=/.../tmp15e_5dd5a8e8347813f5 The new, real temporary directory paths now look like this: TMPDIR=/.../local-spawn-runner.3217503035718074040 RELNOTES: None. PiperOrigin-RevId: 178903361
* Enable local action execution statistics collection when the ↵Gravatar ruperts2017-12-12
| | | | | | | | | | | LocalSpawnRunner uses the process-wrapper to run commands. In particular, record metrics for user and system CPU execution time, block I/O and involuntary context switches. This feature is guarded behind a new option, --experimental_collect_local_action_metrics. RELNOTES: None. PiperOrigin-RevId: 178856077
* Fix: uploading artifacts of failed actions to remote cache stopped working.Gravatar olaola2017-12-11
| | | | | | | | | | | | | | | | To reproduce: run a failing test with --experimental_remote_spawn_cache or with --spawn_strategy=remote and no executor. Expected: test log is uploaded. Desired behavior: - regardless of whether a spawn is cacheable or not, its artifacts should be uploaded to the remote cache. - the spawn result should only be set if the spawn is cacheable *and* the action succeeded. - when executing remotely, the do_not_cache field should be set for non-cacheable spawns, and the remote execution engine should respect it. This CL contains multiple fixes to ensure the above behaviors, and adds a few tests, both end to end and unit tests. Important behavior change: it is no longer assumed that non-cacheable spawns should use a NO_CACHE SpawnCache! The appropriate test case was removed. Instead, an assumption was added that all implementations of SpawnCache should respect the Spawns.mayBeCached(spawn) property. Currently, only NO_CACHE and RemoteSpawnCache exist, and they (now) support it. TESTED=remote build execution backend. WANT_LGTM: philwo,buchgr RELNOTES: None PiperOrigin-RevId: 178617937
* If a temporary directory name clashes in the LocalSpawnRunner, try ↵Gravatar ruperts2017-12-06
| | | | | | | generating another temporary directory name instead of throwing an exception. RELNOTES: None. PiperOrigin-RevId: 178190769
* remote: don't hide non-test failures behind test failures. Fixes #4082Gravatar buchgr2017-11-30
| | | | | | | | | | Bazel should display the root cause of a test failure to the user. For example, if a test could not be executed on a remote executor due to there being no network connection, then it shouldn't display the test as failed but tell the user about the network error. RELNOTES: PiperOrigin-RevId: 177439578
* Successful SpawnResult is not a user error.Gravatar buchgr2017-11-30
| | | | PiperOrigin-RevId: 177428407
* Add a CommandLineBuilder for the process-wrapper embedded tool, and use it ↵Gravatar ruperts2017-11-30
| | | | | | | | | everywhere instead of duplicating process-wrapper --shell_arguments in Blaze. To avoid a cyclic dependency, I broke up exec/local:local into exec/local:local and exec/local:options. RELNOTES: None. PiperOrigin-RevId: 177419268
* Add user and system time to CommandResults, and plumb them into SpawnResults.Gravatar ruperts2017-11-29
| | | | | RELNOTES: None PiperOrigin-RevId: 177290508
* Clean up ExecutionRequirementsGravatar ulfjack2017-11-29
| | | | | | | | | | | | | | | | | | | - remove BaseSpawn.Local; instead, all callers pass in the full set of execution requirements they want to set - disable caching and sandboxing for the symlink tree action - it does not declare outputs, so it can't be cached or sandboxed (fixes #4041) - centralize the existing execution requirements in the ExecutionRequirements class - centralize checking for execution requirements in the Spawn class (it's possible that we may need a more decentralized, extensible design in the future, but for now having them in a single place is simple and effective) - update the documentation - forward the relevant tags to execution requirements in TargetUtils (progress on #3960) - this also contributes to #4153 PiperOrigin-RevId: 177288598
* Simplify SpawnRunner interfaceGravatar ulfjack2017-11-28
| | | | | | | | | | | | | | It turns out that the SUCCESS status is often misunderstood to mean "zero exit", even though this is clearly documented. I've decided to add another status for non-zero exit, and use success only for zero exit to avoid this pitfall. Also, many of the status codes are set, but never used. I decided to reduce the number of status codes to only those that are actually relevant, which simplifies further processing. Instead, we should add a string message for the error case when we need one - we're not using it right now, so I decided not to add that yet. PiperOrigin-RevId: 177129441
* Return SpawnResults using a List instead of a Set.Gravatar ruperts2017-11-21
| | | | | | | | | Currently we don't care about the list order of SpawnResults. However, we may care about the list order later. Also, if the equals() method for SpawnResults is ever changed then it may be problematic to return SpawnResults in a Set. Aside: ActionResults use SpawnResults to calculate cumulative execution times for Actions, and may provide other metrics in future. RELNOTES: None. PiperOrigin-RevId: 176579460
* Add a random number to action temp dirGravatar Changming Sun2017-11-20
| | | | | | | | | | Fix for #4035 @laszlocsomor Closes #4110. PiperOrigin-RevId: 176346381
* Replace all usages of Blaze's Preconditions class with guava.Gravatar tomlu2017-11-09
| | | | | | | | Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own. This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually. PiperOrigin-RevId: 175033526
* Add optional user execution time and system execution time fields to ↵Gravatar ruperts2017-11-06
| | | | | | | TerminationStatus, and also add a TerminationStatus.Builder and tests. RELNOTES: None. PiperOrigin-RevId: 174557303
* Use Durations to store wall, user and system execution time in SpawnResults, ↵Gravatar ruperts2017-11-06
| | | | | | | and make cumulative execution times available in ActionResults. RELNOTES: None PiperOrigin-RevId: 174553272
* Thread FileSystem through to a single Path#delete call site.Gravatar tomlu2017-10-30
| | | | | | This requires a fairly large amount of changes to fundamental objects like BlazeRuntime, Executor, and so on, as well as changing a lot of test code to thread the file system through. I expect future CLs to be much smaller. PiperOrigin-RevId: 173678144
* Fixing #3834, making sure test.log always exists.Gravatar olaola2017-10-18
| | | | | | | | Even if the test action produced no output, which it really shouldn't, Bazel should create an empty test.log file. TESTED=unit tests RELNOTES: Fixes #3834 PiperOrigin-RevId: 172412615
* Actions now have a temp envvar.Gravatar László Csomor2017-10-16
| | | | | | | | | | | | | | | | | | | | | Every build and test action that creates a Spawn will now have platform-specific environment variables for temp directories: - on Windows: TMP and TEMP - on Linux/Darwin: TMPDIR This is particularly important on Windows where e.g. Java programs cannot create temp directories unless there's a valid TMP or TEMP environment variable set. Fixes: - https://github.com/bazelbuild/bazel/issues/1590 - https://github.com/bazelbuild/bazel/issues/2349 - https://github.com/bazelbuild/bazel/issues/2870 Change-Id: Ib758307daf6b3a51b0f71ae5e65e5bb564dad643 PiperOrigin-RevId: 172326371
* Removing the USER variable from test environment, unless explicitly provided.Gravatar olaola2017-10-16
| | | | | | TESTED=added remote test RELNOTES: Fixes #2574 PiperOrigin-RevId: 172294781
* Simplify the SpawnExecException constructorGravatar ulfjack2017-10-16
| | | | | | | Whether or not there was a catastrophic error is stored in the SpawnResult, so we can just use that instead of passing in an additional boolean. PiperOrigin-RevId: 172083752
* In PerActionFileCache, tolerate requests for Artifacts that are not in the ↵Gravatar janakr2017-10-10
| | | | | | cache if we're doing input discovery: input discovery may find new artifacts we don't yet know about. Similarly, in SingleBuildFileCache, use the Artifact's path if it's available, rather than the poor man's route of the execRoot. PiperOrigin-RevId: 171635892
* More SpawnResult-related plumbing changes to Actions, Strategies, ↵Gravatar ruperts2017-10-07
| | | | | | | | | ActionContexts, etc., so that SpawnResult metadata is returned upwards. Note that the TODOs mostly refer to changes that will appear in a subsequent CL (a CL to return SpawnResults, contained in ActionResults, from Actions/AbstractActions). I split off the remaining SpawnResult-related changes into this CL and kept the ActionResult-related changes separate. RELNOTES: None. PiperOrigin-RevId: 171355611
* Fix more linter errorsGravatar ruperts2017-10-06
| | | | | RELNOTES: None. PiperOrigin-RevId: 170955877
* Extract Fileset manifest parsing out of SpawnInputExpanderGravatar ulfjack2017-10-02
| | | | | | | | | | - add a mode to control how to handle relative symlinks - if set, attempt to resolve relative symlinks in manifests by looking for another entry that defines a file with the intended symlink target (we don't do recursive resolution for now) - add more test coverage; fix a bug in handling empty target locations PiperOrigin-RevId: 170691492
* Ensure that SpawnResults make their way back to the SpawnActions that caused ↵Gravatar ruperts2017-09-27
| | | | | | | them to be created. RELNOTES: None. PiperOrigin-RevId: 170058295
* BEP: Add coverage data to test_resultGravatar Klaus Aehlig2017-09-26
| | | | | | | | Include the coverage data file into the list of files repeorted in the test_result event in the BEP. Change-Id: Ia7b653addd5589268ce919b3e0349371dbc18bf2 PiperOrigin-RevId: 169956864
* Move SpawnResult from build.lib.exec into build.lib.actions so that e.g. ↵Gravatar ruperts2017-09-22
| | | | | | | build.lib.actions.SpawnActionContext can import SpawnResult without creating a cyclic dependency. RELNOTES: None. PiperOrigin-RevId: 169642267
* Include all test output files (e.g. undeclared outputs) for non-cached test ↵Gravatar Googler2017-09-22
| | | | | | | | | | | | | | executions and tests with multiple attempts. Previously, non-cached test execution would incorrectly only include test.log and test.xml. Implement this by unifying test output list building in a static TestResult method. For additional output files in multi-attempt tests, we follow the existing convention: Existing example: test.xml => test_attempts/attempt_N.xml New example (all new outputs done similarly): test.outputs/outputs.zip => test_attempts/attempt_N.outputs/outputs.zip RELNOTES: All test output files included for cached, uncached, and multiple attempt tests. PiperOrigin-RevId: 169556608
* Only compute hostname once per server lifetimeGravatar hlopko2017-09-14
| | | | | | | | | | | | NetUtil.getShortHostName can take seconds on mac and on windows (like, 20!), since it performs reverse dns lookup. We already cached hostname for the BazelWorkspaceStatusModule, let's cache it for entire bazel server. Also make sure that users of the method understand it's cached. Fixes #3586. RELNOTES: None. PiperOrigin-RevId: 168691615
* More BUILD file refactorings.Gravatar philwo2017-09-06
| | | | | | | | | Split collect, concurrent, vfs, windows into package-level BUILD files. Move clock classes out of "util", into their own Java package. Move CompactHashSet into its own Java package to break a dependency cycle. Give nestedset and inmemoryfs their own package-level BUILD files. PiperOrigin-RevId: 167702127
* Rename all logger instances to "logger" (instead "LOG" or "log").Gravatar lberki2017-09-05
| | | | | RELNOTES: None. PiperOrigin-RevId: 167505493
* remote: Return exit code 34 for remote caching/execution errors.Gravatar buchgr2017-09-01
| | | | | | | | | | For any errors that are due to failures in the remote caching / execution layers Bazel now returns exit code 34 (ExitCode.REMOTE_ERROR). This includes errors where the remote cache / executor is unreachable or crashes. It does not include errors if the test / build failure is due to user errors i.e. compilation or test failures. PiperOrigin-RevId: 167259236
* In terse test_summary, skip no-status testsGravatar Klaus Aehlig2017-08-30
| | | | | | | | | If there is a build failure, don't clobber the terse test summary by naming all the (usually many) tests that were skipped due to this failure. Change-Id: I6daae3efb1594c2b1018f87a50cf63949a34535b PiperOrigin-RevId: 166983264
* Split "shell", "graph" into their own packages.Gravatar philwo2017-08-29
| | | | PiperOrigin-RevId: 166849610
* Allow CommandLine expansion to throw an exception.Gravatar tomlu2017-08-24
| | | | | | | | | | | | | | | | | | | This paves the way for Skylark-side compact command lines that can fail during expansion. In general, expansion happens in these scenarios: * Action execution expands the command line to execute it. This is fine since we are well equipped already to handle failing actions. * In the analysis phase we expand command lines to investigate whether we need a params file. This could be moved to the execution phase, which would have the benefit of getting params files out of the action graph and saving memory. * Key computation expands the command line. This could be fixed by allowing command lines to compute their own keys (which wouldn't try to expand the command line). This could have the benefit of being more efficient. * Extra actions expand the command line to construct the extra action proto. This could maybe be deferred to the execution phase (writing the extra action), which would also be more memory efficient. For failed key computations, we return a singleton "error" key. This means multiple actions that will fail will map to the same key. These actions will necessarily fail if executed, so we should not need to worry about these ending up in the action cache. If we do manage to make the command line compute its own keys then this will become moot in the future. RELNOTES: None PiperOrigin-RevId: 166266385
* Introduce a new SpawnCache API; add a RemoteSpawnCache implementationGravatar ulfjack2017-08-14
| | | | | | | | | | AbstractSpawnRunner now uses a SpawnCache if one is registered, this allows adding caching to any spawn runner without having to be aware of the implementations. I will delete the old CachedLocalSpawnRunner in a follow-up CL. PiperOrigin-RevId: 165024382
* Refactor persistent workers to use SpawnRunner.Gravatar Benjamin Peterson2017-08-11
| | | | | | | | | | | | | | | | | | | | | | | | | | Change the persistent worker spawn strategy to extend AbstractSpawnStrategy and put the actual logic into WorkerSpawnRunner. WorkerTestStrategy is unaffected. I had to extend SpawnPolicy with a speculating() method. Persistent workers need to know if speculation is happening in order to require sandboxing. Additionally, I added java_test rules for the local runner tests and worker tests. See https://github.com/bazelbuild/bazel/issues/3481. NOTE: ulfjack@ made some changes to this change before merging: - changed Reporter to EventHandler; added TODO about its usage - reverted non-semantic indentation change in AbstractSpawnStrategy - reverted a non-semantic indentation change in WorkerSpawnRunner - updated some internal classes to match - removed catch IOException in WorkerSpawnRunner in some cases, removed verboseFailures flag from WorkerSpawnRunner, updated callers - disable some tests on Windows; we were previously not running them, now that we do, they fail :-( Change-Id: I207b3938f0dc84d374ab052d5030020886451d47 PiperOrigin-RevId: 164965398
* AbstractSpawnStrategy: use ActionExecutionContext.getVerboseFailuresGravatar ulfjack2017-08-11
| | | | | | | ...instead of injecting it through the constructor. Simplify all the callers accordingly. PiperOrigin-RevId: 164955391
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-03 18:31:40 +0000