aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/exec
Commit message (Collapse)AuthorAge
* Remove unused Digest class.Gravatar Googler2018-03-22
| | | | | RELNOTES: None. PiperOrigin-RevId: 190079798
* Begins cleanup to allow ActionFS to be injected into all action executions.Gravatar shahan2018-03-15
| | | | PiperOrigin-RevId: 189244665
* Replace instances of XCode with Xcode which is the proper casing for the name.Gravatar Sergio Campama2018-03-06
| | | | | | Closes #4640. PiperOrigin-RevId: 188022228
* Also get build-runfiles as an ActionInput for the symlink tree spawnGravatar ulfjack2018-03-06
| | | | | | | This isn't strictly necessary since we disable caching and require local execution. PiperOrigin-RevId: 187985476
* Adding a property name to the SpawnRunner. Most runners already had it, I ↵Gravatar olaola2018-02-22
| | | | | | | | | | just add it to the interface, and include it in the SpawnResult. This will be used to categorize/aggregate spawns executed by various runners. Also, minor refinement to the cacheHit property of the SpawnResult with remote execution. RELNOTES: None TESTED=presubmit, next cl PiperOrigin-RevId: 186637978
* Clean up test tag handlingGravatar ulfjack2018-02-21
| | | | PiperOrigin-RevId: 186427907
* Update --flaky_test_attempts to also take a repeatable argument of the formGravatar jcater2018-02-20
| | | | | | | | | regex@attempts, similarly to --runs_per_test. Also re-arrange some option converters to be closer to their options. RELNOTES: flaky_test_attempts supports the regex@attempts syntax, like runs_per_test. PiperOrigin-RevId: 186358437
* Inject the product name in the constructor where neededGravatar ulfjack2018-02-20
| | | | | | ...instead of passing it everywhere so we can rewrite the local env. PiperOrigin-RevId: 186316859
* Provide a hacky way to get an ActionInput from BinToolsGravatar ulfjack2018-02-20
| | | | | | | | | | | | We have two places in internal code that require an ActionInput for an embedded tool. The embedded tool is currently not declared as an input to the corresponding actions, which in turn causes the remote execution to stat the file directly through SingleBuildFileCache. I'm working on removing SingleBuildFileCache, so this needs to be solved. This is the quick-and-dirty fix. A more principled fix might be to check in the relevant tools instead of embedding them into Bazel, but it's also significantly more work. PiperOrigin-RevId: 186292203
* Automated rollback of commit 04757dba0174d22c0a695a7ed5fe511fd13df008.Gravatar laszlocsomor2018-02-16
| | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** There's already a --test_tmpdir flag, and Java tests don't pick up this new one. More info: https://github.com/bazelbuild/bazel/issues/4621#issuecomment-366217321 *** Original change description *** tmpdir,local-exec: implement --local_tmp_root Add new flag called `--local_tmp_root`, which (if specified) tells Bazel what temp directory should locally executed actions use. Fixes https://github.com/bazelbuild/bazel/issues/4621 Related to https://github.com/bazelbuild/bazel/issues/3215 RELNOTES[NEW]: The new "--local_tmp_root=<path>" flag allows specifying the temp directory for locally executed actions. Change-Id: Ice69a5e63d0bf4d3b5c9ef4dbdd1ed1c5025f85e PiperOrigin-RevId: 185982705
* Move BinTools to lib.execGravatar ulfjack2018-02-15
| | | | | | | It's never used during analysis, it shouldn't be used during analysis, so let's not make it available during analysis. PiperOrigin-RevId: 185808384
* tmpdir,local-exec: implement --local_tmp_rootGravatar Laszlo Csomor2018-02-13
| | | | | | | | | | | | | | Add new flag called `--local_tmp_root`, which (if specified) tells Bazel what temp directory should locally executed actions use. Fixes https://github.com/bazelbuild/bazel/issues/4621 Related to https://github.com/bazelbuild/bazel/issues/3215 RELNOTES[NEW]: The new "--local_tmp_root=<path>" flag allows specifying the temp directory for locally executed actions. Change-Id: Ice69a5e63d0bf4d3b5c9ef4dbdd1ed1c5025f85e PiperOrigin-RevId: 185509555
* Remove a few unused fields.Gravatar lberki2018-02-09
| | | | | RELNOTES: None. PiperOrigin-RevId: 185138928
* Remove hard-wired basenames for the test setup and the coverage collection ↵Gravatar lberki2018-02-09
| | | | | | | | | script. Accomplished by creating an explicit attribute for both of them on test actions. RELNOTES: None. PiperOrigin-RevId: 185132460
* Give a reasonable environment for tests run by "blaze run --direct_run".Gravatar lberki2018-02-08
| | | | | | | | | | | | It's not entirely correct, but almost. The code in RunCommand becomes somewhat more confusing. Cleanup change incoming. Fixes #2815. RELNOTES[INC]: "blaze run --direct_run" with tests now gives the test an approximation of the official test environment. PiperOrigin-RevId: 184992651
* BEP: ensure that TestAttempts are reported after their targetGravatar Klaus Aehlig2018-02-07
| | | | | | | | | | Ensure that each test attempt is only reported after the report of the completion of the build of the corresponding test target. Normally this should happen anyway, but due to races on the internal event bus, the order of the report might be messed up. So add an explicit order constraint. Change-Id: I4d325bc31a46dcdf8763ba5416b5135a0978536e PiperOrigin-RevId: 184825306
* AbstractSpawnStrategy should not crash when executing actions with a null owner.Gravatar tomlu2018-02-01
| | | | | RELNOTES: None PiperOrigin-RevId: 184127422
* Move BazelRuleClassProvider.TESTING_SUPPORT to its own class in rules/test.Gravatar lpino2018-01-29
| | | | PiperOrigin-RevId: 183672444
* Fix usages of PathFragment segments that will become inefficient.Gravatar tomlu2018-01-19
| | | | | | | An upcoming replacement to PathFragment will not have efficient segment semantics, causing code to become unnecessarily inefficient. RELNOTES: None PiperOrigin-RevId: 182553098
* Prevent broken cache entries on concurrent file changesGravatar ulfjack2018-01-19
| | | | | | | | | | | | | | | | Local execution has an inherent race condition: if a user modifies a file while an action is executed, then it is impossible for Bazel to tell which version of the file was actually read during action execution. The file may have been modified before or after the tool has read it, or, in the worst case, the tool may have read both the original and the modified version. In addition, the file may be changed back to the original state before Bazel can check the file, so computing the digest before / after may not be sufficient. This is a concern for both local and remote caches, although the cost of poisoning a shared remote cache is significantly higher, and is what has triggered this work. Fixes #3360. We solve this by keeping a reference to the FileContentsProxy, and using that to check for modificaitons before storing the cache entry. We output a warning if this check fails. This change does not increase memory consumption; Java objects are always allocated in multiples of 8 bytes, we use compressed oops, and the FileArtifactValue currently has 12 bytes worth of fields (excl. object overhead), so adding another pointer is effectively free. As a possible performance optimization on purely local builds, we could also consider not computing digests at all, and only use the FileContentsProxy for caching. PiperOrigin-RevId: 182510358
* actions,temp: respect TMPDIR envvarGravatar Laszlo Csomor2018-01-15
| | | | | | | Fixes https://github.com/bazelbuild/bazel/issues/4376 Change-Id: Id78bb0930044626304e54f07735db4d4b2c84720 PiperOrigin-RevId: 181959528
* Plumb exec root through to all spawn runners.Gravatar tomlu2018-01-11
| | | | | | They need this to parse input manifests. Previously we would grab the exec root from the Root, but wish to unsupport this. PiperOrigin-RevId: 181669143
* Adjust semantics of Metadata interface, remove isFile (use getType)Gravatar ulfjack2018-01-10
| | | | | | | | After some consideration, I think it makes sense to always allow a getDigest call, instead of specifying it as disallowed based on type. This is a follow-up CL for a previous CL introducing the getType method, which increased the complexity of the specification. I have a follow-up CL, which is related, namely unknown commit. After that CL, Metadata instances for directories (but not Filesets) also have digests rather than using mtime, which is compatible with the documentation changes made here. Said CL is solving a correctness issue with directory dependencies, which I think we want, and using the digest in the Metadata is a natural way to get correct action cache lookups. PiperOrigin-RevId: 181440548
* Use EmptyActionInput instead of null in SpawnInputExpanderGravatar ulfjack2018-01-08
| | | | | | | | This simplifies some spawn runners, which no longer have to specially handle null; unfortunately, the sandbox runners do not support VirtualActionInput, so they still have to special-case it. PiperOrigin-RevId: 181175408
* Extract LocalEnvProvider members to classes.Gravatar Laszlo Csomor2018-01-04
| | | | | | | | | | | | | | | | | | Create a PosixLocalEnvProvider and WindowsLocalEnvProvider class, with singleton instances for now. This refactoring should not change functionality, it's just a requirement for an upcoming change. That upcoming change is for these classes to respect the client environment's TMPDIR or TMP/TEMP envvars. See https://github.com/bazelbuild/bazel/issues/4376 Change-Id: I032bb6f18adf8af9e43e6bc543c09c58adae3863 PiperOrigin-RevId: 180799936
* Rework implementation of --nobuild_runfile_manifests.Gravatar Benjamin Peterson2018-01-03
| | | | | | | | | | | | | When --nobuild_runfile_manifests is passed, don't create runfiles input or output manifests at all. This seems better than creating fake manifest artifacts that are actually a middleman. Fail fast for local tests and the run command when --nobuild_runfiles_manifests is passed. (These cases were failing with obscure errors before under --nobuild_runfile_manifests-I just improved the messaging. See https://github.com/bazelbuild/bazel/issues/4177.) Change-Id: I351d26f746ecbe47016b58e4662768a5b6a72ff2 PiperOrigin-RevId: 180659571
* Enable local action execution statistics collection for sandboxed actions ↵Gravatar ruperts2017-12-22
| | | | | | | | | | | | | that use either the LinuxSandboxedSpawnRunner or the ProcessWrapperSandboxedSpawnRunner. In particular, record metrics for user and system CPU execution time, block I/O and involuntary context switches. This feature is guarded behind a new option, --experimental_collect_local_sandbox_action_metrics. Note: We still need to enable execution statistics for the DarwinSandboxedSpawnRunner in a later change. RELNOTES: None. PiperOrigin-RevId: 179976217
* Move compulsory ProcessWrapperUtil parameters into its builder() method, to ↵Gravatar ruperts2017-12-20
| | | | | | | be consistent with LinuxSandboxUtil. RELNOTES: None. PiperOrigin-RevId: 179758847
* Remove some unused codeGravatar ulfjack2017-12-19
| | | | PiperOrigin-RevId: 179542482
* Use java.nio.file.Files.createTempDirectory() to create temporary directory ↵Gravatar ruperts2017-12-13
| | | | | | | | | | | | | | | | | in LocalSpawnRunner, to avoid a race condition. Aside: The old, real temporary directory paths looked like this: TMPDIR=/.../tmp15e_5dd5a8e8347813f5 The new, real temporary directory paths now look like this: TMPDIR=/.../local-spawn-runner.3217503035718074040 RELNOTES: None. PiperOrigin-RevId: 178903361
* Enable local action execution statistics collection when the ↵Gravatar ruperts2017-12-12
| | | | | | | | | | | LocalSpawnRunner uses the process-wrapper to run commands. In particular, record metrics for user and system CPU execution time, block I/O and involuntary context switches. This feature is guarded behind a new option, --experimental_collect_local_action_metrics. RELNOTES: None. PiperOrigin-RevId: 178856077
* Fix: uploading artifacts of failed actions to remote cache stopped working.Gravatar olaola2017-12-11
| | | | | | | | | | | | | | | | To reproduce: run a failing test with --experimental_remote_spawn_cache or with --spawn_strategy=remote and no executor. Expected: test log is uploaded. Desired behavior: - regardless of whether a spawn is cacheable or not, its artifacts should be uploaded to the remote cache. - the spawn result should only be set if the spawn is cacheable *and* the action succeeded. - when executing remotely, the do_not_cache field should be set for non-cacheable spawns, and the remote execution engine should respect it. This CL contains multiple fixes to ensure the above behaviors, and adds a few tests, both end to end and unit tests. Important behavior change: it is no longer assumed that non-cacheable spawns should use a NO_CACHE SpawnCache! The appropriate test case was removed. Instead, an assumption was added that all implementations of SpawnCache should respect the Spawns.mayBeCached(spawn) property. Currently, only NO_CACHE and RemoteSpawnCache exist, and they (now) support it. TESTED=remote build execution backend. WANT_LGTM: philwo,buchgr RELNOTES: None PiperOrigin-RevId: 178617937
* If a temporary directory name clashes in the LocalSpawnRunner, try ↵Gravatar ruperts2017-12-06
| | | | | | | generating another temporary directory name instead of throwing an exception. RELNOTES: None. PiperOrigin-RevId: 178190769
* remote: don't hide non-test failures behind test failures. Fixes #4082Gravatar buchgr2017-11-30
| | | | | | | | | | Bazel should display the root cause of a test failure to the user. For example, if a test could not be executed on a remote executor due to there being no network connection, then it shouldn't display the test as failed but tell the user about the network error. RELNOTES: PiperOrigin-RevId: 177439578
* Successful SpawnResult is not a user error.Gravatar buchgr2017-11-30
| | | | PiperOrigin-RevId: 177428407
* Add a CommandLineBuilder for the process-wrapper embedded tool, and use it ↵Gravatar ruperts2017-11-30
| | | | | | | | | everywhere instead of duplicating process-wrapper --shell_arguments in Blaze. To avoid a cyclic dependency, I broke up exec/local:local into exec/local:local and exec/local:options. RELNOTES: None. PiperOrigin-RevId: 177419268
* Add user and system time to CommandResults, and plumb them into SpawnResults.Gravatar ruperts2017-11-29
| | | | | RELNOTES: None PiperOrigin-RevId: 177290508
* Clean up ExecutionRequirementsGravatar ulfjack2017-11-29
| | | | | | | | | | | | | | | | | | | - remove BaseSpawn.Local; instead, all callers pass in the full set of execution requirements they want to set - disable caching and sandboxing for the symlink tree action - it does not declare outputs, so it can't be cached or sandboxed (fixes #4041) - centralize the existing execution requirements in the ExecutionRequirements class - centralize checking for execution requirements in the Spawn class (it's possible that we may need a more decentralized, extensible design in the future, but for now having them in a single place is simple and effective) - update the documentation - forward the relevant tags to execution requirements in TargetUtils (progress on #3960) - this also contributes to #4153 PiperOrigin-RevId: 177288598
* Simplify SpawnRunner interfaceGravatar ulfjack2017-11-28
| | | | | | | | | | | | | | It turns out that the SUCCESS status is often misunderstood to mean "zero exit", even though this is clearly documented. I've decided to add another status for non-zero exit, and use success only for zero exit to avoid this pitfall. Also, many of the status codes are set, but never used. I decided to reduce the number of status codes to only those that are actually relevant, which simplifies further processing. Instead, we should add a string message for the error case when we need one - we're not using it right now, so I decided not to add that yet. PiperOrigin-RevId: 177129441
* Return SpawnResults using a List instead of a Set.Gravatar ruperts2017-11-21
| | | | | | | | | Currently we don't care about the list order of SpawnResults. However, we may care about the list order later. Also, if the equals() method for SpawnResults is ever changed then it may be problematic to return SpawnResults in a Set. Aside: ActionResults use SpawnResults to calculate cumulative execution times for Actions, and may provide other metrics in future. RELNOTES: None. PiperOrigin-RevId: 176579460
* Add a random number to action temp dirGravatar Changming Sun2017-11-20
| | | | | | | | | | Fix for #4035 @laszlocsomor Closes #4110. PiperOrigin-RevId: 176346381
* Replace all usages of Blaze's Preconditions class with guava.Gravatar tomlu2017-11-09
| | | | | | | | Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own. This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually. PiperOrigin-RevId: 175033526
* Add optional user execution time and system execution time fields to ↵Gravatar ruperts2017-11-06
| | | | | | | TerminationStatus, and also add a TerminationStatus.Builder and tests. RELNOTES: None. PiperOrigin-RevId: 174557303
* Use Durations to store wall, user and system execution time in SpawnResults, ↵Gravatar ruperts2017-11-06
| | | | | | | and make cumulative execution times available in ActionResults. RELNOTES: None PiperOrigin-RevId: 174553272
* Thread FileSystem through to a single Path#delete call site.Gravatar tomlu2017-10-30
| | | | | | This requires a fairly large amount of changes to fundamental objects like BlazeRuntime, Executor, and so on, as well as changing a lot of test code to thread the file system through. I expect future CLs to be much smaller. PiperOrigin-RevId: 173678144
* Fixing #3834, making sure test.log always exists.Gravatar olaola2017-10-18
| | | | | | | | Even if the test action produced no output, which it really shouldn't, Bazel should create an empty test.log file. TESTED=unit tests RELNOTES: Fixes #3834 PiperOrigin-RevId: 172412615
* Actions now have a temp envvar.Gravatar László Csomor2017-10-16
| | | | | | | | | | | | | | | | | | | | | Every build and test action that creates a Spawn will now have platform-specific environment variables for temp directories: - on Windows: TMP and TEMP - on Linux/Darwin: TMPDIR This is particularly important on Windows where e.g. Java programs cannot create temp directories unless there's a valid TMP or TEMP environment variable set. Fixes: - https://github.com/bazelbuild/bazel/issues/1590 - https://github.com/bazelbuild/bazel/issues/2349 - https://github.com/bazelbuild/bazel/issues/2870 Change-Id: Ib758307daf6b3a51b0f71ae5e65e5bb564dad643 PiperOrigin-RevId: 172326371
* Removing the USER variable from test environment, unless explicitly provided.Gravatar olaola2017-10-16
| | | | | | TESTED=added remote test RELNOTES: Fixes #2574 PiperOrigin-RevId: 172294781
* Simplify the SpawnExecException constructorGravatar ulfjack2017-10-16
| | | | | | | Whether or not there was a catastrophic error is stored in the SpawnResult, so we can just use that instead of passing in an additional boolean. PiperOrigin-RevId: 172083752
* In PerActionFileCache, tolerate requests for Artifacts that are not in the ↵Gravatar janakr2017-10-10
| | | | | | cache if we're doing input discovery: input discovery may find new artifacts we don't yet know about. Similarly, in SingleBuildFileCache, use the Artifact's path if it's available, rather than the poor man's route of the execRoot. PiperOrigin-RevId: 171635892
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-01 14:31:38 +0000