aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/exec
Commit message (Collapse)AuthorAge
* MetadataProvider now provides ActionInput lookup by exec path.Gravatar shahan2018-06-05
| | | | | | Adds a helper class, ActionInputMap to do this with minimal wrapping overhead. PiperOrigin-RevId: 199391251
* Improve error message slightlyGravatar ulfjack2018-06-04
| | | | PiperOrigin-RevId: 199118944
* Removes most ActionInputFileCache functionality.Gravatar shahan2018-06-01
| | | | | | Actual class to be removed in a later change. PiperOrigin-RevId: 198937695
* Allow passing the relative symlink resolution policy to the SpawnInputExpanderGravatar ulfjack2018-05-30
| | | | PiperOrigin-RevId: 198535546
* Remove special handling of SpawnActionContext in Executor/ActionExecContextGravatar ulfjack2018-05-22
| | | | | | | | Instead, internally look up the correct context by mnemonic. This simplifies all the callers. We still need a little bit of special casing when constructing the action context map. PiperOrigin-RevId: 197572357
* Changing SymlinkTreeStrategy not to use spawns.Gravatar Googler2018-05-18
| | | | | RELNOTES: An internal action for symlinking runfiles will use Command instead of a Spawns. This should have no functional chages; the only user visible consequence should be that the internal action is no longer be included in statistics when calculating processes count. PiperOrigin-RevId: 197131917
* Rename BuildEventConverters to BuildEventContextGravatar ulfjack2018-05-17
| | | | | | | This is in preparation for adding options to the class, which can be used to modify the behavior of build events posted to the BEP. PiperOrigin-RevId: 196997573
* test failure: print stacktrace on I/O errorGravatar Laszlo Csomor2018-05-16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Print the stacktrace of unexpected I/O errors in StandaloneTestStrategy, to ease diagnosing the root cause of the exception. See https://github.com/bazelbuild/bazel/issues/4924 Without the stacktrace, all we see in BuildKite is, for example: ``` ERROR: Caught I/O exception: java.io.IOException: C:/users/b/_bazel_b/bnp8s_vg/execroot/io_bazel/_tmp/b6bda2f0385d1152d3a7f550c6cc1938/_bazel_b/install/23a47abea50baae4d7e032437c1cecc9/_embedded_binaries/embedded_tools/jdk/bin/java.exe (Permission denied) ERROR: C:/build/buildkite-worker-windows-java8-lfl8-1/bazel/google-bazel-presubmit/src/test/py/bazel/BUILD:71:1: Couldn't build file src/test/py/bazel/bazel_windows_test/test.log: failed: unexpected I/O exception: C:/users/b/_bazel_b/bnp8s_vg/execroot/io_bazel/_tmp/b6bda2f0385d1152d3a7f550c6cc1938/_bazel_b/install/23a47abea50baae4d7e032437c1cecc9/_embedded_binaries/embedded_tools/jdk/bin/java.exe (Permission denied) ``` The above log contains no information on what exactly tried accessing java.exe and how, and why it failed. With a stacktrace I'm hoping to shed light on the culprit. Change-Id: I4f3851cd1bc1b2b348217de5b41069591a8f4446 Closes #5207. Change-Id: I792f4a36c1e31ca6332db2dc2d37bd8e597050b3 PiperOrigin-RevId: 196815410
* Refactor TestAttempt event posting codeGravatar ulfjack2018-05-14
| | | | | | | | For flaky tests, Bazel may have cached information about multiple test attempts. In that case, we might want to post all of them on a subsequent cache hit, rather than posting only the passing attempt. We currently subclass TestResult inside Google, which overrides the new getCachedTestAttempts method. PiperOrigin-RevId: 196491575
* Delete the FilesetActionContext and its ImplementationGravatar kush2018-05-04
| | | | | RELNOTES: None PiperOrigin-RevId: 195428899
* Kill Legacy Fileset implementation.Gravatar kush2018-05-04
| | | | | RELNOTES: None PiperOrigin-RevId: 195422399
* Do not spawn local actions when available memory is too lowGravatar Josh Pieper2018-05-04
| | | | | | | | | | | | This effectively subsumes all existing memory estimation, as it no longer matters too much how much memory is assigned to each action. WIP: This is a hack, and if it goes in, should almost certainly be hidden behind a configuration option. Closes #4938. PiperOrigin-RevId: 195378568
* Allow banning symlink action outputs from being uploaded to a remote cache.Gravatar Benjamin Peterson2018-05-03
| | | | | | | | | | | | | | | | | | | This is mostly a roll-forward of 4465dae23de989f1452e93d0a88ac2a289103dd9, which was reverted by fa36d2f48965b127e8fd397348d16e991135bfb6. The main difference is that the new behavior is now gated behind the --noremote_allow_symlink_upload flag. https://docs.google.com/document/d/1gnOYszitgrLVet3sQk-TKGqIcpkkDsc6aw-izoo-d64 is a design proposal to support symlinks in the remote cache, which would render this change moot. I'd like to be able to prevent incorrect cache behavior until that change is implemented, though. This fixes https://github.com/bazelbuild/bazel/issues/4840 (again). Closes #5122. Change-Id: I2136cfe82c2e1a8a9f5856e12a37d42cabd0e299 PiperOrigin-RevId: 195261827
* Report what RemoteSpawnCache is doing.Gravatar Benjamin Peterson2018-05-03
| | | | | | | | | | | | | | | | Post ProgressStatus.CHECKING_CACHE if RemoteSpawnCache is checking the cache. The UI sees CHECKING_CACHE exactly the same as EXECUTING because no UIs currently have any special behavior for actions in cache-lookup state. This is still a UX improvement with --experimental_spawn_cache because EXECUTING is generally more correct than the old action state, which varies from harmless but unhelpful (no known state) to just wrong (C++ compile actions claimed they were doing include scanning during cache lookups). Closes #5130. Change-Id: I77421c3667c180875216f937fe0713f0e9415a7a PiperOrigin-RevId: 195233123
* Print more debugging info when xcode-locator fails.Gravatar philwo2018-05-03
| | | | | | It wasn't very helpful to print only stderr, considering that xcode-locator only ever prints to stdout... PiperOrigin-RevId: 195213838
* Use the in-memory metadata in blaze as the source of truth for Fileset mappingsGravatar kush2018-05-02
| | | | | | | instead of the manifest files. RELNOTES: None PiperOrigin-RevId: 195149880
* Clean up code that directly imports nested classes like Builder, Entry, etc.Gravatar jcater2018-05-01
| | | | PiperOrigin-RevId: 194985259
* Support ParamFileActionInput in LocalSpawnRunner.Gravatar tomlu2018-04-30
| | | | | RELNOTES: None PiperOrigin-RevId: 194798051
* Remove the on-disk caching of Xcode locations.Gravatar jmmv2018-04-28
| | | | | | | | | | | | | | | | | | | | | | | There is no need for the cache to be on disk. Originally, there was a desire to share this cache with other tools... but this never happened. And, actually, because Bazel is in control of what it runs, it can just inject the "cached" values into those tools via flags. Instead, just store the cache in-memory. This avoids having to open and read the cache on every single action that is locally executed on a Mac. Results when building a large iOS app from a clean slate show up to a 1% wall time improvement on my Mac Pro 2013 and a reduction in the variance of the measurements. This change also gets rid of the OS check from the action execution's critical path. There is not much use in checking this: if we instantiate this by mistake, the actual calls will fail. But sometimes we want to actually run this code on non-macOS systems (e.g. for unit-testing with mocked tools), so we should allow that. And this change also ensures that XcodeLocalEnvProviderTest builds and runs... RELNOTES: None. PiperOrigin-RevId: 194681802
* Add execution information to standalone test results.Gravatar Benjamin Peterson2018-04-26
| | | | | | | | | | | | | | | In StandaloneTestStrategy, copy as much information as SpawnResult makes available to us through to both the TestResultData and BEP's TestResult.ExecutionInfo protos. One immediate consequence is that the UI and BEP can tell you whether a test result was cached remotely. I changed Executor.getEventHandler to return an ExtendedEventHandler because it makes this change easier to test. Closes #5081. Change-Id: I94fefdcd2e029c81085076736ad13a4bdf1bae8f PiperOrigin-RevId: 194383009
* SpawnAction.setShellCommand: expect shell pathGravatar laszlocsomor2018-04-23
| | | | | | | | | | | | | | | | | | | | | | | SpawnAction.setShellCommand(String) now expects the shell interpreter's path as an argument. This change enables two things: - rules can report an error if the shell is missing - SpawnAction no longer has to know about a default shell The new ShToolchain class will later also be responsible for retrieving the active shell toolchain (added in https://github.com/bazelbuild/bazel/commit/81ed3add408adb20bddbc3ba1818c65806738dc5). This change brings Bazel closer to not depend on the shell unless it has to (e.g. to run shell scripts). See https://github.com/bazelbuild/bazel/issues/4319 RELNOTES: none PiperOrigin-RevId: 193885943
* Logging the spawn graph on demand. First step on #4891.Gravatar olaola2018-04-19
| | | | | | | | This is a very heavy and slow option by design. It will be enabled only when a user wants to debug their build, most frequently to compare the step-by-step results of two builds. TESTED: manually on various rules, including directories. RELNOTES: None PiperOrigin-RevId: 193539034
* Rename SpawnExecutionPolicy -> SpawnExecutionContext.Gravatar tomlu2018-04-19
| | | | | | | | This class will be used to tie a Spawn to a SpawnRunner, and isn't really a policy object. It will carry state such as the expanded inputs and expanded command line. Currently a context can be passed between different SpawnRunners. This will be addressed independently, so a context is tied to a particular spawn runner. PiperOrigin-RevId: 193501918
* Automated rollback of commit 4465dae23de989f1452e93d0a88ac2a289103dd9.Gravatar buchgr2018-04-18
| | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** The no-cache tag is not respected (see b/77857812) and thus this breaks remote caching for all projects with symlink outputs. *** Original change description *** Only allow regular files and directories spawn outputs to be uploaded to a remote cache. The remote cache protocol only knows about regular files and directories. Currently, during action output upload, symlinks are resolved into regular files. This means cached "executions" of an action may have different output file types than the original execution, which can be a footgun. This CL bans symlinks from cachable spawn outputs and fixes http... *** PiperOrigin-RevId: 193338629
* A slight refactoring of RunCommand.Gravatar lberki2018-04-17
| | | | | RELNOTES: None. PiperOrigin-RevId: 193203414
* Move test_timeout to BuildConfiguration and TargetCompleteEvent.Gravatar Googler2018-04-17
| | | | | | | | - Move test_timeout to BuildConfiguration. - Add test_timeout_seconds field to BEP TargetCompleteEvent. - Deprecate test_timeout field in ExecutionInfo. Data is still written to deprecated field to allow consumer transition. PiperOrigin-RevId: 193192636
* Rollback https://github.com/bazelbuild/bazel/pull/5007Gravatar cparsons2018-04-16
| | | | | | | It breaks downstream rules_nodejs. See https://github.com/bazelbuild/bazel/issues/5028 for details. RELNOTES: None. PiperOrigin-RevId: 193074798
* CommandBuilder: remove useShell and setWorkingDirGravatar Laszlo Csomor2018-04-12
| | | | | | | | | | | | | | | | | | | | | Remove the .useShell method, expect callers to just pass the shell interpreter if they need it. This removes the argument vector transformation heuristic, and stops shelling out to cmd.exe on Windows. Also remove the .setWorkingDir method because callers always had to set the working directory. Instead, the CommandBuilder constructor takes the working directory. Closes #2190. (Test code still shells out to cmd.exe though.) Closes #5007. Change-Id: I545e01c811daaf34913cb585492923da81aa02ee PiperOrigin-RevId: 192611370
* Move shell executable to its own configuration fragment.Gravatar lberki2018-04-06
| | | | | RELNOTES: None. PiperOrigin-RevId: 191861074
* Change profiling to only accept strings for its "description" argument. ↵Gravatar janakr2018-04-01
| | | | | | Profiling can hold onto objects for the duration of the build, and some of those objects may be temporary that should not be persisted. In particular, UnixGlob and its inner classes should not outlive loading and analysis. For the most part, care was taken in this CL to only use strings that required no additional construction, mainly to minimize garbage (retaining references to newly created strings is not as great a concern since only the strings corresponding to the slowest K tasks are retained, for some relatively small values of K). Action descriptions for actually executing actions are eagerly expanded because that work is minimal compared to the work of actually executing an action. PiperOrigin-RevId: 191251488
* BEP: Cached test actions post execution info from TestResultGravatar ulfjack2018-03-30
| | | | | | Currently, the TestResult does not have an execution info, but that may change in the future. PiperOrigin-RevId: 191148241
* Use SpawnResult.getWallTime for test duration if availableGravatar ulfjack2018-03-28
| | | | | | | | Note that the wall time is not always set, especially for cached or remotely executed actions, so we keep the current code around for now. Progress on #4808. PiperOrigin-RevId: 190857756
* Remove categories from Bazel options.Gravatar ccalvarin2018-03-28
| | | | | | | | | These have all had a chance to be categorized with the OptionDocumentationCategory enum, and the help output already uses the enum-grouped format. The "incompatible changes" category has meaning for --all_incompatible_changes and will be removed separately. RELNOTES: None. PiperOrigin-RevId: 190773778
* Compute the list of test output files in TestRunnerActionGravatar ulfjack2018-03-27
| | | | | | This allows us to slightly simplify the renaming logic in StandaloneTestStrategy. PiperOrigin-RevId: 190645674
* Only allow regular files and directories spawn outputs to be uploaded to a ↵Gravatar Benjamin Peterson2018-03-27
| | | | | | | | | | | | | | | | | | | | remote cache. The remote cache protocol only knows about regular files and directories. Currently, during action output upload, symlinks are resolved into regular files. This means cached "executions" of an action may have different output file types than the original execution, which can be a footgun. This CL bans symlinks from cachable spawn outputs and fixes https://github.com/bazelbuild/bazel/issues/4840. The interface of SpawnCache.CacheHandle.store is refactored: 1. The outputs parameter is removed, since that can be retrieved from the underlying Spawn. 2. It can now throw ExecException in order to fail actions. Closes #4902. Change-Id: I0d1d94d48779b970bb5d0840c66a14c189ab0091 PiperOrigin-RevId: 190608852
* Simplify Executor.getSpawnActionContext() by removing mnemonic: spawn is enough.Gravatar Googler2018-03-26
| | | | | RELNOTES: None PiperOrigin-RevId: 190544948
* Simplified ActionContextConsumer by having it operate on a new class which ↵Gravatar Googler2018-03-26
| | | | | | holds a variety of strategy/context maps. PiperOrigin-RevId: 190491357
* Refactor and cleanup the sandboxing code.Gravatar Philipp Wollermann2018-03-23
| | | | | | | | | | | | | - Remove Optional<> where it's not needed. It's nice for return values, but IMHO it was overused in this code (e.g. Optional<List<X>> is an anti-pattern, as the list itself can already signal that it is empty). - Use Bazel's own Path class when dealing with paths, not String or java.io.File. - Move LinuxSandboxUtil into the "sandbox" package. - Remove dead code and unused fields. - Migrate deprecated VFS method calls to their replacements. - Fix a bug in ExecutionStatistics where a FileInputStream was not closed. Closes #4868. PiperOrigin-RevId: 190217476
* Remove unused Digest class.Gravatar Googler2018-03-22
| | | | | RELNOTES: None. PiperOrigin-RevId: 190079798
* Begins cleanup to allow ActionFS to be injected into all action executions.Gravatar shahan2018-03-15
| | | | PiperOrigin-RevId: 189244665
* Replace instances of XCode with Xcode which is the proper casing for the name.Gravatar Sergio Campama2018-03-06
| | | | | | Closes #4640. PiperOrigin-RevId: 188022228
* Also get build-runfiles as an ActionInput for the symlink tree spawnGravatar ulfjack2018-03-06
| | | | | | | This isn't strictly necessary since we disable caching and require local execution. PiperOrigin-RevId: 187985476
* Adding a property name to the SpawnRunner. Most runners already had it, I ↵Gravatar olaola2018-02-22
| | | | | | | | | | just add it to the interface, and include it in the SpawnResult. This will be used to categorize/aggregate spawns executed by various runners. Also, minor refinement to the cacheHit property of the SpawnResult with remote execution. RELNOTES: None TESTED=presubmit, next cl PiperOrigin-RevId: 186637978
* Clean up test tag handlingGravatar ulfjack2018-02-21
| | | | PiperOrigin-RevId: 186427907
* Update --flaky_test_attempts to also take a repeatable argument of the formGravatar jcater2018-02-20
| | | | | | | | | regex@attempts, similarly to --runs_per_test. Also re-arrange some option converters to be closer to their options. RELNOTES: flaky_test_attempts supports the regex@attempts syntax, like runs_per_test. PiperOrigin-RevId: 186358437
* Inject the product name in the constructor where neededGravatar ulfjack2018-02-20
| | | | | | ...instead of passing it everywhere so we can rewrite the local env. PiperOrigin-RevId: 186316859
* Provide a hacky way to get an ActionInput from BinToolsGravatar ulfjack2018-02-20
| | | | | | | | | | | | We have two places in internal code that require an ActionInput for an embedded tool. The embedded tool is currently not declared as an input to the corresponding actions, which in turn causes the remote execution to stat the file directly through SingleBuildFileCache. I'm working on removing SingleBuildFileCache, so this needs to be solved. This is the quick-and-dirty fix. A more principled fix might be to check in the relevant tools instead of embedding them into Bazel, but it's also significantly more work. PiperOrigin-RevId: 186292203
* Automated rollback of commit 04757dba0174d22c0a695a7ed5fe511fd13df008.Gravatar laszlocsomor2018-02-16
| | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** There's already a --test_tmpdir flag, and Java tests don't pick up this new one. More info: https://github.com/bazelbuild/bazel/issues/4621#issuecomment-366217321 *** Original change description *** tmpdir,local-exec: implement --local_tmp_root Add new flag called `--local_tmp_root`, which (if specified) tells Bazel what temp directory should locally executed actions use. Fixes https://github.com/bazelbuild/bazel/issues/4621 Related to https://github.com/bazelbuild/bazel/issues/3215 RELNOTES[NEW]: The new "--local_tmp_root=<path>" flag allows specifying the temp directory for locally executed actions. Change-Id: Ice69a5e63d0bf4d3b5c9ef4dbdd1ed1c5025f85e PiperOrigin-RevId: 185982705
* Move BinTools to lib.execGravatar ulfjack2018-02-15
| | | | | | | It's never used during analysis, it shouldn't be used during analysis, so let's not make it available during analysis. PiperOrigin-RevId: 185808384
* tmpdir,local-exec: implement --local_tmp_rootGravatar Laszlo Csomor2018-02-13
| | | | | | | | | | | | | | Add new flag called `--local_tmp_root`, which (if specified) tells Bazel what temp directory should locally executed actions use. Fixes https://github.com/bazelbuild/bazel/issues/4621 Related to https://github.com/bazelbuild/bazel/issues/3215 RELNOTES[NEW]: The new "--local_tmp_root=<path>" flag allows specifying the temp directory for locally executed actions. Change-Id: Ice69a5e63d0bf4d3b5c9ef4dbdd1ed1c5025f85e PiperOrigin-RevId: 185509555
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-03 20:27:33 +0000