aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/exec
Commit message (Collapse)AuthorAge
* Add easy access to MessageDigests in DigestHashFunctionGravatar ccalvarin2018-08-13
| | | | | | | Move the message-digest cloning to DigestHashFunction and out of Fingerprint, to make it possible to configure Fingerprint to use different hash functions. We keep the default MD5 for now, we'd like it to use the global default but want to isolate the configuration change from any change adding potential contention. RELNOTES: None. PiperOrigin-RevId: 208502993
* Pass Fileset information to the Action filesystem.Gravatar felly2018-08-09
| | | | PiperOrigin-RevId: 208096548
* Automated rollback of commit 186b887e5862c1502010f097e75bfd8d605620b0.Gravatar twerth2018-08-07
| | | | | | *** Reason for rollback *** PiperOrigin-RevId: 207697096
* remote: add support for directory inputs in runfilesGravatar buchgr2018-08-02
| | | | | | | | | | | | | | | | Add support for tree artifacts (ctx.action.declare_directory(...)) in runfiles. Before this change we would throw away the information about the files inside a tree artifact before executing an action. That's fine for local execution where the sandbox just copies/symlinks a directory and doesn't care much what's inside. However, in remote execution we actually need to upload each individual file and so we need to be aware of all individual files not just directories. This change makes it so that this information is made available to a SpawnRunner via the SpawnInputExpander. RELNOTES: None PiperOrigin-RevId: 207091668
* Improve artifact->owner label accounting in two ways. First, don't do the ↵Gravatar janakr2018-07-30
| | | | | | full mapping unless requested. This gets rid of any performance issue for the vast majority of builds. Second, if requested, use a custom data structure so that we don't have to create a full HashSet for artifacts whose only owning labels are their own owner labels. PiperOrigin-RevId: 206610370
* Make the fallback strategy for Bazel's remote execution configurable.Gravatar philwo2018-07-30
| | | | | | | | | RELNOTES: When using Bazel's remote execution feature and Bazel has to fallback to local execution for an action, Bazel used non-sandboxed local execution until now. From this release on, you can use the new flag --remote_local_fallback_strategy=<strategy> to tell Bazel which strategy to use in that case. PiperOrigin-RevId: 206566380
* Add a flag to split test.xml generation into a separate SpawnGravatar ulfjack2018-07-27
| | | | | | | | | | At this time, this is only implemented for the StandaloneTestStrategy. This solves a race condition on Posix-like systems, where we cannot guarantee that the pipes are actually fully flushed to disk when the test process exits, and this can cause the test.xml to be empty, which makes it hard to debug issues. (The test.log files do not show up in normal CI systems, only the test.xml files.) Progress on #4608. PiperOrigin-RevId: 206292179
* Modify --subcommands to allow pretty printing the arguments of a subcommandGravatar ahumesky2018-07-26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | as a list, rather than as a single line (i.e., newline delimited rather than space delimited). Before: SUBCOMMAND: # //src/main/java/com/google/devtools/build/lib:string_util [action 'Building src/main/java/com/google/devtools/build/lib/libstring_util.jar (2 source files) [for host]'] (cd /tmp/devbazel_output_base/execroot/io_bazel && \ exec env - \ LC_CTYPE=en_US.UTF-8 \ external/embedded_jdk/bin/java -XX:+UseParallelOldGC -XX:-CompactStrings '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--patch-module=java.compiler=external/bazel_tools/third_party/java/jdk/langtools/java_compiler.jar' '--patch-module=jdk.compiler=external/bazel_tools/third_party/java/jdk/langtools/jdk_compiler.jar' '--add-opens=java.base/java.nio=ALL-UNNAMED' -jar external/bazel_tools/tools/jdk/JavaBuilder_deploy.jar @bazel-out/host/bin/src/main/java/com/google/devtools/build/lib/libstring_util.jar-2.params) After: SUBCOMMAND: # //src/main/java/com/google/devtools/build/lib:string_util [action 'Building src/main/java/com/google/devtools/build/lib/libstring_util.jar (2 source files) [for host]'] (cd /tmp/devbazel_output_base/execroot/io_bazel && \ exec env - \ LC_CTYPE=en_US.UTF-8 \ external/embedded_jdk/bin/java \ -XX:+UseParallelOldGC \ -XX:-CompactStrings \ '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED' \ '--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' \ '--add-opens=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' \ '--patch-module=java.compiler=external/bazel_tools/third_party/java/jdk/langtools/java_compiler.jar' \ '--patch-module=jdk.compiler=external/bazel_tools/third_party/java/jdk/langtools/jdk_compiler.jar' \ '--add-opens=java.base/java.nio=ALL-UNNAMED' \ -jar \ external/bazel_tools/tools/jdk/JavaBuilder_deploy.jar \ @bazel-out/host/bin/src/main/java/com/google/devtools/build/lib/libstring_util.jar-2.params) RELNOTES: --subcommands can now take a "pretty_print" value ("--subcommands=pretty_print") to print the arguments of subcommands as a list for easier reading. PiperOrigin-RevId: 206213009
* Weave the artifact path resolver through the Spawn execution context.Gravatar felly2018-07-25
| | | | | RELNOTES: None PiperOrigin-RevId: 206063489
* Fileset manifests now propagate Artifact metadata.Gravatar felly2018-07-23
| | | | PiperOrigin-RevId: 205682761
* For all top-level artifacts, track the labels that own them when that is ↵Gravatar janakr2018-07-13
| | | | | | | | available. The owning labels are the labels of the top-level configured targets that requested this artifact to be built (there may be many such targets). In cases where the artifact is added not through a configured target (build-info artifacts and coverage artifacts), the label of the artifact's owner is used. PiperOrigin-RevId: 204432951
* Make the FileArtifactvalue of generated targets available via the ↵Gravatar kush2018-07-11
| | | | | | | | | | | | | | FilesetOutputSymlink whenever available. In this change I'm simply plumbing the FileArtifactValue we requested within RecursiveFilesystemTraversalFunction to the FilesetOutputSymlink. This does not work when the targets are output directories (or symlink to output dirs). The main scenarios this happens is when: 1. Fileset depends on the output dir created by a genrule. 2. Fileset depends on a GoAppengineBinary which creates an output dir. 3. Fileset depends on another Fileset in the non-recommended way (Fileset.entry.files = [<another_fileset>]) instead of the recommended way (FilesetEntry.srcdir = <another_fileset>). RELNOTES: None PiperOrigin-RevId: 204209612
* Omit parts of BEP events (like stdout, stderr) when the associated artifact ↵Gravatar tomlu2018-07-11
| | | | | | | uploader cannot upload a particular file. RELNOTES: None PiperOrigin-RevId: 204167372
* Bazel server, VFS: revert to using asByteSourceGravatar laszlocsomor2018-07-10
| | | | | | | | | | | | | | | | | | commit 59f17d6e0550bf63a0b6ef182e2d63474e058ede updated files to use try-with-resources when dealing with streams. The change also got rid of asByteSource, replacing it with throw-away ByteSource instances wrapping a try-with-resources-guarded InputStream. Doing so was unnecessary though, because all ByteSource methods (except for open[Buffered]InputStream) close the stream. Thanks to @jbduncan to point that out and explain in detail [1]. [1] see comment thread on https://github.com/bazelbuild/bazel/commit/59f17d6e0550bf63a0b6ef182e2d63474e058ede under `FilesetManifest.java` RELNOTES: none PiperOrigin-RevId: 203934830
* Replace instances of Blaze with Bazel.Gravatar Dan Fabulich2018-07-05
| | | | | | | PiperOrigin-RevId: 203300374 Change-Id: Iaa47f870ab2e0cad40a202aad2c7f9430f73c856 PiperOrigin-RevId: 203407392
* Bazel server: ensure InputStreams are closedGravatar laszlocsomor2018-07-05
| | | | | | | | | | | | | | | | | | | Use try-with-resources to ensure InputStreams that we open via FileSystem.InputStream(path) are closed. Eagerly closing InputStreams avoids hanging on to file handles until the garbage collector finalizes the InputStream, meaning Bazel on Windows (and other processes) can delete or mutate these files. Hopefully this avoids intermittent file deletion errors that sometimes occur on Windows. See https://github.com/bazelbuild/bazel/issues/5512 RELNOTES: none PiperOrigin-RevId: 203338148
* Include more information about build events' referenced local files.Gravatar tomlu2018-07-04
| | | | | | | Instead of just a path, events now include information about the type of file (output, source file, stdout/stderr, test logs, etc.). This information can be used by the uploaders to determine a) whether to upload, b) what kind of lease to give the files. RELNOTES: None PiperOrigin-RevId: 203285549
* Refactor TemplateExpansionAction so that various strategies can be used to ↵Gravatar kush2018-06-29
| | | | | | | | | implement it. Also clarify the behavior of the expand_template API in the presence of multiple-substitutions. RELNOTES: None PiperOrigin-RevId: 202719656
* Move HashFunction out of FileSystem, and turn it into a class, instead of an ↵Gravatar ccalvarin2018-06-21
| | | | | | | | | enum. Now that we aren't using enum names for the hash functions, we also accept the standard names, such as SHA-256. RELNOTES: None. PiperOrigin-RevId: 201624286
* Add --materialize_param_files option.Gravatar tomlu2018-06-19
| | | | | | | | | When set, any action parameter files are written locally upon action execution, even when the action is executed remotely. This is mainly useful for debugging. This option is effectively implied by --subcommands and --verbose_failures, as it is likely that the user is debugging actions when using these flags. RELNOTES: Add --materialize_param_files flag to write parameter files even when actions are executed remotely. PiperOrigin-RevId: 201225566
* Refactoring: uses OutputService for ActionFileSystem injectionGravatar shahan2018-06-18
| | | | | | Moves more Action-oriented from skyframe package, which has become very unwieldy, to action package. This is needed to avoid circular dependencies caused when build-base is needed for skyframe. PiperOrigin-RevId: 200996982
* Add a mechanism for build event protocol events to upload filesGravatar ulfjack2018-06-15
| | | | | | | | This should be a no-op, mostly replacing PathConverter with BuildEventArtifactUploader, since none of the implementations perform any upload yet. PiperOrigin-RevId: 200685325
* Inline some ActionContextProvider classes into their modulesGravatar ulfjack2018-06-11
| | | | | | | | | | | | | | | This also gets rid of some boilerplate. The ExecutionTool.addActionContext method has been around for a while, but is underused. There are still a few ActionContextProvider implementations left, which are implementing other functionality besides adding action contexts. As a side effect, this change reduces null build time with a hot server on linux by about a quarter. We were running the linux sandbox twice on every build, which takes about 70ms each (on my machine), with the total null build time around 300ms. PiperOrigin-RevId: 200045145
* Add AutoProfiler-like API to ProfilerGravatar ulfjack2018-06-11
| | | | | | - migrate all startTask/completeTask pairs to the new API PiperOrigin-RevId: 200038703
* Remove ActionContextConsumerGravatar ulfjack2018-06-11
| | | | | | | | Instead, add some simple APIs to ExecutorBuilder and inline all the previous subclasses into their corresponding modules. This removes a bunch of boilerplate. PiperOrigin-RevId: 200017162
* Unify path resolution codepaths.Gravatar felly2018-06-08
| | | | | RELNOTES: None PiperOrigin-RevId: 199880252
* Add status details to the protosGravatar ulfjack2018-06-08
| | | | | | | | Both for the build event stream proto, which contains the TestResult event, and to the test status proto which is used for caching. The new field may be populated in the future. PiperOrigin-RevId: 199846232
* Support basic test functionality in ActionFS.Gravatar felly2018-06-08
| | | | | | | | | ActionFS now allows output files to be created that do not correspond to known Artifacts. Note that tests exercise a greater gamut of filesystem functionality (deleting files, deleting directory trees, moving files, etc.) RELNOTES: None PiperOrigin-RevId: 199809069
* Deletes ActionInputFileCache.Gravatar shahan2018-06-07
| | | | PiperOrigin-RevId: 199732415
* Simplify TestAttempt interfaceGravatar ulfjack2018-06-07
| | | | | | | By always requiring a TestResultData instace, we simplify the callers, which already have to do all the work anyway. PiperOrigin-RevId: 199639965
* Refactoring: makes the code simpler by deleting Metadata and InputArtifactData.Gravatar shahan2018-06-06
| | | | | | (minor) ActionFS now implements MetadataProvider.getInput PiperOrigin-RevId: 199575194
* MetadataProvider now provides ActionInput lookup by exec path.Gravatar shahan2018-06-05
| | | | | | Adds a helper class, ActionInputMap to do this with minimal wrapping overhead. PiperOrigin-RevId: 199391251
* Improve error message slightlyGravatar ulfjack2018-06-04
| | | | PiperOrigin-RevId: 199118944
* Removes most ActionInputFileCache functionality.Gravatar shahan2018-06-01
| | | | | | Actual class to be removed in a later change. PiperOrigin-RevId: 198937695
* Allow passing the relative symlink resolution policy to the SpawnInputExpanderGravatar ulfjack2018-05-30
| | | | PiperOrigin-RevId: 198535546
* Remove special handling of SpawnActionContext in Executor/ActionExecContextGravatar ulfjack2018-05-22
| | | | | | | | Instead, internally look up the correct context by mnemonic. This simplifies all the callers. We still need a little bit of special casing when constructing the action context map. PiperOrigin-RevId: 197572357
* Changing SymlinkTreeStrategy not to use spawns.Gravatar Googler2018-05-18
| | | | | RELNOTES: An internal action for symlinking runfiles will use Command instead of a Spawns. This should have no functional chages; the only user visible consequence should be that the internal action is no longer be included in statistics when calculating processes count. PiperOrigin-RevId: 197131917
* Rename BuildEventConverters to BuildEventContextGravatar ulfjack2018-05-17
| | | | | | | This is in preparation for adding options to the class, which can be used to modify the behavior of build events posted to the BEP. PiperOrigin-RevId: 196997573
* test failure: print stacktrace on I/O errorGravatar Laszlo Csomor2018-05-16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Print the stacktrace of unexpected I/O errors in StandaloneTestStrategy, to ease diagnosing the root cause of the exception. See https://github.com/bazelbuild/bazel/issues/4924 Without the stacktrace, all we see in BuildKite is, for example: ``` ERROR: Caught I/O exception: java.io.IOException: C:/users/b/_bazel_b/bnp8s_vg/execroot/io_bazel/_tmp/b6bda2f0385d1152d3a7f550c6cc1938/_bazel_b/install/23a47abea50baae4d7e032437c1cecc9/_embedded_binaries/embedded_tools/jdk/bin/java.exe (Permission denied) ERROR: C:/build/buildkite-worker-windows-java8-lfl8-1/bazel/google-bazel-presubmit/src/test/py/bazel/BUILD:71:1: Couldn't build file src/test/py/bazel/bazel_windows_test/test.log: failed: unexpected I/O exception: C:/users/b/_bazel_b/bnp8s_vg/execroot/io_bazel/_tmp/b6bda2f0385d1152d3a7f550c6cc1938/_bazel_b/install/23a47abea50baae4d7e032437c1cecc9/_embedded_binaries/embedded_tools/jdk/bin/java.exe (Permission denied) ``` The above log contains no information on what exactly tried accessing java.exe and how, and why it failed. With a stacktrace I'm hoping to shed light on the culprit. Change-Id: I4f3851cd1bc1b2b348217de5b41069591a8f4446 Closes #5207. Change-Id: I792f4a36c1e31ca6332db2dc2d37bd8e597050b3 PiperOrigin-RevId: 196815410
* Refactor TestAttempt event posting codeGravatar ulfjack2018-05-14
| | | | | | | | For flaky tests, Bazel may have cached information about multiple test attempts. In that case, we might want to post all of them on a subsequent cache hit, rather than posting only the passing attempt. We currently subclass TestResult inside Google, which overrides the new getCachedTestAttempts method. PiperOrigin-RevId: 196491575
* Delete the FilesetActionContext and its ImplementationGravatar kush2018-05-04
| | | | | RELNOTES: None PiperOrigin-RevId: 195428899
* Kill Legacy Fileset implementation.Gravatar kush2018-05-04
| | | | | RELNOTES: None PiperOrigin-RevId: 195422399
* Do not spawn local actions when available memory is too lowGravatar Josh Pieper2018-05-04
| | | | | | | | | | | | This effectively subsumes all existing memory estimation, as it no longer matters too much how much memory is assigned to each action. WIP: This is a hack, and if it goes in, should almost certainly be hidden behind a configuration option. Closes #4938. PiperOrigin-RevId: 195378568
* Allow banning symlink action outputs from being uploaded to a remote cache.Gravatar Benjamin Peterson2018-05-03
| | | | | | | | | | | | | | | | | | | This is mostly a roll-forward of 4465dae23de989f1452e93d0a88ac2a289103dd9, which was reverted by fa36d2f48965b127e8fd397348d16e991135bfb6. The main difference is that the new behavior is now gated behind the --noremote_allow_symlink_upload flag. https://docs.google.com/document/d/1gnOYszitgrLVet3sQk-TKGqIcpkkDsc6aw-izoo-d64 is a design proposal to support symlinks in the remote cache, which would render this change moot. I'd like to be able to prevent incorrect cache behavior until that change is implemented, though. This fixes https://github.com/bazelbuild/bazel/issues/4840 (again). Closes #5122. Change-Id: I2136cfe82c2e1a8a9f5856e12a37d42cabd0e299 PiperOrigin-RevId: 195261827
* Report what RemoteSpawnCache is doing.Gravatar Benjamin Peterson2018-05-03
| | | | | | | | | | | | | | | | Post ProgressStatus.CHECKING_CACHE if RemoteSpawnCache is checking the cache. The UI sees CHECKING_CACHE exactly the same as EXECUTING because no UIs currently have any special behavior for actions in cache-lookup state. This is still a UX improvement with --experimental_spawn_cache because EXECUTING is generally more correct than the old action state, which varies from harmless but unhelpful (no known state) to just wrong (C++ compile actions claimed they were doing include scanning during cache lookups). Closes #5130. Change-Id: I77421c3667c180875216f937fe0713f0e9415a7a PiperOrigin-RevId: 195233123
* Print more debugging info when xcode-locator fails.Gravatar philwo2018-05-03
| | | | | | It wasn't very helpful to print only stderr, considering that xcode-locator only ever prints to stdout... PiperOrigin-RevId: 195213838
* Use the in-memory metadata in blaze as the source of truth for Fileset mappingsGravatar kush2018-05-02
| | | | | | | instead of the manifest files. RELNOTES: None PiperOrigin-RevId: 195149880
* Clean up code that directly imports nested classes like Builder, Entry, etc.Gravatar jcater2018-05-01
| | | | PiperOrigin-RevId: 194985259
* Support ParamFileActionInput in LocalSpawnRunner.Gravatar tomlu2018-04-30
| | | | | RELNOTES: None PiperOrigin-RevId: 194798051
* Remove the on-disk caching of Xcode locations.Gravatar jmmv2018-04-28
| | | | | | | | | | | | | | | | | | | | | | | There is no need for the cache to be on disk. Originally, there was a desire to share this cache with other tools... but this never happened. And, actually, because Bazel is in control of what it runs, it can just inject the "cached" values into those tools via flags. Instead, just store the cache in-memory. This avoids having to open and read the cache on every single action that is locally executed on a Mac. Results when building a large iOS app from a clean slate show up to a 1% wall time improvement on my Mac Pro 2013 and a reduction in the variance of the measurements. This change also gets rid of the OS check from the action execution's critical path. There is not much use in checking this: if we instantiate this by mistake, the actual calls will fail. But sometimes we want to actually run this code on non-macOS systems (e.g. for unit-testing with mocked tools), so we should allow that. And this change also ensures that XcodeLocalEnvProviderTest builds and runs... RELNOTES: None. PiperOrigin-RevId: 194681802
generated by cgit v1.2.3 (git 2.39.1) at 2024-06-17 03:46:08 +0000