| Commit message (Collapse) | Author | Age |
|
|
|
|
|
|
|
|
| |
When set, any action parameter files are written locally upon action execution, even when the action is executed remotely. This is mainly useful for debugging.
This option is effectively implied by --subcommands and --verbose_failures, as it is likely that the user is debugging actions when using these flags.
RELNOTES: Add --materialize_param_files flag to write parameter files even when actions are executed remotely.
PiperOrigin-RevId: 201225566
|
|
|
|
|
|
| |
Moves more Action-oriented from skyframe package, which has become very unwieldy, to action package. This is needed to avoid circular dependencies caused when build-base is needed for skyframe.
PiperOrigin-RevId: 200996982
|
|
|
|
|
|
|
|
| |
This should be a no-op, mostly replacing PathConverter with
BuildEventArtifactUploader, since none of the implementations perform any
upload yet.
PiperOrigin-RevId: 200685325
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This also gets rid of some boilerplate. The ExecutionTool.addActionContext
method has been around for a while, but is underused.
There are still a few ActionContextProvider implementations left, which are
implementing other functionality besides adding action contexts.
As a side effect, this change reduces null build time with a hot server on
linux by about a quarter. We were running the linux sandbox twice on every
build, which takes about 70ms each (on my machine), with the total null build
time around 300ms.
PiperOrigin-RevId: 200045145
|
|
|
|
|
|
| |
- migrate all startTask/completeTask pairs to the new API
PiperOrigin-RevId: 200038703
|
|
|
|
|
|
|
|
| |
Instead, add some simple APIs to ExecutorBuilder and inline all the previous
subclasses into their corresponding modules. This removes a bunch of
boilerplate.
PiperOrigin-RevId: 200017162
|
|
|
|
|
| |
RELNOTES: None
PiperOrigin-RevId: 199880252
|
|
|
|
|
|
|
|
| |
Both for the build event stream proto, which contains the TestResult event,
and to the test status proto which is used for caching. The new field may be
populated in the future.
PiperOrigin-RevId: 199846232
|
|
|
|
|
|
|
|
|
| |
ActionFS now allows output files to be created that do not correspond to known Artifacts.
Note that tests exercise a greater gamut of filesystem functionality (deleting files, deleting directory trees, moving files, etc.)
RELNOTES: None
PiperOrigin-RevId: 199809069
|
|
|
|
| |
PiperOrigin-RevId: 199732415
|
|
|
|
|
|
|
| |
By always requiring a TestResultData instace, we simplify the callers, which
already have to do all the work anyway.
PiperOrigin-RevId: 199639965
|
|
|
|
|
|
| |
(minor) ActionFS now implements MetadataProvider.getInput
PiperOrigin-RevId: 199575194
|
|
|
|
|
|
| |
Adds a helper class, ActionInputMap to do this with minimal wrapping overhead.
PiperOrigin-RevId: 199391251
|
|
|
|
| |
PiperOrigin-RevId: 199118944
|
|
|
|
|
|
| |
Actual class to be removed in a later change.
PiperOrigin-RevId: 198937695
|
|
|
|
| |
PiperOrigin-RevId: 198535546
|
|
|
|
|
|
|
|
| |
Instead, internally look up the correct context by mnemonic. This simplifies
all the callers. We still need a little bit of special casing when constructing
the action context map.
PiperOrigin-RevId: 197572357
|
|
|
|
|
| |
RELNOTES: An internal action for symlinking runfiles will use Command instead of a Spawns. This should have no functional chages; the only user visible consequence should be that the internal action is no longer be included in statistics when calculating processes count.
PiperOrigin-RevId: 197131917
|
|
|
|
|
|
|
| |
This is in preparation for adding options to the class, which can be used
to modify the behavior of build events posted to the BEP.
PiperOrigin-RevId: 196997573
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Print the stacktrace of unexpected I/O errors in
StandaloneTestStrategy, to ease diagnosing the
root cause of the exception.
See https://github.com/bazelbuild/bazel/issues/4924
Without the stacktrace, all we see in BuildKite
is, for example:
```
ERROR: Caught I/O exception: java.io.IOException: C:/users/b/_bazel_b/bnp8s_vg/execroot/io_bazel/_tmp/b6bda2f0385d1152d3a7f550c6cc1938/_bazel_b/install/23a47abea50baae4d7e032437c1cecc9/_embedded_binaries/embedded_tools/jdk/bin/java.exe (Permission denied)
ERROR: C:/build/buildkite-worker-windows-java8-lfl8-1/bazel/google-bazel-presubmit/src/test/py/bazel/BUILD:71:1: Couldn't build file src/test/py/bazel/bazel_windows_test/test.log: failed: unexpected I/O exception: C:/users/b/_bazel_b/bnp8s_vg/execroot/io_bazel/_tmp/b6bda2f0385d1152d3a7f550c6cc1938/_bazel_b/install/23a47abea50baae4d7e032437c1cecc9/_embedded_binaries/embedded_tools/jdk/bin/java.exe (Permission denied)
```
The above log contains no information on what
exactly tried accessing java.exe and how, and why
it failed. With a stacktrace I'm hoping to shed
light on the culprit.
Change-Id: I4f3851cd1bc1b2b348217de5b41069591a8f4446
Closes #5207.
Change-Id: I792f4a36c1e31ca6332db2dc2d37bd8e597050b3
PiperOrigin-RevId: 196815410
|
|
|
|
|
|
|
|
| |
For flaky tests, Bazel may have cached information about multiple test attempts. In that case, we might want to post all of them on a subsequent cache hit, rather than posting only the passing attempt.
We currently subclass TestResult inside Google, which overrides the new getCachedTestAttempts method.
PiperOrigin-RevId: 196491575
|
|
|
|
|
| |
RELNOTES: None
PiperOrigin-RevId: 195428899
|
|
|
|
|
| |
RELNOTES: None
PiperOrigin-RevId: 195422399
|
|
|
|
|
|
|
|
|
|
|
|
| |
This effectively subsumes all existing memory estimation, as it no
longer matters too much how much memory is assigned to each action.
WIP: This is a hack, and if it goes in, should almost certainly be
hidden behind a configuration option.
Closes #4938.
PiperOrigin-RevId: 195378568
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is mostly a roll-forward of 4465dae23de989f1452e93d0a88ac2a289103dd9, which
was reverted by fa36d2f48965b127e8fd397348d16e991135bfb6. The main difference is
that the new behavior is now gated behind the --noremote_allow_symlink_upload
flag.
https://docs.google.com/document/d/1gnOYszitgrLVet3sQk-TKGqIcpkkDsc6aw-izoo-d64
is a design proposal to support symlinks in the remote cache, which would render
this change moot. I'd like to be able to prevent incorrect cache behavior until
that change is implemented, though.
This fixes https://github.com/bazelbuild/bazel/issues/4840 (again).
Closes #5122.
Change-Id: I2136cfe82c2e1a8a9f5856e12a37d42cabd0e299
PiperOrigin-RevId: 195261827
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Post ProgressStatus.CHECKING_CACHE if RemoteSpawnCache is checking the cache.
The UI sees CHECKING_CACHE exactly the same as EXECUTING because no UIs
currently have any special behavior for actions in cache-lookup state. This is
still a UX improvement with --experimental_spawn_cache because EXECUTING is
generally more correct than the old action state, which varies from harmless but
unhelpful (no known state) to just wrong (C++ compile actions claimed they were
doing include scanning during cache lookups).
Closes #5130.
Change-Id: I77421c3667c180875216f937fe0713f0e9415a7a
PiperOrigin-RevId: 195233123
|
|
|
|
|
|
| |
It wasn't very helpful to print only stderr, considering that xcode-locator only ever prints to stdout...
PiperOrigin-RevId: 195213838
|
|
|
|
|
|
|
| |
instead of the manifest files.
RELNOTES: None
PiperOrigin-RevId: 195149880
|
|
|
|
| |
PiperOrigin-RevId: 194985259
|
|
|
|
|
| |
RELNOTES: None
PiperOrigin-RevId: 194798051
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There is no need for the cache to be on disk. Originally, there was a
desire to share this cache with other tools... but this never happened.
And, actually, because Bazel is in control of what it runs, it can just
inject the "cached" values into those tools via flags.
Instead, just store the cache in-memory. This avoids having to open and
read the cache on every single action that is locally executed on a Mac.
Results when building a large iOS app from a clean slate show up to a
1% wall time improvement on my Mac Pro 2013 and a reduction in the
variance of the measurements.
This change also gets rid of the OS check from the action execution's
critical path. There is not much use in checking this: if we instantiate
this by mistake, the actual calls will fail. But sometimes we want to
actually run this code on non-macOS systems (e.g. for unit-testing with
mocked tools), so we should allow that. And this change also ensures
that XcodeLocalEnvProviderTest builds and runs...
RELNOTES: None.
PiperOrigin-RevId: 194681802
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In StandaloneTestStrategy, copy as much information as SpawnResult makes
available to us through to both the TestResultData and BEP's
TestResult.ExecutionInfo protos. One immediate consequence is that the UI and
BEP can tell you whether a test result was cached remotely.
I changed Executor.getEventHandler to return an ExtendedEventHandler because it
makes this change easier to test.
Closes #5081.
Change-Id: I94fefdcd2e029c81085076736ad13a4bdf1bae8f
PiperOrigin-RevId: 194383009
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
SpawnAction.setShellCommand(String) now expects
the shell interpreter's path as an argument. This
change enables two things:
- rules can report an error if the shell is
missing
- SpawnAction no longer has to know about a
default shell
The new ShToolchain class will later also be
responsible for retrieving the active shell
toolchain (added in https://github.com/bazelbuild/bazel/commit/81ed3add408adb20bddbc3ba1818c65806738dc5).
This change brings Bazel closer to not depend on
the shell unless it has to (e.g. to run shell
scripts).
See https://github.com/bazelbuild/bazel/issues/4319
RELNOTES: none
PiperOrigin-RevId: 193885943
|
|
|
|
|
|
|
|
| |
This is a very heavy and slow option by design. It will be enabled only when a user wants to debug their build, most frequently to compare the step-by-step results of two builds.
TESTED: manually on various rules, including directories.
RELNOTES: None
PiperOrigin-RevId: 193539034
|
|
|
|
|
|
|
|
| |
This class will be used to tie a Spawn to a SpawnRunner, and isn't really a policy object. It will carry state such as the expanded inputs and expanded command line.
Currently a context can be passed between different SpawnRunners. This will be addressed independently, so a context is tied to a particular spawn runner.
PiperOrigin-RevId: 193501918
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
The no-cache tag is not respected (see b/77857812) and thus this breaks remote caching for all projects with symlink outputs.
*** Original change description ***
Only allow regular files and directories spawn outputs to be uploaded to a remote cache.
The remote cache protocol only knows about regular files and
directories. Currently, during action output upload, symlinks are
resolved into regular files. This means cached "executions" of an
action may have different output file types than the original
execution, which can be a footgun. This CL bans symlinks from cachable
spawn outputs and fixes http...
***
PiperOrigin-RevId: 193338629
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 193203414
|
|
|
|
|
|
|
|
| |
- Move test_timeout to BuildConfiguration.
- Add test_timeout_seconds field to BEP TargetCompleteEvent.
- Deprecate test_timeout field in ExecutionInfo. Data is still written to deprecated field to allow consumer transition.
PiperOrigin-RevId: 193192636
|
|
|
|
|
|
|
| |
It breaks downstream rules_nodejs. See https://github.com/bazelbuild/bazel/issues/5028 for details.
RELNOTES: None.
PiperOrigin-RevId: 193074798
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Remove the .useShell method, expect callers to
just pass the shell interpreter if they need it.
This removes the argument vector transformation
heuristic, and stops shelling out to cmd.exe on
Windows.
Also remove the .setWorkingDir method because
callers always had to set the working directory.
Instead, the CommandBuilder constructor takes the
working directory.
Closes #2190. (Test code still shells out to
cmd.exe though.)
Closes #5007.
Change-Id: I545e01c811daaf34913cb585492923da81aa02ee
PiperOrigin-RevId: 192611370
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 191861074
|
|
|
|
|
|
| |
Profiling can hold onto objects for the duration of the build, and some of those objects may be temporary that should not be persisted. In particular, UnixGlob and its inner classes should not outlive loading and analysis. For the most part, care was taken in this CL to only use strings that required no additional construction, mainly to minimize garbage (retaining references to newly created strings is not as great a concern since only the strings corresponding to the slowest K tasks are retained, for some relatively small values of K). Action descriptions for actually executing actions are eagerly expanded because that work is minimal compared to the work of actually executing an action.
PiperOrigin-RevId: 191251488
|
|
|
|
|
|
| |
Currently, the TestResult does not have an execution info, but that may change in the future.
PiperOrigin-RevId: 191148241
|
|
|
|
|
|
|
|
| |
Note that the wall time is not always set, especially for cached or remotely executed actions, so we keep the current code around for now.
Progress on #4808.
PiperOrigin-RevId: 190857756
|
|
|
|
|
|
|
|
|
| |
These have all had a chance to be categorized with the OptionDocumentationCategory enum, and the help output already uses the enum-grouped format.
The "incompatible changes" category has meaning for --all_incompatible_changes and will be removed separately.
RELNOTES: None.
PiperOrigin-RevId: 190773778
|
|
|
|
|
|
| |
This allows us to slightly simplify the renaming logic in StandaloneTestStrategy.
PiperOrigin-RevId: 190645674
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
remote cache.
The remote cache protocol only knows about regular files and
directories. Currently, during action output upload, symlinks are
resolved into regular files. This means cached "executions" of an
action may have different output file types than the original
execution, which can be a footgun. This CL bans symlinks from cachable
spawn outputs and fixes https://github.com/bazelbuild/bazel/issues/4840.
The interface of SpawnCache.CacheHandle.store is refactored:
1. The outputs parameter is removed, since that can be retrieved from the underlying Spawn.
2. It can now throw ExecException in order to fail actions.
Closes #4902.
Change-Id: I0d1d94d48779b970bb5d0840c66a14c189ab0091
PiperOrigin-RevId: 190608852
|
|
|
|
|
| |
RELNOTES: None
PiperOrigin-RevId: 190544948
|
|
|
|
|
|
| |
holds a variety of strategy/context maps.
PiperOrigin-RevId: 190491357
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Remove Optional<> where it's not needed. It's nice for return values, but IMHO it was overused in this code (e.g. Optional<List<X>> is an anti-pattern, as the list itself can already signal that it is empty).
- Use Bazel's own Path class when dealing with paths, not String or java.io.File.
- Move LinuxSandboxUtil into the "sandbox" package.
- Remove dead code and unused fields.
- Migrate deprecated VFS method calls to their replacements.
- Fix a bug in ExecutionStatistics where a FileInputStream was not closed.
Closes #4868.
PiperOrigin-RevId: 190217476
|