| Commit message (Collapse) | Author | Age |
|
|
|
|
|
|
| |
An upcoming replacement to PathFragment will not have efficient segment semantics, causing code to become unnecessarily inefficient.
RELNOTES: None
PiperOrigin-RevId: 182553098
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Local execution has an inherent race condition: if a user modifies a file while an action is executed, then it is impossible for Bazel to tell which version of the file was actually read during action execution. The file may have been modified before or after the tool has read it, or, in the worst case, the tool may have read both the original and the modified version. In addition, the file may be changed back to the original state before Bazel can check the file, so computing the digest before / after may not be sufficient.
This is a concern for both local and remote caches, although the cost of poisoning a shared remote cache is significantly higher, and is what has triggered this work.
Fixes #3360.
We solve this by keeping a reference to the FileContentsProxy, and using that to check for modificaitons before storing the cache entry. We output a warning if this check fails.
This change does not increase memory consumption; Java objects are always allocated in multiples of 8 bytes, we use compressed oops, and the FileArtifactValue currently has 12 bytes worth of fields (excl. object overhead), so adding another pointer is effectively free.
As a possible performance optimization on purely local builds, we could also consider not computing digests at all, and only use the FileContentsProxy for caching.
PiperOrigin-RevId: 182510358
|
|
|
|
|
|
|
| |
Fixes https://github.com/bazelbuild/bazel/issues/4376
Change-Id: Id78bb0930044626304e54f07735db4d4b2c84720
PiperOrigin-RevId: 181959528
|
|
|
|
|
|
| |
They need this to parse input manifests. Previously we would grab the exec root from the Root, but wish to unsupport this.
PiperOrigin-RevId: 181669143
|
|
|
|
|
|
|
|
| |
After some consideration, I think it makes sense to always allow a getDigest call, instead of specifying it as disallowed based on type. This is a follow-up CL for a previous CL introducing the getType method, which increased the complexity of the specification.
I have a follow-up CL, which is related, namely unknown commit. After that CL, Metadata instances for directories (but not Filesets) also have digests rather than using mtime, which is compatible with the documentation changes made here. Said CL is solving a correctness issue with directory dependencies, which I think we want, and using the digest in the Metadata is a natural way to get correct action cache lookups.
PiperOrigin-RevId: 181440548
|
|
|
|
|
|
|
|
| |
This simplifies some spawn runners, which no longer have to specially handle
null; unfortunately, the sandbox runners do not support VirtualActionInput,
so they still have to special-case it.
PiperOrigin-RevId: 181175408
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Create a PosixLocalEnvProvider and
WindowsLocalEnvProvider class, with singleton
instances for now.
This refactoring should not change functionality,
it's just a requirement for an upcoming change.
That upcoming change is for these classes to
respect the client environment's TMPDIR or
TMP/TEMP envvars.
See https://github.com/bazelbuild/bazel/issues/4376
Change-Id: I032bb6f18adf8af9e43e6bc543c09c58adae3863
PiperOrigin-RevId: 180799936
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When --nobuild_runfile_manifests is passed, don't create runfiles
input or output manifests at all. This seems better than creating fake
manifest artifacts that are actually a middleman. Fail fast for local
tests and the run command when --nobuild_runfiles_manifests is
passed. (These cases were failing with obscure errors before under
--nobuild_runfile_manifests-I just improved the messaging. See
https://github.com/bazelbuild/bazel/issues/4177.)
Change-Id: I351d26f746ecbe47016b58e4662768a5b6a72ff2
PiperOrigin-RevId: 180659571
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
that use either the LinuxSandboxedSpawnRunner or the ProcessWrapperSandboxedSpawnRunner.
In particular, record metrics for user and system CPU execution time, block I/O and involuntary context switches.
This feature is guarded behind a new option, --experimental_collect_local_sandbox_action_metrics.
Note: We still need to enable execution statistics for the DarwinSandboxedSpawnRunner in a later change.
RELNOTES: None.
PiperOrigin-RevId: 179976217
|
|
|
|
|
|
|
| |
be consistent with LinuxSandboxUtil.
RELNOTES: None.
PiperOrigin-RevId: 179758847
|
|
|
|
| |
PiperOrigin-RevId: 179542482
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
in LocalSpawnRunner, to avoid a race condition.
Aside:
The old, real temporary directory paths looked like this:
TMPDIR=/.../tmp15e_5dd5a8e8347813f5
The new, real temporary directory paths now look like this:
TMPDIR=/.../local-spawn-runner.3217503035718074040
RELNOTES: None.
PiperOrigin-RevId: 178903361
|
|
|
|
|
|
|
|
|
|
|
| |
LocalSpawnRunner uses the process-wrapper to run commands.
In particular, record metrics for user and system CPU execution time, block I/O and involuntary context switches.
This feature is guarded behind a new option, --experimental_collect_local_action_metrics.
RELNOTES: None.
PiperOrigin-RevId: 178856077
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
To reproduce: run a failing test with --experimental_remote_spawn_cache or with --spawn_strategy=remote and no executor. Expected: test log is uploaded.
Desired behavior:
- regardless of whether a spawn is cacheable or not, its artifacts should be uploaded to the remote cache.
- the spawn result should only be set if the spawn is cacheable *and* the action succeeded.
- when executing remotely, the do_not_cache field should be set for non-cacheable spawns, and the remote execution engine should respect it.
This CL contains multiple fixes to ensure the above behaviors, and adds a few tests, both end to end and unit tests. Important behavior change: it is no longer assumed that non-cacheable spawns should use a NO_CACHE SpawnCache! The appropriate test case was removed. Instead, an assumption was added that all implementations of SpawnCache should respect the Spawns.mayBeCached(spawn) property. Currently, only NO_CACHE and RemoteSpawnCache exist, and they (now) support it.
TESTED=remote build execution backend.
WANT_LGTM: philwo,buchgr
RELNOTES: None
PiperOrigin-RevId: 178617937
|
|
|
|
|
|
|
| |
generating another temporary directory name instead of throwing an exception.
RELNOTES: None.
PiperOrigin-RevId: 178190769
|
|
|
|
|
|
|
|
|
|
| |
Bazel should display the root cause of a test failure to the user. For
example, if a test could not be executed on a remote executor due to
there being no network connection, then it shouldn't display the test
as failed but tell the user about the network error.
RELNOTES:
PiperOrigin-RevId: 177439578
|
|
|
|
| |
PiperOrigin-RevId: 177428407
|
|
|
|
|
|
|
|
|
| |
everywhere instead of duplicating process-wrapper --shell_arguments in Blaze.
To avoid a cyclic dependency, I broke up exec/local:local into exec/local:local and exec/local:options.
RELNOTES: None.
PiperOrigin-RevId: 177419268
|
|
|
|
|
| |
RELNOTES: None
PiperOrigin-RevId: 177290508
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- remove BaseSpawn.Local; instead, all callers pass in the full set of
execution requirements they want to set
- disable caching and sandboxing for the symlink tree action - it does not
declare outputs, so it can't be cached or sandboxed (fixes #4041)
- centralize the existing execution requirements in the ExecutionRequirements
class
- centralize checking for execution requirements in the Spawn class
(it's possible that we may need a more decentralized, extensible design in
the future, but for now having them in a single place is simple and
effective)
- update the documentation
- forward the relevant tags to execution requirements in TargetUtils (progress
on #3960)
- this also contributes to #4153
PiperOrigin-RevId: 177288598
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It turns out that the SUCCESS status is often misunderstood to mean "zero exit",
even though this is clearly documented. I've decided to add another status for
non-zero exit, and use success only for zero exit to avoid this pitfall.
Also, many of the status codes are set, but never used. I decided to reduce the
number of status codes to only those that are actually relevant, which
simplifies further processing. Instead, we should add a string message for the
error case when we need one - we're not using it right now, so I decided not to
add that yet.
PiperOrigin-RevId: 177129441
|
|
|
|
|
|
|
|
|
| |
Currently we don't care about the list order of SpawnResults. However, we may care about the list order later. Also, if the equals() method for SpawnResults is ever changed then it may be problematic to return SpawnResults in a Set.
Aside: ActionResults use SpawnResults to calculate cumulative execution times for Actions, and may provide other metrics in future.
RELNOTES: None.
PiperOrigin-RevId: 176579460
|
|
|
|
|
|
|
|
|
|
| |
Fix for #4035
@laszlocsomor
Closes #4110.
PiperOrigin-RevId: 176346381
|
|
|
|
|
|
|
|
| |
Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own.
This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually.
PiperOrigin-RevId: 175033526
|
|
|
|
|
|
|
| |
TerminationStatus, and also add a TerminationStatus.Builder and tests.
RELNOTES: None.
PiperOrigin-RevId: 174557303
|
|
|
|
|
|
|
| |
and make cumulative execution times available in ActionResults.
RELNOTES: None
PiperOrigin-RevId: 174553272
|
|
|
|
|
|
| |
This requires a fairly large amount of changes to fundamental objects like BlazeRuntime, Executor, and so on, as well as changing a lot of test code to thread the file system through. I expect future CLs to be much smaller.
PiperOrigin-RevId: 173678144
|
|
|
|
|
|
|
|
| |
Even if the test action produced no output, which it really shouldn't, Bazel should create an empty test.log file.
TESTED=unit tests
RELNOTES: Fixes #3834
PiperOrigin-RevId: 172412615
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Every build and test action that creates a Spawn
will now have platform-specific environment
variables for temp directories:
- on Windows: TMP and TEMP
- on Linux/Darwin: TMPDIR
This is particularly important on Windows where
e.g. Java programs cannot create temp directories
unless there's a valid TMP or TEMP environment
variable set.
Fixes:
- https://github.com/bazelbuild/bazel/issues/1590
- https://github.com/bazelbuild/bazel/issues/2349
- https://github.com/bazelbuild/bazel/issues/2870
Change-Id: Ib758307daf6b3a51b0f71ae5e65e5bb564dad643
PiperOrigin-RevId: 172326371
|
|
|
|
|
|
| |
TESTED=added remote test
RELNOTES: Fixes #2574
PiperOrigin-RevId: 172294781
|
|
|
|
|
|
|
| |
Whether or not there was a catastrophic error is stored in the SpawnResult, so
we can just use that instead of passing in an additional boolean.
PiperOrigin-RevId: 172083752
|
|
|
|
|
|
| |
cache if we're doing input discovery: input discovery may find new artifacts we don't yet know about. Similarly, in SingleBuildFileCache, use the Artifact's path if it's available, rather than the poor man's route of the execRoot.
PiperOrigin-RevId: 171635892
|
|
|
|
|
|
|
|
|
| |
ActionContexts, etc., so that SpawnResult metadata is returned upwards.
Note that the TODOs mostly refer to changes that will appear in a subsequent CL (a CL to return SpawnResults, contained in ActionResults, from Actions/AbstractActions). I split off the remaining SpawnResult-related changes into this CL and kept the ActionResult-related changes separate.
RELNOTES: None.
PiperOrigin-RevId: 171355611
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 170955877
|
|
|
|
|
|
|
|
|
|
| |
- add a mode to control how to handle relative symlinks
- if set, attempt to resolve relative symlinks in manifests by looking for
another entry that defines a file with the intended symlink target (we don't
do recursive resolution for now)
- add more test coverage; fix a bug in handling empty target locations
PiperOrigin-RevId: 170691492
|
|
|
|
|
|
|
| |
them to be created.
RELNOTES: None.
PiperOrigin-RevId: 170058295
|
|
|
|
|
|
|
|
| |
Include the coverage data file into the list of files repeorted in the
test_result event in the BEP.
Change-Id: Ia7b653addd5589268ce919b3e0349371dbc18bf2
PiperOrigin-RevId: 169956864
|
|
|
|
|
|
|
| |
build.lib.actions.SpawnActionContext can import SpawnResult without creating a cyclic dependency.
RELNOTES: None.
PiperOrigin-RevId: 169642267
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
executions and tests with multiple attempts. Previously, non-cached test execution would incorrectly only include test.log and test.xml.
Implement this by unifying test output list building in a static TestResult method.
For additional output files in multi-attempt tests, we follow the existing convention:
Existing example: test.xml => test_attempts/attempt_N.xml
New example (all new outputs done similarly): test.outputs/outputs.zip => test_attempts/attempt_N.outputs/outputs.zip
RELNOTES: All test output files included for cached, uncached, and multiple attempt tests.
PiperOrigin-RevId: 169556608
|
|
|
|
|
|
|
|
|
|
|
|
| |
NetUtil.getShortHostName can take seconds on mac and on windows (like, 20!),
since it performs reverse dns lookup. We already cached hostname for the
BazelWorkspaceStatusModule, let's cache it for entire bazel server. Also make
sure that users of the method understand it's cached.
Fixes #3586.
RELNOTES: None.
PiperOrigin-RevId: 168691615
|
|
|
|
|
|
|
|
|
| |
Split collect, concurrent, vfs, windows into package-level BUILD files.
Move clock classes out of "util", into their own Java package.
Move CompactHashSet into its own Java package to break a dependency cycle.
Give nestedset and inmemoryfs their own package-level BUILD files.
PiperOrigin-RevId: 167702127
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 167505493
|
|
|
|
|
|
|
|
|
|
| |
For any errors that are due to failures in the remote caching /
execution layers Bazel now returns exit code 34 (ExitCode.REMOTE_ERROR).
This includes errors where the remote cache / executor is unreachable or
crashes. It does not include errors if the test / build failure is due
to user errors i.e. compilation or test failures.
PiperOrigin-RevId: 167259236
|
|
|
|
|
|
|
|
|
| |
If there is a build failure, don't clobber the terse test summary
by naming all the (usually many) tests that were skipped due to
this failure.
Change-Id: I6daae3efb1594c2b1018f87a50cf63949a34535b
PiperOrigin-RevId: 166983264
|
|
|
|
| |
PiperOrigin-RevId: 166849610
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This paves the way for Skylark-side compact command lines that can fail during expansion.
In general, expansion happens in these scenarios:
* Action execution expands the command line to execute it. This is fine since we are well equipped already to handle failing actions.
* In the analysis phase we expand command lines to investigate whether we need a params file. This could be moved to the execution phase, which would have the benefit of getting params files out of the action graph and saving memory.
* Key computation expands the command line. This could be fixed by allowing command lines to compute their own keys (which wouldn't try to expand the command line). This could have the benefit of being more efficient.
* Extra actions expand the command line to construct the extra action proto. This could maybe be deferred to the execution phase (writing the extra action), which would also be more memory efficient.
For failed key computations, we return a singleton "error" key. This means multiple actions that will fail will map to the same key. These actions will necessarily fail if executed, so we should not need to worry about these ending up in the action cache. If we do manage to make the command line compute its own keys then this will become moot in the future.
RELNOTES: None
PiperOrigin-RevId: 166266385
|
|
|
|
|
|
|
|
|
|
| |
AbstractSpawnRunner now uses a SpawnCache if one is registered, this allows
adding caching to any spawn runner without having to be aware of the
implementations.
I will delete the old CachedLocalSpawnRunner in a follow-up CL.
PiperOrigin-RevId: 165024382
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Change the persistent worker spawn strategy to extend
AbstractSpawnStrategy and put the actual logic into
WorkerSpawnRunner. WorkerTestStrategy is unaffected.
I had to extend SpawnPolicy with a speculating() method. Persistent
workers need to know if speculation is happening in order to require
sandboxing.
Additionally, I added java_test rules for the local runner tests and
worker tests. See https://github.com/bazelbuild/bazel/issues/3481.
NOTE: ulfjack@ made some changes to this change before merging:
- changed Reporter to EventHandler; added TODO about its usage
- reverted non-semantic indentation change in AbstractSpawnStrategy
- reverted a non-semantic indentation change in WorkerSpawnRunner
- updated some internal classes to match
- removed catch IOException in WorkerSpawnRunner in some cases,
removed verboseFailures flag from WorkerSpawnRunner, updated callers
- disable some tests on Windows; we were previously not running them,
now that we do, they fail :-(
Change-Id: I207b3938f0dc84d374ab052d5030020886451d47
PiperOrigin-RevId: 164965398
|
|
|
|
|
|
|
| |
...instead of injecting it through the constructor. Simplify all the callers
accordingly.
PiperOrigin-RevId: 164955391
|
|
|
|
|
|
|
|
|
|
|
| |
This isn't ideal, but is the fastest way to split analysis, execution, and
rules into separate java libraries, with lib.skyframe still being in the
analysis library. Ideally, we'd split up the lib.skyframe package, move the
analysis stuff to the analysis library, the execution stuff to the execution
library, and so on. That wouldn't require us moving OutputService out of
lib.exec.
PiperOrigin-RevId: 164856998
|