| Commit message (Collapse) | Author | Age |
|
|
|
|
| |
RELNOTES: None
PiperOrigin-RevId: 194798051
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There is no need for the cache to be on disk. Originally, there was a
desire to share this cache with other tools... but this never happened.
And, actually, because Bazel is in control of what it runs, it can just
inject the "cached" values into those tools via flags.
Instead, just store the cache in-memory. This avoids having to open and
read the cache on every single action that is locally executed on a Mac.
Results when building a large iOS app from a clean slate show up to a
1% wall time improvement on my Mac Pro 2013 and a reduction in the
variance of the measurements.
This change also gets rid of the OS check from the action execution's
critical path. There is not much use in checking this: if we instantiate
this by mistake, the actual calls will fail. But sometimes we want to
actually run this code on non-macOS systems (e.g. for unit-testing with
mocked tools), so we should allow that. And this change also ensures
that XcodeLocalEnvProviderTest builds and runs...
RELNOTES: None.
PiperOrigin-RevId: 194681802
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In StandaloneTestStrategy, copy as much information as SpawnResult makes
available to us through to both the TestResultData and BEP's
TestResult.ExecutionInfo protos. One immediate consequence is that the UI and
BEP can tell you whether a test result was cached remotely.
I changed Executor.getEventHandler to return an ExtendedEventHandler because it
makes this change easier to test.
Closes #5081.
Change-Id: I94fefdcd2e029c81085076736ad13a4bdf1bae8f
PiperOrigin-RevId: 194383009
|
|
|
|
|
|
|
|
| |
This is a very heavy and slow option by design. It will be enabled only when a user wants to debug their build, most frequently to compare the step-by-step results of two builds.
TESTED: manually on various rules, including directories.
RELNOTES: None
PiperOrigin-RevId: 193539034
|
|
|
|
|
|
|
|
| |
This class will be used to tie a Spawn to a SpawnRunner, and isn't really a policy object. It will carry state such as the expanded inputs and expanded command line.
Currently a context can be passed between different SpawnRunners. This will be addressed independently, so a context is tied to a particular spawn runner.
PiperOrigin-RevId: 193501918
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
The no-cache tag is not respected (see b/77857812) and thus this breaks remote caching for all projects with symlink outputs.
*** Original change description ***
Only allow regular files and directories spawn outputs to be uploaded to a remote cache.
The remote cache protocol only knows about regular files and
directories. Currently, during action output upload, symlinks are
resolved into regular files. This means cached "executions" of an
action may have different output file types than the original
execution, which can be a footgun. This CL bans symlinks from cachable
spawn outputs and fixes http...
***
PiperOrigin-RevId: 193338629
|
|
|
|
|
|
| |
Profiling can hold onto objects for the duration of the build, and some of those objects may be temporary that should not be persisted. In particular, UnixGlob and its inner classes should not outlive loading and analysis. For the most part, care was taken in this CL to only use strings that required no additional construction, mainly to minimize garbage (retaining references to newly created strings is not as great a concern since only the strings corresponding to the slowest K tasks are retained, for some relatively small values of K). Action descriptions for actually executing actions are eagerly expanded because that work is minimal compared to the work of actually executing an action.
PiperOrigin-RevId: 191251488
|
|
|
|
| |
PiperOrigin-RevId: 190779535
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 190617155
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
remote cache.
The remote cache protocol only knows about regular files and
directories. Currently, during action output upload, symlinks are
resolved into regular files. This means cached "executions" of an
action may have different output file types than the original
execution, which can be a footgun. This CL bans symlinks from cachable
spawn outputs and fixes https://github.com/bazelbuild/bazel/issues/4840.
The interface of SpawnCache.CacheHandle.store is refactored:
1. The outputs parameter is removed, since that can be retrieved from the underlying Spawn.
2. It can now throw ExecException in order to fail actions.
Closes #4902.
Change-Id: I0d1d94d48779b970bb5d0840c66a14c189ab0091
PiperOrigin-RevId: 190608852
|
|
|
|
|
| |
RELNOTES: None
PiperOrigin-RevId: 190544948
|
|
|
|
|
|
| |
holds a variety of strategy/context maps.
PiperOrigin-RevId: 190491357
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 190079798
|
|
|
|
| |
PiperOrigin-RevId: 189244665
|
|
|
|
|
|
| |
Closes #4640.
PiperOrigin-RevId: 188022228
|
|
|
|
|
|
|
| |
This isn't strictly necessary since we disable caching and require local
execution.
PiperOrigin-RevId: 187985476
|
|
|
|
| |
PiperOrigin-RevId: 187397314
|
|
|
|
|
|
|
|
|
|
|
| |
...so that it can use that path to compute other directories in the
output user base, in particular the default location for caches.
The first cache we will add is the hash-index cache for downloads
of external archives, but a spawn cache might be added later in the
output user base as well.
Change-Id: I24b1c33235c8f76ec008ecb1789163de2b2a45be
PiperOrigin-RevId: 187164275
|
|
|
|
|
|
| |
...instead of passing it everywhere so we can rewrite the local env.
PiperOrigin-RevId: 186316859
|
|
|
|
|
|
| |
AbstractAction.
PiperOrigin-RevId: 186316435
|
|
|
|
|
|
|
|
|
| |
- Make sure that we correctly resolve symlinks starting with ./ and ../. This
is currently failing, although it's working at head.
From my reading, https://github.com/bazelbuild/bazel/commit/a729b9b4c3d7844a7d44934bf3365f92633c0a60 changes PathFragment.getRelative to always
normalize the return value, which it wasn't doing before.
PiperOrigin-RevId: 186289431
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
There's already a --test_tmpdir flag, and Java tests don't pick up this new one. More info: https://github.com/bazelbuild/bazel/issues/4621#issuecomment-366217321
*** Original change description ***
tmpdir,local-exec: implement --local_tmp_root
Add new flag called `--local_tmp_root`, which (if
specified) tells Bazel what temp directory should
locally executed actions use.
Fixes https://github.com/bazelbuild/bazel/issues/4621
Related to https://github.com/bazelbuild/bazel/issues/3215
RELNOTES[NEW]: The new "--local_tmp_root=<path>" flag allows specifying the temp directory for locally executed actions.
Change-Id: Ice69a5e63d0bf4d3b5c9ef4dbdd1ed1c5025f85e
PiperOrigin-RevId: 185982705
|
|
|
|
|
|
|
|
| |
The BVTC is primarily intended for analysis tests, and those should not have
access to BinTools. However, there are a few subclasses that misuse it for
integration tests, and those need to create their own BinTools instances now.
PiperOrigin-RevId: 185862672
|
|
|
|
|
|
|
| |
It's never used during analysis, it shouldn't be used during analysis, so let's
not make it available during analysis.
PiperOrigin-RevId: 185808384
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add new flag called `--local_tmp_root`, which (if
specified) tells Bazel what temp directory should
locally executed actions use.
Fixes https://github.com/bazelbuild/bazel/issues/4621
Related to https://github.com/bazelbuild/bazel/issues/3215
RELNOTES[NEW]: The new "--local_tmp_root=<path>" flag allows specifying the temp directory for locally executed actions.
Change-Id: Ice69a5e63d0bf4d3b5c9ef4dbdd1ed1c5025f85e
PiperOrigin-RevId: 185509555
|
|
|
|
|
|
|
|
|
|
|
| |
This class represents a root (such as a package path or an output root) used for file lookups and artifacts. It is meant to be as opaque as possible in order to hide the user's environment from sky keys and sky functions.
Roots are used by RootedPaths and ArtifactRoots.
This CL attempts to make the minimum number of modifications necessary to change RootedPath and ArtifactRoot to use these fields. Deprecated methods and invasive accessors are permitted to minimise the risk of any observable changes.
RELNOTES: None
PiperOrigin-RevId: 182271759
|
|
|
|
|
|
| |
This is slightly more descriptive, and we will potentially want to use the name Root for a broader concept shared between ArtifactRoot and RootedPath.
PiperOrigin-RevId: 182082367
|
|
|
|
|
|
|
| |
Fixes https://github.com/bazelbuild/bazel/issues/4376
Change-Id: Id78bb0930044626304e54f07735db4d4b2c84720
PiperOrigin-RevId: 181959528
|
|
|
|
|
|
| |
This is no longer used.
PiperOrigin-RevId: 181754475
|
|
|
|
| |
PiperOrigin-RevId: 181686136
|
|
|
|
|
|
| |
They need this to parse input manifests. Previously we would grab the exec root from the Root, but wish to unsupport this.
PiperOrigin-RevId: 181669143
|
|
|
|
|
|
|
|
|
| |
more cases.
Part of #4128.
Change-Id: Ife5e4581f91ac07931d193ed5eaa256aab3ad047
PiperOrigin-RevId: 180826445
|
|
|
|
|
|
|
|
| |
I no longer believe that it is worthwhile to remove the file system instance from Path. Measurements indicate that I can simply replace the Path implementation in one go and just leave the instance on the Path class.
This CL was done entirely using IDE refactoring, so should be safe.
PiperOrigin-RevId: 180287162
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fixes #4322, #4306.
*** Reason for rollback ***
Introduces a deadlock (see https://github.com/bazelbuild/bazel/issues/4322)
*** Original change description ***
Make FileSystem operate on LocalPath instead of Path.
PiperOrigin-RevId: 179549866
|
|
|
|
|
|
|
| |
bound (because of interference).
RELNOTES: None.
PiperOrigin-RevId: 179229423
|
|
|
|
| |
PiperOrigin-RevId: 179082062
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
in LocalSpawnRunner, to avoid a race condition.
Aside:
The old, real temporary directory paths looked like this:
TMPDIR=/.../tmp15e_5dd5a8e8347813f5
The new, real temporary directory paths now look like this:
TMPDIR=/.../local-spawn-runner.3217503035718074040
RELNOTES: None.
PiperOrigin-RevId: 178903361
|
|
|
|
|
|
|
|
|
|
|
| |
LocalSpawnRunner uses the process-wrapper to run commands.
In particular, record metrics for user and system CPU execution time, block I/O and involuntary context switches.
This feature is guarded behind a new option, --experimental_collect_local_action_metrics.
RELNOTES: None.
PiperOrigin-RevId: 178856077
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
To reproduce: run a failing test with --experimental_remote_spawn_cache or with --spawn_strategy=remote and no executor. Expected: test log is uploaded.
Desired behavior:
- regardless of whether a spawn is cacheable or not, its artifacts should be uploaded to the remote cache.
- the spawn result should only be set if the spawn is cacheable *and* the action succeeded.
- when executing remotely, the do_not_cache field should be set for non-cacheable spawns, and the remote execution engine should respect it.
This CL contains multiple fixes to ensure the above behaviors, and adds a few tests, both end to end and unit tests. Important behavior change: it is no longer assumed that non-cacheable spawns should use a NO_CACHE SpawnCache! The appropriate test case was removed. Instead, an assumption was added that all implementations of SpawnCache should respect the Spawns.mayBeCached(spawn) property. Currently, only NO_CACHE and RemoteSpawnCache exist, and they (now) support it.
TESTED=remote build execution backend.
WANT_LGTM: philwo,buchgr
RELNOTES: None
PiperOrigin-RevId: 178617937
|
|
|
|
|
|
|
| |
generating another temporary directory name instead of throwing an exception.
RELNOTES: None.
PiperOrigin-RevId: 178190769
|
|
|
|
|
|
|
| |
Part of #4128.
Change-Id: Id822d3ae6f8daf7c92a75bd8bd28590d4f625845
PiperOrigin-RevId: 177905460
|
|
|
|
|
|
|
|
|
|
| |
Bazel should display the root cause of a test failure to the user. For
example, if a test could not be executed on a remote executor due to
there being no network connection, then it shouldn't display the test
as failed but tell the user about the network error.
RELNOTES:
PiperOrigin-RevId: 177439578
|
|
|
|
|
|
|
| |
This key context can be used by actions to share partial key computations, for instance when computing MD5s for nested sets.
RELNOTES: None
PiperOrigin-RevId: 177359607
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- remove BaseSpawn.Local; instead, all callers pass in the full set of
execution requirements they want to set
- disable caching and sandboxing for the symlink tree action - it does not
declare outputs, so it can't be cached or sandboxed (fixes #4041)
- centralize the existing execution requirements in the ExecutionRequirements
class
- centralize checking for execution requirements in the Spawn class
(it's possible that we may need a more decentralized, extensible design in
the future, but for now having them in a single place is simple and
effective)
- update the documentation
- forward the relevant tags to execution requirements in TargetUtils (progress
on #3960)
- this also contributes to #4153
PiperOrigin-RevId: 177288598
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It turns out that the SUCCESS status is often misunderstood to mean "zero exit",
even though this is clearly documented. I've decided to add another status for
non-zero exit, and use success only for zero exit to avoid this pitfall.
Also, many of the status codes are set, but never used. I decided to reduce the
number of status codes to only those that are actually relevant, which
simplifies further processing. Instead, we should add a string message for the
error case when we need one - we're not using it right now, so I decided not to
add that yet.
PiperOrigin-RevId: 177129441
|
|
|
|
|
|
|
|
|
| |
Currently we don't care about the list order of SpawnResults. However, we may care about the list order later. Also, if the equals() method for SpawnResults is ever changed then it may be problematic to return SpawnResults in a Set.
Aside: ActionResults use SpawnResults to calculate cumulative execution times for Actions, and may provide other metrics in future.
RELNOTES: None.
PiperOrigin-RevId: 176579460
|
|
|
|
|
|
|
|
|
|
| |
Fix for #4035
@laszlocsomor
Closes #4110.
PiperOrigin-RevId: 176346381
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 175153541
|
|
|
|
|
|
|
| |
and make cumulative execution times available in ActionResults.
RELNOTES: None
PiperOrigin-RevId: 174553272
|
|
|
|
|
|
| |
This requires a fairly large amount of changes to fundamental objects like BlazeRuntime, Executor, and so on, as well as changing a lot of test code to thread the file system through. I expect future CLs to be much smaller.
PiperOrigin-RevId: 173678144
|