| Commit message (Collapse) | Author | Age |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Prior to this cl CompileCommandLine would (almost) unconditionally emit -c and
-o flags. This cl removes this logic and relies on crosstool to emit these
flags. This is another small step towards platform independent C++ rules.
Memory use is not affected, since the build variables used by this cl are already
exposed, this cl just forces crosstools to use it.
Encore of https://github.com/bazelbuild/bazel/commit/f26e8694ae78599b3e2004e3360eaf3443fa53a6.
RELNOTES: None.
PiperOrigin-RevId: 184981106
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
native skyframe implementation was actually quite incorrect in this case: It was adding a skyframe dependency on a FileValue for the output file. Without a transitive dependency on the source files and actions that determine the output file's state, this could never work (and explains why the incremental build would fail). Instead, we now depend on the Artifact corresponding to the output file instead.
This change updates the business logic RecursiveFilesystemTraversalFunction. This approach keeps the business logic of Fileset filesystem traversal centralized in RFTF.
To avoid making weird recursive Skyframe nodes in the output tree, we inline Skyframe dependencies and do direct filesystem operations over the output tree.
There are now three states we can be in when looking up a file:
1. Source file: As before, make a skyframe dep on the corresponding file
2. Top-level file of an output tree: Make a dep on the corresponding Artifact
3. Recursive file under an output directory: Do direct filesystem operations. It doesn't make sense to make Skyframe nodes corresponding to these files. In the future, I think we should consider failing fast on this case.
RELNOTES: None
PiperOrigin-RevId: 184556044
|
|
|
|
|
|
| |
SpecialArtifact.
PiperOrigin-RevId: 184539696
|
|
|
|
| |
PiperOrigin-RevId: 184144301
|
|
|
|
|
| |
RELNOTES: None
PiperOrigin-RevId: 183727976
|
|
|
|
|
|
|
|
| |
This interface makes it clearer in the type system exactly how items that go into a CustomCommandLine are turned into strings.
It is a preparatory change to allow command line fingerprints to be more cheaply calculated, but it is valuable in itself from a code quality standpoint.
PiperOrigin-RevId: 183274022
|
|
|
|
|
|
| |
owner and fix up BuildConfigurationValue.Key. ConfiguredTargetKey is going to need some modifications to AutoCodec: probably the long-awaited static "create" method.
PiperOrigin-RevId: 182630181
|
|
|
|
|
|
|
| |
An upcoming replacement to PathFragment will not have efficient segment semantics, causing code to become unnecessarily inefficient.
RELNOTES: None
PiperOrigin-RevId: 182553098
|
|
|
|
|
|
|
| |
* Use an enum instead of booleans and null fields having special meaning.
* Remove the exec root from ArtifactRoot. This is only used to calculate artifact exec paths, which can be done from the root's exec path and the root relative path.
PiperOrigin-RevId: 182411710
|
|
|
|
|
|
|
|
| |
An absolute root accepts any absolute path fragments and simply returns it.
This concept replaces the FileSystem root directory concept.
PiperOrigin-RevId: 182400471
|
|
|
|
|
|
|
|
|
|
|
| |
This class represents a root (such as a package path or an output root) used for file lookups and artifacts. It is meant to be as opaque as possible in order to hide the user's environment from sky keys and sky functions.
Roots are used by RootedPaths and ArtifactRoots.
This CL attempts to make the minimum number of modifications necessary to change RootedPath and ArtifactRoot to use these fields. Deprecated methods and invasive accessors are permitted to minimise the risk of any observable changes.
RELNOTES: None
PiperOrigin-RevId: 182271759
|
|
|
|
|
|
| |
Post to the Skyframe Environment instead.
PiperOrigin-RevId: 182096559
|
|
|
|
|
|
| |
This is slightly more descriptive, and we will potentially want to use the name Root for a broader concept shared between ArtifactRoot and RootedPath.
PiperOrigin-RevId: 182082367
|
|
|
|
|
|
|
|
| |
This method violates the invariant that derived roots are never equal to the exec root. Only source roots can be equal to the exec root.
Note that this method was only used in tests, so this CL should be completely safe as long as its tests pass.
PiperOrigin-RevId: 181998483
|
|
|
|
|
|
| |
In this case, the invariant violated is creating derived roots at the exec root.
PiperOrigin-RevId: 181994080
|
|
|
|
| |
PiperOrigin-RevId: 181975994
|
|
|
|
|
|
| |
This is no longer used.
PiperOrigin-RevId: 181754475
|
|
|
|
|
|
| |
removing the layer of indirection of getting SkyKey out of ActionLookupKey, which uses more memory for no reason.
PiperOrigin-RevId: 181658615
|
|
|
|
|
|
|
|
| |
After some consideration, I think it makes sense to always allow a getDigest call, instead of specifying it as disallowed based on type. This is a follow-up CL for a previous CL introducing the getType method, which increased the complexity of the specification.
I have a follow-up CL, which is related, namely unknown commit. After that CL, Metadata instances for directories (but not Filesets) also have digests rather than using mtime, which is compatible with the documentation changes made here. Said CL is solving a correctness issue with directory dependencies, which I think we want, and using the digest in the Metadata is a natural way to get correct action cache lookups.
PiperOrigin-RevId: 181440548
|
|
|
|
|
|
|
|
| |
This simplifies some spawn runners, which no longer have to specially handle
null; unfortunately, the sandbox runners do not support VirtualActionInput,
so they still have to special-case it.
PiperOrigin-RevId: 181175408
|
|
|
|
|
|
|
|
|
| |
more cases.
Part of #4128.
Change-Id: Ife5e4581f91ac07931d193ed5eaa256aab3ad047
PiperOrigin-RevId: 180826445
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Change FileType to no longer assume it operates on just the base name (it can now be given a full path).
* Move the responsibility to specific classes (Artifact, Path, PathFragment) to decide how they want to offer up a string that includes the file name.
* Flip the order in which users are expected to check Artifact type, from FileType#matches(Artifact) to Artifact#isFileType(FileType). This looks natural and should encourage developers to use efficient file type checking methods.
* Change CppCompileAction to use the new API.
RELNOTES: None
PiperOrigin-RevId: 179903239
|
|
|
|
|
|
|
|
| |
This is in preparation for merging FileArtifactValue and FileStateValue.
Progress on #3360.
PiperOrigin-RevId: 179832948
|
|
|
|
|
|
| |
same information and is more useful, since it's practically a SkyKey.
PiperOrigin-RevId: 179727105
|
|
|
|
|
|
| |
And fix the tests that were doing this.
PiperOrigin-RevId: 179548691
|
|
|
|
| |
PiperOrigin-RevId: 179213789
|
|
|
|
|
|
|
| |
Tests on my local machine show that the Hyperthreading multiplier is significantly hurting our builds times.
RELNOTES: Set build jobs equivalent to number of logical processors by default. Should improve build times significantly.
PiperOrigin-RevId: 178795078
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Tests on my local machine show that the Hyperthreading multiplier is significantly hurting our builds times. Running with 8 logical cores is 11% faster than running with 4.8 cores when doing builds on my Mac.
macOS 10.12
MacBookPro 2017 (1 CPU, 4 core, 8 hyperthreaded)
1434.259 with --jobs = 5
1274.459 with --jobs = 8
I'll do some similar tests on Linux to see if it makes sense there.
PiperOrigin-RevId: 178654477
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 178618888
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
To reproduce: run a failing test with --experimental_remote_spawn_cache or with --spawn_strategy=remote and no executor. Expected: test log is uploaded.
Desired behavior:
- regardless of whether a spawn is cacheable or not, its artifacts should be uploaded to the remote cache.
- the spawn result should only be set if the spawn is cacheable *and* the action succeeded.
- when executing remotely, the do_not_cache field should be set for non-cacheable spawns, and the remote execution engine should respect it.
This CL contains multiple fixes to ensure the above behaviors, and adds a few tests, both end to end and unit tests. Important behavior change: it is no longer assumed that non-cacheable spawns should use a NO_CACHE SpawnCache! The appropriate test case was removed. Instead, an assumption was added that all implementations of SpawnCache should respect the Spawns.mayBeCached(spawn) property. Currently, only NO_CACHE and RemoteSpawnCache exist, and they (now) support it.
TESTED=remote build execution backend.
WANT_LGTM: philwo,buchgr
RELNOTES: None
PiperOrigin-RevId: 178617937
|
|
|
|
|
|
|
|
|
|
|
|
| |
MiddlemanFactory wants to check if the middleman inputs set is empty
or singleton. A NestedSet can answer both queries efficiently if the
right APIs are used.
My ultimate goal here is to avoid the expansion of runfiles artifacts
nested sets until execution.
Change-Id: I29a269df757ef41b1410bbb492cf24c926df6114
PiperOrigin-RevId: 178600943
|
|
|
|
|
|
|
| |
statistics to SpawnResults, and add cumulative getters for these in ActionResults.
RELNOTES: None.
PiperOrigin-RevId: 178426142
|
|
|
|
| |
PiperOrigin-RevId: 178116776
|
|
|
|
|
|
|
| |
Part of #4128.
Change-Id: Id822d3ae6f8daf7c92a75bd8bd28590d4f625845
PiperOrigin-RevId: 177905460
|
|
|
|
|
|
|
| |
each Action.
RELNOTES: None.
PiperOrigin-RevId: 177652741
|
|
|
|
| |
PiperOrigin-RevId: 177453738
|
|
|
|
|
|
|
|
|
|
|
| |
Slow read: a 1253924941-byte read from image.tar.gz took 5396ms.
is 221 MB/s, which is smokin' fast, not slow :P
Fixes: #3967
Change-Id: Ieca7464e0a670ade52f80172408334c9fc3a5b52
PiperOrigin-RevId: 177438364
|
|
|
|
| |
PiperOrigin-RevId: 177428407
|
|
|
|
|
|
|
| |
This key context can be used by actions to share partial key computations, for instance when computing MD5s for nested sets.
RELNOTES: None
PiperOrigin-RevId: 177359607
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 177326265
|
|
|
|
|
| |
RELNOTES: None
PiperOrigin-RevId: 177290508
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- remove BaseSpawn.Local; instead, all callers pass in the full set of
execution requirements they want to set
- disable caching and sandboxing for the symlink tree action - it does not
declare outputs, so it can't be cached or sandboxed (fixes #4041)
- centralize the existing execution requirements in the ExecutionRequirements
class
- centralize checking for execution requirements in the Spawn class
(it's possible that we may need a more decentralized, extensible design in
the future, but for now having them in a single place is simple and
effective)
- update the documentation
- forward the relevant tags to execution requirements in TargetUtils (progress
on #3960)
- this also contributes to #4153
PiperOrigin-RevId: 177288598
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It turns out that the SUCCESS status is often misunderstood to mean "zero exit",
even though this is clearly documented. I've decided to add another status for
non-zero exit, and use success only for zero exit to avoid this pitfall.
Also, many of the status codes are set, but never used. I decided to reduce the
number of status codes to only those that are actually relevant, which
simplifies further processing. Instead, we should add a string message for the
error case when we need one - we're not using it right now, so I decided not to
add that yet.
PiperOrigin-RevId: 177129441
|
|
|
|
|
|
|
|
|
| |
Currently we don't care about the list order of SpawnResults. However, we may care about the list order later. Also, if the equals() method for SpawnResults is ever changed then it may be problematic to return SpawnResults in a Set.
Aside: ActionResults use SpawnResults to calculate cumulative execution times for Actions, and may provide other metrics in future.
RELNOTES: None.
PiperOrigin-RevId: 176579460
|
|
|
|
|
| |
Change-Id: I28eee0bfdcdfc832c44a7a0af4d3c6f443e39a17
PiperOrigin-RevId: 176092054
|
|
|
|
|
|
|
|
|
|
| |
The build event protocol reports certain actions that are particularly
important to the user, in particular failed ones. Also report the precise
command line of those actions, if available. This is useful for IDEs
wanting to report errors in a more precise way.
Change-Id: I04224185ea46c608ad0dfcdb2e772071ca0736b3
PiperOrigin-RevId: 175203502
|
|
|
|
|
|
|
|
|
| |
Knowing the kind of action that was reported (and hence likely failed)
helps users quickly understand what possibly went wrong. So report this
information.
Change-Id: Ie4869234df895f404ac5d2ff6f05026e4e82ef7e
PiperOrigin-RevId: 175043006
|
|
|
|
|
|
|
|
| |
Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own.
This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually.
PiperOrigin-RevId: 175033526
|
|
|
|
|
|
|
| |
and make cumulative execution times available in ActionResults.
RELNOTES: None
PiperOrigin-RevId: 174553272
|
|
|
|
|
|
|
| |
Since the problem or the means to fix it is usually on the user side.
RELNOTES=none
PiperOrigin-RevId: 174348233
|