| Commit message (Collapse) | Author | Age |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The new RemoteExecutionClient class only performs remote execution, and
nothing else; all higher-level functions, local rety, etc. will live outside
of the client.
In order to add unit tests, I had to add another layer of indirection between
the Grpc{RemoteExecutor,ActionCache} and GRPC, since GRPC generates final,
non-mockable classes. While a testing approach that uses a fake server can
also get some test coverage (as in GrpcActionCacheTest), it doesn't allow us
to test the full range of bad things that can happen at the GRPC layer.
The cloned implementation uses a single GRPC channel, as was recommended to
me by Jakob, who worked on GRPC. A single channel should be sufficiently
scalable, it's thread-safe, and it performs chunking internally. On the
server-side, the requests from a single channel can be dispatched to a thread
pool, so this should not be a blocker for server-side parallelism.
I also changed it to throw an exception whenever anything bad happens - this
makes it much more obvious if there's still bug in this code; the old code
silently swallows many errors, falling back to local execution, which papers
over many issues.
Furthermore, we now return a RemoteExecutionResult to indicate whether the
action ran at all (regardless of exit code), as well as the exit code.
All in all, this implementation is closer to the production code we're using
internally, although quite a few things are still missing.
The cloned implementation is not hooked up to RemoteSpawnStrategy yet. It
also does not support combining remote caching with local execution, but note
that RemoteSpawnStrategy regressed in that respect and currently also does
not support that mode.
PiperOrigin-RevId: 151578409
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The simple distributed caching in Bazel used ConcurrentMap as a blob store. It is incorrect to use an overloaded interface for this purpose.
This change defines a SimpleBlobStore interface that only has put(), get() and containsKey()
methods and allows a simple implementation of a blob store as remote cache for Bazel.
Also updated documentation to summarize the options available in the remote spwan strategy.
There is no functional change.
TESTED=shell integration tests
--
Change-Id: Iedff0bc4f06c4a93c398c53801014d998c3df13b
Reviewed-on: https://cr.bazel.build/9330
PiperOrigin-RevId: 151439467
MOS_MIGRATED_REVID=151439467
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
execution.
If you're feeling like you've already seen this, that's correct, these were the exact contents of commit e860316559eac366d47923a8eb4b5489a661aa35... and then, on Nov 15, something unclear happened and the code disappeared! Perhaps it was the result of a faulty sync. In any case, nobody noticed, and the CL went in. It was later rolloed back and resubmitted, but the crucial code changes were gone.
TESTED=local server with profiling for SHA1 specifically
RELNOTES: n/a
--
PiperOrigin-RevId: 151139685
MOS_MIGRATED_REVID=151139685
|
|
|
|
|
|
|
|
|
|
| |
This fixes remote test execution.
Fixes #1593.
--
PiperOrigin-RevId: 151030133
MOS_MIGRATED_REVID=151030133
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
line. This will be used during development to test new toolchains in docker
containers.
Example usage: --experimental_remote_platform_override='entry:{ name:"a" value:"b" } entry:{ name:"c" value:"d" }'
TESTED=local server
--
PiperOrigin-RevId: 149933081
MOS_MIGRATED_REVID=149933081
|
|
|
|
|
|
|
|
|
|
| |
It's silly that we require every spawn strategy to do this individually, and
the new spawn scheduler will fix this. However, it's useful to add this for
debugging.
--
PiperOrigin-RevId: 149743992
MOS_MIGRATED_REVID=149743992
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 149110466
MOS_MIGRATED_REVID=149110466
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Move all local resource acquisition to where local execution actually happens.
Don't attempt to acquire resources per action, but only for individual spawns.
This significantly simplifies the code.
The downside is that we don't account for action-level work anymore. In
general, actions should not perform any process execution themselves, but
always delegate such work to a SpawnStrategy implementation.
This change makes sure that every Spawn has local resources set in a way that
is consistent with the previous state.
However, there are two actions - Fileset and FileWrite -, which are not spawns,
and so we now don't limit their concurrent execution anymore. For Fileset, all
work is done in a custom Fileset-specific thread pool, so this shouldn't be a
problem. I'm not sure about FileWriteAction.
--
PiperOrigin-RevId: 149012600
MOS_MIGRATED_REVID=149012600
|
|
|
|
|
|
|
|
| |
execution. Usually it is enabled, and will be triggered whenever a remote_cache is specified, but a remote_worker is not; however, this flag allows to specifically disable it for the cases where remote_worker is defined, but something went wrong and we were forced to execute locally. This is useful to save time when the remote API does not actually support setting the remote action result.
--
PiperOrigin-RevId: 149007755
MOS_MIGRATED_REVID=149007755
|
|
|
|
|
|
|
|
|
|
| |
Specifying both options can cause OOM on OSX.
--
Change-Id: I52daf194a8840f9e63f1d537f13152e53f8436a7
Reviewed-on: https://cr.bazel.build/8220
PiperOrigin-RevId: 145079331
MOS_MIGRATED_REVID=145079331
|
|
|
|
|
|
|
|
|
|
| |
This allows Bazel to talk to multiple instances of the server, if these exist, enabling server-side parallelism (due to using separate gRPC channels).
TESTED: internally and local server
--
PiperOrigin-RevId: 142262973
MOS_MIGRATED_REVID=142262973
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 141817345
MOS_MIGRATED_REVID=141817345
|
|
|
|
|
|
|
|
| |
execution cache.
--
PiperOrigin-RevId: 141807596
MOS_MIGRATED_REVID=141807596
|
|
|
|
|
|
|
|
| |
Helps debugging.
--
PiperOrigin-RevId: 141802189
MOS_MIGRATED_REVID=141802189
|
|
|
|
|
|
|
|
|
|
| |
an ActionFileInputCache for SHA1 digests used with remote execution.
I missed an important test failure -- turned out we have a remote execution
test that does not live under lib/remote.
--
MOS_MIGRATED_REVID=140135277
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
--
MOS_MIGRATED_REVID=139640949
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=139613925
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=133584935
|
|
|
|
|
|
|
| |
in places other than the sandbox code.
--
MOS_MIGRATED_REVID=132436150
|
|
|
|
|
|
|
|
|
|
|
|
| |
If the user interrupts the build, they probably don't want local
fallback to happen.
Also clean up an unused method and parameter.
--
Change-Id: I6bf80fa110bbba911b0743f24c25240c208c98d1
Reviewed-on: https://bazel-review.googlesource.com/5470
MOS_MIGRATED_REVID=130612791
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
ActionInputFileCache:
Change getDigest() to return the underlying byte[16] owned by each
FileArtifactValue.
Remove throws clause from getInputFromDigest(); this should be an
in-memory operation, and no implementation actually throws.
PerActionFileCache:
Invert mapping from artifact to digest only if needed. Remove interner,
as it was used only for the reverse map keys, not the returned values.
This should be a significant cpu savings as eagerly constructing the
reverse maps was a noticeable hotspot.
--
MOS_MIGRATED_REVID=127972359
|
|
|
|
|
|
|
|
|
| |
BlazeRuntime#getProductName() or a reference to TestConstants.PRODUCT_NAME for tests.
This CL prepares the codebase in order to delete the constant.
--
MOS_MIGRATED_REVID=122993568
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This change implements a remote worker that executes work (build or test).
Bazel will be a client of the remote worker. The communication uses gRPC
and Netty as transport.
A single remote worker has little advantage over running locally. Additional
infrastructure is needed to run workers on multiple machines and distributing
the work among them.
This change provides the basic building blocks for a distributed build farm.
(Mainly reformatting changes compared to https://bazel-review.googlesource.com/3110, some BUILD file changes.)
--
Change-Id: If7d285444ef42a6823b59443af17b61b04b9ce6a
Reviewed-on: https://bazel-review.googlesource.com/#/c/3110/
MOS_MIGRATED_REVID=122376861
|
|
|
|
|
|
|
| |
Except in action execution logic (ActionExecutionFunction, SkyframeActionExecutor, etc.), switch Action interface references to either ActionAnalysisMetadata if possible or ActionExecutionMetadata.
--
MOS_MIGRATED_REVID=120723431
|
|
|
|
|
|
|
|
|
| |
--sandbox_debug and --verbose_failures are on. See discussion in #1049.
RELNOTES:
--
MOS_MIGRATED_REVID=119635080
|
|
|
|
|
|
|
| |
willExecuteRemotely(). Remove unused parameter "mnemonic".
--
MOS_MIGRATED_REVID=115666410
|
|
This patch implements distributed caching for Bazel using Hazelcast.
Hazelcast is used as a key value store that stores content of files
indexed by the digest of the file. The cache also stores the list of files
for an action. The key in this case is the digest from the key of the action
and the list of files.
In this change I also added the interface for remote execution. The
implementation will be added in a subsequent patch.
This change is only the first in a series of changes related to distributed
caching and remote execution. I plan to revise the APIs and implementation
in subsequent changes.
--
Change-Id: I569285d6149a4e9f8ba2362682c07a9f1e1943b7
Reviewed-on: https://bazel-review.googlesource.com/#/c/2760/
MOS_MIGRATED_REVID=114325038
|