| Commit message (Collapse) | Author | Age |
|---|
| |
|
|
|
|
|
|
| |
This observably removes any ill effect of CAS transience.
Closes #5229.
PiperOrigin-RevId: 204010317
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This change introduces concurrent downloads of action outputs
for remote caching/execution. So far, for an action we would
download one output after the other which isn't as bad as it
sounds as we would typically run dozens or hundreds of actions
in parallel. However, for actions with a lot of outputs or graphs
that allow limited parallelism we expect this change to positively
impact performance.
Note, that with this change the AbstractRemoteActionCache will
attempt to always download all outputs concurrently. The actual
parallelism is controlled by the underlying network transport.
The gRPC transport currently enforces no limits on the concurrent
calls, which should be fine given that all calls are multiplexed
on a single network connection. The HTTP/1.1 transport also
enforces no parallelism by default, but I have added the
--remote_max_connections=INT flag which allows to specify an upper
bound on the number of network connections to be open concurrently.
I have introduced this flag as a defensive mechanism for users
who's environment might enforce an upper bound on the number of open
connections, as with this change its possible for the number of
concurrently open connections to dramatically increase (from
NumParallelActions to NumParallelActions * SumParallelActionOutputs).
A side effect of this change is that it puts the infrastructure
for retries and circuit breaking for the HttpBlobStore in place.
RELNOTES: None
PiperOrigin-RevId: 199005510
|
| |
|
|
|
|
|
|
|
| |
- Replace the existing Retrier with Retrier2.
- Rename Retrier2 to Retrier and remove the old Retrier + RetryException
class.
RELNOTES: None.
PiperOrigin-RevId: 177835070
|
| |
|
|
|
|
|
|
| |
Adding unit tests.
TESTED=unit tests
RELNOTES: None
PiperOrigin-RevId: 170750220
|
| |
|
|
|
|
| |
TESTED=remote worker, triggered some errors
RELNOTES: fixes #3305, helps #3356
PiperOrigin-RevId: 161722997
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Make the RemoteSpawnRunner match RemoteSpawnStrategy functionality, including
local fallback, remote caching, and execution. This is done so we can
actually finish the migration to the SpawnRunner API, for which I've had a
pending change for several months now.
- Never throw StatusRuntimeException from the GrpcRemoteCache or the
GrpcExecutor. We almost never do, since the Retrier catches them implcitly,
so a number of catch blocks were already unreachable. Carefully document
the cases where we still need to handle it.
- RemoteSpawnStrategy / RemoteSpawnRunner no longer catch gRPC-specific
exceptions; they should be able to handle any reasonable remote caching /
execution implementation (except we don't have a common interface for
GrpcRemoteExecutor yet), with no dependency on gRPC as such. Note that the
RemoteSpawnStrategy class will actually go away after the SpawnRunner
migration (eventually).
- However, ensure that we _are_ actually throwing CacheNotFoundException;
the retrier implicitly catches that also, so we need to manually unwrap
from RetryException.
- Don't call into the EventHandler from RemoteSpawnStrategy; instead, throw
an exception with the message, and let the higher levels handle the
reporting (we only allow this for exception + local fallback, for which
there's no good reporting API right now).
PiperOrigin-RevId: 161195666
|
|
|
retry strategy may need tuning.
Other behavior changes: swallowing gRPC CANCELLED errors when the thread is interrupted, as these are expected and just make debugging difficult. Also, distinguishing between the gRPC DEADLINE_EXCEEDED caused by the actual command timing out on the server vs. other causes (the former should not be retriable, while the latter should retry).
TESTED=unit tests, remote worker on Bazel
PiperOrigin-RevId: 160605830
|
George Gensure
buchgr
olaola
ulfjack