| Commit message (Collapse) | Author | Age |
|---|
| |
|
|
|
|
| |
Add support for directory trees as artifacts. Closes #4011.
PiperOrigin-RevId: 179691001
|
| |
|
|
|
|
|
|
| |
This is because I want to add another remote execution related tool, the remote_client, which will use the Remote Execution API to fetch blobs from a remote cache. I will use this tool as part of end-to-end tests for remote execution.
TESTED=remote integration tests, presubmit
RELNOTES: None
PiperOrigin-RevId: 177995895
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Refactor the FileSystem class to include the hash function as an
instance field. This allows us to have a different hash function
per FileSystem and removes technical debt, as currently that's
somewhat accomplished by a horrible hack that has a static method
to set the hash function for all FileSystem instances.
The FileSystem's default hash function remains MD5.
RELNOTES: None
PiperOrigin-RevId: 177479772
|
| |
|
|
|
|
|
|
|
|
| |
@ulfjack @buchgr - I'm resubmitting https://github.com/bazelbuild/bazel/pull/3984 on behalf of @mterring to get past CLA issues that are holding it up from merging.
This is a temporary fix for the issue https://github.com/bazelbuild/bazel/issues/3891 while we wait for https://github.com/bazelbuild/bazel/pull/4011 to be reviewed and tested
Closes #4188.
PiperOrigin-RevId: 177276751
|
| |
|
|
|
|
|
|
|
| |
It's a left over. It has already been removed for file
downloads and uploads and is now still used for the
action cache up- and downloads.
Change-Id: Ic65ad0d34cfda75774131094de916e78b8835ff0
PiperOrigin-RevId: 173364883
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This means we no longer keep large action inputs or outputs in memory
during upload.
I adjusted the SimpleBlogStore interface to require clients to pass in
the InputStream length. That allows us to always set Content-length on
uploads. It's polite to do so, so that the server may, e.g.,
preallocate space for the blob.
Fixes https://github.com/bazelbuild/bazel/issues/3250.
Change-Id: I944c9dbc35fa2fa80dce523b0133ea9757bb3973
PiperOrigin-RevId: 172433522
|
| |
|
|
|
|
|
|
|
| |
tests fails, we still want to be able to download the logs and other outputs from CAS.
This fixes a bug introduced by https://github.com/bazelbuild/bazel/commit/562fcf9f5dfd14daea718f77da95b43b1400689b. To reproduce: run a failing test vs a BES service, the test log would not be uploaded.
TESTED=unit tests
PiperOrigin-RevId: 169143428
|
| |
|
|
|
|
| |
Update the callers that only need getMetadata to use the new interface.
PiperOrigin-RevId: 167992239
|
| |
|
|
|
| |
RELNOTES: None
PiperOrigin-RevId: 166841380
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The remote http caching client now prefixes the URL of an action cache
request with 'ac' and cas request with 'cas'. This is useful
as servers might want to distinquish between the action cache and the
cas.
Before this change:
(GET|PUT) /f141ae2d23d0f976599e678da1c51d82fedaf8b1
After this change:
(GET|PUT) /ac/f141ae2d23d0f976599e678da1c51d82fedaf8b1
(GET|PUT) /cas/f141ae2d23d0f976599e678da1c51d82fedaf8b1
This change will likely break any HTTP caching service that assumed the
old URL scheme.
TESTED: Manually using https://github.com/buchgr/bazel-remote
RELNOTES: The remote HTTP/1.1 caching client (--remote_rest_cache) now
distinquishes between action cache and CAS. The request URL for the
action cache is prefixed with 'ac' and the URL for the CAS
is prefixed with 'cas'.
PiperOrigin-RevId: 166831997
|
| |
|
|
|
|
|
|
|
|
| |
If a remote download fails, delete any output files that might have
already been created. Else, this might intefere with a subsequent
locally executed actions that expects none of its output files to
exist. See #3452.
Change-Id: I467a97d05606c586aa257326213940a37dad9dd5
PiperOrigin-RevId: 163336093
|
| |
|
|
|
|
|
|
| |
Also added tests specifically for the output, to ensure we don't break it again.
TESTED=remote worker, unit tests
RELNOTES: fixes #3380
PiperOrigin-RevId: 163283558
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously, it was allocating in-memory buffers for upload, which caused it to
run out of memory on large file uploads (or on many small uploads running
simultaneously).
Unfortunately, we don't create the temporary file in the right location (due
to separation of the BlobStore from the ByteStreamServer), so we copy the file
again to write it to the OnDiskBlobStore, which isn't ideal. There'll need to
be another BlobStore API change to make that work, when the InMemoryBlobStore
is gone.
PiperOrigin-RevId: 160974550
|
| |
|
|
|
|
|
| |
This avoid having to load all the data into memory at once in most cases,
especially for large files.
PiperOrigin-RevId: 160954187
|
| |
|
|
|
|
|
|
|
|
| |
Also extend the API to throw exceptions rather than having to wrap or swallow.
This is in preparation for adding yet another implementation using an on-disk
cache. Having the RemoteWorker keep everything in memory is not good for my
sanity.
PiperOrigin-RevId: 160871586
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Connection pooling is a useful optimization for REST caches that are
far away as it avoids constantly redoing the TCP handshake. It also
prevents large builds from exhausting the local interface's source
ports through tens of thousands of one-transaction connections.
The default connection pool size of 20 is fairly arbitrary. Users
probably want to set this to something close to the value of --jobs.
We introduce some generic infrastructure for closing remote cache
instances and use it to cleanly shutdown the connection pool between
builds.
Change-Id: I73adc29ecae15cc10a1217ffbaa483892bcd4f9a
PiperOrigin-RevId: 160264681
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- merge all the inputs upload functionality into a single ensureInputsPresent
method
- merge all of the action result upload functionality into a single upload
method
- merge all the download functionality into a single download method
This significantly simplifies the caller of this interface, and opens the door
for additional performance improvements in implementations which now have more
control over the upload / download flows; in particular, in the gRPC case, we
can upload stdout / stderr using the existing chunker - upload of stdout /
stderr is no longer serialized with file upload.
In particular, the CachedLocalSpawnRunner test becomes much simpler, since it
no longer needs to handle the previous more complex upload code path.
PiperOrigin-RevId: 160260161
|
| |
|
|
|
|
|
|
|
| |
https://docs.google.com/document/d/1AaGk7fOPByEvpAbqeXIyE8HX_A3_axxNnvroblTZ_6s/edit
Also refactored away the various *Interface* files, no need since unit testing can be done with mocking the appropriate gRPC Impl classes directly (see tests). This also fixes the RemoteSpawnRunner, which should use different objects for remote caching and remote execution, the same way RemoteSpawnStrategy does.
RELNOTES: n/a
PiperOrigin-RevId: 158473700
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Tests basically always finish with non-existent files, and this is not an
error on the server side. Instead, the client (Bazel) checks whether all the
files it expects are present.
The test didn't catch this because it was silently falling back to local
execution.
Non-existent files were leading to an IOException, which caused the remote
worker to log nothing, and silently return an error with no output. Log
errors in the remote worker to make future debugging easier.
Fixes #2887.
PiperOrigin-RevId: 157400131
|
| |
|
|
|
|
|
|
| |
The SpawnInputExpander can return null action inputs to indicate that we
should create an empty file at the corresponding location, without a
corresponding input file.
PiperOrigin-RevId: 154832564
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The new RemoteExecutionClient class only performs remote execution, and
nothing else; all higher-level functions, local rety, etc. will live outside
of the client.
In order to add unit tests, I had to add another layer of indirection between
the Grpc{RemoteExecutor,ActionCache} and GRPC, since GRPC generates final,
non-mockable classes. While a testing approach that uses a fake server can
also get some test coverage (as in GrpcActionCacheTest), it doesn't allow us
to test the full range of bad things that can happen at the GRPC layer.
The cloned implementation uses a single GRPC channel, as was recommended to
me by Jakob, who worked on GRPC. A single channel should be sufficiently
scalable, it's thread-safe, and it performs chunking internally. On the
server-side, the requests from a single channel can be dispatched to a thread
pool, so this should not be a blocker for server-side parallelism.
I also changed it to throw an exception whenever anything bad happens - this
makes it much more obvious if there's still bug in this code; the old code
silently swallows many errors, falling back to local execution, which papers
over many issues.
Furthermore, we now return a RemoteExecutionResult to indicate whether the
action ran at all (regardless of exit code), as well as the exit code.
All in all, this implementation is closer to the production code we're using
internally, although quite a few things are still missing.
The cloned implementation is not hooked up to RemoteSpawnStrategy yet. It
also does not support combining remote caching with local execution, but note
that RemoteSpawnStrategy regressed in that respect and currently also does
not support that mode.
PiperOrigin-RevId: 151578409
|
|
|
The simple distributed caching in Bazel used ConcurrentMap as a blob store. It is incorrect to use an overloaded interface for this purpose.
This change defines a SimpleBlobStore interface that only has put(), get() and containsKey()
methods and allows a simple implementation of a blob store as remote cache for Bazel.
Also updated documentation to summarize the options available in the remote spwan strategy.
There is no functional change.
TESTED=shell integration tests
--
Change-Id: Iedff0bc4f06c4a93c398c53801014d998c3df13b
Reviewed-on: https://cr.bazel.build/9330
PiperOrigin-RevId: 151439467
MOS_MIGRATED_REVID=151439467
|
Hadrien Chauvin
olaola
buchgr
Rahul Malik
Benjamin Peterson
ulfjack
Alpha Lam