aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/remote/RemoteModule.java
Commit message (Collapse)AuthorAge
* remote: add a ByteStreamBuildEventArtifactUploaderGravatar buchgr2018-07-12
| | | | | | | | | | | | | | | This change allows local files referenced by the BEP/BES protocol to be uploaded to a ByteStream gRPC service. The ByteStreamUploader is now implicitly also used by the BES module which has a different lifecycle than the remote module. We introduce reference counting to ensure that the channel is closed after its no longer needed. This also fixes a bug where we currently leak one socket per remote build until the Bazel server is shut down. RELNOTES: None PiperOrigin-RevId: 204275316
* Pass options to BuildEventArtifactUploaderFactory.Gravatar tomlu2018-07-11
| | | | | | | The artifact uploaders may need command-level options. RELNOTES: None PiperOrigin-RevId: 204151808
* Retry ensureInputsPresent/execute/downloadGravatar George Gensure2018-07-10
| | | | | | | | This observably removes any ill effect of CAS transience. Closes #5229. PiperOrigin-RevId: 204010317
* bep: introduce BuildEventArtifactUploaderFactoryGravatar buchgr2018-07-09
| | | | | | | | | | | | | | | | There can be multiple BuildEventTransports active at the same time and we need to ensure that each transport gets its own BuildEventArtifactUploader as these transports might have different lifecycles. We do that by introducing another level of indirection via the BuildEventArtifactUploaderFactory. BlazeModules now register a factory object instead of an uploader. In addition, the BuildEventArtifactUploader gets a shutdown() method that allows to free any resources associated with it. PiperOrigin-RevId: 203752092
* Include more information about build events' referenced local files.Gravatar tomlu2018-07-04
| | | | | | | Instead of just a path, events now include information about the type of file (output, source file, stdout/stderr, test logs, etc.). This information can be used by the uploaders to determine a) whether to upload, b) what kind of lease to give the files. RELNOTES: None PiperOrigin-RevId: 203285549
* bep: Make the BuildEventArtifactUploader async.Gravatar buchgr2018-06-29
| | | | | | | | | | This changes the BuildEventArtifactUploader to an async interface, thereby no longer potentially delaying event delivery over the eventbus. Additionally, the BES transport is changed to start uploading local files immediately as the events are delivered. RELNOTES: None PiperOrigin-RevId: 202694121
* Move HashFunction out of FileSystem, and turn it into a class, instead of an ↵Gravatar ccalvarin2018-06-21
| | | | | | | | | enum. Now that we aren't using enum names for the hash functions, we also accept the standard names, such as SHA-256. RELNOTES: None. PiperOrigin-RevId: 201624286
* Add a mechanism for build event protocol events to upload filesGravatar ulfjack2018-06-15
| | | | | | | | This should be a no-op, mostly replacing PathConverter with BuildEventArtifactUploader, since none of the implementations perform any upload yet. PiperOrigin-RevId: 200685325
* remote: concurrent blob downloads. Fixes #5215Gravatar buchgr2018-06-02
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This change introduces concurrent downloads of action outputs for remote caching/execution. So far, for an action we would download one output after the other which isn't as bad as it sounds as we would typically run dozens or hundreds of actions in parallel. However, for actions with a lot of outputs or graphs that allow limited parallelism we expect this change to positively impact performance. Note, that with this change the AbstractRemoteActionCache will attempt to always download all outputs concurrently. The actual parallelism is controlled by the underlying network transport. The gRPC transport currently enforces no limits on the concurrent calls, which should be fine given that all calls are multiplexed on a single network connection. The HTTP/1.1 transport also enforces no parallelism by default, but I have added the --remote_max_connections=INT flag which allows to specify an upper bound on the number of network connections to be open concurrently. I have introduced this flag as a defensive mechanism for users who's environment might enforce an upper bound on the number of open connections, as with this change its possible for the number of concurrently open connections to dramatically increase (from NumParallelActions to NumParallelActions * SumParallelActionOutputs). A side effect of this change is that it puts the infrastructure for retries and circuit breaking for the HttpBlobStore in place. RELNOTES: None PiperOrigin-RevId: 199005510
* Enable --experimental_remote_spawn_cache by defaultGravatar ulfjack2018-05-28
| | | | | | | | | | | | | | | | | | | | | - It is now an error to specify the gRPC remote execution backend in combination with a local disk or HTTP-based cache. - It is now an error to specify both local disk and HTTP-based caches. Note that before this CL, enabling the local disk cache silently disabled remote execution - we now give an error in that case. With these combination no longer being accepted, remote execution being enabled now means that we only create a RemoteSpawnRunner, and don't provide a SpawnCache. This is not a semantic change - we never created both. In principle, it should be possible for users to combine local execution with remote caching for actions that are marked local or no-remote, and still use remote execution otherwise. However, Bazel cannot currently express this combination of execution strategies. RELNOTES: The --experimental_remote_spawn_cache flag is now enabled by default, and remote caching no longer needs --*_strategy=remote flags (it will fail if they are specified). PiperOrigin-RevId: 198280398
* Remove AbruptException(ExitCode exitCode) constructor.Gravatar lpino2018-05-17
| | | | | | | | | This constructor was creating an Exception with a null message leading to possible NullPointerExceptions in a few places in our codebase. The call sites have been replaced with calls to AbruptException(String message, ExitCode exitCode) with a meaningful message. PiperOrigin-RevId: 196973540
* Allow banning symlink action outputs from being uploaded to a remote cache.Gravatar Benjamin Peterson2018-05-03
| | | | | | | | | | | | | | | | | | | This is mostly a roll-forward of 4465dae23de989f1452e93d0a88ac2a289103dd9, which was reverted by fa36d2f48965b127e8fd397348d16e991135bfb6. The main difference is that the new behavior is now gated behind the --noremote_allow_symlink_upload flag. https://docs.google.com/document/d/1gnOYszitgrLVet3sQk-TKGqIcpkkDsc6aw-izoo-d64 is a design proposal to support symlinks in the remote cache, which would render this change moot. I'd like to be able to prevent incorrect cache behavior until that change is implemented, though. This fixes https://github.com/bazelbuild/bazel/issues/4840 (again). Closes #5122. Change-Id: I2136cfe82c2e1a8a9f5856e12a37d42cabd0e299 PiperOrigin-RevId: 195261827
* This change adds timestamps (call start and end time) to the remote grpc log.Gravatar Googler2018-04-23
| | | | PiperOrigin-RevId: 193937177
* This CL adds log entry protos and logging handlers for Watch, ↵Gravatar Googler2018-04-13
| | | | | | FindMissingBlobs, GetActionResult so that they are logged. unknown commit must be submitted before this for Watch calls to be logged properly. PiperOrigin-RevId: 192794535
* This change adds the writing of the remote execution log to a file behind an ↵Gravatar Googler2018-03-29
| | | | | | experimental flag. It also adds a logging handler for Execute calls so that they are logged. PiperOrigin-RevId: 190991493
* Propagating and printing server logs on demand.Gravatar olaola2018-03-21
| | | | | | | WANT_LGTM=all TESTED=RBE, unit tests RELNOTES: None PiperOrigin-RevId: 189938345
* remote: Add interceptor for logging gRPC calls during remote execution/cachingGravatar Googler2018-03-05
| | | | | | | | | | This provides a io.grpc.ClientInterceptor implementation that can be used to log gRPC call information. The interceptor can select a logging handler to use based on the gRPC method being called (Watch, Execute, Write, etc) to build a LogEntry, which can then be logged after the call has finished. Unit tests for the interceptor are included. In this change, the interceptor is never invoked, nor are there any handlers implemented for any gRPC methods. The interceptor also never tries to log any entries. To avoid circular dependency issues (Remote library will depend on logger which depends on remote library for utils), I've factored out the utility classes from the remote library into their own directory/package as part of this change. PiperOrigin-RevId: 187926516
* remote: add directory support for remote caching and executionGravatar Hadrien Chauvin2017-12-20
| | | | | | Add support for directory trees as artifacts. Closes #4011. PiperOrigin-RevId: 179691001
* remote: Add support for Google Cloud Storage.Gravatar Jakob Buchgraber2017-12-18
| | | | | | | | | | | | | | | | | | | | | | Add support for Google Cloud Storage (GCS) as a HTTP caching backend. This commit mainly adds the infrastructure necessary to authenticate to GCS. Using GCS as a caching backend works as follows: 1) Create a new GCS bucket. 2) Create a service account that can read and write to the GCS bucket and generate a JSON credentials file for it. 3) Invoke Bazel as follows: bazel build --remote_rest_cache=https://storage.googleapis.com/<bucket> --auth_enabled --auth_scope=https://www.googleapis.com/auth/devstorage.read_write --auth_credentials=</path/to/creds.json> I'll add simplification's and docs in a subsequent commit. Change-Id: Ie827d7946a2193b97ea7d9aa72eb15f09de2164d PiperOrigin-RevId: 179406380
* remote: Replace Retrier with Retrier2.Gravatar buchgr2017-12-04
| | | | | | | | | - Replace the existing Retrier with Retrier2. - Rename Retrier2 to Retrier and remove the old Retrier + RetryException class. RELNOTES: None. PiperOrigin-RevId: 177835070
* Refactor the FileSystem API to allow for different hash functions.Gravatar buchgr2017-11-30
| | | | | | | | | | | | | Refactor the FileSystem class to include the hash function as an instance field. This allows us to have a different hash function per FileSystem and removes technical debt, as currently that's somewhat accomplished by a horrible hack that has a static method to set the hash function for all FileSystem instances. The FileSystem's default hash function remains MD5. RELNOTES: None PiperOrigin-RevId: 177479772
* Internal changeGravatar David Ostrovsky2017-09-25
| | | | PiperOrigin-RevId: 169874059
* Passing Bazel metadata in gRPC headers.Gravatar olaola2017-09-21
| | | | | | TESTED=unit tests RELNOTES: none PiperOrigin-RevId: 169395919
* grpc: Consolidate gRPC code from BES and Remote Execution. Fixes #3460, #3486Gravatar buchgr2017-08-04
| | | | | | | | | | | BES and Remote Execution have separate implementations of gRPC channel creation, authentication and TLS. We should merge them, to avoid duplication and bugs. One such bug is #3640, where the BES code had a different implementation for Google Application Default Credentials. RELNOTES: The Build Event Service (BES) client now properly supports Google Applicaton Default Credentials. PiperOrigin-RevId: 164253879
* Merge handleOptions into beforeCommandGravatar ulfjack2017-07-17
| | | | | | | Now that we have the options before calling beforeCommand, there's no need for a separate handleOptions method in the BlazeModule API. Remove it. PiperOrigin-RevId: 162002300
* remote: Improve error handling for --remote_* cmd line flags. Fixes #3361, #3358Gravatar buchgr2017-07-14
| | | | | | | | | | | - Move flag handling into RemoteModule to fail as early as possible. - Make error messages from flag handling human readable. - Fix a bug where remote execution would only support TLS with a root certificate being specified. - If a remote executor without a remote cache is specified, assume the remote cache to be the same as the executor. PiperOrigin-RevId: 161946029
* Rename GrpcActionCache to GrpcRemoteCacheGravatar ulfjack2017-07-05
| | | | PiperOrigin-RevId: 160872755
* Change CAS URI to use the "bytestream" scheme instead of being scheme-lessGravatar ulfjack2017-06-22
| | | | | | This makes the URIs absolute, which makes them easier to process server-side. PiperOrigin-RevId: 159694335
* Adding support for SHA256 for remote execution. Switch remote execution to useGravatar olaola2017-06-20
| | | | | | | | | the currently defined hash function for blobs. Some refactoring. Adding an option to set the hash function in the remote worker, defaulting to the current behavior (unfortunately it is a build option, have not found a clean way to specify it at runtime). BUG=62622420 TESTED=remote worker RELNOTES: none PiperOrigin-RevId: 159473116
* Rewrite StandaloneSpawnStrategy to use LocalSpawnRunnerGravatar ulfjack2017-06-19
| | | | PiperOrigin-RevId: 159221067
* Simplify BlazeModule.beforeCommandGravatar ulfjack2017-06-14
| | | | | | | | | | | Don't pass the Command annotation explicitly, but add it to CommandEnvironment instead; most modules don't need it in the first place, so it was a lot of boilerplate for not much. Also change it so that the command is passed to the constructor. Add some documentation to the beforeCommand method. PiperOrigin-RevId: 158847128
* Fix bug in URI computation in RemoteModuleGravatar ulfjack2017-06-13
| | | | | | | | | | | Also add tests for the CAS path converter. I've also changed the code to explicitly inject the RemoteOptions into the CasPathConverter - note that the class is now static, so it no longer has access to the fields in the RemoteModule class. This change will need to be cherry-picked into 0.5.2. PiperOrigin-RevId: 158816408
* Remote execution: post CAS links to the BEP instead of local file URIsGravatar ulfjack2017-06-09
| | | | | | | | | | | | Also use DigestUtils for performance, which has a static cache of the digests. This isn't ideal, but necessary for now to avoid computing the digests multiple times. We'll need to clean this up afterwards by including the digests in the posted events, and making them available to the PathConverter. I have some changes towards that (as well as some submitted already), but it requires quite some refactoring. RELNOTES: Bazel posts links to the CAS to the BEP if remote caching / execution is enabled PiperOrigin-RevId: 158513165
* Remote tests should not depend on lib:runtimeGravatar buchgr2017-05-31
| | | | | | | | Move AuthAndTLSOptions to its own package, so that tests/remote no longer depends on lib:runtime. RELNOTES: None. PiperOrigin-RevId: 157469629
* Remote+BES: Stabilize command line flags.Gravatar buchgr2017-05-22
| | | | | | | | | | | | | | | | | | | | Update the command line flags used by remote execution/caching as well as the build event service (BES). Major changes: - Remote execution/caching and BES share flags for authentication and TLS. - Removed API Key authentication from BES, as it's not being used. - Add TLS support to BES upload. - Add --bes_project_id flag. If set, the value is propagated as part of BES lifecycle events. For reviewers: Start your review at CommonRemoteAndBesOptions, BuildEventServiceOptions and RemoteOptions. The other changes are mostly automatic IDE renames of fields and flag updates in shell script tests. RELNOTES: None. PiperOrigin-RevId: 156553857
* Refactor simple distributed caching support Gravatar Alpha Lam2017-03-28
| | | | | | | | | | | | | | | | | | | The simple distributed caching in Bazel used ConcurrentMap as a blob store. It is incorrect to use an overloaded interface for this purpose. This change defines a SimpleBlobStore interface that only has put(), get() and containsKey() methods and allows a simple implementation of a blob store as remote cache for Bazel. Also updated documentation to summarize the options available in the remote spwan strategy. There is no functional change. TESTED=shell integration tests -- Change-Id: Iedff0bc4f06c4a93c398c53801014d998c3df13b Reviewed-on: https://cr.bazel.build/9330 PiperOrigin-RevId: 151439467 MOS_MIGRATED_REVID=151439467
* Move some classes to lib.exec in preparation for layering enforcementGravatar Ulf Adams2016-12-21
| | | | | | | | | | | | This change is part of the mu-bazel effort, which aims to build a minimally useful Bazel binary with most extraneous functionality removed. As part of that, we want to enforce layering of packages. In particular, lib.actions must not depend on lib.rules or lib.exec. lib.rules must not depend on lib.exec. Moving these classes is a necessary step to enforce that layering. -- PiperOrigin-RevId: 142668172 MOS_MIGRATED_REVID=142668172
* Creating separate instances of CAS and execution handlers for every action. ↵Gravatar Ola Rozenfeld2016-12-16
| | | | | | | | | | This allows Bazel to talk to multiple instances of the server, if these exist, enabling server-side parallelism (due to using separate gRPC channels). TESTED: internally and local server -- PiperOrigin-RevId: 142262973 MOS_MIGRATED_REVID=142262973
* Attempt #2 to submit commit e860316559eac366d47923a8eb4b5489a661aa35: Using ↵Gravatar Ola Rozenfeld2016-11-24
| | | | | | | | | | an ActionFileInputCache for SHA1 digests used with remote execution. I missed an important test failure -- turned out we have a remote execution test that does not live under lib/remote. -- MOS_MIGRATED_REVID=140135277
* Rollback of commit e860316559eac366d47923a8eb4b5489a661aa35.Gravatar Alex Humesky2016-11-21
| | | | | | | *** Reason for rollback *** -- MOS_MIGRATED_REVID=139640949
* Using an ActionFileInputCache for SHA1 digests used with remote execution.Gravatar Ola Rozenfeld2016-11-21
| | | | | -- MOS_MIGRATED_REVID=139613925
* Refactor the module API to use the builder pattern for executor creation.Gravatar Ulf Adams2016-11-02
| | | | | | | This significantly simplifies several of our modules. -- MOS_MIGRATED_REVID=137713119
* Basic implementation of a remote gRPC based cache.Gravatar Ola Rozenfeld2016-09-21
| | | | | | | TODO during review: add A LOT more tests! -- MOS_MIGRATED_REVID=133702188
* Description redacted.Gravatar Ola Rozenfeld2016-09-20
| | | | | -- MOS_MIGRATED_REVID=133584935
* Stop catching InterruptedException in remote execution.Gravatar Janak Ramakrishnan2016-08-18
| | | | | | | | | | | | If the user interrupts the build, they probably don't want local fallback to happen. Also clean up an unused method and parameter. -- Change-Id: I6bf80fa110bbba911b0743f24c25240c208c98d1 Reviewed-on: https://bazel-review.googlesource.com/5470 MOS_MIGRATED_REVID=130612791
* Rename MemcacheActionCache to ConcurrentMapActionCacheGravatar Dan Fabulich2016-07-22
| | | | | | | -- Change-Id: I7c36fcfee46f4340aa6d17c03b6599beef45f524 Reviewed-on: https://bazel-review.git.corp.google.com/#/c/4062/1 MOS_MIGRATED_REVID=128167039
* Simple REST URL cacheGravatar Dan Fabulich2016-07-22
| | | | | | | -- Change-Id: Ied34278c63c74aaf2db5ad90d5f076b3bddbe396 Reviewed-on: https://bazel-review.git.corp.google.com/#/c/4061/ MOS_MIGRATED_REVID=128165927
* Add support for Hazelcast configuration fileGravatar Dan Fabulich2016-07-18
| | | | | | | -- Change-Id: I3255a14a60b7ae7749c49d5a885d92f4f19ec84f Reviewed-on: https://bazel-review.git.corp.google.com/#/c/3980/ MOS_MIGRATED_REVID=127537367
* Split BlazeModule.getCommandOptions into two; unify all implementations.Gravatar Ulf Adams2016-06-23
| | | | | | | | | | | Several modules now explicitly add common command options. Of the remaining ones, most add options to the build command, except one, which adds options to query. They now all use the canonical implementation. Also updated the documentation to clarify what this method actually does. -- MOS_MIGRATED_REVID=125560058
* Prototype for remote execution using gRPC and Netty transportGravatar Alpha Lam2016-05-16
| | | | | | | | | | | | | | | | | | | This change implements a remote worker that executes work (build or test). Bazel will be a client of the remote worker. The communication uses gRPC and Netty as transport. A single remote worker has little advantage over running locally. Additional infrastructure is needed to run workers on multiple machines and distributing the work among them. This change provides the basic building blocks for a distributed build farm. (Mainly reformatting changes compared to https://bazel-review.googlesource.com/3110, some BUILD file changes.) -- Change-Id: If7d285444ef42a6823b59443af17b61b04b9ce6a Reviewed-on: https://bazel-review.googlesource.com/#/c/3110/ MOS_MIGRATED_REVID=122376861
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-10 06:18:42 +0000