aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/remote/RemoteModule.java
Commit message (Collapse)AuthorAge
* Remove AbruptException(ExitCode exitCode) constructor.Gravatar lpino2018-05-17
| | | | | | | | | This constructor was creating an Exception with a null message leading to possible NullPointerExceptions in a few places in our codebase. The call sites have been replaced with calls to AbruptException(String message, ExitCode exitCode) with a meaningful message. PiperOrigin-RevId: 196973540
* Allow banning symlink action outputs from being uploaded to a remote cache.Gravatar Benjamin Peterson2018-05-03
| | | | | | | | | | | | | | | | | | | This is mostly a roll-forward of 4465dae23de989f1452e93d0a88ac2a289103dd9, which was reverted by fa36d2f48965b127e8fd397348d16e991135bfb6. The main difference is that the new behavior is now gated behind the --noremote_allow_symlink_upload flag. https://docs.google.com/document/d/1gnOYszitgrLVet3sQk-TKGqIcpkkDsc6aw-izoo-d64 is a design proposal to support symlinks in the remote cache, which would render this change moot. I'd like to be able to prevent incorrect cache behavior until that change is implemented, though. This fixes https://github.com/bazelbuild/bazel/issues/4840 (again). Closes #5122. Change-Id: I2136cfe82c2e1a8a9f5856e12a37d42cabd0e299 PiperOrigin-RevId: 195261827
* This change adds timestamps (call start and end time) to the remote grpc log.Gravatar Googler2018-04-23
| | | | PiperOrigin-RevId: 193937177
* This CL adds log entry protos and logging handlers for Watch, ↵Gravatar Googler2018-04-13
| | | | | | FindMissingBlobs, GetActionResult so that they are logged. unknown commit must be submitted before this for Watch calls to be logged properly. PiperOrigin-RevId: 192794535
* This change adds the writing of the remote execution log to a file behind an ↵Gravatar Googler2018-03-29
| | | | | | experimental flag. It also adds a logging handler for Execute calls so that they are logged. PiperOrigin-RevId: 190991493
* Propagating and printing server logs on demand.Gravatar olaola2018-03-21
| | | | | | | WANT_LGTM=all TESTED=RBE, unit tests RELNOTES: None PiperOrigin-RevId: 189938345
* remote: Add interceptor for logging gRPC calls during remote execution/cachingGravatar Googler2018-03-05
| | | | | | | | | | This provides a io.grpc.ClientInterceptor implementation that can be used to log gRPC call information. The interceptor can select a logging handler to use based on the gRPC method being called (Watch, Execute, Write, etc) to build a LogEntry, which can then be logged after the call has finished. Unit tests for the interceptor are included. In this change, the interceptor is never invoked, nor are there any handlers implemented for any gRPC methods. The interceptor also never tries to log any entries. To avoid circular dependency issues (Remote library will depend on logger which depends on remote library for utils), I've factored out the utility classes from the remote library into their own directory/package as part of this change. PiperOrigin-RevId: 187926516
* remote: add directory support for remote caching and executionGravatar Hadrien Chauvin2017-12-20
| | | | | | Add support for directory trees as artifacts. Closes #4011. PiperOrigin-RevId: 179691001
* remote: Add support for Google Cloud Storage.Gravatar Jakob Buchgraber2017-12-18
| | | | | | | | | | | | | | | | | | | | | | Add support for Google Cloud Storage (GCS) as a HTTP caching backend. This commit mainly adds the infrastructure necessary to authenticate to GCS. Using GCS as a caching backend works as follows: 1) Create a new GCS bucket. 2) Create a service account that can read and write to the GCS bucket and generate a JSON credentials file for it. 3) Invoke Bazel as follows: bazel build --remote_rest_cache=https://storage.googleapis.com/<bucket> --auth_enabled --auth_scope=https://www.googleapis.com/auth/devstorage.read_write --auth_credentials=</path/to/creds.json> I'll add simplification's and docs in a subsequent commit. Change-Id: Ie827d7946a2193b97ea7d9aa72eb15f09de2164d PiperOrigin-RevId: 179406380
* remote: Replace Retrier with Retrier2.Gravatar buchgr2017-12-04
| | | | | | | | | - Replace the existing Retrier with Retrier2. - Rename Retrier2 to Retrier and remove the old Retrier + RetryException class. RELNOTES: None. PiperOrigin-RevId: 177835070
* Refactor the FileSystem API to allow for different hash functions.Gravatar buchgr2017-11-30
| | | | | | | | | | | | | Refactor the FileSystem class to include the hash function as an instance field. This allows us to have a different hash function per FileSystem and removes technical debt, as currently that's somewhat accomplished by a horrible hack that has a static method to set the hash function for all FileSystem instances. The FileSystem's default hash function remains MD5. RELNOTES: None PiperOrigin-RevId: 177479772
* Internal changeGravatar David Ostrovsky2017-09-25
| | | | PiperOrigin-RevId: 169874059
* Passing Bazel metadata in gRPC headers.Gravatar olaola2017-09-21
| | | | | | TESTED=unit tests RELNOTES: none PiperOrigin-RevId: 169395919
* grpc: Consolidate gRPC code from BES and Remote Execution. Fixes #3460, #3486Gravatar buchgr2017-08-04
| | | | | | | | | | | BES and Remote Execution have separate implementations of gRPC channel creation, authentication and TLS. We should merge them, to avoid duplication and bugs. One such bug is #3640, where the BES code had a different implementation for Google Application Default Credentials. RELNOTES: The Build Event Service (BES) client now properly supports Google Applicaton Default Credentials. PiperOrigin-RevId: 164253879
* Merge handleOptions into beforeCommandGravatar ulfjack2017-07-17
| | | | | | | Now that we have the options before calling beforeCommand, there's no need for a separate handleOptions method in the BlazeModule API. Remove it. PiperOrigin-RevId: 162002300
* remote: Improve error handling for --remote_* cmd line flags. Fixes #3361, #3358Gravatar buchgr2017-07-14
| | | | | | | | | | | - Move flag handling into RemoteModule to fail as early as possible. - Make error messages from flag handling human readable. - Fix a bug where remote execution would only support TLS with a root certificate being specified. - If a remote executor without a remote cache is specified, assume the remote cache to be the same as the executor. PiperOrigin-RevId: 161946029
* Rename GrpcActionCache to GrpcRemoteCacheGravatar ulfjack2017-07-05
| | | | PiperOrigin-RevId: 160872755
* Change CAS URI to use the "bytestream" scheme instead of being scheme-lessGravatar ulfjack2017-06-22
| | | | | | This makes the URIs absolute, which makes them easier to process server-side. PiperOrigin-RevId: 159694335
* Adding support for SHA256 for remote execution. Switch remote execution to useGravatar olaola2017-06-20
| | | | | | | | | the currently defined hash function for blobs. Some refactoring. Adding an option to set the hash function in the remote worker, defaulting to the current behavior (unfortunately it is a build option, have not found a clean way to specify it at runtime). BUG=62622420 TESTED=remote worker RELNOTES: none PiperOrigin-RevId: 159473116
* Rewrite StandaloneSpawnStrategy to use LocalSpawnRunnerGravatar ulfjack2017-06-19
| | | | PiperOrigin-RevId: 159221067
* Simplify BlazeModule.beforeCommandGravatar ulfjack2017-06-14
| | | | | | | | | | | Don't pass the Command annotation explicitly, but add it to CommandEnvironment instead; most modules don't need it in the first place, so it was a lot of boilerplate for not much. Also change it so that the command is passed to the constructor. Add some documentation to the beforeCommand method. PiperOrigin-RevId: 158847128
* Fix bug in URI computation in RemoteModuleGravatar ulfjack2017-06-13
| | | | | | | | | | | Also add tests for the CAS path converter. I've also changed the code to explicitly inject the RemoteOptions into the CasPathConverter - note that the class is now static, so it no longer has access to the fields in the RemoteModule class. This change will need to be cherry-picked into 0.5.2. PiperOrigin-RevId: 158816408
* Remote execution: post CAS links to the BEP instead of local file URIsGravatar ulfjack2017-06-09
| | | | | | | | | | | | Also use DigestUtils for performance, which has a static cache of the digests. This isn't ideal, but necessary for now to avoid computing the digests multiple times. We'll need to clean this up afterwards by including the digests in the posted events, and making them available to the PathConverter. I have some changes towards that (as well as some submitted already), but it requires quite some refactoring. RELNOTES: Bazel posts links to the CAS to the BEP if remote caching / execution is enabled PiperOrigin-RevId: 158513165
* Remote tests should not depend on lib:runtimeGravatar buchgr2017-05-31
| | | | | | | | Move AuthAndTLSOptions to its own package, so that tests/remote no longer depends on lib:runtime. RELNOTES: None. PiperOrigin-RevId: 157469629
* Remote+BES: Stabilize command line flags.Gravatar buchgr2017-05-22
| | | | | | | | | | | | | | | | | | | | Update the command line flags used by remote execution/caching as well as the build event service (BES). Major changes: - Remote execution/caching and BES share flags for authentication and TLS. - Removed API Key authentication from BES, as it's not being used. - Add TLS support to BES upload. - Add --bes_project_id flag. If set, the value is propagated as part of BES lifecycle events. For reviewers: Start your review at CommonRemoteAndBesOptions, BuildEventServiceOptions and RemoteOptions. The other changes are mostly automatic IDE renames of fields and flag updates in shell script tests. RELNOTES: None. PiperOrigin-RevId: 156553857
* Refactor simple distributed caching support Gravatar Alpha Lam2017-03-28
| | | | | | | | | | | | | | | | | | | The simple distributed caching in Bazel used ConcurrentMap as a blob store. It is incorrect to use an overloaded interface for this purpose. This change defines a SimpleBlobStore interface that only has put(), get() and containsKey() methods and allows a simple implementation of a blob store as remote cache for Bazel. Also updated documentation to summarize the options available in the remote spwan strategy. There is no functional change. TESTED=shell integration tests -- Change-Id: Iedff0bc4f06c4a93c398c53801014d998c3df13b Reviewed-on: https://cr.bazel.build/9330 PiperOrigin-RevId: 151439467 MOS_MIGRATED_REVID=151439467
* Move some classes to lib.exec in preparation for layering enforcementGravatar Ulf Adams2016-12-21
| | | | | | | | | | | | This change is part of the mu-bazel effort, which aims to build a minimally useful Bazel binary with most extraneous functionality removed. As part of that, we want to enforce layering of packages. In particular, lib.actions must not depend on lib.rules or lib.exec. lib.rules must not depend on lib.exec. Moving these classes is a necessary step to enforce that layering. -- PiperOrigin-RevId: 142668172 MOS_MIGRATED_REVID=142668172
* Creating separate instances of CAS and execution handlers for every action. ↵Gravatar Ola Rozenfeld2016-12-16
| | | | | | | | | | This allows Bazel to talk to multiple instances of the server, if these exist, enabling server-side parallelism (due to using separate gRPC channels). TESTED: internally and local server -- PiperOrigin-RevId: 142262973 MOS_MIGRATED_REVID=142262973
* Attempt #2 to submit commit e860316559eac366d47923a8eb4b5489a661aa35: Using ↵Gravatar Ola Rozenfeld2016-11-24
| | | | | | | | | | an ActionFileInputCache for SHA1 digests used with remote execution. I missed an important test failure -- turned out we have a remote execution test that does not live under lib/remote. -- MOS_MIGRATED_REVID=140135277
* Rollback of commit e860316559eac366d47923a8eb4b5489a661aa35.Gravatar Alex Humesky2016-11-21
| | | | | | | *** Reason for rollback *** -- MOS_MIGRATED_REVID=139640949
* Using an ActionFileInputCache for SHA1 digests used with remote execution.Gravatar Ola Rozenfeld2016-11-21
| | | | | -- MOS_MIGRATED_REVID=139613925
* Refactor the module API to use the builder pattern for executor creation.Gravatar Ulf Adams2016-11-02
| | | | | | | This significantly simplifies several of our modules. -- MOS_MIGRATED_REVID=137713119
* Basic implementation of a remote gRPC based cache.Gravatar Ola Rozenfeld2016-09-21
| | | | | | | TODO during review: add A LOT more tests! -- MOS_MIGRATED_REVID=133702188
* Description redacted.Gravatar Ola Rozenfeld2016-09-20
| | | | | -- MOS_MIGRATED_REVID=133584935
* Stop catching InterruptedException in remote execution.Gravatar Janak Ramakrishnan2016-08-18
| | | | | | | | | | | | If the user interrupts the build, they probably don't want local fallback to happen. Also clean up an unused method and parameter. -- Change-Id: I6bf80fa110bbba911b0743f24c25240c208c98d1 Reviewed-on: https://bazel-review.googlesource.com/5470 MOS_MIGRATED_REVID=130612791
* Rename MemcacheActionCache to ConcurrentMapActionCacheGravatar Dan Fabulich2016-07-22
| | | | | | | -- Change-Id: I7c36fcfee46f4340aa6d17c03b6599beef45f524 Reviewed-on: https://bazel-review.git.corp.google.com/#/c/4062/1 MOS_MIGRATED_REVID=128167039
* Simple REST URL cacheGravatar Dan Fabulich2016-07-22
| | | | | | | -- Change-Id: Ied34278c63c74aaf2db5ad90d5f076b3bddbe396 Reviewed-on: https://bazel-review.git.corp.google.com/#/c/4061/ MOS_MIGRATED_REVID=128165927
* Add support for Hazelcast configuration fileGravatar Dan Fabulich2016-07-18
| | | | | | | -- Change-Id: I3255a14a60b7ae7749c49d5a885d92f4f19ec84f Reviewed-on: https://bazel-review.git.corp.google.com/#/c/3980/ MOS_MIGRATED_REVID=127537367
* Split BlazeModule.getCommandOptions into two; unify all implementations.Gravatar Ulf Adams2016-06-23
| | | | | | | | | | | Several modules now explicitly add common command options. Of the remaining ones, most add options to the build command, except one, which adds options to query. They now all use the canonical implementation. Also updated the documentation to clarify what this method actually does. -- MOS_MIGRATED_REVID=125560058
* Prototype for remote execution using gRPC and Netty transportGravatar Alpha Lam2016-05-16
| | | | | | | | | | | | | | | | | | | This change implements a remote worker that executes work (build or test). Bazel will be a client of the remote worker. The communication uses gRPC and Netty as transport. A single remote worker has little advantage over running locally. Additional infrastructure is needed to run workers on multiple machines and distributing the work among them. This change provides the basic building blocks for a distributed build farm. (Mainly reformatting changes compared to https://bazel-review.googlesource.com/3110, some BUILD file changes.) -- Change-Id: If7d285444ef42a6823b59443af17b61b04b9ce6a Reviewed-on: https://bazel-review.googlesource.com/#/c/3110/ MOS_MIGRATED_REVID=122376861
* Move most BlazeRuntime methods related to workspace directories to CommandEnv.Gravatar Ulf Adams2016-03-30
| | | | | -- MOS_MIGRATED_REVID=118561661
* Implement distributed caching for BazelGravatar Alpha Lam2016-02-10
This patch implements distributed caching for Bazel using Hazelcast. Hazelcast is used as a key value store that stores content of files indexed by the digest of the file. The cache also stores the list of files for an action. The key in this case is the digest from the key of the action and the list of files. In this change I also added the interface for remote execution. The implementation will be added in a subsequent patch. This change is only the first in a series of changes related to distributed caching and remote execution. I plan to revise the APIs and implementation in subsequent changes. -- Change-Id: I569285d6149a4e9f8ba2362682c07a9f1e1943b7 Reviewed-on: https://bazel-review.googlesource.com/#/c/2760/ MOS_MIGRATED_REVID=114325038
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-05 15:38:39 +0000