aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/remote/BUILD
Commit message (Collapse)AuthorAge
* remote: Rewrite the HTTP caching client in Netty. Fixes #4481Gravatar buchgr2018-01-26
| | | | | | | | | | | | | | * This puts in the foundation of HTTP/2 support for remote caching. * Allows us to remove the Apache HTTP library as a dependency, reducing the Bazel binary size by 1MiB. On fast networks (i.e. GCE to GCS) we can see a >2x speed improvement for TLS throughput. Even from my workstation to GCS I get significant build time improvements when using Netty's TLS 18s vs 12s. Closes #4481. PiperOrigin-RevId: 183411787
* Remove some unnecessary depsGravatar Ulf Adams2018-01-15
| | | | | | | | | Change-Id: Ib8dd9265b18fa0915f52427567845105fcdfa295 Closes #4447. Change-Id: Ib8dd9265b18fa0915f52427567845105fcdfa295 PiperOrigin-RevId: 181943004
* Have the RemoteSpawnRunner use the execution platform present in the Spawn ↵Gravatar John Cater2018-01-10
| | | | | | | | | | | to get the remote execution properties. Fixes #4128. This reverts commit 3ce42ef3074ee6d3ac7d9968381c8c0a51d9d38d. Change-Id: I8b9ad5099f6334c2488a22baf05d0b273e10f776 PiperOrigin-RevId: 181550828
* Automated rollback of commit 43f45b58acf10beadbb1221b71dfa06fa1341510.Gravatar jcater2017-12-21
| | | | | | | | | | | | | | | *** Reason for rollback *** Breaks some remote execution clients. *** Original change description *** Have the RemoteSpawnRunner use the execution platform present in the Spawn to get the remote execution properties. Fixes #4128. Change-Id: I7e71caef2465204d2dd8225448d54e52366807e6 PiperOrigin-RevId: 179847553
* Have the RemoteSpawnRunner use the execution platform present in the Spawn ↵Gravatar John Cater2017-12-19
| | | | | | | | | to get the remote execution properties. Fixes #4128. Change-Id: I7e71caef2465204d2dd8225448d54e52366807e6 PiperOrigin-RevId: 179595126
* remote: Add support for Google Cloud Storage.Gravatar Jakob Buchgraber2017-12-18
| | | | | | | | | | | | | | | | | | | | | | Add support for Google Cloud Storage (GCS) as a HTTP caching backend. This commit mainly adds the infrastructure necessary to authenticate to GCS. Using GCS as a caching backend works as follows: 1) Create a new GCS bucket. 2) Create a service account that can read and write to the GCS bucket and generate a JSON credentials file for it. 3) Invoke Bazel as follows: bazel build --remote_rest_cache=https://storage.googleapis.com/<bucket> --auth_enabled --auth_scope=https://www.googleapis.com/auth/devstorage.read_write --auth_credentials=</path/to/creds.json> I'll add simplification's and docs in a subsequent commit. Change-Id: Ie827d7946a2193b97ea7d9aa72eb15f09de2164d PiperOrigin-RevId: 179406380
* Add a CommandLineBuilder for the process-wrapper embedded tool, and use it ↵Gravatar ruperts2017-11-30
| | | | | | | | | everywhere instead of duplicating process-wrapper --shell_arguments in Blaze. To avoid a cyclic dependency, I broke up exec/local:local into exec/local:local and exec/local:options. RELNOTES: None. PiperOrigin-RevId: 177419268
* Remove hazelcast dependency from BazelGravatar Alpha Lam2017-10-30
| | | | | | | | | | | | This change removes Bazel's dependency on Hazelcast. This will help to reduce size of the Bazel binary and simplify the usage of remote cache. However Hazelcast library is still kept in the repository and still being used by remote_worker. It is useful as a REST server to allow integration testing with the remote rest cache functionality. Change-Id: Ia21b970cedaec84bc6c13e839509d838acb5756f PiperOrigin-RevId: 173880600
* Passing Bazel metadata in gRPC headers.Gravatar olaola2017-09-21
| | | | | | TESTED=unit tests RELNOTES: none PiperOrigin-RevId: 169395919
* More BUILD file refactorings.Gravatar philwo2017-09-06
| | | | | | | | | Split collect, concurrent, vfs, windows into package-level BUILD files. Move clock classes out of "util", into their own Java package. Move CompactHashSet into its own Java package to break a dependency cycle. Give nestedset and inmemoryfs their own package-level BUILD files. PiperOrigin-RevId: 167702127
* Extract authandtls, buildeventservice, buildeventstream into package-level ↵Gravatar philwo2017-09-04
| | | | | | | | BUILD files. Replace all ":relative" labels with "//absolute:path" labels for easier search & replace. PiperOrigin-RevId: 167500985
* Make the @Option annotation depend on the java version of the tagging enums.Gravatar ccalvarin2017-07-18
| | | | | | | The option filters proto dependency can be removed from the OptionsParser. This is in response to option parser users that want to avoid the bazel-internal proto file in their dependencies. RELNOTES: None. PiperOrigin-RevId: 162249778
* Reimplement RemoteSpawnStrategy on top of RemoteSpawnRunnerGravatar ulfjack2017-07-10
| | | | | | | | | | | | | | | | | It is intentional that RemoteSpawnStrategy no longer contains any remote execution specific code. My intent is to merge all SpawnStrategy implementations into a single class (similar to the new RemoteSpawnStrategy), and delegate all the specific work to SpawnRunner implementations. However, we're not there yet, and we still need to be able to look up SpawnStrategy implementations by name through the annotations, so we still need separate classes for now. We might also want to have a shared test suite for all SpawnRunner instances that checks for basic compliance with the specification. Progress on #1531. PiperOrigin-RevId: 161377751
* Move the SimpleBlobStore and implementations to a subpackageGravatar ulfjack2017-07-05
| | | | | | | | | | Also extend the API to throw exceptions rather than having to wrap or swallow. This is in preparation for adding yet another implementation using an on-disk cache. Having the RemoteWorker keep everything in memory is not good for my sanity. PiperOrigin-RevId: 160871586
* Add the default category and tag to all options.Gravatar ccalvarin2017-06-28
| | | | | | | | Move the default from the annotation to every mention. This makes the incompleteness explicit. Will add the defaults to test targets in a separate change. Once all dependencies are cleaned up, the Option annotation will no longer allow options without the documentationCategory or effectTag, to prevent new options being added without categories while we migrate to the new option categorization. PiperOrigin-RevId: 160281252
* Rewrite StandaloneSpawnStrategy to use LocalSpawnRunnerGravatar ulfjack2017-06-19
| | | | PiperOrigin-RevId: 159221067
* Switching to Watcher API instead of wait_for_completion, in preparation forGravatar olaola2017-06-14
| | | | | | | | | deprecating the wait_for_completion field. Note on errors: in the RemoteWorker, I currently handle all errors as onError of the watch call. Other options are: pass them as the operation error field, and pass some of them as the onError of the execute call. For now, I'm just using the simplest option; the Bazel client is ready to handle all possible options. RELNOTES: none PiperOrigin-RevId: 158974207
* Remote execution: post CAS links to the BEP instead of local file URIsGravatar ulfjack2017-06-09
| | | | | | | | | | | | Also use DigestUtils for performance, which has a static cache of the digests. This isn't ideal, but necessary for now to avoid computing the digests multiple times. We'll need to clean this up afterwards by including the digests in the posted events, and making them available to the PathConverter. I have some changes towards that (as well as some submitted already), but it requires quite some refactoring. RELNOTES: Bazel posts links to the CAS to the BEP if remote caching / execution is enabled PiperOrigin-RevId: 158513165
* Switching Bazel to use the new remote execution API: ↵Gravatar olaola2017-06-09
| | | | | | | | | https://docs.google.com/document/d/1AaGk7fOPByEvpAbqeXIyE8HX_A3_axxNnvroblTZ_6s/edit Also refactored away the various *Interface* files, no need since unit testing can be done with mocking the appropriate gRPC Impl classes directly (see tests). This also fixes the RemoteSpawnRunner, which should use different objects for remote caching and remote execution, the same way RemoteSpawnStrategy does. RELNOTES: n/a PiperOrigin-RevId: 158473700
* protobuf: Use bazel's native java_proto_library rules.Gravatar Jakob Buchgraber2017-06-06
| | | | | | | | | Additional changes: - Introduce a Skylark macro java_library_srcs that provides the source jars of a java_*_library rule. - Remove bazel's own java_proto_library implementation. Change-Id: I18f2259bc75ca0fb32dcd8a6a857c609bd2c7773 PiperOrigin-RevId: 158146210
* Remote tests should not depend on lib:runtimeGravatar buchgr2017-05-31
| | | | | | | | Move AuthAndTLSOptions to its own package, so that tests/remote no longer depends on lib:runtime. RELNOTES: None. PiperOrigin-RevId: 157469629
* OnePlatform auth support for Bazel, in preparation for next version of the API.Gravatar olaola2017-04-20
| | | | | | TESTED: local server RELNOTES: n/a PiperOrigin-RevId: 153599636
* Open source LocalSpawnRunnerGravatar ulfjack2017-04-07
| | | | | | | | | The LocalSpawnRunner is a non-sandboxed local execution implementation, which will replace the current StandaloneSpawnStrategy. The code has been around for a long time and has seen a lot of bugfixes. It also supports local prefetching, which is required for Google. I have a follow-up change to make it support Windows, so it's not a drop-in replacement for StandaloneSpawnStrategy yet. PiperOrigin-RevId: 152486973
* Use //third_party/protobuf:protobuf_java instead of ↵Gravatar Carmi Grushko2017-02-09
| | | | | | | | | | //third_party/protobuf:protobuf to refer to the Java proto runtime. This is the name in the upstream protobuf repo. -- PiperOrigin-RevId: 146949832 MOS_MIGRATED_REVID=146949832
* Implementing directory Merkle trees for remote execution/caching (seeGravatar Ola Rozenfeld2016-09-14
| | | | | | | | | | | | | | | https://docs.google.com/document/d/1hh63AzKlwcOJN6jBZzY3GNPffzww-JKx1015DfFKM6g/edit#). A directory is represented by recursive structure of TreeNodes, which are interned in a single TreeNodeRepository. Common subtrees are shared between various parents. In the change we introduce only creating the tree from an existing list of ActionInputs; in the future, we should refactor to populate the TreeNodeRepository alongside (or maybe even within) the ArtifactFactory. The plan is for the TreeNodeRepository to become a new centralized location for all Artifact data to be computed and cached. -- MOS_MIGRATED_REVID=133080315
* Prototype for remote execution using gRPC and Netty transportGravatar Alpha Lam2016-05-16
| | | | | | | | | | | | | | | | | | | This change implements a remote worker that executes work (build or test). Bazel will be a client of the remote worker. The communication uses gRPC and Netty as transport. A single remote worker has little advantage over running locally. Additional infrastructure is needed to run workers on multiple machines and distributing the work among them. This change provides the basic building blocks for a distributed build farm. (Mainly reformatting changes compared to https://bazel-review.googlesource.com/3110, some BUILD file changes.) -- Change-Id: If7d285444ef42a6823b59443af17b61b04b9ce6a Reviewed-on: https://bazel-review.googlesource.com/#/c/3110/ MOS_MIGRATED_REVID=122376861
* Add Python protobuf to Bazel third_party and move all protobuf targets intoGravatar David Chen2016-03-17
| | | | | | | //third_party/protobuf. -- MOS_MIGRATED_REVID=117336377
* Rename proto dependencies to include a _java component in them.Gravatar Julio Merino2016-02-23
| | | | | | | | While doing this, homogenize the two dependencies on build.proto (via build_proto and build_proto_v2) into a single build_java_proto. -- MOS_MIGRATED_REVID=115328219
* Implement distributed caching for BazelGravatar Alpha Lam2016-02-10
This patch implements distributed caching for Bazel using Hazelcast. Hazelcast is used as a key value store that stores content of files indexed by the digest of the file. The cache also stores the list of files for an action. The key in this case is the digest from the key of the action and the list of files. In this change I also added the interface for remote execution. The implementation will be added in a subsequent patch. This change is only the first in a series of changes related to distributed caching and remote execution. I plan to revise the APIs and implementation in subsequent changes. -- Change-Id: I569285d6149a4e9f8ba2362682c07a9f1e1943b7 Reviewed-on: https://bazel-review.googlesource.com/#/c/2760/ MOS_MIGRATED_REVID=114325038
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-08 23:15:48 +0000