aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/remote/RemoteOptions.java
Commit message (Collapse)AuthorAge
* Make the fallback strategy for Bazel's remote execution configurable.Gravatar philwo2018-07-30
| | | | | | | | | RELNOTES: When using Bazel's remote execution feature and Bazel has to fallback to local execution for an action, Bazel used non-sandboxed local execution until now. From this release on, you can use the new flag --remote_local_fallback_strategy=<strategy> to tell Bazel which strategy to use in that case. PiperOrigin-RevId: 206566380
* remote: unix socket support for remote http cachingGravatar buchgr2018-07-11
| | | | | | | | | | | This adds support for Unix sockets to Bazel for the remote http cache. See corresponding issue #5098 for discussion. RELNOTES: Introduce the --remote_cache_proxy flag, which allows for remote http caching to connect via a unix domain socket. PiperOrigin-RevId: 204111667
* remote: limit number of open tcp connections by default. Fixes #5491Gravatar buchgr2018-07-02
| | | | | | | | | | | | | | | | This change limits the number of open tcp connections by default to 100 for remote caching. We have had error reports where some use cases Bazel would open so many TCP connections that it crashed/ran out of sockets. The max. number of TCP connections can still be adjusted by specifying --remote_max_connections. See also #5047. RELNOTES: In remote caching we limit the number of open TCP connections to 100 by default. The number can be adjusted by specifying the --remote_max_connections flag. PiperOrigin-RevId: 202958838
* remote: concurrent blob downloads. Fixes #5215Gravatar buchgr2018-06-02
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This change introduces concurrent downloads of action outputs for remote caching/execution. So far, for an action we would download one output after the other which isn't as bad as it sounds as we would typically run dozens or hundreds of actions in parallel. However, for actions with a lot of outputs or graphs that allow limited parallelism we expect this change to positively impact performance. Note, that with this change the AbstractRemoteActionCache will attempt to always download all outputs concurrently. The actual parallelism is controlled by the underlying network transport. The gRPC transport currently enforces no limits on the concurrent calls, which should be fine given that all calls are multiplexed on a single network connection. The HTTP/1.1 transport also enforces no parallelism by default, but I have added the --remote_max_connections=INT flag which allows to specify an upper bound on the number of network connections to be open concurrently. I have introduced this flag as a defensive mechanism for users who's environment might enforce an upper bound on the number of open connections, as with this change its possible for the number of concurrently open connections to dramatically increase (from NumParallelActions to NumParallelActions * SumParallelActionOutputs). A side effect of this change is that it puts the infrastructure for retries and circuit breaking for the HttpBlobStore in place. RELNOTES: None PiperOrigin-RevId: 199005510
* Enable --experimental_remote_spawn_cache by defaultGravatar ulfjack2018-05-28
| | | | | | | | | | | | | | | | | | | | | - It is now an error to specify the gRPC remote execution backend in combination with a local disk or HTTP-based cache. - It is now an error to specify both local disk and HTTP-based caches. Note that before this CL, enabling the local disk cache silently disabled remote execution - we now give an error in that case. With these combination no longer being accepted, remote execution being enabled now means that we only create a RemoteSpawnRunner, and don't provide a SpawnCache. This is not a semantic change - we never created both. In principle, it should be possible for users to combine local execution with remote caching for actions that are marked local or no-remote, and still use remote execution otherwise. However, Bazel cannot currently express this combination of execution strategies. RELNOTES: The --experimental_remote_spawn_cache flag is now enabled by default, and remote caching no longer needs --*_strategy=remote flags (it will fail if they are specified). PiperOrigin-RevId: 198280398
* Allow banning symlink action outputs from being uploaded to a remote cache.Gravatar Benjamin Peterson2018-05-03
| | | | | | | | | | | | | | | | | | | This is mostly a roll-forward of 4465dae23de989f1452e93d0a88ac2a289103dd9, which was reverted by fa36d2f48965b127e8fd397348d16e991135bfb6. The main difference is that the new behavior is now gated behind the --noremote_allow_symlink_upload flag. https://docs.google.com/document/d/1gnOYszitgrLVet3sQk-TKGqIcpkkDsc6aw-izoo-d64 is a design proposal to support symlinks in the remote cache, which would render this change moot. I'd like to be able to prevent incorrect cache behavior until that change is implemented, though. This fixes https://github.com/bazelbuild/bazel/issues/4840 (again). Closes #5122. Change-Id: I2136cfe82c2e1a8a9f5856e12a37d42cabd0e299 PiperOrigin-RevId: 195261827
* remote: introduce --disk_cache_flagGravatar David Ostrovsky2018-05-02
| | | | | | | | | | | | | | | Consolidate the --experimental_local_disk_cache and --experimental_local_disk_cache_path flags into a single --disk_cache= flag. Also, create the cache directory if it doesn't exist. RELNOTES: We replaced the --experimental_local_disk_cache and --experimental_local_disk_cache_path flags into a single --disk_cache flag. Additionally, Bazel now tries to create the disk cache directory if it doesn't exist. Closes #5119. PiperOrigin-RevId: 195070550
* This change adds the writing of the remote execution log to a file behind an ↵Gravatar Googler2018-03-29
| | | | | | experimental flag. It also adds a logging handler for Execute calls so that they are logged. PiperOrigin-RevId: 190991493
* Remove categories from Bazel options.Gravatar ccalvarin2018-03-28
| | | | | | | | | These have all had a chance to be categorized with the OptionDocumentationCategory enum, and the help output already uses the enum-grouped format. The "incompatible changes" category has meaning for --all_incompatible_changes and will be removed separately. RELNOTES: None. PiperOrigin-RevId: 190773778
* Disable concurrent change detectionGravatar ulfjack2018-01-22
| | | | | | | | Regress on #3360. We have reports of Bazel outputting warnings for generated files, which I have been able to reproduce. Apparently, Bazel gets stuck with an old FileContentsProxy for generated files, and is unable to recover. PiperOrigin-RevId: 182772324
* Prevent broken cache entries on concurrent file changesGravatar ulfjack2018-01-19
| | | | | | | | | | | | | | | | Local execution has an inherent race condition: if a user modifies a file while an action is executed, then it is impossible for Bazel to tell which version of the file was actually read during action execution. The file may have been modified before or after the tool has read it, or, in the worst case, the tool may have read both the original and the modified version. In addition, the file may be changed back to the original state before Bazel can check the file, so computing the digest before / after may not be sufficient. This is a concern for both local and remote caches, although the cost of poisoning a shared remote cache is significantly higher, and is what has triggered this work. Fixes #3360. We solve this by keeping a reference to the FileContentsProxy, and using that to check for modificaitons before storing the cache entry. We output a warning if this check fails. This change does not increase memory consumption; Java objects are always allocated in multiples of 8 bytes, we use compressed oops, and the FileArtifactValue currently has 12 bytes worth of fields (excl. object overhead), so adding another pointer is effectively free. As a possible performance optimization on purely local builds, we could also consider not computing digests at all, and only use the FileContentsProxy for caching. PiperOrigin-RevId: 182510358
* Have the RemoteSpawnRunner use the execution platform present in the Spawn ↵Gravatar John Cater2018-01-10
| | | | | | | | | | | to get the remote execution properties. Fixes #4128. This reverts commit 3ce42ef3074ee6d3ac7d9968381c8c0a51d9d38d. Change-Id: I8b9ad5099f6334c2488a22baf05d0b273e10f776 PiperOrigin-RevId: 181550828
* Automated rollback of commit 43f45b58acf10beadbb1221b71dfa06fa1341510.Gravatar jcater2017-12-21
| | | | | | | | | | | | | | | *** Reason for rollback *** Breaks some remote execution clients. *** Original change description *** Have the RemoteSpawnRunner use the execution platform present in the Spawn to get the remote execution properties. Fixes #4128. Change-Id: I7e71caef2465204d2dd8225448d54e52366807e6 PiperOrigin-RevId: 179847553
* Have the RemoteSpawnRunner use the execution platform present in the Spawn ↵Gravatar John Cater2017-12-19
| | | | | | | | | to get the remote execution properties. Fixes #4128. Change-Id: I7e71caef2465204d2dd8225448d54e52366807e6 PiperOrigin-RevId: 179595126
* remote: Rename --remote_rest_cache to --remote_http_cacheGravatar Jakob Buchgraber2017-12-19
| | | | | | | | | | | Call it what it is. RELNOTES: --remote_rest_cache was renamed to --remote_http_cache. Both options keep working in this release, but --remote_rest_cache will be removed in the next release. Change-Id: I9e0b947f2184e0d543e7e19c5c33b6aa851d47d2 PiperOrigin-RevId: 179542826
* remote: deprecate --remote_rest_cache_pool_sizeGravatar Jakob Buchgraber2017-12-19
| | | | | | | | There's no point in having this option. We'll use as many TCP connections as we'll need. Fewer options FTW. Change-Id: I502eadd6a3a35040c7eda05ef49320b273ac26ad PiperOrigin-RevId: 179533022
* Minor fixes to spacing in help messagesGravatar Mike Morearty2017-11-29
| | | | | | | | A few help messages were missing a space after the period at the end of a sentence. Closes #4094. PiperOrigin-RevId: 177282132
* Remove hazelcast dependency from BazelGravatar Alpha Lam2017-10-30
| | | | | | | | | | | | This change removes Bazel's dependency on Hazelcast. This will help to reduce size of the Bazel binary and simplify the usage of remote cache. However Hazelcast library is still kept in the repository and still being used by remote_worker. It is useful as a REST server to allow integration testing with the remote rest cache functionality. Change-Id: Ia21b970cedaec84bc6c13e839509d838acb5756f PiperOrigin-RevId: 173880600
* Internal changeGravatar David Ostrovsky2017-09-25
| | | | PiperOrigin-RevId: 169874059
* Introduce a new SpawnCache API; add a RemoteSpawnCache implementationGravatar ulfjack2017-08-14
| | | | | | | | | | AbstractSpawnRunner now uses a SpawnCache if one is registered, this allows adding caching to any spawn runner without having to be aware of the implementations. I will delete the old CachedLocalSpawnRunner in a follow-up CL. PiperOrigin-RevId: 165024382
* Make the @Option annotation depend on the java version of the tagging enums.Gravatar ccalvarin2017-07-18
| | | | | | | The option filters proto dependency can be removed from the OptionsParser. This is in response to option parser users that want to avoid the bazel-internal proto file in their dependencies. RELNOTES: None. PiperOrigin-RevId: 162249778
* Move the remote initialization code out of RemoteSpawnStrategyGravatar ulfjack2017-07-07
| | | | | | | | This is in preparation for making it use the RemoteSpawnRunner, as part of which it will no longer need to do that. Also, Java style says you shouldn't do work in the constructor, and it's better dependency injection. PiperOrigin-RevId: 161071134
* Enforce the new category and effect tags.Gravatar ccalvarin2017-07-03
| | | | | | | | | | All options need to explicitly list their category and effect. If they are uncategorized, this makes the lack of information obvious. Remove defaults from the annotation to enforce this. Also enforce the sanity check that no option should have UNKNOWN or NO_OP effects listed with other effect tags. Includes some last default sets for options I missed in the previous mass-setting change, and some that were added since. PiperOrigin-RevId: 160641861
* Implement retry logic for the gRPC calls in remote execution and caching. TheGravatar olaola2017-06-30
| | | | | | | | retry strategy may need tuning. Other behavior changes: swallowing gRPC CANCELLED errors when the thread is interrupted, as these are expected and just make debugging difficult. Also, distinguishing between the gRPC DEADLINE_EXCEEDED caused by the actual command timing out on the server vs. other causes (the former should not be retriable, while the latter should retry). TESTED=unit tests, remote worker on Bazel PiperOrigin-RevId: 160605830
* Add the default category and tag to all options.Gravatar ccalvarin2017-06-28
| | | | | | | | Move the default from the annotation to every mention. This makes the incompleteness explicit. Will add the defaults to test targets in a separate change. Once all dependencies are cleaned up, the Option annotation will no longer allow options without the documentationCategory or effectTag, to prevent new options being added without categories while we migrate to the new option categorization. PiperOrigin-RevId: 160281252
* Enable connection pooling for the remote REST cacheGravatar Benjamin Peterson2017-06-27
| | | | | | | | | | | | | | | | | Connection pooling is a useful optimization for REST caches that are far away as it avoids constantly redoing the TCP handshake. It also prevents large builds from exhausting the local interface's source ports through tens of thousands of one-transaction connections. The default connection pool size of 20 is fairly arbitrary. Users probably want to set this to something close to the value of --jobs. We introduce some generic infrastructure for closing remote cache instances and use it to cleanly shutdown the connection pool between builds. Change-Id: I73adc29ecae15cc10a1217ffbaa483892bcd4f9a PiperOrigin-RevId: 160264681
* Disable local fallback by defaultGravatar ulfjack2017-06-09
| | | | | | We don't want people to get used to this. PiperOrigin-RevId: 158508816
* Switching Bazel to use the new remote execution API: ↵Gravatar olaola2017-06-09
| | | | | | | | | https://docs.google.com/document/d/1AaGk7fOPByEvpAbqeXIyE8HX_A3_axxNnvroblTZ_6s/edit Also refactored away the various *Interface* files, no need since unit testing can be done with mocking the appropriate gRPC Impl classes directly (see tests). This also fixes the RemoteSpawnRunner, which should use different objects for remote caching and remote execution, the same way RemoteSpawnStrategy does. RELNOTES: n/a PiperOrigin-RevId: 158473700
* Remote+BES: Stabilize command line flags.Gravatar buchgr2017-05-22
| | | | | | | | | | | | | | | | | | | | Update the command line flags used by remote execution/caching as well as the build event service (BES). Major changes: - Remote execution/caching and BES share flags for authentication and TLS. - Removed API Key authentication from BES, as it's not being used. - Add TLS support to BES upload. - Add --bes_project_id flag. If set, the value is propagated as part of BES lifecycle events. For reviewers: Start your review at CommonRemoteAndBesOptions, BuildEventServiceOptions and RemoteOptions. The other changes are mostly automatic IDE renames of fields and flag updates in shell script tests. RELNOTES: None. PiperOrigin-RevId: 156553857
* Enabling chunking of outputs in the RemoteWorker downloadBlob. This fixes ↵Gravatar olaola2017-04-24
| | | | | | | | | the bug that the RemoteWorker would fail to upload outputs that were too big to send in a single gRPC message. Fixes #2822. RELNOTES: n/a PiperOrigin-RevId: 153710733
* OnePlatform auth support for Bazel, in preparation for next version of the API.Gravatar olaola2017-04-20
| | | | | | TESTED: local server RELNOTES: n/a PiperOrigin-RevId: 153599636
* Adding a temporary flag to Bazel to allow Platform override from the commandGravatar Ola Rozenfeld2017-03-14
| | | | | | | | | | | | | line. This will be used during development to test new toolchains in docker containers. Example usage: --experimental_remote_platform_override='entry:{ name:"a" value:"b" } entry:{ name:"c" value:"d" }' TESTED=local server -- PiperOrigin-RevId: 149933081 MOS_MIGRATED_REVID=149933081
* Adding a small flag allowing to control remote caching without remote ↵Gravatar Ola Rozenfeld2017-03-03
| | | | | | | | execution. Usually it is enabled, and will be triggered whenever a remote_cache is specified, but a remote_worker is not; however, this flag allows to specifically disable it for the cases where remote_worker is defined, but something went wrong and we were forced to execute locally. This is useful to save time when the remote API does not actually support setting the remote action result. -- PiperOrigin-RevId: 149007755 MOS_MIGRATED_REVID=149007755
* Add a flag for disabling local fallback for actions that failed remotely.Gravatar Ola Rozenfeld2016-12-13
| | | | | | -- PiperOrigin-RevId: 141817345 MOS_MIGRATED_REVID=141817345
* Debugging flag (will rarely be used by actual users) that disables remoteGravatar Ola Rozenfeld2016-12-13
| | | | | | | | execution cache. -- PiperOrigin-RevId: 141807596 MOS_MIGRATED_REVID=141807596
* Basic implementation of a remote gRPC based cache.Gravatar Ola Rozenfeld2016-09-21
| | | | | | | TODO during review: add A LOT more tests! -- MOS_MIGRATED_REVID=133702188
* Description redacted.Gravatar Ola Rozenfeld2016-09-20
| | | | | -- MOS_MIGRATED_REVID=133584935
* Simple REST URL cacheGravatar Dan Fabulich2016-07-22
| | | | | | | -- Change-Id: Ied34278c63c74aaf2db5ad90d5f076b3bddbe396 Reviewed-on: https://bazel-review.git.corp.google.com/#/c/4061/ MOS_MIGRATED_REVID=128165927
* Add support for Hazelcast configuration fileGravatar Dan Fabulich2016-07-18
| | | | | | | -- Change-Id: I3255a14a60b7ae7749c49d5a885d92f4f19ec84f Reviewed-on: https://bazel-review.git.corp.google.com/#/c/3980/ MOS_MIGRATED_REVID=127537367
* Remote worker to listen to a specific port for HazelcastGravatar Alpha Lam2016-05-20
| | | | | | | | | | | | This change allows starting remote worker with an embedded hazelcast server that listens to a specific port. This allows test to run reliably. Fixes #1271. -- Change-Id: I0bb36af5837f83cad3d76b8acb50f89cd599ee87 Reviewed-on: https://bazel-review.googlesource.com/c/3650/ MOS_MIGRATED_REVID=122829898
* Prototype for remote execution using gRPC and Netty transportGravatar Alpha Lam2016-05-16
| | | | | | | | | | | | | | | | | | | This change implements a remote worker that executes work (build or test). Bazel will be a client of the remote worker. The communication uses gRPC and Netty as transport. A single remote worker has little advantage over running locally. Additional infrastructure is needed to run workers on multiple machines and distributing the work among them. This change provides the basic building blocks for a distributed build farm. (Mainly reformatting changes compared to https://bazel-review.googlesource.com/3110, some BUILD file changes.) -- Change-Id: If7d285444ef42a6823b59443af17b61b04b9ce6a Reviewed-on: https://bazel-review.googlesource.com/#/c/3110/ MOS_MIGRATED_REVID=122376861
* Implement distributed caching for BazelGravatar Alpha Lam2016-02-10
This patch implements distributed caching for Bazel using Hazelcast. Hazelcast is used as a key value store that stores content of files indexed by the digest of the file. The cache also stores the list of files for an action. The key in this case is the digest from the key of the action and the list of files. In this change I also added the interface for remote execution. The implementation will be added in a subsequent patch. This change is only the first in a series of changes related to distributed caching and remote execution. I plan to revise the APIs and implementation in subsequent changes. -- Change-Id: I569285d6149a4e9f8ba2362682c07a9f1e1943b7 Reviewed-on: https://bazel-review.googlesource.com/#/c/2760/ MOS_MIGRATED_REVID=114325038
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-07 12:18:25 +0000