aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/remote
Commit message (Collapse)AuthorAge
* Automated rollback of commit 43f45b58acf10beadbb1221b71dfa06fa1341510.Gravatar jcater2017-12-21
| | | | | | | | | | | | | | | *** Reason for rollback *** Breaks some remote execution clients. *** Original change description *** Have the RemoteSpawnRunner use the execution platform present in the Spawn to get the remote execution properties. Fixes #4128. Change-Id: I7e71caef2465204d2dd8225448d54e52366807e6 PiperOrigin-RevId: 179847553
* remote: fix download of output directoriesGravatar buchgr2017-12-20
| | | | | | | | Fix a bug where Bazel would crash if two Directory protos had the same hash. RELNOTES: Remote Caching and Execution support output directories. PiperOrigin-RevId: 179731040
* remote: add directory support for remote caching and executionGravatar Hadrien Chauvin2017-12-20
| | | | | | Add support for directory trees as artifacts. Closes #4011. PiperOrigin-RevId: 179691001
* remote: delete dead codeGravatar Jakob Buchgraber2017-12-20
| | | | | Change-Id: Iacbba3eced0abc0dcfd7311a0f07da48cbaba6e4 PiperOrigin-RevId: 179664071
* Have the RemoteSpawnRunner use the execution platform present in the Spawn ↵Gravatar John Cater2017-12-19
| | | | | | | | | to get the remote execution properties. Fixes #4128. Change-Id: I7e71caef2465204d2dd8225448d54e52366807e6 PiperOrigin-RevId: 179595126
* remote: Rename --remote_rest_cache to --remote_http_cacheGravatar Jakob Buchgraber2017-12-19
| | | | | | | | | | | Call it what it is. RELNOTES: --remote_rest_cache was renamed to --remote_http_cache. Both options keep working in this release, but --remote_rest_cache will be removed in the next release. Change-Id: I9e0b947f2184e0d543e7e19c5c33b6aa851d47d2 PiperOrigin-RevId: 179542826
* remote: deprecate --remote_rest_cache_pool_sizeGravatar Jakob Buchgraber2017-12-19
| | | | | | | | There's no point in having this option. We'll use as many TCP connections as we'll need. Fewer options FTW. Change-Id: I502eadd6a3a35040c7eda05ef49320b273ac26ad PiperOrigin-RevId: 179533022
* remote: update READMEGravatar buchgr2017-12-18
| | | | | | | | - Mention sandboxing for remote caching/execution. - Mention buchgr/bazel-remote-cache as a remote caching backend. RELNOTES: None. PiperOrigin-RevId: 179515966
* remote: fix out-of-date documentationGravatar Jakob Buchgraber2017-12-18
| | | | | | | The action cache prefix is "ac", not "actioncache". Change-Id: I841a026133ab3b4ec5b58a0cf29252dae49434fe PiperOrigin-RevId: 179437730
* remote: fix rest cache documentation. Fixes #4246Gravatar buchgr2017-12-18
| | | | | | | | Describe the HTTP protocol in some detail. Also improve the wording of remote caching a bit. RELNOTES: None. PiperOrigin-RevId: 179433738
* remote: Add support for Google Cloud Storage.Gravatar Jakob Buchgraber2017-12-18
| | | | | | | | | | | | | | | | | | | | | | Add support for Google Cloud Storage (GCS) as a HTTP caching backend. This commit mainly adds the infrastructure necessary to authenticate to GCS. Using GCS as a caching backend works as follows: 1) Create a new GCS bucket. 2) Create a service account that can read and write to the GCS bucket and generate a JSON credentials file for it. 3) Invoke Bazel as follows: bazel build --remote_rest_cache=https://storage.googleapis.com/<bucket> --auth_enabled --auth_scope=https://www.googleapis.com/auth/devstorage.read_write --auth_credentials=</path/to/creds.json> I'll add simplification's and docs in a subsequent commit. Change-Id: Ie827d7946a2193b97ea7d9aa72eb15f09de2164d PiperOrigin-RevId: 179406380
* Fix: uploading artifacts of failed actions to remote cache stopped working.Gravatar olaola2017-12-11
| | | | | | | | | | | | | | | | To reproduce: run a failing test with --experimental_remote_spawn_cache or with --spawn_strategy=remote and no executor. Expected: test log is uploaded. Desired behavior: - regardless of whether a spawn is cacheable or not, its artifacts should be uploaded to the remote cache. - the spawn result should only be set if the spawn is cacheable *and* the action succeeded. - when executing remotely, the do_not_cache field should be set for non-cacheable spawns, and the remote execution engine should respect it. This CL contains multiple fixes to ensure the above behaviors, and adds a few tests, both end to end and unit tests. Important behavior change: it is no longer assumed that non-cacheable spawns should use a NO_CACHE SpawnCache! The appropriate test case was removed. Instead, an assumption was added that all implementations of SpawnCache should respect the Spawns.mayBeCached(spawn) property. Currently, only NO_CACHE and RemoteSpawnCache exist, and they (now) support it. TESTED=remote build execution backend. WANT_LGTM: philwo,buchgr RELNOTES: None PiperOrigin-RevId: 178617937
* Moving the RemoteWorker into tools/remote directory.Gravatar olaola2017-12-05
| | | | | | | | This is because I want to add another remote execution related tool, the remote_client, which will use the Remote Execution API to fetch blobs from a remote cache. I will use this tool as part of end-to-end tests for remote execution. TESTED=remote integration tests, presubmit RELNOTES: None PiperOrigin-RevId: 177995895
* remote: Have --remote_rest_cache respect --remote_timeout.Gravatar buchgr2017-12-05
| | | | | RELNOTES: The --remote_rest_cache flag now respects --remote_timeout. PiperOrigin-RevId: 177926523
* remote: Replace Retrier with Retrier2.Gravatar buchgr2017-12-04
| | | | | | | | | - Replace the existing Retrier with Retrier2. - Rename Retrier2 to Retrier and remove the old Retrier + RetryException class. RELNOTES: None. PiperOrigin-RevId: 177835070
* Change Bazel's default hash function to SHA256. Fixes #3193Gravatar buchgr2017-12-03
| | | | | | | RELNOTES: Bazel's default hash function was changed from MD5 to SHA256. In particular, this affects users of remote caching and execution, as all hashes will be SHA256 by default. PiperOrigin-RevId: 177740702
* Refactor the FileSystem API to allow for different hash functions.Gravatar buchgr2017-11-30
| | | | | | | | | | | | | Refactor the FileSystem class to include the hash function as an instance field. This allows us to have a different hash function per FileSystem and removes technical debt, as currently that's somewhat accomplished by a horrible hack that has a static method to set the hash function for all FileSystem instances. The FileSystem's default hash function remains MD5. RELNOTES: None PiperOrigin-RevId: 177479772
* remote: don't hide non-test failures behind test failures. Fixes #4082Gravatar buchgr2017-11-30
| | | | | | | | | | Bazel should display the root cause of a test failure to the user. For example, if a test could not be executed on a remote executor due to there being no network connection, then it shouldn't display the test as failed but tell the user about the network error. RELNOTES: PiperOrigin-RevId: 177439578
* Add a CommandLineBuilder for the process-wrapper embedded tool, and use it ↵Gravatar ruperts2017-11-30
| | | | | | | | | everywhere instead of duplicating process-wrapper --shell_arguments in Blaze. To avoid a cyclic dependency, I broke up exec/local:local into exec/local:local and exec/local:options. RELNOTES: None. PiperOrigin-RevId: 177419268
* RestBlobStore: add missing CAS_PREFIXGravatar nicolov2017-11-29
| | | | | | | | | According to the SimpleBlobStore interface `containsKey` should look up blobs in the CAS. However, the URL it requests is missing `CAS_PREFIX`, so it will never find anything. Closes #4118. PiperOrigin-RevId: 177313658
* Clean up ExecutionRequirementsGravatar ulfjack2017-11-29
| | | | | | | | | | | | | | | | | | | - remove BaseSpawn.Local; instead, all callers pass in the full set of execution requirements they want to set - disable caching and sandboxing for the symlink tree action - it does not declare outputs, so it can't be cached or sandboxed (fixes #4041) - centralize the existing execution requirements in the ExecutionRequirements class - centralize checking for execution requirements in the Spawn class (it's possible that we may need a more decentralized, extensible design in the future, but for now having them in a single place is simple and effective) - update the documentation - forward the relevant tags to execution requirements in TargetUtils (progress on #3960) - this also contributes to #4153 PiperOrigin-RevId: 177288598
* Minor fixes to spacing in help messagesGravatar Mike Morearty2017-11-29
| | | | | | | | A few help messages were missing a space after the period at the end of a sentence. Closes #4094. PiperOrigin-RevId: 177282132
* Change error on adding a directory to the remote cache to be non-fatalGravatar Rahul Malik2017-11-29
| | | | | | | | | | @ulfjack @buchgr - I'm resubmitting https://github.com/bazelbuild/bazel/pull/3984 on behalf of @mterring to get past CLA issues that are holding it up from merging. This is a temporary fix for the issue https://github.com/bazelbuild/bazel/issues/3891 while we wait for https://github.com/bazelbuild/bazel/pull/4011 to be reviewed and tested Closes #4188. PiperOrigin-RevId: 177276751
* Simplify SpawnRunner interfaceGravatar ulfjack2017-11-28
| | | | | | | | | | | | | | It turns out that the SUCCESS status is often misunderstood to mean "zero exit", even though this is clearly documented. I've decided to add another status for non-zero exit, and use success only for zero exit to avoid this pitfall. Also, many of the status codes are set, but never used. I decided to reduce the number of status codes to only those that are actually relevant, which simplifies further processing. Instead, we should add a string message for the error case when we need one - we're not using it right now, so I decided not to add that yet. PiperOrigin-RevId: 177129441
* Adding checks for empty metadata and digest.Gravatar olaola2017-11-17
| | | | | | | | This will avoid the uninformative NullPointerException for this case (#3913). The underlying cause is data dependency on directories, which we want to disable. In the future, it will be allowed, but in a more explicit form (e.g. trailing slash). TESTED=no RELNOTES: None PiperOrigin-RevId: 176109375
* More resilient to remote spawn cache errorsGravatar Alpha Lam2017-11-16
| | | | | | | | | | Bazel used to fail the entire build if there is a single error with remote spawn cache. This could happen on unreliable network or when server is under load. This change will allow Bazel to fallback to local execution with a warning. Change-Id: I0bd41dc862b33a2ab2ab772d8ce9c943476edbf4 PiperOrigin-RevId: 175951699
* Replace all usages of Blaze's Preconditions class with guava.Gravatar tomlu2017-11-09
| | | | | | | | Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own. This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually. PiperOrigin-RevId: 175033526
* Remove hazelcast dependency from BazelGravatar Alpha Lam2017-10-30
| | | | | | | | | | | | This change removes Bazel's dependency on Hazelcast. This will help to reduce size of the Bazel binary and simplify the usage of remote cache. However Hazelcast library is still kept in the repository and still being used by remote_worker. It is useful as a REST server to allow integration testing with the remote rest cache functionality. Change-Id: Ia21b970cedaec84bc6c13e839509d838acb5756f PiperOrigin-RevId: 173880600
* remote: remove maximum blob size check.Gravatar Jakob Buchgraber2017-10-25
| | | | | | | | | It's a left over. It has already been removed for file downloads and uploads and is now still used for the action cache up- and downloads. Change-Id: Ic65ad0d34cfda75774131094de916e78b8835ff0 PiperOrigin-RevId: 173364883
* Properly attaching context for remote uploads in RemoteSpawnCache.Gravatar olaola2017-10-20
| | | | | | | | Fixes #3930. Also added tests. TESTED=added tests RELNOTES: Fixing regression to --experimental_remote_spawn_cache PiperOrigin-RevId: 172852740
* Stream rest cache file uploads.Gravatar Benjamin Peterson2017-10-18
| | | | | | | | | | | | | | | This means we no longer keep large action inputs or outputs in memory during upload. I adjusted the SimpleBlogStore interface to require clients to pass in the InputStream length. That allows us to always set Content-length on uploads. It's polite to do so, so that the server may, e.g., preallocate space for the blob. Fixes https://github.com/bazelbuild/bazel/issues/3250. Change-Id: I944c9dbc35fa2fa80dce523b0133ea9757bb3973 PiperOrigin-RevId: 172433522
* Fixing #3834, making sure test.log always exists.Gravatar olaola2017-10-18
| | | | | | | | Even if the test action produced no output, which it really shouldn't, Bazel should create an empty test.log file. TESTED=unit tests RELNOTES: Fixes #3834 PiperOrigin-RevId: 172412615
* Removing local fallback on failed action commands.Gravatar olaola2017-10-18
| | | | | | | | We should only fall back if a remote execution error occurred, not if the command itself failed. TESTED=better unit tests RELNOTES: None PiperOrigin-RevId: 172406687
* Simplify the SpawnExecException constructorGravatar ulfjack2017-10-16
| | | | | | | Whether or not there was a catastrophic error is stored in the SpawnResult, so we can just use that instead of passing in an additional boolean. PiperOrigin-RevId: 172083752
* Fixing displayed retry attempts one-off error, when the error is non-retriable.Gravatar olaola2017-10-06
| | | | | | | | Adding unit tests. TESTED=unit tests RELNOTES: None PiperOrigin-RevId: 170750220
* Using only one GetMissingBlobs RPC per action instead of two.Gravatar olaola2017-09-29
| | | | | | | | This reduces the # of round-trips, improving overall latency, although I didn't profile by how much. TESTED=with remote execution RELNOTES: None PiperOrigin-RevId: 170484009
* Making RESOURCE_EXHAUSTED a retriable error, per guideine in: ↵Gravatar olaola2017-09-29
| | | | | | | | https://cloud.google.com/pubsub/docs/reference/error-codes TESTED=not really RELNOTES: None PiperOrigin-RevId: 170455553
* Checking both old and new error fields on remote execute operation.Gravatar olaola2017-09-29
| | | | | | | The old field is the error on Operation proto. The new field is the ExecuteResponse status field. Note that the new field will also allow us to fetch logs for timing out tests, resolving a TODO, but this is not yet done is this change. PiperOrigin-RevId: 170370676
* Internal changeGravatar David Ostrovsky2017-09-25
| | | | PiperOrigin-RevId: 169874059
* Move SpawnResult from build.lib.exec into build.lib.actions so that e.g. ↵Gravatar ruperts2017-09-22
| | | | | | | build.lib.actions.SpawnActionContext can import SpawnResult without creating a cyclic dependency. RELNOTES: None. PiperOrigin-RevId: 169642267
* Removing deprecated field total_input_file_count.Gravatar olaola2017-09-21
| | | | | | | | | See API Revisions design doc: https://docs.google.com/document/d/12c3oAPgedckLpue2yj0xGgJTEOyUm4mXWWBJ157J-8I/edit#heading=h.llz6ymkp07b1 The BuildInfo is now sent via RequestMetadata. TESTED=we never used this field PiperOrigin-RevId: 169432884
* Passing Bazel metadata in gRPC headers.Gravatar olaola2017-09-21
| | | | | | TESTED=unit tests RELNOTES: none PiperOrigin-RevId: 169395919
* Uploading failed action outputs to the remote cache, because even if the ↵Gravatar olaola2017-09-19
| | | | | | | | | tests fails, we still want to be able to download the logs and other outputs from CAS. This fixes a bug introduced by https://github.com/bazelbuild/bazel/commit/562fcf9f5dfd14daea718f77da95b43b1400689b. To reproduce: run a failing test vs a BES service, the test log would not be uploaded. TESTED=unit tests PiperOrigin-RevId: 169143428
* Delete the unused CachedLocalSpawnRunner, which is superseeded by SpawnCacheGravatar ulfjack2017-09-12
| | | | PiperOrigin-RevId: 168359681
* remote: Add new retrier with support for circuit breakingGravatar buchgr2017-09-12
| | | | | | | | | | | | | | | | | | | | | | | | | Add a generic retrier implementation (Retrier2) that can be configured by plugging in a backoff strategy, a function to decide on retriable errors and a circuit breaker. A concrete implementation is added via RemoteRetrier that mostly is a copy of the code of the existing Retrier. Retrier2 adds support for circuit breaking [1]. It allows the retrier to reject execution when failure rates are high. The remote execution code will use this to gently switch between local and remote execution/caching if the latter experiences lots of failures. Retrier2 is also useful when not used with gRPC. We need retriers for the HTTP caching interface too. All the code added in this CL is unused, to keep reviews managable. In a follow up CL, I will switch the code to use the new Retrier and delete the old retrier. [1] https://martinfowler.com/bliki/CircuitBreaker.html PiperOrigin-RevId: 168355597
* Automatic code cleanup.Gravatar cushon2017-09-11
| | | | PiperOrigin-RevId: 168141877
* ActionInputFileCache: move getMetadata to a new super-interfaceGravatar ulfjack2017-09-11
| | | | | | Update the callers that only need getMetadata to use the new interface. PiperOrigin-RevId: 167992239
* More BUILD file refactorings.Gravatar philwo2017-09-06
| | | | | | | | | Split collect, concurrent, vfs, windows into package-level BUILD files. Move clock classes out of "util", into their own Java package. Move CompactHashSet into its own Java package to break a dependency cycle. Give nestedset and inmemoryfs their own package-level BUILD files. PiperOrigin-RevId: 167702127
* Extract authandtls, buildeventservice, buildeventstream into package-level ↵Gravatar philwo2017-09-04
| | | | | | | | BUILD files. Replace all ":relative" labels with "//absolute:path" labels for easier search & replace. PiperOrigin-RevId: 167500985
* remote: Return exit code 34 for remote caching/execution errors.Gravatar buchgr2017-09-01
| | | | | | | | | | For any errors that are due to failures in the remote caching / execution layers Bazel now returns exit code 34 (ExitCode.REMOTE_ERROR). This includes errors where the remote cache / executor is unreachable or crashes. It does not include errors if the test / build failure is due to user errors i.e. compilation or test failures. PiperOrigin-RevId: 167259236
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-07 04:35:23 +0000