aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/remote
Commit message (Collapse)AuthorAge
...
* Inject the product name in the constructor where neededGravatar ulfjack2018-02-20
| | | | | | ...instead of passing it everywhere so we can rewrite the local env. PiperOrigin-RevId: 186316859
* Fix env for local fallback (from remote exec) on Windows and LinuxGravatar ulfjack2018-02-20
| | | | PiperOrigin-RevId: 186309181
* Fix typo in cache download failure warningGravatar Alex Ryan2018-02-20
| | | | | | Closes #4646. PiperOrigin-RevId: 186301842
* Propagating whether there was a cache hit in the SpawnResult.Gravatar olaola2018-02-20
| | | | | | | | So far, nobody uses it, but I want to start using this field soon. TESTED=unit test RELNOTES: None PiperOrigin-RevId: 186290375
* Replace path implementation.Gravatar tomlu2018-02-08
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Path and PathFragment have been replaced with String-based implementations. They are pretty similar, but each method is dissimilar enough that I did not feel sharing code was appropriate. A summary of changes: PATH ==== * Subsumes LocalPath (deleted, its tests repurposed) * Use a simple string to back Path * Path instances are no longer interned; Reference equality will no longer work * Always normalized (same as before) * Some operations will now be slower, like instance compares (which were previously just a reference check) * Multiple identical paths will now consume more memory since they are not interned PATH FRAGMENT ============= * Use a simple string to back PathFragment * No more segment arrays with interned strings * Always normalized * Remove isNormalized * Replace some isNormalizied uses with containsUpLevelReferences() to check if path fragments try to escape their scope * To check if user input is normalized, supply static methods on PathFragment to validate the string before constructing a PathFragment * Because PathFragments are always normalized, we have to replace checks for literal "." from PathFragment#getPathString to PathFragment#getSafePathString. The latter returns "." for the empty string. * The previous implementation supported efficient segment semantics (segment count, iterating over segments). This is now expensive since we do longer have a segment array. ARTIFACT ======== * Remove Path instance. It is instead dynamically constructed on request. This is necessary to avoid this CL becoming a memory regression. RELNOTES: None PiperOrigin-RevId: 185062932
* remote: Add support for HTTP Basic AuthGravatar Jakob Buchgraber2018-02-08
| | | | | | Closes #4609. PiperOrigin-RevId: 185032751
* User-friendlier representation of a missing digest.Gravatar olaola2018-02-08
| | | | | | | | I moved it into DigestUtil preemptively in case we switch to binary instead of hex representation. TESTED=manually RELNOTES: None PiperOrigin-RevId: 185007558
* Fixing #4585: broken re-execution of orphaned actions.Gravatar olaola2018-02-08
| | | | | | | | This is an important regression, we will want to patch the fix into 0.10 TESTED=fixed unit test, with A/B testing RELNOTES: Resolved an issue where a failure in the remote cache would not trigger local re-execution of an action. PiperOrigin-RevId: 184991670
* remote: correct link to remote caching documentation.Gravatar buchgr2018-02-05
| | | | | RELNOTES: None PiperOrigin-RevId: 184518931
* site/docs: add documentation for remote caching.Gravatar Jakob Buchgraber2018-02-02
| | | | | | Closes #4540. PiperOrigin-RevId: 184349872
* remote: Rewrite the HTTP caching client in Netty. Fixes #4481Gravatar buchgr2018-01-26
| | | | | | | | | | | | | | * This puts in the foundation of HTTP/2 support for remote caching. * Allows us to remove the Apache HTTP library as a dependency, reducing the Bazel binary size by 1MiB. On fast networks (i.e. GCE to GCS) we can see a >2x speed improvement for TLS throughput. Even from my workstation to GCS I get significant build time improvements when using Netty's TLS 18s vs 12s. Closes #4481. PiperOrigin-RevId: 183411787
* Disable concurrent change detectionGravatar ulfjack2018-01-22
| | | | | | | | Regress on #3360. We have reports of Bazel outputting warnings for generated files, which I have been able to reproduce. Apparently, Bazel gets stuck with an old FileContentsProxy for generated files, and is unable to recover. PiperOrigin-RevId: 182772324
* Prevent broken cache entries on concurrent file changesGravatar ulfjack2018-01-19
| | | | | | | | | | | | | | | | Local execution has an inherent race condition: if a user modifies a file while an action is executed, then it is impossible for Bazel to tell which version of the file was actually read during action execution. The file may have been modified before or after the tool has read it, or, in the worst case, the tool may have read both the original and the modified version. In addition, the file may be changed back to the original state before Bazel can check the file, so computing the digest before / after may not be sufficient. This is a concern for both local and remote caches, although the cost of poisoning a shared remote cache is significantly higher, and is what has triggered this work. Fixes #3360. We solve this by keeping a reference to the FileContentsProxy, and using that to check for modificaitons before storing the cache entry. We output a warning if this check fails. This change does not increase memory consumption; Java objects are always allocated in multiples of 8 bytes, we use compressed oops, and the FileArtifactValue currently has 12 bytes worth of fields (excl. object overhead), so adding another pointer is effectively free. As a possible performance optimization on purely local builds, we could also consider not computing digests at all, and only use the FileContentsProxy for caching. PiperOrigin-RevId: 182510358
* actions,temp: respect TMPDIR envvarGravatar Laszlo Csomor2018-01-15
| | | | | | | Fixes https://github.com/bazelbuild/bazel/issues/4376 Change-Id: Id78bb0930044626304e54f07735db4d4b2c84720 PiperOrigin-RevId: 181959528
* Remove some unnecessary depsGravatar Ulf Adams2018-01-15
| | | | | | | | | Change-Id: Ib8dd9265b18fa0915f52427567845105fcdfa295 Closes #4447. Change-Id: Ib8dd9265b18fa0915f52427567845105fcdfa295 PiperOrigin-RevId: 181943004
* Plumb exec root through to all spawn runners.Gravatar tomlu2018-01-11
| | | | | | They need this to parse input manifests. Previously we would grab the exec root from the Root, but wish to unsupport this. PiperOrigin-RevId: 181669143
* Have the RemoteSpawnRunner use the execution platform present in the Spawn ↵Gravatar John Cater2018-01-10
| | | | | | | | | | | to get the remote execution properties. Fixes #4128. This reverts commit 3ce42ef3074ee6d3ac7d9968381c8c0a51d9d38d. Change-Id: I8b9ad5099f6334c2488a22baf05d0b273e10f776 PiperOrigin-RevId: 181550828
* Use EmptyActionInput instead of null in SpawnInputExpanderGravatar ulfjack2018-01-08
| | | | | | | | This simplifies some spawn runners, which no longer have to specially handle null; unfortunately, the sandbox runners do not support VirtualActionInput, so they still have to special-case it. PiperOrigin-RevId: 181175408
* Automated rollback of commit 43f45b58acf10beadbb1221b71dfa06fa1341510.Gravatar jcater2017-12-21
| | | | | | | | | | | | | | | *** Reason for rollback *** Breaks some remote execution clients. *** Original change description *** Have the RemoteSpawnRunner use the execution platform present in the Spawn to get the remote execution properties. Fixes #4128. Change-Id: I7e71caef2465204d2dd8225448d54e52366807e6 PiperOrigin-RevId: 179847553
* remote: fix download of output directoriesGravatar buchgr2017-12-20
| | | | | | | | Fix a bug where Bazel would crash if two Directory protos had the same hash. RELNOTES: Remote Caching and Execution support output directories. PiperOrigin-RevId: 179731040
* remote: add directory support for remote caching and executionGravatar Hadrien Chauvin2017-12-20
| | | | | | Add support for directory trees as artifacts. Closes #4011. PiperOrigin-RevId: 179691001
* remote: delete dead codeGravatar Jakob Buchgraber2017-12-20
| | | | | Change-Id: Iacbba3eced0abc0dcfd7311a0f07da48cbaba6e4 PiperOrigin-RevId: 179664071
* Have the RemoteSpawnRunner use the execution platform present in the Spawn ↵Gravatar John Cater2017-12-19
| | | | | | | | | to get the remote execution properties. Fixes #4128. Change-Id: I7e71caef2465204d2dd8225448d54e52366807e6 PiperOrigin-RevId: 179595126
* remote: Rename --remote_rest_cache to --remote_http_cacheGravatar Jakob Buchgraber2017-12-19
| | | | | | | | | | | Call it what it is. RELNOTES: --remote_rest_cache was renamed to --remote_http_cache. Both options keep working in this release, but --remote_rest_cache will be removed in the next release. Change-Id: I9e0b947f2184e0d543e7e19c5c33b6aa851d47d2 PiperOrigin-RevId: 179542826
* remote: deprecate --remote_rest_cache_pool_sizeGravatar Jakob Buchgraber2017-12-19
| | | | | | | | There's no point in having this option. We'll use as many TCP connections as we'll need. Fewer options FTW. Change-Id: I502eadd6a3a35040c7eda05ef49320b273ac26ad PiperOrigin-RevId: 179533022
* remote: update READMEGravatar buchgr2017-12-18
| | | | | | | | - Mention sandboxing for remote caching/execution. - Mention buchgr/bazel-remote-cache as a remote caching backend. RELNOTES: None. PiperOrigin-RevId: 179515966
* remote: fix out-of-date documentationGravatar Jakob Buchgraber2017-12-18
| | | | | | | The action cache prefix is "ac", not "actioncache". Change-Id: I841a026133ab3b4ec5b58a0cf29252dae49434fe PiperOrigin-RevId: 179437730
* remote: fix rest cache documentation. Fixes #4246Gravatar buchgr2017-12-18
| | | | | | | | Describe the HTTP protocol in some detail. Also improve the wording of remote caching a bit. RELNOTES: None. PiperOrigin-RevId: 179433738
* remote: Add support for Google Cloud Storage.Gravatar Jakob Buchgraber2017-12-18
| | | | | | | | | | | | | | | | | | | | | | Add support for Google Cloud Storage (GCS) as a HTTP caching backend. This commit mainly adds the infrastructure necessary to authenticate to GCS. Using GCS as a caching backend works as follows: 1) Create a new GCS bucket. 2) Create a service account that can read and write to the GCS bucket and generate a JSON credentials file for it. 3) Invoke Bazel as follows: bazel build --remote_rest_cache=https://storage.googleapis.com/<bucket> --auth_enabled --auth_scope=https://www.googleapis.com/auth/devstorage.read_write --auth_credentials=</path/to/creds.json> I'll add simplification's and docs in a subsequent commit. Change-Id: Ie827d7946a2193b97ea7d9aa72eb15f09de2164d PiperOrigin-RevId: 179406380
* Fix: uploading artifacts of failed actions to remote cache stopped working.Gravatar olaola2017-12-11
| | | | | | | | | | | | | | | | To reproduce: run a failing test with --experimental_remote_spawn_cache or with --spawn_strategy=remote and no executor. Expected: test log is uploaded. Desired behavior: - regardless of whether a spawn is cacheable or not, its artifacts should be uploaded to the remote cache. - the spawn result should only be set if the spawn is cacheable *and* the action succeeded. - when executing remotely, the do_not_cache field should be set for non-cacheable spawns, and the remote execution engine should respect it. This CL contains multiple fixes to ensure the above behaviors, and adds a few tests, both end to end and unit tests. Important behavior change: it is no longer assumed that non-cacheable spawns should use a NO_CACHE SpawnCache! The appropriate test case was removed. Instead, an assumption was added that all implementations of SpawnCache should respect the Spawns.mayBeCached(spawn) property. Currently, only NO_CACHE and RemoteSpawnCache exist, and they (now) support it. TESTED=remote build execution backend. WANT_LGTM: philwo,buchgr RELNOTES: None PiperOrigin-RevId: 178617937
* Moving the RemoteWorker into tools/remote directory.Gravatar olaola2017-12-05
| | | | | | | | This is because I want to add another remote execution related tool, the remote_client, which will use the Remote Execution API to fetch blobs from a remote cache. I will use this tool as part of end-to-end tests for remote execution. TESTED=remote integration tests, presubmit RELNOTES: None PiperOrigin-RevId: 177995895
* remote: Have --remote_rest_cache respect --remote_timeout.Gravatar buchgr2017-12-05
| | | | | RELNOTES: The --remote_rest_cache flag now respects --remote_timeout. PiperOrigin-RevId: 177926523
* remote: Replace Retrier with Retrier2.Gravatar buchgr2017-12-04
| | | | | | | | | - Replace the existing Retrier with Retrier2. - Rename Retrier2 to Retrier and remove the old Retrier + RetryException class. RELNOTES: None. PiperOrigin-RevId: 177835070
* Change Bazel's default hash function to SHA256. Fixes #3193Gravatar buchgr2017-12-03
| | | | | | | RELNOTES: Bazel's default hash function was changed from MD5 to SHA256. In particular, this affects users of remote caching and execution, as all hashes will be SHA256 by default. PiperOrigin-RevId: 177740702
* Refactor the FileSystem API to allow for different hash functions.Gravatar buchgr2017-11-30
| | | | | | | | | | | | | Refactor the FileSystem class to include the hash function as an instance field. This allows us to have a different hash function per FileSystem and removes technical debt, as currently that's somewhat accomplished by a horrible hack that has a static method to set the hash function for all FileSystem instances. The FileSystem's default hash function remains MD5. RELNOTES: None PiperOrigin-RevId: 177479772
* remote: don't hide non-test failures behind test failures. Fixes #4082Gravatar buchgr2017-11-30
| | | | | | | | | | Bazel should display the root cause of a test failure to the user. For example, if a test could not be executed on a remote executor due to there being no network connection, then it shouldn't display the test as failed but tell the user about the network error. RELNOTES: PiperOrigin-RevId: 177439578
* Add a CommandLineBuilder for the process-wrapper embedded tool, and use it ↵Gravatar ruperts2017-11-30
| | | | | | | | | everywhere instead of duplicating process-wrapper --shell_arguments in Blaze. To avoid a cyclic dependency, I broke up exec/local:local into exec/local:local and exec/local:options. RELNOTES: None. PiperOrigin-RevId: 177419268
* RestBlobStore: add missing CAS_PREFIXGravatar nicolov2017-11-29
| | | | | | | | | According to the SimpleBlobStore interface `containsKey` should look up blobs in the CAS. However, the URL it requests is missing `CAS_PREFIX`, so it will never find anything. Closes #4118. PiperOrigin-RevId: 177313658
* Clean up ExecutionRequirementsGravatar ulfjack2017-11-29
| | | | | | | | | | | | | | | | | | | - remove BaseSpawn.Local; instead, all callers pass in the full set of execution requirements they want to set - disable caching and sandboxing for the symlink tree action - it does not declare outputs, so it can't be cached or sandboxed (fixes #4041) - centralize the existing execution requirements in the ExecutionRequirements class - centralize checking for execution requirements in the Spawn class (it's possible that we may need a more decentralized, extensible design in the future, but for now having them in a single place is simple and effective) - update the documentation - forward the relevant tags to execution requirements in TargetUtils (progress on #3960) - this also contributes to #4153 PiperOrigin-RevId: 177288598
* Minor fixes to spacing in help messagesGravatar Mike Morearty2017-11-29
| | | | | | | | A few help messages were missing a space after the period at the end of a sentence. Closes #4094. PiperOrigin-RevId: 177282132
* Change error on adding a directory to the remote cache to be non-fatalGravatar Rahul Malik2017-11-29
| | | | | | | | | | @ulfjack @buchgr - I'm resubmitting https://github.com/bazelbuild/bazel/pull/3984 on behalf of @mterring to get past CLA issues that are holding it up from merging. This is a temporary fix for the issue https://github.com/bazelbuild/bazel/issues/3891 while we wait for https://github.com/bazelbuild/bazel/pull/4011 to be reviewed and tested Closes #4188. PiperOrigin-RevId: 177276751
* Simplify SpawnRunner interfaceGravatar ulfjack2017-11-28
| | | | | | | | | | | | | | It turns out that the SUCCESS status is often misunderstood to mean "zero exit", even though this is clearly documented. I've decided to add another status for non-zero exit, and use success only for zero exit to avoid this pitfall. Also, many of the status codes are set, but never used. I decided to reduce the number of status codes to only those that are actually relevant, which simplifies further processing. Instead, we should add a string message for the error case when we need one - we're not using it right now, so I decided not to add that yet. PiperOrigin-RevId: 177129441
* Adding checks for empty metadata and digest.Gravatar olaola2017-11-17
| | | | | | | | This will avoid the uninformative NullPointerException for this case (#3913). The underlying cause is data dependency on directories, which we want to disable. In the future, it will be allowed, but in a more explicit form (e.g. trailing slash). TESTED=no RELNOTES: None PiperOrigin-RevId: 176109375
* More resilient to remote spawn cache errorsGravatar Alpha Lam2017-11-16
| | | | | | | | | | Bazel used to fail the entire build if there is a single error with remote spawn cache. This could happen on unreliable network or when server is under load. This change will allow Bazel to fallback to local execution with a warning. Change-Id: I0bd41dc862b33a2ab2ab772d8ce9c943476edbf4 PiperOrigin-RevId: 175951699
* Replace all usages of Blaze's Preconditions class with guava.Gravatar tomlu2017-11-09
| | | | | | | | Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own. This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually. PiperOrigin-RevId: 175033526
* Remove hazelcast dependency from BazelGravatar Alpha Lam2017-10-30
| | | | | | | | | | | | This change removes Bazel's dependency on Hazelcast. This will help to reduce size of the Bazel binary and simplify the usage of remote cache. However Hazelcast library is still kept in the repository and still being used by remote_worker. It is useful as a REST server to allow integration testing with the remote rest cache functionality. Change-Id: Ia21b970cedaec84bc6c13e839509d838acb5756f PiperOrigin-RevId: 173880600
* remote: remove maximum blob size check.Gravatar Jakob Buchgraber2017-10-25
| | | | | | | | | It's a left over. It has already been removed for file downloads and uploads and is now still used for the action cache up- and downloads. Change-Id: Ic65ad0d34cfda75774131094de916e78b8835ff0 PiperOrigin-RevId: 173364883
* Properly attaching context for remote uploads in RemoteSpawnCache.Gravatar olaola2017-10-20
| | | | | | | | Fixes #3930. Also added tests. TESTED=added tests RELNOTES: Fixing regression to --experimental_remote_spawn_cache PiperOrigin-RevId: 172852740
* Stream rest cache file uploads.Gravatar Benjamin Peterson2017-10-18
| | | | | | | | | | | | | | | This means we no longer keep large action inputs or outputs in memory during upload. I adjusted the SimpleBlogStore interface to require clients to pass in the InputStream length. That allows us to always set Content-length on uploads. It's polite to do so, so that the server may, e.g., preallocate space for the blob. Fixes https://github.com/bazelbuild/bazel/issues/3250. Change-Id: I944c9dbc35fa2fa80dce523b0133ea9757bb3973 PiperOrigin-RevId: 172433522
* Fixing #3834, making sure test.log always exists.Gravatar olaola2017-10-18
| | | | | | | | Even if the test action produced no output, which it really shouldn't, Bazel should create an empty test.log file. TESTED=unit tests RELNOTES: Fixes #3834 PiperOrigin-RevId: 172412615
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-08 18:17:40 +0000