aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/remote
Commit message (Collapse)AuthorAge
* Automated rollback of commit 4465dae23de989f1452e93d0a88ac2a289103dd9.Gravatar buchgr2018-04-18
| | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** The no-cache tag is not respected (see b/77857812) and thus this breaks remote caching for all projects with symlink outputs. *** Original change description *** Only allow regular files and directories spawn outputs to be uploaded to a remote cache. The remote cache protocol only knows about regular files and directories. Currently, during action output upload, symlinks are resolved into regular files. This means cached "executions" of an action may have different output file types than the original execution, which can be a footgun. This CL bans symlinks from cachable spawn outputs and fixes http... *** PiperOrigin-RevId: 193338629
* This CL adds log entry protos and logging handlers for bytestream Read and ↵Gravatar Googler2018-04-16
| | | | | | Write so that they are logged. I'm open to suggestions for the logging format for these calls, since we don't want to log the actual contents of reads/writes because of their size. PiperOrigin-RevId: 193047886
* This CL adds log entry protos and logging handlers for Watch, ↵Gravatar Googler2018-04-13
| | | | | | FindMissingBlobs, GetActionResult so that they are logged. unknown commit must be submitted before this for Watch calls to be logged properly. PiperOrigin-RevId: 192794535
* Don't warn about /dev/null having been modifiedGravatar Mike Morearty2018-04-12
| | | | | | | | Fixes #4647 Closes #5003. PiperOrigin-RevId: 192576694
* remote/http: properly complete user promiseGravatar Jakob Buchgraber2018-04-10
| | | | | | | | Fixes #4976, #4935 Closes #4991. PiperOrigin-RevId: 192269206
* Consume the rest of the response stream for Watch RPC before returning. This ↵Gravatar Googler2018-04-09
| | | | | | | | necessary for the call to close correctly (e.g. for listeners to receive Status/trailers). RELNOTES: PiperOrigin-RevId: 192185329
* Enable bulk writes in the HttpBlobStoreGravatar ulfjack2018-04-01
| | | | | | | | | | Second attempt of https://github.com/bazelbuild/bazel/commit/0654620304728a5aecadd58138e96c41135d24e7, which I am rolling back. The problem is that FilterOutputStream.write is just plain wrong and we shouldn't inherit FilterOutputStream at all, but instead do it manually (which actually requires less code). This was a performance regression in https://github.com/bazelbuild/bazel/commit/deccc485603c004daad959fd747f1c0c9efc4f00. Fixed #4944. PiperOrigin-RevId: 191215696
* Automated rollback of commit 0654620304728a5aecadd58138e96c41135d24e7.Gravatar ulfjack2018-03-30
| | | | | | | | | | | | | | | | *** Reason for rollback *** Not a proper fix. *** Original change description *** Enable bulk writes in the HttpBlobStore This was a performance regression in https://github.com/bazelbuild/bazel/commit/deccc485603c004daad959fd747f1c0c9efc4f00. Fixed #4944. PiperOrigin-RevId: 191133416
* Enable bulk writes in the HttpBlobStoreGravatar ulfjack2018-03-30
| | | | | | | | This was a performance regression in https://github.com/bazelbuild/bazel/commit/deccc485603c004daad959fd747f1c0c9efc4f00. Fixed #4944. PiperOrigin-RevId: 191109352
* This change adds the writing of the remote execution log to a file behind an ↵Gravatar Googler2018-03-29
| | | | | | experimental flag. It also adds a logging handler for Execute calls so that they are logged. PiperOrigin-RevId: 190991493
* Remove categories from Bazel options.Gravatar ccalvarin2018-03-28
| | | | | | | | | These have all had a chance to be categorized with the OptionDocumentationCategory enum, and the help output already uses the enum-grouped format. The "incompatible changes" category has meaning for --all_incompatible_changes and will be removed separately. RELNOTES: None. PiperOrigin-RevId: 190773778
* Remove unused execRoot parameter from buildActionGravatar George Gensure2018-03-27
| | | | | | Closes #4916. PiperOrigin-RevId: 190662077
* Ensure Runner name is always set.Gravatar Googler2018-03-27
| | | | | RELNOTES: None. PiperOrigin-RevId: 190617155
* Only allow regular files and directories spawn outputs to be uploaded to a ↵Gravatar Benjamin Peterson2018-03-27
| | | | | | | | | | | | | | | | | | | | remote cache. The remote cache protocol only knows about regular files and directories. Currently, during action output upload, symlinks are resolved into regular files. This means cached "executions" of an action may have different output file types than the original execution, which can be a footgun. This CL bans symlinks from cachable spawn outputs and fixes https://github.com/bazelbuild/bazel/issues/4840. The interface of SpawnCache.CacheHandle.store is refactored: 1. The outputs parameter is removed, since that can be retrieved from the underlying Spawn. 2. It can now throw ExecException in order to fail actions. Closes #4902. Change-Id: I0d1d94d48779b970bb5d0840c66a14c189ab0091 PiperOrigin-RevId: 190608852
* Local Disk Cache: Only write to disk if target hash doesn't already exist.Gravatar robinnabel2018-03-22
| | | | | | | | | | Under Windows, the default permissions used when writing to the local disk cache prevent the files from being overwritten. This CR adds a check: if the target file already exists, return early. This is a performance improvement and will fix the error described above as existing files will no longer need to be overwritten. Similar features have been implemented on the remote gRPC cache implementations, see bazel issue #4789. Closes #4886. PiperOrigin-RevId: 190060233
* remote/http: properly support read timeoutGravatar Jakob Buchgraber2018-03-22
| | | | | | | | | | Also, remove unused SO_TIMEOUT. Fixes #4890 cc @benjaminp Closes #4895. PiperOrigin-RevId: 190051030
* Automatic code cleanup.Gravatar Googler2018-03-21
| | | | PiperOrigin-RevId: 189970123
* Propagating and printing server logs on demand.Gravatar olaola2018-03-21
| | | | | | | WANT_LGTM=all TESTED=RBE, unit tests RELNOTES: None PiperOrigin-RevId: 189938345
* Propagating remote results, including stdout/err, to Bazel on execution ↵Gravatar olaola2018-03-16
| | | | | | | | | timeouts. The refactoring to have an Exception that contains partial results will also be used in the next CL, in order to propagate and save remote server logs. RELNOTES: None PiperOrigin-RevId: 189344465
* remote/http: support refresh of oauth2 tokens in the remote cache.Gravatar Jakob Buchgraber2018-03-10
| | | | | | Closes #4622. PiperOrigin-RevId: 188595430
* Remove unnecessary I/O operations from handling remotely executed actions ↵Gravatar olaola2018-03-08
| | | | | | | | (fixes performance regression #4749). Also adding Skylark tests for input/output directories. TESTED=locally RELNOTES: Fix performance regression PiperOrigin-RevId: 188346410
* update to MethodDescriptor methods instead of fieldsGravatar Carl Mastrangelo2018-03-08
| | | | | | | | @buchgr Closes #4790. PiperOrigin-RevId: 188332795
* Don't display remote exec failed message on interrupt. Fixes #4783Gravatar olaola2018-03-06
| | | | | | TESTED=manually RELNOTES: None PiperOrigin-RevId: 188079436
* Replace instances of XCode with Xcode which is the proper casing for the name.Gravatar Sergio Campama2018-03-06
| | | | | | Closes #4640. PiperOrigin-RevId: 188022228
* remote: Add interceptor for logging gRPC calls during remote execution/cachingGravatar Googler2018-03-05
| | | | | | | | | | This provides a io.grpc.ClientInterceptor implementation that can be used to log gRPC call information. The interceptor can select a logging handler to use based on the gRPC method being called (Watch, Execute, Write, etc) to build a LogEntry, which can then be logged after the call has finished. Unit tests for the interceptor are included. In this change, the interceptor is never invoked, nor are there any handlers implemented for any gRPC methods. The interceptor also never tries to log any entries. To avoid circular dependency issues (Remote library will depend on logger which depends on remote library for utils), I've factored out the utility classes from the remote library into their own directory/package as part of this change. PiperOrigin-RevId: 187926516
* Adding a property name to the SpawnRunner. Most runners already had it, I ↵Gravatar olaola2018-02-22
| | | | | | | | | | just add it to the interface, and include it in the SpawnResult. This will be used to categorize/aggregate spawns executed by various runners. Also, minor refinement to the cacheHit property of the SpawnResult with remote execution. RELNOTES: None TESTED=presubmit, next cl PiperOrigin-RevId: 186637978
* Bugfix: spawn output directories are not passed to remote execution.Gravatar olaola2018-02-21
| | | | | | | | This bug was invisible to out tests, because our RemoteWorker will work regardless of whether the output was listed as a file or as a directory. TESTED=same tests RELNOTES: None PiperOrigin-RevId: 186518904
* Inject the product name in the constructor where neededGravatar ulfjack2018-02-20
| | | | | | ...instead of passing it everywhere so we can rewrite the local env. PiperOrigin-RevId: 186316859
* Fix env for local fallback (from remote exec) on Windows and LinuxGravatar ulfjack2018-02-20
| | | | PiperOrigin-RevId: 186309181
* Fix typo in cache download failure warningGravatar Alex Ryan2018-02-20
| | | | | | Closes #4646. PiperOrigin-RevId: 186301842
* Propagating whether there was a cache hit in the SpawnResult.Gravatar olaola2018-02-20
| | | | | | | | So far, nobody uses it, but I want to start using this field soon. TESTED=unit test RELNOTES: None PiperOrigin-RevId: 186290375
* Replace path implementation.Gravatar tomlu2018-02-08
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Path and PathFragment have been replaced with String-based implementations. They are pretty similar, but each method is dissimilar enough that I did not feel sharing code was appropriate. A summary of changes: PATH ==== * Subsumes LocalPath (deleted, its tests repurposed) * Use a simple string to back Path * Path instances are no longer interned; Reference equality will no longer work * Always normalized (same as before) * Some operations will now be slower, like instance compares (which were previously just a reference check) * Multiple identical paths will now consume more memory since they are not interned PATH FRAGMENT ============= * Use a simple string to back PathFragment * No more segment arrays with interned strings * Always normalized * Remove isNormalized * Replace some isNormalizied uses with containsUpLevelReferences() to check if path fragments try to escape their scope * To check if user input is normalized, supply static methods on PathFragment to validate the string before constructing a PathFragment * Because PathFragments are always normalized, we have to replace checks for literal "." from PathFragment#getPathString to PathFragment#getSafePathString. The latter returns "." for the empty string. * The previous implementation supported efficient segment semantics (segment count, iterating over segments). This is now expensive since we do longer have a segment array. ARTIFACT ======== * Remove Path instance. It is instead dynamically constructed on request. This is necessary to avoid this CL becoming a memory regression. RELNOTES: None PiperOrigin-RevId: 185062932
* remote: Add support for HTTP Basic AuthGravatar Jakob Buchgraber2018-02-08
| | | | | | Closes #4609. PiperOrigin-RevId: 185032751
* User-friendlier representation of a missing digest.Gravatar olaola2018-02-08
| | | | | | | | I moved it into DigestUtil preemptively in case we switch to binary instead of hex representation. TESTED=manually RELNOTES: None PiperOrigin-RevId: 185007558
* Fixing #4585: broken re-execution of orphaned actions.Gravatar olaola2018-02-08
| | | | | | | | This is an important regression, we will want to patch the fix into 0.10 TESTED=fixed unit test, with A/B testing RELNOTES: Resolved an issue where a failure in the remote cache would not trigger local re-execution of an action. PiperOrigin-RevId: 184991670
* remote: correct link to remote caching documentation.Gravatar buchgr2018-02-05
| | | | | RELNOTES: None PiperOrigin-RevId: 184518931
* site/docs: add documentation for remote caching.Gravatar Jakob Buchgraber2018-02-02
| | | | | | Closes #4540. PiperOrigin-RevId: 184349872
* remote: Rewrite the HTTP caching client in Netty. Fixes #4481Gravatar buchgr2018-01-26
| | | | | | | | | | | | | | * This puts in the foundation of HTTP/2 support for remote caching. * Allows us to remove the Apache HTTP library as a dependency, reducing the Bazel binary size by 1MiB. On fast networks (i.e. GCE to GCS) we can see a >2x speed improvement for TLS throughput. Even from my workstation to GCS I get significant build time improvements when using Netty's TLS 18s vs 12s. Closes #4481. PiperOrigin-RevId: 183411787
* Disable concurrent change detectionGravatar ulfjack2018-01-22
| | | | | | | | Regress on #3360. We have reports of Bazel outputting warnings for generated files, which I have been able to reproduce. Apparently, Bazel gets stuck with an old FileContentsProxy for generated files, and is unable to recover. PiperOrigin-RevId: 182772324
* Prevent broken cache entries on concurrent file changesGravatar ulfjack2018-01-19
| | | | | | | | | | | | | | | | Local execution has an inherent race condition: if a user modifies a file while an action is executed, then it is impossible for Bazel to tell which version of the file was actually read during action execution. The file may have been modified before or after the tool has read it, or, in the worst case, the tool may have read both the original and the modified version. In addition, the file may be changed back to the original state before Bazel can check the file, so computing the digest before / after may not be sufficient. This is a concern for both local and remote caches, although the cost of poisoning a shared remote cache is significantly higher, and is what has triggered this work. Fixes #3360. We solve this by keeping a reference to the FileContentsProxy, and using that to check for modificaitons before storing the cache entry. We output a warning if this check fails. This change does not increase memory consumption; Java objects are always allocated in multiples of 8 bytes, we use compressed oops, and the FileArtifactValue currently has 12 bytes worth of fields (excl. object overhead), so adding another pointer is effectively free. As a possible performance optimization on purely local builds, we could also consider not computing digests at all, and only use the FileContentsProxy for caching. PiperOrigin-RevId: 182510358
* actions,temp: respect TMPDIR envvarGravatar Laszlo Csomor2018-01-15
| | | | | | | Fixes https://github.com/bazelbuild/bazel/issues/4376 Change-Id: Id78bb0930044626304e54f07735db4d4b2c84720 PiperOrigin-RevId: 181959528
* Remove some unnecessary depsGravatar Ulf Adams2018-01-15
| | | | | | | | | Change-Id: Ib8dd9265b18fa0915f52427567845105fcdfa295 Closes #4447. Change-Id: Ib8dd9265b18fa0915f52427567845105fcdfa295 PiperOrigin-RevId: 181943004
* Plumb exec root through to all spawn runners.Gravatar tomlu2018-01-11
| | | | | | They need this to parse input manifests. Previously we would grab the exec root from the Root, but wish to unsupport this. PiperOrigin-RevId: 181669143
* Have the RemoteSpawnRunner use the execution platform present in the Spawn ↵Gravatar John Cater2018-01-10
| | | | | | | | | | | to get the remote execution properties. Fixes #4128. This reverts commit 3ce42ef3074ee6d3ac7d9968381c8c0a51d9d38d. Change-Id: I8b9ad5099f6334c2488a22baf05d0b273e10f776 PiperOrigin-RevId: 181550828
* Use EmptyActionInput instead of null in SpawnInputExpanderGravatar ulfjack2018-01-08
| | | | | | | | This simplifies some spawn runners, which no longer have to specially handle null; unfortunately, the sandbox runners do not support VirtualActionInput, so they still have to special-case it. PiperOrigin-RevId: 181175408
* Automated rollback of commit 43f45b58acf10beadbb1221b71dfa06fa1341510.Gravatar jcater2017-12-21
| | | | | | | | | | | | | | | *** Reason for rollback *** Breaks some remote execution clients. *** Original change description *** Have the RemoteSpawnRunner use the execution platform present in the Spawn to get the remote execution properties. Fixes #4128. Change-Id: I7e71caef2465204d2dd8225448d54e52366807e6 PiperOrigin-RevId: 179847553
* remote: fix download of output directoriesGravatar buchgr2017-12-20
| | | | | | | | Fix a bug where Bazel would crash if two Directory protos had the same hash. RELNOTES: Remote Caching and Execution support output directories. PiperOrigin-RevId: 179731040
* remote: add directory support for remote caching and executionGravatar Hadrien Chauvin2017-12-20
| | | | | | Add support for directory trees as artifacts. Closes #4011. PiperOrigin-RevId: 179691001
* remote: delete dead codeGravatar Jakob Buchgraber2017-12-20
| | | | | Change-Id: Iacbba3eced0abc0dcfd7311a0f07da48cbaba6e4 PiperOrigin-RevId: 179664071
* Have the RemoteSpawnRunner use the execution platform present in the Spawn ↵Gravatar John Cater2017-12-19
| | | | | | | | | to get the remote execution properties. Fixes #4128. Change-Id: I7e71caef2465204d2dd8225448d54e52366807e6 PiperOrigin-RevId: 179595126
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-04 16:17:08 +0000