aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/remote
Commit message (Collapse)AuthorAge
* Enabling chunking of outputs in the RemoteWorker downloadBlob. This fixes ↵Gravatar olaola2017-04-24
| | | | | | | | | the bug that the RemoteWorker would fail to upload outputs that were too big to send in a single gRPC message. Fixes #2822. RELNOTES: n/a PiperOrigin-RevId: 153710733
* Also download stdout & stderr in case of a cache hitGravatar ulfjack2017-04-20
| | | | | | Fixes #1413. PiperOrigin-RevId: 153684106
* OnePlatform auth support for Bazel, in preparation for next version of the API.Gravatar olaola2017-04-20
| | | | | | TESTED: local server RELNOTES: n/a PiperOrigin-RevId: 153599636
* Fix RemoteSpawnStrategy to upload stdout/stderr to the remote cacheGravatar ulfjack2017-04-19
| | | | | | | | | | This is already fixed in the CachedLocalSpawnRunner, with tests there, which will replace RemoteSpawnStrategy in the near future. For now, I'd like to get this in in time for 0.5.0 to get test caching working. Fixes #1413. PiperOrigin-RevId: 153486592
* Write stdout and stderr to the remote cacheGravatar ulfjack2017-04-18
| | | | | | | | | | Only write a cache entry when the spawn executed successfully, and with a 0 exit code. In the test, we only check that uploadFileContents is called exactly twice. Progress on #1413. PiperOrigin-RevId: 153458240
* Re-enabling the remote caching without remote execution code path, which wasGravatar olaola2017-04-12
| | | | | | | | accidentally regressed. TESTED=local RemoteWorker without work_path RELNOTES: n/a PiperOrigin-RevId: 152806430
* Open source LocalSpawnRunnerGravatar ulfjack2017-04-07
| | | | | | | | | The LocalSpawnRunner is a non-sandboxed local execution implementation, which will replace the current StandaloneSpawnStrategy. The code has been around for a long time and has seen a lot of bugfixes. It also supports local prefetching, which is required for Google. I have a follow-up change to make it support Windows, so it's not a drop-in replacement for StandaloneSpawnStrategy yet. PiperOrigin-RevId: 152486973
* Refactoring the chunking logic out of GrpcActionCache, and cleaning it up. AlsoGravatar olaola2017-04-06
| | | | | | | | | | | | added tests. This should not conflict with any of Ulf's upcoming CLs. I need it in preparation for the next CL, which fixes the bug that I don't limit the message size according to grpcChunkSizeBytes and don't do chunking in RemoteWorker.java, causing it to fail on uploading large outputs. TESTED=unit / integration tests RELNOTES: n/a PiperOrigin-RevId: 152385956
* Automated g4 rollback of commit 520f947580634cb9b62020fba4c93d5e78044ccf.Gravatar mstaib2017-04-06
| | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Broke mac_jdk7_test. *** Original change description *** Refactoring the chunking logic out of GrpcActionCache, and cleaning it up. Also added tests. This should not conflict with any of Ulf's upcoming CLs. I need it in preparation for the next CL, which fixes the bug that I don't limit the message size according to grpcChunkSizeBytes and don't do chunking in RemoteWorker.java, causing it to fail on uploading large outputs. TESTED=unit / integration tests RELNOTES: n/a PiperOrigin-RevId: 152270056
* Refactoring the chunking logic out of GrpcActionCache, and cleaning it up. AlsoGravatar olaola2017-04-06
| | | | | | | | | | | | added tests. This should not conflict with any of Ulf's upcoming CLs. I need it in preparation for the next CL, which fixes the bug that I don't limit the message size according to grpcChunkSizeBytes and don't do chunking in RemoteWorker.java, causing it to fail on uploading large outputs. TESTED=unit / integration tests RELNOTES: n/a PiperOrigin-RevId: 152258232
* Expand the SpawnRunner/SpawnResult interfaces to cover more functionalityGravatar ulfjack2017-04-06
| | | | | | | | | | | | | The intention is for the SpawnRunner interface to be the single, unified interface for running Spawns, so it needs to cover both execution and error reporting functionality for all current implementations, some of which are internal to Google. Note in particular the unified status code - it reports success if the subprocess was executed regardless of its exit code, which is reported separately. PiperOrigin-RevId: 152252975
* Add a new CachedLocalSpawnRunner to implement remote caching + local execGravatar ulfjack2017-04-05
| | | | | | | | | | | | | | | | | The new class wraps an existing SpawnRunner and adds remote caching. Ideally, the wrapped runner should be local and sandboxed, but this is not currently enforced. The new class is not hooked up to anything yet. The added test indicates that the RemoteActionCache interface is still more complex than necessary - in particular, we should merge downloadAllResults and downloadBlobs (for stdout/stderr) into a single method, and also change the upload to a single combined method in a similar way instead of two calls. Doing so allows the RemoteActionCache implementation more leeway in how it wants to implement these, potentially improving parallelism and performance. One step towards #1413. PiperOrigin-RevId: 152245644
* Refactor all ctor callsites of PathFragment to instead call a static ↵Gravatar nharmata2017-04-05
| | | | | | | | | | | | 'create' method. This paves the way for changing PathFragment to e.g. an abstract class with multiple subclasses. This way we can split out the windows-specific stuff into one of these concrete classes, making the code more readable and also saving memory (since the shallow heap size of the NonWindowsPathFragment subclass will hopefully be smaller than that of the current PathFragment). This also lets us pursue gc churn optimizations. We can now do interning in PathFragment#create and can also get rid of unnecessary intermediate PathFragment allocations. RELNOTES: None PiperOrigin-RevId: 152145768
* Extract a SpawnRunner interfaceGravatar ulfjack2017-04-04
| | | | | | | | | | | | | | | | | | | | | The RemoteSpawnRunner now implements the SpawnRunner interface. Note that Google's internal implementations were also retrofitted, and SpawnRunner is intended as a stable interface; that's also why I decided to move all params into SpawnExecutionPolicy, which is, unfortunately, not quite done yet. The specification of SpawnRunner is also still incomplete. In particular, it is still missing execution info keys, as well as inputs and outputs handling. This is a step towards unifying all SpawnStrategy implementations, with the SpawnRunner implementations performing the actual Spawn execution. There should be no user-visible semantic changes to the code, but one small fix: - GrpcActionCache was trying to download files even if there were none PiperOrigin-RevId: 152105696
* Clone the remote execution implementation into a new classGravatar ulfjack2017-03-29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The new RemoteExecutionClient class only performs remote execution, and nothing else; all higher-level functions, local rety, etc. will live outside of the client. In order to add unit tests, I had to add another layer of indirection between the Grpc{RemoteExecutor,ActionCache} and GRPC, since GRPC generates final, non-mockable classes. While a testing approach that uses a fake server can also get some test coverage (as in GrpcActionCacheTest), it doesn't allow us to test the full range of bad things that can happen at the GRPC layer. The cloned implementation uses a single GRPC channel, as was recommended to me by Jakob, who worked on GRPC. A single channel should be sufficiently scalable, it's thread-safe, and it performs chunking internally. On the server-side, the requests from a single channel can be dispatched to a thread pool, so this should not be a blocker for server-side parallelism. I also changed it to throw an exception whenever anything bad happens - this makes it much more obvious if there's still bug in this code; the old code silently swallows many errors, falling back to local execution, which papers over many issues. Furthermore, we now return a RemoteExecutionResult to indicate whether the action ran at all (regardless of exit code), as well as the exit code. All in all, this implementation is closer to the production code we're using internally, although quite a few things are still missing. The cloned implementation is not hooked up to RemoteSpawnStrategy yet. It also does not support combining remote caching with local execution, but note that RemoteSpawnStrategy regressed in that respect and currently also does not support that mode. PiperOrigin-RevId: 151578409
* Refactor simple distributed caching support Gravatar Alpha Lam2017-03-28
| | | | | | | | | | | | | | | | | | | The simple distributed caching in Bazel used ConcurrentMap as a blob store. It is incorrect to use an overloaded interface for this purpose. This change defines a SimpleBlobStore interface that only has put(), get() and containsKey() methods and allows a simple implementation of a blob store as remote cache for Bazel. Also updated documentation to summarize the options available in the remote spwan strategy. There is no functional change. TESTED=shell integration tests -- Change-Id: Iedff0bc4f06c4a93c398c53801014d998c3df13b Reviewed-on: https://cr.bazel.build/9330 PiperOrigin-RevId: 151439467 MOS_MIGRATED_REVID=151439467
* Deja-vu: Using an ActionInputFileCache for SHA1 digests used with remote ↵Gravatar Ola Rozenfeld2017-03-27
| | | | | | | | | | | | | execution. If you're feeling like you've already seen this, that's correct, these were the exact contents of commit e860316559eac366d47923a8eb4b5489a661aa35... and then, on Nov 15, something unclear happened and the code disappeared! Perhaps it was the result of a faulty sync. In any case, nobody noticed, and the CL went in. It was later rolloed back and resubmitted, but the crucial code changes were gone. TESTED=local server with profiling for SHA1 specifically RELNOTES: n/a -- PiperOrigin-RevId: 151139685 MOS_MIGRATED_REVID=151139685
* Use SpawnInputExpander in the remote spawn strategy to expand runfiles treesGravatar Ulf Adams2017-03-24
| | | | | | | | | | This fixes remote test execution. Fixes #1593. -- PiperOrigin-RevId: 151030133 MOS_MIGRATED_REVID=151030133
* Adding a more descriptive error message to ConcurrentMapActionCacheGravatar Googler2017-03-23
| | | | | | -- PiperOrigin-RevId: 150930281 MOS_MIGRATED_REVID=150930281
* Adding a temporary flag to Bazel to allow Platform override from the commandGravatar Ola Rozenfeld2017-03-14
| | | | | | | | | | | | | line. This will be used during development to test new toolchains in docker containers. Example usage: --experimental_remote_platform_override='entry:{ name:"a" value:"b" } entry:{ name:"c" value:"d" }' TESTED=local server -- PiperOrigin-RevId: 149933081 MOS_MIGRATED_REVID=149933081
* Also report the spawn from the remote strategyGravatar Ulf Adams2017-03-10
| | | | | | | | | | It's silly that we require every spawn strategy to do this individually, and the new spawn scheduler will fix this. However, it's useful to add this for debugging. -- PiperOrigin-RevId: 149743992 MOS_MIGRATED_REVID=149743992
* Merging remote_cache and remote_worker into a single binary. Gravatar Ola Rozenfeld2017-03-09
| | | | | | | | | | | | | | | It can still be used as only a cache server, or only a worker with a wrapper of Hazelcast, so no functionality is lost, but it is now simpler to use in local testing / prototyping. Changed README files appropriately. TESTED=locally -- Change-Id: I3fdff9d434ce8cae5a6a700df0cb9f5bc364b60c Reviewed-on: https://cr.bazel.build/9253 PiperOrigin-RevId: 149569790 MOS_MIGRATED_REVID=149569790
* Remove all the action resource estimation codeGravatar Ulf Adams2017-03-06
| | | | | | -- PiperOrigin-RevId: 149110466 MOS_MIGRATED_REVID=149110466
* Rationalize local resource acquisitionGravatar Ulf Adams2017-03-03
| | | | | | | | | | | | | | | | | | | | | | Move all local resource acquisition to where local execution actually happens. Don't attempt to acquire resources per action, but only for individual spawns. This significantly simplifies the code. The downside is that we don't account for action-level work anymore. In general, actions should not perform any process execution themselves, but always delegate such work to a SpawnStrategy implementation. This change makes sure that every Spawn has local resources set in a way that is consistent with the previous state. However, there are two actions - Fileset and FileWrite -, which are not spawns, and so we now don't limit their concurrent execution anymore. For Fileset, all work is done in a custom Fileset-specific thread pool, so this shouldn't be a problem. I'm not sure about FileWriteAction. -- PiperOrigin-RevId: 149012600 MOS_MIGRATED_REVID=149012600
* Adding a small flag allowing to control remote caching without remote ↵Gravatar Ola Rozenfeld2017-03-03
| | | | | | | | execution. Usually it is enabled, and will be triggered whenever a remote_cache is specified, but a remote_worker is not; however, this flag allows to specifically disable it for the cases where remote_worker is defined, but something went wrong and we were forced to execute locally. This is useful to save time when the remote API does not actually support setting the remote action result. -- PiperOrigin-RevId: 149007755 MOS_MIGRATED_REVID=149007755
* More documentation for distributed caching and remote execution Gravatar Alpha Lam2017-02-27
| | | | | | | | | | | | | | Adds step by step documentation for remote caching feature and document how to run the remote execution demo. Fixes a small issue with interacting with WebDAV module of NGINX and Apache using --rest_cache_url feature. -- Change-Id: I5e98aa6707bf502eab0801ba637eae575ba26d42 Reviewed-on: https://cr.bazel.build/9031 PiperOrigin-RevId: 148546592 MOS_MIGRATED_REVID=148546592
* Use //third_party/protobuf:protobuf_java instead of ↵Gravatar Carmi Grushko2017-02-09
| | | | | | | | | | //third_party/protobuf:protobuf to refer to the Java proto runtime. This is the name in the upstream protobuf repo. -- PiperOrigin-RevId: 146949832 MOS_MIGRATED_REVID=146949832
* Fixing reliance on same order for downloading multiple blobs.Gravatar Ola Rozenfeld2017-02-03
| | | | | | | | RELNOTES: n/a -- PiperOrigin-RevId: 146396318 MOS_MIGRATED_REVID=146396318
* Make --hazelcast_node and --remote_cache options mutually exclusive Gravatar Marcin Maliszkiewicz2017-01-20
| | | | | | | | | | Specifying both options can cause OOM on OSX. -- Change-Id: I52daf194a8840f9e63f1d537f13152e53f8436a7 Reviewed-on: https://cr.bazel.build/8220 PiperOrigin-RevId: 145079331 MOS_MIGRATED_REVID=145079331
* Move some classes to lib.exec in preparation for layering enforcementGravatar Ulf Adams2016-12-21
| | | | | | | | | | | | This change is part of the mu-bazel effort, which aims to build a minimally useful Bazel binary with most extraneous functionality removed. As part of that, we want to enforce layering of packages. In particular, lib.actions must not depend on lib.rules or lib.exec. lib.rules must not depend on lib.exec. Moving these classes is a necessary step to enforce that layering. -- PiperOrigin-RevId: 142668172 MOS_MIGRATED_REVID=142668172
* Creating separate instances of CAS and execution handlers for every action. ↵Gravatar Ola Rozenfeld2016-12-16
| | | | | | | | | | This allows Bazel to talk to multiple instances of the server, if these exist, enabling server-side parallelism (due to using separate gRPC channels). TESTED: internally and local server -- PiperOrigin-RevId: 142262973 MOS_MIGRATED_REVID=142262973
* Fix formatting of README.mdGravatar Tom Payne2016-12-16
| | | | | | | | | | | A number of the shorter commands were not displayed in a code block, making them trickier to copy and paste into a console. Closes #2257. -- Reviewed-on: https://github.com/bazelbuild/bazel/pull/2257 PiperOrigin-RevId: 142262893 MOS_MIGRATED_REVID=142262893
* Add a flag for disabling local fallback for actions that failed remotely.Gravatar Ola Rozenfeld2016-12-13
| | | | | | -- PiperOrigin-RevId: 141817345 MOS_MIGRATED_REVID=141817345
* Debugging flag (will rarely be used by actual users) that disables remoteGravatar Ola Rozenfeld2016-12-13
| | | | | | | | execution cache. -- PiperOrigin-RevId: 141807596 MOS_MIGRATED_REVID=141807596
* Printing the stack trace of remote failures on --verbose_failures.Gravatar Ola Rozenfeld2016-12-13
| | | | | | | | Helps debugging. -- PiperOrigin-RevId: 141802189 MOS_MIGRATED_REVID=141802189
* Attempt #2 to submit commit e860316559eac366d47923a8eb4b5489a661aa35: Using ↵Gravatar Ola Rozenfeld2016-11-24
| | | | | | | | | | an ActionFileInputCache for SHA1 digests used with remote execution. I missed an important test failure -- turned out we have a remote execution test that does not live under lib/remote. -- MOS_MIGRATED_REVID=140135277
* Introduce BlazeInterners, a Blaze-specific wrapper around Guava's Interners ↵Gravatar Nathan Harmata2016-11-24
| | | | | | | | | | | that makes an appropriate call to Interners.InternerBuilder#concurrencyLevel. For current readers of this CL, I used this class everywhere in the Blaze codebase. For future readers of this CL, this class should be used to create an Interner in the Blaze codebase. -- MOS_MIGRATED_REVID=140063271
* Rollback of commit e860316559eac366d47923a8eb4b5489a661aa35.Gravatar Alex Humesky2016-11-21
| | | | | | | *** Reason for rollback *** -- MOS_MIGRATED_REVID=139640949
* Using an ActionFileInputCache for SHA1 digests used with remote execution.Gravatar Ola Rozenfeld2016-11-21
| | | | | -- MOS_MIGRATED_REVID=139613925
* Upload empty files to remote execution. I was too eager to optimize empty filesGravatar Ola Rozenfeld2016-11-08
| | | | | | | | away -- Bazel should still work with remote execution servers that don't have that optimization. -- MOS_MIGRATED_REVID=138384785
* Refactor the module API to use the builder pattern for executor creation.Gravatar Ulf Adams2016-11-02
| | | | | | | This significantly simplifies several of our modules. -- MOS_MIGRATED_REVID=137713119
* Syncing up the latest protos with the internal version. Adding UNSUPPORTEDGravatar Ola Rozenfeld2016-10-24
| | | | | | | option to setting remote execution cache. -- MOS_MIGRATED_REVID=137027196
* Adding a remote gRPC cache server to wrap Hazelcast. Useful to test theGravatar Ola Rozenfeld2016-09-21
| | | | | | | prototype. -- MOS_MIGRATED_REVID=133704420
* Basic implementation of a remote gRPC based cache.Gravatar Ola Rozenfeld2016-09-21
| | | | | | | TODO during review: add A LOT more tests! -- MOS_MIGRATED_REVID=133702188
* Description redacted.Gravatar Ola Rozenfeld2016-09-20
| | | | | -- MOS_MIGRATED_REVID=133584935
* Improving getAllDigests to remove duplicates.Gravatar Ola Rozenfeld2016-09-20
| | | | | -- MOS_MIGRATED_REVID=133580990
* Cosmetic changes to the remote execution proto, and fixing a minor bug inGravatar Ola Rozenfeld2016-09-15
| | | | | | | ContentDigests function. -- MOS_MIGRATED_REVID=133256094
* Implementing directory Merkle trees for remote execution/caching (seeGravatar Ola Rozenfeld2016-09-14
| | | | | | | | | | | | | | | https://docs.google.com/document/d/1hh63AzKlwcOJN6jBZzY3GNPffzww-JKx1015DfFKM6g/edit#). A directory is represented by recursive structure of TreeNodes, which are interned in a single TreeNodeRepository. Common subtrees are shared between various parents. In the change we introduce only creating the tree from an existing list of ActionInputs; in the future, we should refactor to populate the TreeNodeRepository alongside (or maybe even within) the ArtifactFactory. The plan is for the TreeNodeRepository to become a new centralized location for all Artifact data to be computed and cached. -- MOS_MIGRATED_REVID=133080315
* Adding utility functions to compute ContentDigests for various types; inGravatar Ola Rozenfeld2016-09-12
| | | | | | | | | | | particular, ActionKeys, which are a type of ContentDigest. The digests are a SHA1 of the contents. For proto messages, for now we compute the digest using the serialized form of the message, which is something we will probably want to rethink in the future. -- MOS_MIGRATED_REVID=132690105
* Refactoring getTimeoutSeconds of a Spawn into Spaws.java, becaused it is usedGravatar Googler2016-09-08
| | | | | | | in places other than the sandbox code. -- MOS_MIGRATED_REVID=132436150
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-16 18:01:04 +0000