aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/test/java/com/google/devtools/build/lib/exec
Commit message (Collapse)AuthorAge
* Introduce a new SpawnCache API; add a RemoteSpawnCache implementationGravatar ulfjack2017-08-14
| | | | | | | | | | AbstractSpawnRunner now uses a SpawnCache if one is registered, this allows adding caching to any spawn runner without having to be aware of the implementations. I will delete the old CachedLocalSpawnRunner in a follow-up CL. PiperOrigin-RevId: 165024382
* Refactor persistent workers to use SpawnRunner.Gravatar Benjamin Peterson2017-08-11
| | | | | | | | | | | | | | | | | | | | | | | | | | Change the persistent worker spawn strategy to extend AbstractSpawnStrategy and put the actual logic into WorkerSpawnRunner. WorkerTestStrategy is unaffected. I had to extend SpawnPolicy with a speculating() method. Persistent workers need to know if speculation is happening in order to require sandboxing. Additionally, I added java_test rules for the local runner tests and worker tests. See https://github.com/bazelbuild/bazel/issues/3481. NOTE: ulfjack@ made some changes to this change before merging: - changed Reporter to EventHandler; added TODO about its usage - reverted non-semantic indentation change in AbstractSpawnStrategy - reverted a non-semantic indentation change in WorkerSpawnRunner - updated some internal classes to match - removed catch IOException in WorkerSpawnRunner in some cases, removed verboseFailures flag from WorkerSpawnRunner, updated callers - disable some tests on Windows; we were previously not running them, now that we do, they fail :-( Change-Id: I207b3938f0dc84d374ab052d5030020886451d47 PiperOrigin-RevId: 164965398
* Unify input prefetchingGravatar ulfjack2017-08-10
| | | | | | | All prefetching now goes through AbstractSpawnStrategy's implementation of SpawnExecutionPolicy. Make sure the sandbox runners also do this consistently. PiperOrigin-RevId: 164836877
* Implement timeouts on top of Java ProcessGravatar ulfjack2017-08-10
| | | | PiperOrigin-RevId: 164827022
* Rewrite the Command APIGravatar ulfjack2017-08-10
| | | | | | | | | | | | | | | | | | | | | | | | Important: the simplified API now defaults to forwarding interrupts to subprocesses. I did audit all the call sites, and I think this is a safe change to make. - Properly support timeouts with all implementations - Simplify the API - only provide two flavours of blocking calls, which require no input and forward interrupts; this is the most common usage - provide a number of async calls, which optionally takes input, and a flag whether to forward interrupts - only support input streams, no byte arrays or other 'convenience features' that are rarely needed and unnecessarily increase the surface area - use java.time.Duration to specify timeout; for consistency, interpret a timeout of <= 0 as no timeout (i.e., including rather than excluding 0) - KillableObserver and subclasses are no longer part of the public API, but still used to implement timeouts if the Subprocess.Factory does not support them - Update the documentation for Command - Update all callers; most callers now use the simplified API PiperOrigin-RevId: 164716782
* Use java.time.Duration for timeoutsGravatar ulfjack2017-08-09
| | | | PiperOrigin-RevId: 164577062
* Extend the SpawnRunner APIGravatar ulfjack2017-07-17
| | | | | | | | | | | | | - add an id for logging; this allows us to correlate log entries for the same spawn from multiple spawn runner implementations in the future - add a prefetch method to the SpawnExecutionPolicy; better than relying on the ActionInputPrefetcher being injected in the constructor - add a name parameter to the report method; this is in preparation for a single unified SpawnStrategy implementation - it's basically the last bit of difference between SandboxStrategy and RemoteSpawnStrategy; they're otherwise equivalent (if not identical) PiperOrigin-RevId: 162194684
* LocalSpawnRunner now always logs as itselfGravatar ulfjack2017-07-14
| | | | PiperOrigin-RevId: 161944209
* BEP: Report configuration for all actionsGravatar Klaus Aehlig2017-07-14
| | | | | | | | | | Whenever we report an action in the build event protocol that has a configuration associated with it, report the configuration as well in the protocol (and not only the checksum, if the configuration is not that of one of the top-level configured targets). Change-Id: I9b085d5381b3c3509b4f3b99c8a77bc8fba6abfe PiperOrigin-RevId: 161789745
* Rewrite all the sandbox strategy implementationsGravatar ulfjack2017-07-12
| | | | | | | | | | | | | | - Make use of existing abstractions like SpawnRunner and SpawnExecutionPolicy. - Instead of having the *Strategy create a *Runner, and then call back into SandboxStrategy, create a single SandboxContainer which contains the full command line, environment, and everything needed to create and delete the sandbox directory. - Do all the work in SandboxStrategy, including creation and deletion of the sandbox directory. - Use SpawnResult instead of throwing, catching, and rethrowing. - Simplify the control flow a bit. PiperOrigin-RevId: 161644979
* Simplify the ActionInputFileCacheGravatar ulfjack2017-07-06
| | | | | | | | | Add a single getMetadata method (matching MetadataHandler), and rewrite everything in those terms. This is in preparation for merging ActionInputFileCache and MetadataHandler. PiperOrigin-RevId: 161053535
* Add a #getBytes() method to DeterministicWriter that returns a ByteString. ↵Gravatar janakr2017-06-30
| | | | | | | | By default it just delegates to the existing #writeOutputFile, but implementations may choose to override if they have easy access to a ByteString. Also change some DeterministicWriter implementations that do have easy access to the ByteString. PiperOrigin-RevId: 160550028
* Use getopt to parse process-wrapper's command-line.Gravatar philwo2017-06-22
| | | | | | | | This will allow us to add new and optional flags like selecting a strategy used to spawn / wait for the child process. No one except Bazel should be calling "process-wrapper" and I couldn't find any references, so this breaking change should be fine. PiperOrigin-RevId: 159685867
* Adding support for SHA256 for remote execution. Switch remote execution to useGravatar olaola2017-06-20
| | | | | | | | | the currently defined hash function for blobs. Some refactoring. Adding an option to set the hash function in the remote worker, defaulting to the current behavior (unfortunately it is a build option, have not found a clean way to specify it at runtime). BUG=62622420 TESTED=remote worker RELNOTES: none PiperOrigin-RevId: 159473116
* Extract ActionContext to a top-level classGravatar ulfjack2017-06-19
| | | | PiperOrigin-RevId: 159423459
* Rewrite StandaloneSpawnStrategy to use LocalSpawnRunnerGravatar ulfjack2017-06-19
| | | | PiperOrigin-RevId: 159221067
* Fix local execution to not throw an exceptionGravatar ulfjack2017-06-14
| | | | | | | | The SpawnInputExpander returns null for empty files, but the ActionInputPrefetcher does not expect null values, and implementations may throw NPE. PiperOrigin-RevId: 158960425
* Extract the MacOS/XCode env rewrite logic into lib.exec.appleGravatar ulfjack2017-06-13
| | | | | | | | | | | | | | | | | | | Also add an interface to allow injecting that logic into LocalSpawnRunner; this is in preparation for rewriting StandaloneSpawnStrategy to use LocalSpawnRunner. At the same time, this reduces the dependencies from exec / standalone to rules.apple, which is a prerequisite for micro-Bazel. There's a small semantic change hidden here - we now only set the new XCodeLocalEnvProvider if we're actually running on Darwin, so we no longer fail execution on non-Darwin platforms if XCODE_VERSION_OVERRIDE or APPLE_SDK_VERSION_OVERRIDE is set. As a result, I moved the corresponding test from StandaloneSpawnStrategyTest to the new XCodeLocalEnvProviderTest. While I'm at it, also open source DottedVersionTest and CacheManagerTest. PiperOrigin-RevId: 158829077
* Add some tests to the LocalSpawnRunner, and make some small tweaksGravatar ulfjack2017-06-08
| | | | | | | | | - Only manually check the timeout if the process wrapper is not used - Set the timeout correctly; the process API uses milliseconds - flush the error output stream after writing - return SIGALRM as exit code for timeout cases PiperOrigin-RevId: 158374246
* Migrate Java tests to Truth.Gravatar lberki2017-05-30
| | | | | | RELNOTES: None. PiperOrigin-RevId: 157446717
* Automated g4 rollback of commit c4134802dd15d6ef5cca6521f6bf6aac395ee2ad.Gravatar kchodorow2017-05-23
| | | | | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Roll forward of directory name change *** Original change description *** Automated g4 rollback of commit 1d9e1ac90197b1d3d7b137ba3c1ada67bb9ba31b. *** Reason for rollback *** Breaks //src/test/shell/integration:force_delete_output_test *** Original change description *** Symlink output directories to the correct directory name If the workspace directory is /path/to/my/proj and the name in the WORKSPACE file is "floop", this will symlink the output directories to output_base/execroot/floop instead of output_base/execroot/proj. More prep for #1262, fixes #1681. PiperOrigin-RevId: 156892980
* Implement logging in LocalSpawnRunnerGravatar ulfjack2017-04-18
| | | | | | | | This is a prerequisite for Google to use it; there's no consistent logging strategy yet, but Google internally requires logging, and it needs to be somewhat backwards compatible. PiperOrigin-RevId: 153454160
* Fix the SpawnInputExpander to compute the manifest path correctlyGravatar ulfjack2017-04-10
| | | | PiperOrigin-RevId: 152663008
* LocalSpawnRunner: on Windows, don't use the process wrapper for nowGravatar ulfjack2017-04-07
| | | | | This is ported from StandaloneSpawnStrategy. PiperOrigin-RevId: 152493898
* Open source LocalSpawnRunnerGravatar ulfjack2017-04-07
| | | | | | | | | The LocalSpawnRunner is a non-sandboxed local execution implementation, which will replace the current StandaloneSpawnStrategy. The code has been around for a long time and has seen a lot of bugfixes. It also supports local prefetching, which is required for Google. I have a follow-up change to make it support Windows, so it's not a drop-in replacement for StandaloneSpawnStrategy yet. PiperOrigin-RevId: 152486973
* Refactor all ctor callsites of PathFragment to instead call a static ↵Gravatar nharmata2017-04-05
| | | | | | | | | | | | 'create' method. This paves the way for changing PathFragment to e.g. an abstract class with multiple subclasses. This way we can split out the windows-specific stuff into one of these concrete classes, making the code more readable and also saving memory (since the shallow heap size of the NonWindowsPathFragment subclass will hopefully be smaller than that of the current PathFragment). This also lets us pursue gc churn optimizations. We can now do interning in PathFragment#create and can also get rid of unnecessary intermediate PathFragment allocations. RELNOTES: None PiperOrigin-RevId: 152145768
* Automated g4 rollback of commit 1d9e1ac90197b1d3d7b137ba3c1ada67bb9ba31b.Gravatar hlopko2017-04-04
| | | | | | | | | | | | | | | | | | *** Reason for rollback *** Breaks //src/test/shell/integration:force_delete_output_test *** Original change description *** Symlink output directories to the correct directory name If the workspace directory is /path/to/my/proj and the name in the WORKSPACE file is "floop", this will symlink the output directories to output_base/execroot/floop instead of output_base/execroot/proj. More prep for #1262, fixes #1681. PiperOrigin-RevId: 152126545
* Symlink output directories to the correct directory nameGravatar kchodorow2017-03-31
| | | | | | | | | | If the workspace directory is /path/to/my/proj and the name in the WORKSPACE file is "floop", this will symlink the output directories to output_base/execroot/floop instead of output_base/execroot/proj. More prep for #1262, fixes #1681. PiperOrigin-RevId: 151712384
* Add SpawnInputExpander helper class to arrange runfiles for spawn strategiesGravatar Ulf Adams2017-03-21
| | | | | | | | | | | | | | | | | | | | | This new class is a combination of SpawnHelper and our internal code; the plan is to migrate all spawn strategies to the new class. The strict flag should be enabled by default, but that's a breaking change, so we need to do it later. - Use it in SandboxStrategy. - Add ActionInput.getExecPath to return a PathFragment; this avoids lots of back and forth between path fragments and strings. This is a step towards #1593. The previous attempt was missing a one-line patch in StandaloneTestStrategy, which broke all tests with sandboxing. StandaloneTestStrategy was fixed in a separate change, so this should be safe now. -- PiperOrigin-RevId: 150733457 MOS_MIGRATED_REVID=150733457
* Add ActionInput.getExecPath, which returns a PathFragmentGravatar Ulf Adams2017-03-21
| | | | | | | | | | | | Update some callers to use getExecPath, which generally results in less intermediate garbage generation (almost all callers need a PathFragment, not a String). Another small step towards #1593. -- PiperOrigin-RevId: 150631279 MOS_MIGRATED_REVID=150631279
* Rollback of commit 59180a4ea66b1395b5b85defd732859ecae919ea.Gravatar Yue Gan2017-03-20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Break bazel-tests and many other jobs on CI. http://ci.bazel.io/job/bazel-tests/BAZEL_VERSION=HEAD,PLATFORM_NAME=linux-x86_64/651/console *** Original change description *** Add SpawnInputExpander helper class to arrange runfiles for spawn strategies This new class is a combination of SpawnHelper and our internal code; the plan is to migrate all spawn strategies to the new class. The strict flag should be enabled by default, but that's a breaking change, so we need to do it later. - Use it in SandboxStrategy. - Add ActionInput.getExecPath to return a PathFragment; this avoids lots of back and forth between path fragments and strings. This is a step towards #159... *** -- PiperOrigin-RevId: 150610616 MOS_MIGRATED_REVID=150610616
* Add SpawnInputExpander helper class to arrange runfiles for spawn strategiesGravatar Ulf Adams2017-03-17
| | | | | | | | | | | | | | | | | This new class is a combination of SpawnHelper and our internal code; the plan is to migrate all spawn strategies to the new class. The strict flag should be enabled by default, but that's a breaking change, so we need to do it later. - Use it in SandboxStrategy. - Add ActionInput.getExecPath to return a PathFragment; this avoids lots of back and forth between path fragments and strings. This is a step towards #1593. -- PiperOrigin-RevId: 150427021 MOS_MIGRATED_REVID=150427021
* Global cleanup change.Gravatar Googler2017-03-03
| | | | | | -- PiperOrigin-RevId: 149067322 MOS_MIGRATED_REVID=149067322
* Provide more reporting options to SkyFunctions Gravatar Klaus Aehlig2017-02-27
| | | | | | | | | | | | | With more specific information to be reported by Skyfunctions, e.g., to inform the build-event protocol on missing files, the EventHandler interface is no longer enough. Therefore, provide an enriched context for reporting events. -- Change-Id: I2d06166fe4d5b9054e24ad8c752fafc039e3f9f8 Reviewed-on: https://cr.bazel.build/8794 PiperOrigin-RevId: 148463437 MOS_MIGRATED_REVID=148463437
* Add a flag to print the effective (Spawn)ActionContexts for debugging.Gravatar Philipp Wollermann2017-02-01
| | | | | | | | | | | | | | | | | | | | | | | | | By specifying the flag "--debug_print_action_contexts", Bazel will print the contents of the internal SpawnActionContext and ContextMap maps, which allows developers to see which kind of actions are run using which strategy. Example output of Bazel at HEAD: $ ./output/bazel build --debug_print_action_contexts INFO: SpawnActionContextMap: "" = LinuxSandboxedStrategy INFO: SpawnActionContextMap: "Closure" = WorkerSpawnStrategy INFO: SpawnActionContextMap: "Javac" = WorkerSpawnStrategy INFO: ContextMap: Context = BazelWorkspaceStatusActionContext INFO: ContextMap: CppCompileActionContext = SpawnGccStrategy INFO: ContextMap: CppLinkActionContext = SpawnLinkStrategy INFO: ContextMap: FileWriteActionContext = FileWriteStrategy INFO: ContextMap: FilesetActionContext = FilesetActionContextImpl INFO: ContextMap: IncludeScanningContext = DummyIncludeScanningContext INFO: ContextMap: SpawnActionContext = LinuxSandboxedStrategy INFO: ContextMap: SymlinkTreeActionContext = SymlinkTreeStrategy INFO: ContextMap: TestActionContext = ExclusiveTestStrategy (Can you spot the bug found by this feature here? The default TestActionContext is ExclusiveTestStrategy, which is probably not what we want.) -- PiperOrigin-RevId: 146233390 MOS_MIGRATED_REVID=146233390
* Move some classes to lib.exec in preparation for layering enforcementGravatar Ulf Adams2016-12-21
| | | | | | | | | | | | This change is part of the mu-bazel effort, which aims to build a minimally useful Bazel binary with most extraneous functionality removed. As part of that, we want to enforce layering of packages. In particular, lib.actions must not depend on lib.rules or lib.exec. lib.rules must not depend on lib.exec. Moving these classes is a necessary step to enforce that layering. -- PiperOrigin-RevId: 142668172 MOS_MIGRATED_REVID=142668172
* Remove some more occurences of deprecated getExecRoot() callsGravatar Kristina Chodorow2016-09-15
| | | | | -- MOS_MIGRATED_REVID=133257532
* Add the repository name as a parameter to the output path functionsGravatar Kristina Chodorow2016-08-12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This doesn't do anything yet, it's in preparation for the execroot rearranging change. The execroot will have one bazel-out per repo, so it'll look like: execroot/ repo1/ bazel-out/ local-fastbuild/ bin/ repo2/ bazel-out/ local-fastbuild/ bin/ genfiles/ repo3/ bazel-out/ local-fastbuild/ testlogs/ and so on. Thus, any output path (getBinDirectory() & friends) needs to know what the repo name is. This changes so many places in the code I thought it would be good to do separately, then just flip the functionality in the execroot-rearranging commit. While I was poking around, I changed all of the refs I could from getPackageRelativeArtifact() to getBin/GenfilesArtifact(), so that 1) rule implementation don't have to know as much about roots and 2) they'll be more isolated from other output dir changes. `bazel info` and similar just return roots for the main repository. The only "change" is passing around a target label in the Java rules. Continues work on #1262. -- MOS_MIGRATED_REVID=129985336
* Use byte[] rather than ByteString for file digests.Gravatar Googler2016-07-20
| | | | | | | | | | | | | | | | | ActionInputFileCache: Change getDigest() to return the underlying byte[16] owned by each FileArtifactValue. Remove throws clause from getInputFromDigest(); this should be an in-memory operation, and no implementation actually throws. PerActionFileCache: Invert mapping from artifact to digest only if needed. Remove interner, as it was used only for the reverse map keys, not the returned values. This should be a significant cpu savings as eagerly constructing the reverse maps was a noticeable hotspot. -- MOS_MIGRATED_REVID=127972359
* Separate runfiles middlemen into two layers: one that is a middleman for the ↵Gravatar Lukacs Berki2016-01-26
| | | | | | | | | | | files itself, and one that contains this middleman and the runfiles output manifest. This is in preparation for Windows symlink support in runfiles manifests: on Windows, we'll need the target files created before we can create the runfiles symlinks for them, but we can't do that by adding the current runfiles middleman to the inputs of SymlinkTreeAction because it would create a cycle (output manifest -> SymlinkTreeAction -> runfiles manifest -> MiddlemanAction -> output manifest) The alternative is to just add the artifacts to the inputs of SymlinkTreeAction, but that would create a bunch of extra edges and, more importantly, reverse edges in the action graph which we don't want. -- MOS_MIGRATED_REVID=113041344
* Add test case to SingleBuildFileCache for unreadable filesGravatar Michajlo Matijkiw2016-01-22
| | | | | -- MOS_MIGRATED_REVID=112713014
* Migrated base test classes to JUnit 4 and deleted their temporary *ForJunit4 ↵Gravatar Florian Weikert2015-12-07
| | | | | | | | | | | | versions: devtools/build/lib/analysis/util/AnalysisTestCase.java devtools/build/lib/analysis/util/BuildViewTestCase.java devtools/build/lib/packages/util/PackageLoadingTestCase.java devtools/build/lib/testutil/FoundationTestCase.java -- MOS_MIGRATED_REVID=109560679
* Rename AbstractDistributorDriverTest to UserUtilsTestGravatar Michajlo Matijkiw2015-12-04
| | | | | | | Old name is vestigial, new home makes more sense. -- MOS_MIGRATED_REVID=109410381
* Migrated tests in devtools/build/lib/exec to JUnit 4.Gravatar Florian Weikert2015-12-02
| | | | | -- MOS_MIGRATED_REVID=109205207
* Migrated remaining tests in lib/exec to JUnit 4.Gravatar Florian Weikert2015-11-30
| | | | | -- MOS_MIGRATED_REVID=108794613
* Open-source tests from lib/execGravatar Dmitry Lomov2015-11-20
| | | | | -- MOS_MIGRATED_REVID=108255623
* Replace "Google Inc" with "The Bazel Authors" in every copyright notice.Gravatar Lukacs Berki2015-10-21
| | | | | -- MOS_MIGRATED_REVID=105949508
* Implement "More flexible strategy flag for Bazel".Gravatar Philipp Wollermann2015-06-08
| | | | | -- MOS_MIGRATED_REVID=95427512
* Description redacted.Gravatar Nathan Harmata2015-03-05
| | | | | -- MOS_MIGRATED_REVID=87698104
* Get runfiles prefix from the WORKSPACE fileGravatar Kristina Chodorow2015-03-05
| | | | | -- MOS_MIGRATED_REVID=87513766
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-07 07:52:03 +0000