aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/runtime/CommandEnvironment.java
Commit message (Collapse)AuthorAge
* Passing Bazel metadata in gRPC headers.Gravatar olaola2017-09-21
| | | | | | TESTED=unit tests RELNOTES: none PiperOrigin-RevId: 169395919
* Inform SkyframeExecutor when a command starts.Gravatar janakr2017-09-19
| | | | PiperOrigin-RevId: 169179218
* Remove unnecessary injection of precomputed values. Precomputed values are ↵Gravatar janakr2017-09-19
| | | | | | injected in BlazeCommandDispatcher#execExclusively. PiperOrigin-RevId: 169122571
* Move OutputService to lib.skyframeGravatar ulfjack2017-08-11
| | | | | | | | | | | This isn't ideal, but is the fastest way to split analysis, execution, and rules into separate java libraries, with lib.skyframe still being in the analysis library. Ideally, we'd split up the lib.skyframe package, move the analysis stuff to the analysis library, the execution stuff to the execution library, and so on. That wouldn't require us moving OutputService out of lib.exec. PiperOrigin-RevId: 164856998
* Remove superfluous resetEvaluator() call in setupPackageCache(). If ↵Gravatar felly2017-07-25
| | | | | | necessary, the reset occurs before the command starts. PiperOrigin-RevId: 162973194
* Merge handleOptions into beforeCommandGravatar ulfjack2017-07-17
| | | | | | | Now that we have the options before calling beforeCommand, there's no need for a separate handleOptions method in the BlazeModule API. Remove it. PiperOrigin-RevId: 162002300
* Create top-level configs dynamically: period.Gravatar gregce2017-07-06
| | | | | | | | | | | | | | | Before this change, top-level configs were still built statically, then cloned into dynamic configs to run the actual build. After this change, no static configuration should ever be created in a dynamically configured build. We're still keeping support for --experimental_dynamic_configs=off a bit longer to provide easy reversion on any bad surprises. But barring that we'll quickly move toward making --experimental_dynamic_configs=off a no-op, then ripping out Bazel's static configuration logic. PiperOrigin-RevId: 161005150
* BlazeCommandDispatcher: all options parsing and editing in one placeGravatar ulfjack2017-06-26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The invocation policy must be the last step in determining the active options for the current command. Take precautions to prevent accidental options editing of options after the policy is applied by declaring the options as an OptionsProvider (interface has no mutating methods); technically, this could be circumvented by casting to OptionsParser, but it's at least safer than before. As part of this, I'm moving the BlazeCommand.editOptions call to before the invocation policy, and also making minor changes to the CommandEnvironment. Commands that expect to have the last word on options could therefore see options changes after this commit. There are three commands that override editOptions: 1. TestCommand Current behavior: if test_output=streamed is set, also sets --test_sharding_strategy=disabled and --test_strategy=exclusive. Overriding --test_sharding_strategy is not a concern; in fact, maybe we shouldn't be setting that in the first place, since it can cause tests to timeout (timeout is usually applied per shard, so a 10-way sharded test will very likely timeout). If you override the test_strategy to local or standalone, then you may get interleaved stdout / stderr output from tests that run in parallel - a bit surprising, but no showstopper. If you override the test_strategy to remote, you won't get streamed output, because no remote strategy currently supports that. You may get interleaved output, if multiple tests finish very close to each other. There are no correctness concerns, it's just a slightly worse user experience. The code is safe in all cases, AFAICT. We could consider changing streamed to not require serialized execution, and instead use something like a lock - the first test to produce output gets streamed, and all others get delayed until the lock is released, but could still execute concurrently. However, it's unlikely that that's the desired behavior for most users. 2. CoverageCommand Current behavior: sets --collect_code_coverage, increases default --test_timeout. bazel coverage --nocollect_code_coverage is effectively bazel test. You can actually now run bazel test --collect_code_coverage --test_timeout and it will behave exactly the same way as bazel coverage. It's mostly a convenience thing. Note that CoverageCommand inherits TestCommand, so the case above also applies. 3. MobileInstallCommand Current behavior: sets --aspects, and --output_groups. If you override the aspect or output_groups, then the command will fail or run an old binary from a previous build. Not what you'd expect, but no correctness concerns. Summary: IMO, the impact on the existing commands is minor. The advantage of this change is that it's more secure, and gives us an escape hatch if a command ever overrides options in a way that turns out to be wrong or broken in some way. RELNOTES: None. PiperOrigin-RevId: 160114902
* Simplify BlazeModule.beforeCommandGravatar ulfjack2017-06-14
| | | | | | | | | | | Don't pass the Command annotation explicitly, but add it to CommandEnvironment instead; most modules don't need it in the first place, so it was a lot of boilerplate for not much. Also change it so that the command is passed to the constructor. Add some documentation to the beforeCommand method. PiperOrigin-RevId: 158847128
* Track the test environment in Skyframe, like the action environmentGravatar ulfjack2017-06-09
| | | | | | | | Instead of passing a client env into the test strategies, use the same mechansim as --action_env, by depending on the right set of Skyframe nodes that correspond to client env entries. PiperOrigin-RevId: 158401670
* Automated g4 rollback of commit c4134802dd15d6ef5cca6521f6bf6aac395ee2ad.Gravatar kchodorow2017-05-23
| | | | | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Roll forward of directory name change *** Original change description *** Automated g4 rollback of commit 1d9e1ac90197b1d3d7b137ba3c1ada67bb9ba31b. *** Reason for rollback *** Breaks //src/test/shell/integration:force_delete_output_test *** Original change description *** Symlink output directories to the correct directory name If the workspace directory is /path/to/my/proj and the name in the WORKSPACE file is "floop", this will symlink the output directories to output_base/execroot/floop instead of output_base/execroot/proj. More prep for #1262, fixes #1681. PiperOrigin-RevId: 156892980
* Inform the GotOptionsEvent of invocation policy, which is no longer passed ↵Gravatar ccalvarin2017-05-15
| | | | | | as part of the startup arguments. PiperOrigin-RevId: 156090009
* Store Skylark command-line flags in SkyframeGravatar brandjon2017-04-28
| | | | | | | | | This is the first of two CLs for making command line options able to affect the Skylark interpreter. It introduces SkylarkSemanticsOptions, and stores it as a precomputed (injected) value in Skyframe. The next CL will read these options from Skyframe when constructing the Skylark environment. This CL affects the dataflow from command/test initialization to Skyframe. Some code paths, like those used for testing, use the default SkylarkSemanticsOptions and therefore won't be able to use (for example) --incompatible_* flags. The call sites to update were found by searching for uses of defaultVisibility and working upward from there. RELNOTES: None PiperOrigin-RevId: 154432058
* Add repository override optionGravatar kchodorow2017-04-20
| | | | | | | | | | | RELNOTES: Adds a --override_repository option that takes a repository name and path. This forces Bazel to use the directory at that path for the repository. Example usage: `--override_repository=foo=/home/user/gitroot/foo`. Fixes #1266 PiperOrigin-RevId: 153599291
* PiperOrigin-RevId: 152483983Gravatar lpino2017-04-07
|
* When tracking the critical path, if not keeping incremental state, don't ↵Gravatar janakr2017-04-06
| | | | | | | | keep references to actions indefinitely. Instead, once an action is finished executing, keep just some metadata about it. This allows actions to be unconditionally dropped when running with --batch, --discard_analysis_cache, and --keep_going, even if profiling is enabled. The additional fields here add between 8 and 12 bytes per component, and we have one component per action. This additional penalty is only incurred when we are already saving memory, so I think it's ok. The full penalty will be realized only towards the end of the build, when every action has started executing at least once. Users can still specify --noexperimental_enable_critical_path_profiling if they want to squeeze even more memory out. PiperOrigin-RevId: 152328870
* Move action_outs directory to under the output path.Gravatar janakr2017-04-06
| | | | PiperOrigin-RevId: 152307322
* Automated g4 rollback of commit 1d9e1ac90197b1d3d7b137ba3c1ada67bb9ba31b.Gravatar hlopko2017-04-04
| | | | | | | | | | | | | | | | | | *** Reason for rollback *** Breaks //src/test/shell/integration:force_delete_output_test *** Original change description *** Symlink output directories to the correct directory name If the workspace directory is /path/to/my/proj and the name in the WORKSPACE file is "floop", this will symlink the output directories to output_base/execroot/floop instead of output_base/execroot/proj. More prep for #1262, fixes #1681. PiperOrigin-RevId: 152126545
* If --batch, --keep_going, --discard_analysis_cache, and the new ↵Gravatar janakr2017-03-31
| | | | | | | | | | | | | | | --noexperimental_enable_critical_path_profiling flags are all specified, then Bazel will delete Actions from ActionLookupValues as they are executed in order to save memory. Because an already-run action may output an artifact that is only requested later in the build, we need to maintain a way for the artifact to look up the action. But in most cases we don't need to keep the action itself, just its output metadata. Some actions unfortunately are needed post-execution, and so we special-case them. Also includes dependency change with description: Move action out of key. This keeps action references from polluting the graph -- actions are just stored in one SkyValue, instead of being present in SkyKeys. This does mean additional memory used: we have a separate ActionLookupData object per Action executed. That may reach ~24M for million-action builds. PiperOrigin-RevId: 151756383
* Symlink output directories to the correct directory nameGravatar kchodorow2017-03-31
| | | | | | | | | | If the workspace directory is /path/to/my/proj and the name in the WORKSPACE file is "floop", this will symlink the output directories to output_base/execroot/floop instead of output_base/execroot/proj. More prep for #1262, fixes #1681. PiperOrigin-RevId: 151712384
* Env.vars: server won't ignore the client envGravatar Dmitry Lomov2017-03-08
| | | | | | | | | | | | | | | | | | Make the --ignore_client_env flag a no-op. The client will pass --client_env flags to the server even in --batch mode. This simplifies the code as well as ensuring that the server always uses the up-do-date client environment. We'll gradually get rid of all System.getenv calls in the server, because the server should always respect the client env. Roll forward of 149403129 with fixes. -- PiperOrigin-RevId: 149435060 MOS_MIGRATED_REVID=149435060
* Rollback of commit 94d8f4e9c8c8757aa7ab7c1a7c3e9afb34039127.Gravatar Klaus Aehlig2017-03-07
| | | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** broke //src/test/shell/bazel:bazel_bootstrap_distfile_test *** Original change description *** Env.vars: server won't ignore the client env Make the --ignore_client_env flag a no-op. The client will pass --client_env flags to the server even in --batch mode. This simplifies the code as well as ensuring that the server always uses the up-do-date client environment. We'll gradually get rid of all System.getenv calls in the server, because the server should always respect the client env. -- PiperOrigin-RevId: 149416602 MOS_MIGRATED_REVID=149416602
* Env.vars: server won't ignore the client envGravatar Laszlo Csomor2017-03-07
| | | | | | | | | | | | | | | | Make the --ignore_client_env flag a no-op. The client will pass --client_env flags to the server even in --batch mode. This simplifies the code as well as ensuring that the server always uses the up-do-date client environment. We'll gradually get rid of all System.getenv calls in the server, because the server should always respect the client env. -- PiperOrigin-RevId: 149403129 MOS_MIGRATED_REVID=149403129
* Provide more reporting options to SkyFunctions Gravatar Klaus Aehlig2017-02-27
| | | | | | | | | | | | | With more specific information to be reported by Skyfunctions, e.g., to inform the build-event protocol on missing files, the EventHandler interface is no longer enough. Therefore, provide an enriched context for reporting events. -- Change-Id: I2d06166fe4d5b9054e24ad8c752fafc039e3f9f8 Reviewed-on: https://cr.bazel.build/8794 PiperOrigin-RevId: 148463437 MOS_MIGRATED_REVID=148463437
* Introduce a Skyframe function: ActionEnvironmentFunction Gravatar Damien Martin-Guillerez2017-02-08
| | | | | | | | | | | | | | | | | | ActionEnvironmentFunction returns the list of environment variable with the one overwritten by --action_env being replaced. This let other Skyframe function declares dependency to any value of the environment and being influenced by the --action_env flag. This will be used to declare dependency of remote repositories on environment variables (step 3 of https://bazel.build/designs/2016/10/18/repository-invalidation.html) -- Change-Id: I1ed3fb6f48e8e17d4d64c903fccecb6ed7596350 Reviewed-on: https://cr.bazel.build/7974 PiperOrigin-RevId: 146918603 MOS_MIGRATED_REVID=146918603
* Merge the action input prefetcher into the ExecutorBuilder.Gravatar Ulf Adams2017-01-13
| | | | | | | | | This should be a no-op change, primarily intended to improve the BlazeModule API. The code simplification in the distributor code path is incidental. -- PiperOrigin-RevId: 144441458 MOS_MIGRATED_REVID=144441458
* Skylark repositories: propage value from --action_env to repository_ctx.environ Gravatar Damien Martin-Guillerez2017-01-10
| | | | | | | | | | | | | | | | | | | | | | The environment is now computed with a mixture of the client environment and the values specified by the --action_env flag. If a user want to overwrite its environment for skylark repository, they can do `--action_env FOO=BAR` and the repository will see FOO as having the value BAR, whichever value is set in the client environment. Also propagate it to all repository functions, and deduplicate the way the client environment is passed to repository functions. Design doc: https://bazel.build/designs/2016/10/18/repository-invalidation.html [step 1] RELNOTES[INC]: repository_ctx environment is now affected by --action_env flag (value from the client environment will be replaced by value given on the command line through --action_env). -- Change-Id: I131a9695439aa9949d5001f820e2ae450e41332f Reviewed-on: https://cr.bazel.build/7971 PiperOrigin-RevId: 144091492 MOS_MIGRATED_REVID=144091492
* Don't get output service or action input prefetcher if not running a "build" ↵Gravatar Janak Ramakrishnan2016-11-18
| | | | | | | command, to save time on the critical path of a build. -- MOS_MIGRATED_REVID=139477157
* Stop including the size of the action cache file in our crash message. I've ↵Gravatar Janak Ramakrishnan2016-11-17
| | | | | | | never seen a case it was useful, and it adds to startup latency by putting a file stat on the critical path of every command. -- MOS_MIGRATED_REVID=139467038
* Merge all action input prefetchers into a single one.Gravatar Ulf Adams2016-11-16
| | | | | | | | | It seems unnecessary to pass these lists around, so this should simplify the APIs. Also, I want to move the action input prefetcher setup to the serverInit call. -- MOS_MIGRATED_REVID=139304928
* Add a GenSignatureActionContext to inject the command line info.Gravatar Ulf Adams2016-11-15
| | | | | | | | | This is needed to fulfill the contract of the gensignature rule / the corresponding audit trail protobuf. Note that the action is uncachable and unconditional to execute. -- MOS_MIGRATED_REVID=139178114
* Refactor the module API to use the builder pattern for executor creation.Gravatar Ulf Adams2016-11-02
| | | | | | | This significantly simplifies several of our modules. -- MOS_MIGRATED_REVID=137713119
* Make --watchfs a common command option.Gravatar Ulf Adams2016-10-14
| | | | | | | | | | | | | | | | | | | | | | | | | Adding an options parameter to DiffAwareness#getCurrentView seems like the simplest way to achieve that. Alternatives considered: 1. Making the diff awareness modules stateful. However, I did not want to do so as I've also been working on improving the module API to reduce state, or at least to have a proper lifecycle management for any necessary state. 2. Making the watchFs flag a constructor parameter. However, that would also invalidate any implementations that don't use the flag (of which we have several). 3. Only passing in a single boolean flag instead of an options class provider; however, this is a more principled, futureproof API, which allows other modules / awareness implementations to use their own options. RELNOTES: --watchfs is now a command option; the startup option of the same name is deprecated. I.e., use bazel build --watchfs, not blaze --watchfs build. -- MOS_MIGRATED_REVID=136154395
* Rollback of commit 2891ec527eed27d0c6460d66f51cb66a43373b6a.Gravatar Damien Martin-Guillerez2016-10-14
| | | | | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Causes our integration tests on Darwin to time out *** Original change description *** Make --watchfs a common command option. Adding an options parameter to DiffAwareness#getCurrentView seems like the simplest way to achieve that. Alternatives considered: 1. Making the diff awareness modules stateful. However, I did not want to do so as I've also been working on improving the module API to reduce state, or at least to have a proper lifecycle management for any necessary state. 2. Making the watchFs flag a constructor parameter. However, that would also invalidate any implement... *** -- MOS_MIGRATED_REVID=136070807
* Make --watchfs a common command option.Gravatar Ulf Adams2016-10-13
| | | | | | | | | | | | | | | | | | | | | | | | | Adding an options parameter to DiffAwareness#getCurrentView seems like the simplest way to achieve that. Alternatives considered: 1. Making the diff awareness modules stateful. However, I did not want to do so as I've also been working on improving the module API to reduce state, or at least to have a proper lifecycle management for any necessary state. 2. Making the watchFs flag a constructor parameter. However, that would also invalidate any implementations that don't use the flag (of which we have several). 3. Only passing in a single boolean flag instead of an options class provider; however, this is a more principled, futureproof API, which allows other modules / awareness implementations to use their own options. RELNOTES: --watchfs is now a command option; the startup option of the same name is deprecated. I.e., use bazel build --watchfs, not blaze --watchfs build. -- MOS_MIGRATED_REVID=136026835
* Enable Bazel commands to exit at any time.Gravatar Michael Staib2016-10-05
| | | | | | | | | | | | | This is the first step on a journey toward allowing commands to AbruptExit wherever they please, similar to how the user can press Ctrl+C at any time and we (should) bail out as fast as we can. By interrupting the command's main thread, we at least offer the command the ability to see that an error requiring a bail has happened, and it should trigger at potentially more locations, rather than just between phases. -- MOS_MIGRATED_REVID=135152330
* Add an info item to show the currently inherited client environmentGravatar Klaus Aehlig2016-09-21
| | | | | | | | | | | | | Add 'bazel info client-env' which outputs entries for the configuration file that would freeze the current client environment. The main intended use case is to use 'bazel info client-env >> .bazelrc' to keep the project reproducible once a suitable value for the environment variables that used to be taken from the client environment has been found. -- Change-Id: Ib4d14dd824d223f335a4d4de04ee21c4a3ec4d83 Reviewed-on: https://bazel-review.googlesource.com/#/c/6112 MOS_MIGRATED_REVID=133699234
* Remove some more occurences of deprecated getExecRoot() callsGravatar Kristina Chodorow2016-09-15
| | | | | -- MOS_MIGRATED_REVID=133257532
* Track client environment in SkyframeGravatar Klaus Aehlig2016-09-14
| | | | | | | | | | | | | ...to determine which actions have to be recomputed based on changes to the client environment. Note that this change does it the simple way and reconsideres all actions on a changed client environment, while still only reexecuting those, where the part that was inherited from the environment actually did change. -- Change-Id: Ie1116d094642165e5e959447a6fcf49d19b37d6e Reviewed-on: https://bazel-review.googlesource.com/#/c/5431 MOS_MIGRATED_REVID=133010705
* Only call BlazeModule.handleOptions when we're done editing them.Gravatar Ulf Adams2016-09-06
| | | | | | | | | Previously, it was possible for us to edit the options in command.editOptions, after reporting them to the modules, though I don't have any examples of that happening. I found this just looking through the code. -- MOS_MIGRATED_REVID=132310873
* Add a framework for prefetching input files (in case they come from remote ↵Gravatar Philipp Wollermann2016-08-25
| | | | | | | file systems). -- MOS_MIGRATED_REVID=131280794
* Support setting the build id via the client environmentGravatar Klaus Aehlig2016-08-16
| | | | | | | | | | | | | | | | Normally, Bazel generates a new UUID for every build invocation. In some use cases, however, the invocation of Bazel is part of a larger build process or otherwise controlled by a different tool. In this case, it is useful to tell Bazel the identifier of the build to make it fit with other identifiers generated by the controlling process. To achieve this, the variable BAZEL_INTERNAL_INVOCATION_ID from the client environment is inspected and if it is set and its value is a syntactically correct UUID, this UUID will be used as the identifier for the build. It is in the responsibility of the caller to ensure the id is sufficiently unique if that environment variable is set. -- MOS_MIGRATED_REVID=130079446
* Remove unused methodGravatar Kristina Chodorow2016-07-21
| | | | | -- MOS_MIGRATED_REVID=128080879
* Various fixes for gRPC mode:Gravatar Lukacs Berki2016-07-07
| | | | | | | | - Use the abrupt exit protocol to return the correct exit code when the server dies unexpectedly in gRPC mode - Report the command waiting time correctly even if the waiting is done within the server -- MOS_MIGRATED_REVID=126780186
* Post BlazeDirectories as part of CommandStartedEvent.Gravatar Ulf Adams2016-07-06
| | | | | | | | We have some modules that assume 1:1 correspondence between servers and workspaces, which I'd like to fix. -- MOS_MIGRATED_REVID=126628292
* Make "bazel info" load only the packages that are necessary for its job.Gravatar Lukacs Berki2016-06-01
| | | | | -- MOS_MIGRATED_REVID=123723401
* When we don't have an OutputService, precompute and cache the ↵Gravatar Nathan Harmata2016-05-02
| | | | | | | | | | | assumed-to-be-immutable filesystem type of output_base. When we do have an OutputService, defer computation of the OutputService's dynamic filesystem type until we actually need it at the start of a "build" or "test" command. This saves ~450us on each invocation, according to my benchmarks. -- MOS_MIGRATED_REVID=121156323
* Inject the BlazeDirectories in SkyframeExecutor.Gravatar Ulf Adams2016-04-15
| | | | | | | | | The BlazeDirectories are also needed for loading the WORKSPACE file, so inject them as part of preparePackageLoading rather than in createConfigurations, which is too late. -- MOS_MIGRATED_REVID=119931633
* Move BlazeRuntime.initCommand to BlazeWorkspace.Gravatar Ulf Adams2016-03-30
| | | | | -- MOS_MIGRATED_REVID=118568284
* Move most of the workspace-handling code from BlazeRuntime to a new class.Gravatar Ulf Adams2016-03-30
| | | | | -- MOS_MIGRATED_REVID=118563271
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-04 10:08:41 +0000