| Commit message (Collapse) | Author | Age |
|
|
|
|
|
| |
- migrate all startTask/completeTask pairs to the new API
PiperOrigin-RevId: 200038703
|
|
|
|
|
|
|
|
|
| |
This constructor was creating an Exception with a null message leading to possible
NullPointerExceptions in a few places in our codebase. The call sites have
been replaced with calls to AbruptException(String message, ExitCode exitCode) with
a meaningful message.
PiperOrigin-RevId: 196973540
|
|
|
|
|
|
| |
CommonCommandOptions to better facilitate certain kinds of benchmarking.
PiperOrigin-RevId: 196695342
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
child of the process where the Blaze client itself was run.
Limitations:
- Untested on Windows; it should work because ExecuteProgram() is implemented there, too, but since Windows doesn't support exec(), there is at least one process in between
Progress towards #2815.
RELNOTES[NEW]: The new "--direct_run" flag on "blaze run" lets one run interactive binaries.
PiperOrigin-RevId: 184528845
|
|
|
|
|
|
|
|
|
| |
New name clears the namespace a 2nd flag that will wipe the build graph after the build. The old name would be confusing as it could easily apply to that, and so needs to be more specifically just about tracking state in the first place. The new flag can be clearly separate and about keeping state after the build.
Partial roll forward of https://github.com/bazelbuild/bazel/commit/9321316b34767b06c3071b2cf2a4de189874fcce, with fixes to documentation that are still relevant.
RELNOTES: Rename --keep_incrementality_data to --track_incremental_state
PiperOrigin-RevId: 179078292
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
https://github.com/bazelbuild/bazel/commit/c6b6dbadd0a93936c51154b25abc5fbba8f2d1af)
We accepted these by environment variable largely because setting it via invocation policy would require changing invocation policy for each command, which had caused the Bazel server to restart, loosing incremental state. This is fixed: changing invocation policy no longer causes Bazel to restart its servers, so accept these as normal options.
We will soon no longer accept these flags by environment variable, but will accept both for a transition period, so that nobody relying on these values is broken by a single release. To inform users of this environment variable, anyone setting the environment variable without the flag will receive a warning but the value will be kept. The following release will no longer accept an environment variable.
Note on format: invocation_id we accept only clean UUIDs, but for build_request_id, to help differentiate otherwise undifferentiable id types, we accept arbitrary prefixes before the UUID. The user is responsible for picking prefixes that are sane.
RELNOTES: None.
PiperOrigin-RevId: 179063120
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Design change, 2 boolean flags instead of 1 enum flag
*** Original change description ***
Add --incremental_state_retention_strategy
This option is intended to replace some of the uses of --batch. It lets users specify that builds should not be incremental, and how eagerly to discard the state that is kept around for incrementality. Note that for both values discard_eargerly and keep_for_life_of_build, the build graph is kept around until the next build. This may change.
Will add tests for keep_for_life_of_build in a later change, for now it will warn that that feature is experimen...
***
ROLLBACK_OF=178661777
RELNOTES: None.
PiperOrigin-RevId: 178681472
|
|
|
|
|
|
|
|
|
| |
This option is intended to replace some of the uses of --batch. It lets users specify that builds should not be incremental, and how eagerly to discard the state that is kept around for incrementality. Note that for both values discard_eargerly and keep_for_life_of_build, the build graph is kept around until the next build. This may change.
Will add tests for keep_for_life_of_build in a later change, for now it will warn that that feature is experimental.
RELNOTES: --[no]keep_incrementality_data is gone, replaced by the enum-valued --incremental_state_retention_strategy
PiperOrigin-RevId: 178661777
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
memory-saving non-incremental mode independent of --batch and --discard_analysis_cache.
A command run with --nokeep_incrementality_data will discard data that would be needed for incremental builds. Subsequent commands can be sent to the same server, but they will not get the benefit of incrementality from this command. However, if --keep_incrementality_data is specified on a subsequent command, the commands after that will get the benefits of incrementality.
There are two benefits to not being dependent on --batch. First, this allows Bazel servers to be run in extreme memory-saving mode without the startup penalties (JVM startup, JITting) that --batch execution imposes. Second, this allows Bazel developers to inspect the state of a Bazel server after an extreme memory-saving build.
In order to avoid discarding data unnecessarily (for instance, on a "bazel info used-heap-size-after-gc" or "bazel dump --skyframe=summary") the actual resetting of the graph is done lazily, right before its use in SequencedSkyframeExecutor#sync. This is morally a partial rollback of https://github.com/bazelbuild/bazel/commit/98cd82cbdcac7c48164a611c5a9aa8fc2f1720ef.
For now, our tests specify all of the flags. After this change sticks, I plan to get rid of the --batch flag from these tests, which should allow for some clean-ups. Eventually --batch and --discard_analysis_cache may not imply that we don't keep incremental state: we can require that it be specified explicitly.
PiperOrigin-RevId: 175335075
|
|
|
|
|
|
|
|
| |
Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own.
This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually.
PiperOrigin-RevId: 175033526
|
|
|
|
|
|
|
|
| |
Not a totally clean rollback because of the intervening https://github.com/bazelbuild/bazel/commit/b5158a9a677b149e04e844c40a01e9a9dde40783.
*** Reason for rollback ***
PiperOrigin-RevId: 173957187
|
|
|
|
|
|
| |
This requires a fairly large amount of changes to fundamental objects like BlazeRuntime, Executor, and so on, as well as changing a lot of test code to thread the file system through. I expect future CLs to be much smaller.
PiperOrigin-RevId: 173678144
|
|
|
|
|
|
|
|
|
| |
CommonCommandOptions.
client_env and default_override stand apart from the other common options. They can be very long, and should only be used by the environment setup. They should ideally not be passed by option at all, but in the RunRequest proto message, but for now, isolate them so that they don't clutter the common command options, which are passed all over the place.
RELNOTES: None.
PiperOrigin-RevId: 173592980
|
|
|
|
|
|
|
|
|
|
|
| |
We accepted these by environment variable largely because setting it via invocation policy would require changing invocation policy for each command, which had caused the Bazel server to restart, loosing incremental state. This is fixed: changing invocation policy no longer causes Bazel to restart its servers, so accept these as normal options.
We will soon no longer accept these flags by environment variable, but will accept both for a transition period, so that nobody relying on these values is broken by a single release. To inform users of this environment variable, anyone setting the environment variable without the flag will receive a warning but the value will be kept. The following release will no longer accept an environment variable.
Note on format: invocation_id we accept only clean UUIDs, but for build_request_id, to help differentiate otherwise undifferentiable id types, we accept arbitrary prefixes before the UUID. The user is responsible for picking prefixes that are sane.
RELNOTES: None.
PiperOrigin-RevId: 173432904
|
|
|
|
|
|
| |
test to make sure we are using the expected type of node entries when discarding/keeping graph edges.
PiperOrigin-RevId: 173131307
|
|
|
|
|
|
|
|
|
| |
SkylarkSemanticsCodec
Note that the syntax package and its test package still depend indirectly on the options parser via other Bazel-specific packages.
RELNOTES: None
PiperOrigin-RevId: 171342823
|
|
|
|
|
|
|
|
|
|
| |
BAZEL_INTERNAL_BUILD_REQUEST_ID.
I added a check for build request id to be a valid UUID, and it broke them.
TESTED=tap
RELNOTES: None
PiperOrigin-RevId: 170886183
|
|
|
|
|
|
| |
TESTED=unit tests
RELNOTES: none
PiperOrigin-RevId: 169395919
|
|
|
|
| |
PiperOrigin-RevId: 169179218
|
|
|
|
|
|
| |
injected in BlazeCommandDispatcher#execExclusively.
PiperOrigin-RevId: 169122571
|
|
|
|
|
|
|
|
|
|
|
| |
This isn't ideal, but is the fastest way to split analysis, execution, and
rules into separate java libraries, with lib.skyframe still being in the
analysis library. Ideally, we'd split up the lib.skyframe package, move the
analysis stuff to the analysis library, the execution stuff to the execution
library, and so on. That wouldn't require us moving OutputService out of
lib.exec.
PiperOrigin-RevId: 164856998
|
|
|
|
|
|
| |
necessary, the reset occurs before the command starts.
PiperOrigin-RevId: 162973194
|
|
|
|
|
|
|
| |
Now that we have the options before calling beforeCommand, there's no need for
a separate handleOptions method in the BlazeModule API. Remove it.
PiperOrigin-RevId: 162002300
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Before this change, top-level configs were still built statically,
then cloned into dynamic configs to run the actual build.
After this change, no static configuration should ever be created in
a dynamically configured build.
We're still keeping support for --experimental_dynamic_configs=off a
bit longer to provide easy reversion on any bad surprises. But barring
that we'll quickly move toward making --experimental_dynamic_configs=off
a no-op, then ripping out Bazel's static configuration logic.
PiperOrigin-RevId: 161005150
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The invocation policy must be the last step in determining the active options
for the current command. Take precautions to prevent accidental options editing
of options after the policy is applied by declaring the options as an
OptionsProvider (interface has no mutating methods); technically, this could
be circumvented by casting to OptionsParser, but it's at least safer than
before.
As part of this, I'm moving the BlazeCommand.editOptions call to before the
invocation policy, and also making minor changes to the CommandEnvironment.
Commands that expect to have the last word on options could therefore see
options changes after this commit.
There are three commands that override editOptions:
1. TestCommand
Current behavior: if test_output=streamed is set, also sets
--test_sharding_strategy=disabled and --test_strategy=exclusive.
Overriding --test_sharding_strategy is not a concern; in fact, maybe we
shouldn't be setting that in the first place, since it can cause tests to
timeout (timeout is usually applied per shard, so a 10-way sharded test will
very likely timeout). If you override the test_strategy to local or standalone,
then you may get interleaved stdout / stderr output from tests that run in
parallel - a bit surprising, but no showstopper. If you override the
test_strategy to remote, you won't get streamed output, because no remote
strategy currently supports that. You may get interleaved output, if multiple
tests finish very close to each other.
There are no correctness concerns, it's just a slightly worse user experience.
The code is safe in all cases, AFAICT.
We could consider changing streamed to not require serialized execution, and
instead use something like a lock - the first test to produce output gets
streamed, and all others get delayed until the lock is released, but could
still execute concurrently. However, it's unlikely that that's the desired
behavior for most users.
2. CoverageCommand
Current behavior: sets --collect_code_coverage, increases default
--test_timeout.
bazel coverage --nocollect_code_coverage is effectively bazel test. You can
actually now run bazel test --collect_code_coverage --test_timeout and it will
behave exactly the same way as bazel coverage. It's mostly a convenience thing.
Note that CoverageCommand inherits TestCommand, so the case above also applies.
3. MobileInstallCommand
Current behavior: sets --aspects, and --output_groups.
If you override the aspect or output_groups, then the command will fail or run
an old binary from a previous build. Not what you'd expect, but no correctness
concerns.
Summary:
IMO, the impact on the existing commands is minor. The advantage of this change
is that it's more secure, and gives us an escape hatch if a command ever
overrides options in a way that turns out to be wrong or broken in some way.
RELNOTES: None.
PiperOrigin-RevId: 160114902
|
|
|
|
|
|
|
|
|
|
|
| |
Don't pass the Command annotation explicitly, but add it to CommandEnvironment
instead; most modules don't need it in the first place, so it was a lot of
boilerplate for not much. Also change it so that the command is passed to the
constructor.
Add some documentation to the beforeCommand method.
PiperOrigin-RevId: 158847128
|
|
|
|
|
|
|
|
| |
Instead of passing a client env into the test strategies, use the same
mechansim as --action_env, by depending on the right set of Skyframe nodes that
correspond to client env entries.
PiperOrigin-RevId: 158401670
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Roll forward of directory name change
*** Original change description ***
Automated g4 rollback of commit 1d9e1ac90197b1d3d7b137ba3c1ada67bb9ba31b.
*** Reason for rollback ***
Breaks //src/test/shell/integration:force_delete_output_test
*** Original change description ***
Symlink output directories to the correct directory name
If the workspace directory is /path/to/my/proj and the name in the WORKSPACE
file is "floop", this will symlink the output directories to
output_base/execroot/floop instead of output_base/execroot/proj.
More prep for #1262, fixes #1681.
PiperOrigin-RevId: 156892980
|
|
|
|
|
|
| |
as part of the startup arguments.
PiperOrigin-RevId: 156090009
|
|
|
|
|
|
|
|
|
| |
This is the first of two CLs for making command line options able to affect the Skylark interpreter. It introduces SkylarkSemanticsOptions, and stores it as a precomputed (injected) value in Skyframe. The next CL will read these options from Skyframe when constructing the Skylark environment.
This CL affects the dataflow from command/test initialization to Skyframe. Some code paths, like those used for testing, use the default SkylarkSemanticsOptions and therefore won't be able to use (for example) --incompatible_* flags. The call sites to update were found by searching for uses of defaultVisibility and working upward from there.
RELNOTES: None
PiperOrigin-RevId: 154432058
|
|
|
|
|
|
|
|
|
|
|
| |
RELNOTES: Adds a --override_repository option that takes a repository
name and path. This forces Bazel to use the directory at that path
for the repository. Example usage:
`--override_repository=foo=/home/user/gitroot/foo`.
Fixes #1266
PiperOrigin-RevId: 153599291
|
| |
|
|
|
|
|
|
|
|
| |
keep references to actions indefinitely. Instead, once an action is finished executing, keep just some metadata about it. This allows actions to be unconditionally dropped when running with --batch, --discard_analysis_cache, and --keep_going, even if profiling is enabled.
The additional fields here add between 8 and 12 bytes per component, and we have one component per action. This additional penalty is only incurred when we are already saving memory, so I think it's ok. The full penalty will be realized only towards the end of the build, when every action has started executing at least once. Users can still specify --noexperimental_enable_critical_path_profiling if they want to squeeze even more memory out.
PiperOrigin-RevId: 152328870
|
|
|
|
| |
PiperOrigin-RevId: 152307322
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Breaks //src/test/shell/integration:force_delete_output_test
*** Original change description ***
Symlink output directories to the correct directory name
If the workspace directory is /path/to/my/proj and the name in the WORKSPACE
file is "floop", this will symlink the output directories to
output_base/execroot/floop instead of output_base/execroot/proj.
More prep for #1262, fixes #1681.
PiperOrigin-RevId: 152126545
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
--noexperimental_enable_critical_path_profiling flags are all specified, then Bazel will delete Actions from ActionLookupValues as they are executed in order to save memory.
Because an already-run action may output an artifact that is only requested later in the build, we need to maintain a way for the artifact to look up the action. But in most cases we don't need to keep the action itself, just its output metadata.
Some actions unfortunately are needed post-execution, and so we special-case them.
Also includes dependency change with description:
Move action out of key. This keeps action references from polluting the graph -- actions are just stored in one SkyValue, instead of being present in SkyKeys.
This does mean additional memory used: we have a separate ActionLookupData object per Action executed. That may reach ~24M for million-action builds.
PiperOrigin-RevId: 151756383
|
|
|
|
|
|
|
|
|
|
| |
If the workspace directory is /path/to/my/proj and the name in the WORKSPACE
file is "floop", this will symlink the output directories to
output_base/execroot/floop instead of output_base/execroot/proj.
More prep for #1262, fixes #1681.
PiperOrigin-RevId: 151712384
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Make the --ignore_client_env flag a no-op.
The client will pass --client_env flags to the
server even in --batch mode. This simplifies the
code as well as ensuring that the server always
uses the up-do-date client environment.
We'll gradually get rid of all System.getenv calls
in the server, because the server should always
respect the client env.
Roll forward of 149403129 with fixes.
--
PiperOrigin-RevId: 149435060
MOS_MIGRATED_REVID=149435060
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
broke //src/test/shell/bazel:bazel_bootstrap_distfile_test
*** Original change description ***
Env.vars: server won't ignore the client env
Make the --ignore_client_env flag a no-op.
The client will pass --client_env flags to the
server even in --batch mode. This simplifies the
code as well as ensuring that the server always
uses the up-do-date client environment.
We'll gradually get rid of all System.getenv calls
in the server, because the server should always
respect the client env.
--
PiperOrigin-RevId: 149416602
MOS_MIGRATED_REVID=149416602
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Make the --ignore_client_env flag a no-op.
The client will pass --client_env flags to the
server even in --batch mode. This simplifies the
code as well as ensuring that the server always
uses the up-do-date client environment.
We'll gradually get rid of all System.getenv calls
in the server, because the server should always
respect the client env.
--
PiperOrigin-RevId: 149403129
MOS_MIGRATED_REVID=149403129
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
With more specific information to be reported by Skyfunctions, e.g.,
to inform the build-event protocol on missing files, the EventHandler
interface is no longer enough. Therefore, provide an enriched context
for reporting events.
--
Change-Id: I2d06166fe4d5b9054e24ad8c752fafc039e3f9f8
Reviewed-on: https://cr.bazel.build/8794
PiperOrigin-RevId: 148463437
MOS_MIGRATED_REVID=148463437
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
ActionEnvironmentFunction returns the list of environment
variable with the one overwritten by --action_env being
replaced. This let other Skyframe function declares
dependency to any value of the environment and being
influenced by the --action_env flag.
This will be used to declare dependency of remote repositories
on environment variables (step 3 of
https://bazel.build/designs/2016/10/18/repository-invalidation.html)
--
Change-Id: I1ed3fb6f48e8e17d4d64c903fccecb6ed7596350
Reviewed-on: https://cr.bazel.build/7974
PiperOrigin-RevId: 146918603
MOS_MIGRATED_REVID=146918603
|
|
|
|
|
|
|
|
|
| |
This should be a no-op change, primarily intended to improve the BlazeModule
API. The code simplification in the distributor code path is incidental.
--
PiperOrigin-RevId: 144441458
MOS_MIGRATED_REVID=144441458
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The environment is now computed with a mixture of the client environment and
the values specified by the --action_env flag. If a user want to overwrite its
environment for skylark repository, they can do `--action_env FOO=BAR` and
the repository will see FOO as having the value BAR, whichever value is
set in the client environment.
Also propagate it to all repository functions, and deduplicate the way
the client environment is passed to repository functions.
Design doc: https://bazel.build/designs/2016/10/18/repository-invalidation.html [step 1]
RELNOTES[INC]: repository_ctx environment is now affected by --action_env flag (value from the
client environment will be replaced by value given on the command line through --action_env).
--
Change-Id: I131a9695439aa9949d5001f820e2ae450e41332f
Reviewed-on: https://cr.bazel.build/7971
PiperOrigin-RevId: 144091492
MOS_MIGRATED_REVID=144091492
|
|
|
|
|
|
|
| |
command, to save time on the critical path of a build.
--
MOS_MIGRATED_REVID=139477157
|
|
|
|
|
|
|
| |
never seen a case it was useful, and it adds to startup latency by putting a file stat on the critical path of every command.
--
MOS_MIGRATED_REVID=139467038
|
|
|
|
|
|
|
|
|
| |
It seems unnecessary to pass these lists around, so this should simplify the
APIs. Also, I want to move the action input prefetcher setup to the serverInit
call.
--
MOS_MIGRATED_REVID=139304928
|
|
|
|
|
|
|
|
|
| |
This is needed to fulfill the contract of the gensignature rule / the
corresponding audit trail protobuf. Note that the action is uncachable and
unconditional to execute.
--
MOS_MIGRATED_REVID=139178114
|
|
|
|
|
|
|
| |
This significantly simplifies several of our modules.
--
MOS_MIGRATED_REVID=137713119
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adding an options parameter to DiffAwareness#getCurrentView seems like the
simplest way to achieve that.
Alternatives considered:
1. Making the diff awareness modules stateful. However, I did not want to do so
as I've also been working on improving the module API to reduce state, or at
least to have a proper lifecycle management for any necessary state.
2. Making the watchFs flag a constructor parameter. However, that would also
invalidate any implementations that don't use the flag (of which we have
several).
3. Only passing in a single boolean flag instead of an options class provider;
however, this is a more principled, futureproof API, which allows other
modules / awareness implementations to use their own options.
RELNOTES: --watchfs is now a command option; the startup option of the same
name is deprecated. I.e., use bazel build --watchfs, not blaze --watchfs
build.
--
MOS_MIGRATED_REVID=136154395
|