| Commit message (Collapse) | Author | Age |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We are currently tracking the action environment (computed from --action_env),
and the test action environment (computed from --test_env, and not including
the action env) as two pairs of fields, and a lot of callers haven't been
updated to handle both parts (TestRunnerAction does, but many 'normal' actions
don't). However, both parts have always both be handled, and moving them into
a single abstraction makes it harder to accidentally miss one part.
We'll subsequently need to update all the action implementations to use the
new abstraction, and also update the Skylark API (or bypass it and always apply
the action environment for all actions, except we don't know which
configuration to use).
PiperOrigin-RevId: 160386181
|
|
|
|
|
|
|
|
| |
Move the default from the annotation to every mention. This makes the incompleteness explicit. Will add the defaults to test targets in a separate change.
Once all dependencies are cleaned up, the Option annotation will no longer allow options without the documentationCategory or effectTag, to prevent new options being added without categories while we migrate to the new option categorization.
PiperOrigin-RevId: 160281252
|
|
|
|
|
|
|
|
| |
This will allow us to add new and optional flags like selecting a strategy used to spawn / wait for the child process.
No one except Bazel should be calling "process-wrapper" and I couldn't find any references, so this breaking change should be fine.
PiperOrigin-RevId: 159685867
|
|
|
|
| |
PiperOrigin-RevId: 159423459
|
|
|
|
|
|
|
| |
Move everything to ActionExecutionContext, and drop Executor whereever possible.
This clarifies the API, makes it simpler to test, and simplifies the code.
PiperOrigin-RevId: 159414816
|
|
|
|
| |
PiperOrigin-RevId: 159221067
|
|
|
|
|
|
|
|
| |
The SpawnInputExpander returns null for empty files, but the
ActionInputPrefetcher does not expect null values, and implementations may
throw NPE.
PiperOrigin-RevId: 158960425
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Also add an interface to allow injecting that logic into LocalSpawnRunner; this
is in preparation for rewriting StandaloneSpawnStrategy to use
LocalSpawnRunner.
At the same time, this reduces the dependencies from exec / standalone to
rules.apple, which is a prerequisite for micro-Bazel.
There's a small semantic change hidden here - we now only set the new
XCodeLocalEnvProvider if we're actually running on Darwin, so we no longer
fail execution on non-Darwin platforms if XCODE_VERSION_OVERRIDE or
APPLE_SDK_VERSION_OVERRIDE is set. As a result, I moved the corresponding test
from StandaloneSpawnStrategyTest to the new XCodeLocalEnvProviderTest.
While I'm at it, also open source DottedVersionTest and CacheManagerTest.
PiperOrigin-RevId: 158829077
|
|
|
|
|
|
|
|
| |
Instead of passing a client env into the test strategies, use the same
mechansim as --action_env, by depending on the right set of Skyframe nodes that
correspond to client env entries.
PiperOrigin-RevId: 158401670
|
|
|
|
|
|
|
|
|
| |
- Only manually check the timeout if the process wrapper is not used
- Set the timeout correctly; the process API uses milliseconds
- flush the error output stream after writing
- return SIGALRM as exit code for timeout cases
PiperOrigin-RevId: 158374246
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
py_test cannot find it's data file at runtime. The reason is it's
running in a wrong directory.
The test directory is set to runfiles directory by test-setup.sh, but on
Windows, python binary unzip itself to another temp directory which
test-setup.sh doesn't know.
So let the python stub template switch to the correct runfiles directory
if RUN_UNDER_RUNFILES = 1
Fixed https://github.com/bazelbuild/bazel/issues/3134
Change-Id: If5dbee811330372d86484ebd871ea55d84bc29a8
PiperOrigin-RevId: 158299041
|
|
|
|
|
|
|
| |
- tag status results that are likely due to user behavior
- add a cache hit flag
PiperOrigin-RevId: 158240265
|
|
|
|
|
|
|
| |
Remove SpawnExecutionPolicy.shouldPrefetchInputsForLocalExecution in favor of
a static helper method in Spawns.
PiperOrigin-RevId: 158119993
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Rollforward with fix.
*** Original change description ***
Automated g4 rollback of commit c78c947e6a8cbb323304f872a3dcabb989a3d76b.
*** Reason for rollback ***
Breaks android targets in the nightly - see []
*** Original change description ***
Do not retain transitive data in AndroidLocalTestBase...
***
ROLLBACK_OF=156745610
RELNOTES: None
PiperOrigin-RevId: 157028029
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Breaks android targets in the nightly - see []
*** Original change description ***
Do not retain transitive data in AndroidLocalTestBase.
The argument strings and artifacts were both transitive and flattened, causing O(N^2) memory consumption.
PiperOrigin-RevId: 156745610
|
|
|
|
|
|
| |
The argument strings and artifacts were both transitive and flattened, causing O(N^2) memory consumption.
PiperOrigin-RevId: 156083738
|
|
|
|
|
|
|
|
|
|
| |
BinTools.getExecPath() returns an execroot-relative path, which should
be usable in a spawn as-is. Absolutizing it is unnecessary. Using a
relative path also prevents the execroot from getting into the action
key for runfiles building actions.
Change-Id: Ie04d1bac841e41679d97064dc6ac1f0561a21eef
PiperOrigin-RevId: 155914912
|
|
|
|
|
|
|
|
|
|
|
|
| |
the general collection interface (e.g. List) with an immutable type (e.g. ImmutableList).
For constant field declarations, you should use the immutable type (such as ImmutableList) instead of the general collection interface type (such as List). This communicates to your callers important semantic guarantees ([]
For more info, see:[]
Cleanup change automatically generated by error-prone refactoring //third_party/java_src/error_prone/project/core/src/main/java/com/google/errorprone/bugpatterns:MutableConstantField_refactoring on targets //third_party/bazel/...
PiperOrigin-RevId: 155305768
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This lets users specify that their test needs a minimum of <n> CPU cores
to run and not be flaky. Example for a reservation of 4 CPUs:
sh_test(
name = "test",
size = "large",
srcs = ["test.sh"],
tags = ["cpu:4"],
)
This could also be used by remote execution strategies to tune their
resource adjustment.
RELNOTES: You can increase the CPU reservation for tests by adding a "cpu:<n>" (e.g. "cpu:4" for four cores) tag to their rule in a BUILD file. This can be used if tests would otherwise overwhelm your system if there's too much parallelism.
PiperOrigin-RevId: 154856091
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bazel always miscounted the number of passes that a test was run,
resulting in confusing output like this:
philwo@philwo:~/src/errortest$ bazel test //tests:fail
[...]
//tests:fail FAILED in 1 out of 2 in 0.1s
ERROR .tests/fail
It shows "1 out of 2" even though just one pass happened.
With this fix, the output is correct:
philwo@philwo:~/src/errortest$ bazel test //tests:fail
[...]
//tests:fail FAILED in 0.1s
ERROR .tests/fail
Relevant to #2855.
PiperOrigin-RevId: 154043240
|
|
|
|
|
|
| |
Fixes #2853.
PiperOrigin-RevId: 153730500
|
|
|
|
|
|
|
|
| |
This is a prerequisite for Google to use it; there's no consistent logging
strategy yet, but Google internally requires logging, and it needs to be
somewhat backwards compatible.
PiperOrigin-RevId: 153454160
|
|
|
|
| |
PiperOrigin-RevId: 153444516
|
|
|
|
|
|
|
|
|
| |
These are two different concepts. Do not remove category overload compatibility in this CL, to keep this change limited to converting the current uses of category.
With some flyby formatting fixes on affected OptionsBases.
RELNOTES: None.
PiperOrigin-RevId: 153390002
|
|
|
|
| |
PiperOrigin-RevId: 152663008
|
|
|
|
|
| |
This is ported from StandaloneSpawnStrategy.
PiperOrigin-RevId: 152493898
|
|
|
|
|
|
|
|
|
| |
The LocalSpawnRunner is a non-sandboxed local execution implementation, which
will replace the current StandaloneSpawnStrategy. The code has been around for
a long time and has seen a lot of bugfixes. It also supports local prefetching,
which is required for Google. I have a follow-up change to make it support
Windows, so it's not a drop-in replacement for StandaloneSpawnStrategy yet.
PiperOrigin-RevId: 152486973
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The intention is for the SpawnRunner interface to be the single, unified
interface for running Spawns, so it needs to cover both execution and error
reporting functionality for all current implementations, some of which are
internal to Google.
Note in particular the unified status code - it reports success if the
subprocess was executed regardless of its exit code, which is reported
separately.
PiperOrigin-RevId: 152252975
|
|
|
|
|
|
|
|
|
|
|
|
| |
'create' method.
This paves the way for changing PathFragment to e.g. an abstract class with multiple subclasses. This way we can split out the windows-specific stuff into one of these concrete classes, making the code more readable and also saving memory (since the shallow heap size of the NonWindowsPathFragment subclass will hopefully be smaller than that of the current PathFragment).
This also lets us pursue gc churn optimizations. We can now do interning in PathFragment#create and can also get rid of unnecessary intermediate PathFragment allocations.
RELNOTES: None
PiperOrigin-RevId: 152145768
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The RemoteSpawnRunner now implements the SpawnRunner interface.
Note that Google's internal implementations were also retrofitted, and
SpawnRunner is intended as a stable interface; that's also why I decided to
move all params into SpawnExecutionPolicy, which is, unfortunately, not quite
done yet.
The specification of SpawnRunner is also still incomplete. In particular, it
is still missing execution info keys, as well as inputs and outputs handling.
This is a step towards unifying all SpawnStrategy implementations, with the
SpawnRunner implementations performing the actual Spawn execution.
There should be no user-visible semantic changes to the code, but one small
fix:
- GrpcActionCache was trying to download files even if there were none
PiperOrigin-RevId: 152105696
|
|
|
|
|
|
|
|
|
|
| |
information is useful, the critical path computer retains references to objects that could otherwise be cleared to save memory.
This change is probably not worth submitting on its own -- the benefit it provides is too slight. But my follow-up change unknown commit needs this option to be effective -- the critical path currently hangs on to references to every action in the graph, so we can't drop references to actions if it's enabled.
The critical path could probably be reworked in the future to not hang onto those references.
PiperOrigin-RevId: 151747605
|
|
|
|
|
|
|
|
|
| |
Record the starting times of test actions, so that they can be reported
in the build event protocol.
Change-Id: I28e8d7d6ad39d91f4ffdd8a6161a5fc30f9a39b8
PiperOrigin-RevId: 151724760
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The new RemoteExecutionClient class only performs remote execution, and
nothing else; all higher-level functions, local rety, etc. will live outside
of the client.
In order to add unit tests, I had to add another layer of indirection between
the Grpc{RemoteExecutor,ActionCache} and GRPC, since GRPC generates final,
non-mockable classes. While a testing approach that uses a fake server can
also get some test coverage (as in GrpcActionCacheTest), it doesn't allow us
to test the full range of bad things that can happen at the GRPC layer.
The cloned implementation uses a single GRPC channel, as was recommended to
me by Jakob, who worked on GRPC. A single channel should be sufficiently
scalable, it's thread-safe, and it performs chunking internally. On the
server-side, the requests from a single channel can be dispatched to a thread
pool, so this should not be a blocker for server-side parallelism.
I also changed it to throw an exception whenever anything bad happens - this
makes it much more obvious if there's still bug in this code; the old code
silently swallows many errors, falling back to local execution, which papers
over many issues.
Furthermore, we now return a RemoteExecutionResult to indicate whether the
action ran at all (regardless of exit code), as well as the exit code.
All in all, this implementation is closer to the production code we're using
internally, although quite a few things are still missing.
The cloned implementation is not hooked up to RemoteSpawnStrategy yet. It
also does not support combining remote caching with local execution, but note
that RemoteSpawnStrategy regressed in that respect and currently also does
not support that mode.
PiperOrigin-RevId: 151578409
|
|
|
|
|
|
|
|
| |
--
Change-Id: I07c15f7d232a3e9363ebedfb9b5523999630c401
Reviewed-on: https://cr.bazel.build/9450
PiperOrigin-RevId: 150887776
MOS_MIGRATED_REVID=150887776
|
|
|
|
|
|
|
|
| |
Should fix #2721.
--
PiperOrigin-RevId: 150866145
MOS_MIGRATED_REVID=150866145
|
|
|
|
|
|
|
|
|
|
|
| |
Using /dev/null was causing it to create symlinks to /dev/null, which
breaks Python programs. I didn't catch this earlier, because my machine had
an old linux kernel installed that didn't support sandboxing. There's an
existing integration test, which just broke in our CI.
--
PiperOrigin-RevId: 150750032
MOS_MIGRATED_REVID=150750032
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This new class is a combination of SpawnHelper and our internal code; the
plan is to migrate all spawn strategies to the new class. The strict flag
should be enabled by default, but that's a breaking change, so we need to do
it later.
- Use it in SandboxStrategy.
- Add ActionInput.getExecPath to return a PathFragment; this avoids lots of
back and forth between path fragments and strings.
This is a step towards #1593.
The previous attempt was missing a one-line patch in StandaloneTestStrategy,
which broke all tests with sandboxing. StandaloneTestStrategy was fixed in a
separate change, so this should be safe now.
--
PiperOrigin-RevId: 150733457
MOS_MIGRATED_REVID=150733457
|
|
|
|
|
|
|
|
|
|
| |
Add preconditions to enforce this and remove some now unnecessary code.
A small step towards #1593.
--
PiperOrigin-RevId: 150625693
MOS_MIGRATED_REVID=150625693
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Break bazel-tests and many other jobs on CI.
http://ci.bazel.io/job/bazel-tests/BAZEL_VERSION=HEAD,PLATFORM_NAME=linux-x86_64/651/console
*** Original change description ***
Add SpawnInputExpander helper class to arrange runfiles for spawn strategies
This new class is a combination of SpawnHelper and our internal code; the
plan is to migrate all spawn strategies to the new class. The strict flag
should be enabled by default, but that's a breaking change, so we need to do
it later.
- Use it in SandboxStrategy.
- Add ActionInput.getExecPath to return a PathFragment; this avoids lots of
back and forth between path fragments and strings.
This is a step towards #159...
***
--
PiperOrigin-RevId: 150610616
MOS_MIGRATED_REVID=150610616
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Use a unique TEST_TMPDIR for each running test on Windows, too.
The old behavior caused shell tests to share the same TEST_TMPDIR, which
caused conflicts when multiple tests were run in parallel.
Set Bazel's TEST_TMPDIR root on Windows to c:/temp by default in
TestStrategy. This avoids tripping the 260-character limit for paths.
We still have to keep the override in testenv.sh for the time being,
because Bazel's tests are often run with the latest released Bazel and
thus we have to wait for a new Bazel release before we can remove the
TEST_TMPDIR override from testenv.sh. Once that is done, the flag
--test_tmpdir should also magically start working on Windows.
Move the Python detection logic out of bazel_windows_example_test.sh
into testenv.sh. This is better than copying that code into every new
test's set_up method.
--
PiperOrigin-RevId: 150433265
MOS_MIGRATED_REVID=150433265
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This new class is a combination of SpawnHelper and our internal code; the
plan is to migrate all spawn strategies to the new class. The strict flag
should be enabled by default, but that's a breaking change, so we need to do
it later.
- Use it in SandboxStrategy.
- Add ActionInput.getExecPath to return a PathFragment; this avoids lots of
back and forth between path fragments and strings.
This is a step towards #1593.
--
PiperOrigin-RevId: 150427021
MOS_MIGRATED_REVID=150427021
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This change modifies DigestUtils to add a cache of (path, inode, mtime,
size) to the digest of the file for those digests that are computed by
reading the file contents.
The cache itself is optional because relying on file metadata to cache
the file digests could lead to correctness issues. Enabling the cache
is exposed via a new (undocumented) --cache_computed_file_digests flag
that we can use post-release to tune the built-in values if they prove
to be incorrect or problematic.
For Bazel, enable this cache unconditionally because the rest of
Bazel already relies on mtimes and other file metadata to determine
changes to files.
The rationale for this change is performance: once we have lost the
in-memory file metadata (e.g. because of a flag flip), Bazel has to
redigest all of the output files to recompute action cache keys. For
a pathological case of rebuilding a large app with 5GB of outputs and
then flipping the --[no]check_visibility flag on the command line, we
get the following numbers before this change:
____Elapsed time: 11.170s, Critical Path: 8.34s
____Elapsed time: 11.027s, Critical Path: 8.20s
____Elapsed time: 11.084s, Critical Path: 7.46s
____Elapsed time: 11.051s, Critical Path: 6.61s
____Elapsed time: 11.211s, Critical Path: 7.81s
____Elapsed time: 10.884s, Critical Path: 8.20s
____Elapsed time: 11.385s, Critical Path: 8.12s
____Elapsed time: 11.723s, Critical Path: 8.18s
____Elapsed time: 11.327s, Critical Path: 7.73s
____Elapsed time: 11.028s, Critical Path: 7.89s
And after this change:
____Elapsed time: 4.294s, Critical Path: 0.27s
____Elapsed time: 4.376s, Critical Path: 0.83s
____Elapsed time: 8.083s, Critical Path: 0.52s
____Elapsed time: 4.302s, Critical Path: 0.64s
____Elapsed time: 4.282s, Critical Path: 0.37s
____Elapsed time: 4.219s, Critical Path: 0.61s
____Elapsed time: 4.214s, Critical Path: 0.97s
____Elapsed time: 4.185s, Critical Path: 0.71s
____Elapsed time: 7.962s, Critical Path: 4.30s
____Elapsed time: 4.149s, Critical Path: 1.03s
--
PiperOrigin-RevId: 150351444
MOS_MIGRATED_REVID=150351444
|
|
|
|
|
|
|
|
|
|
| |
For tests, also provide information on the time the test took.
--
Change-Id: I8e71391e4dd97627d6293159a0cbb0d922683af7
Reviewed-on: https://cr.bazel.build/9214
PiperOrigin-RevId: 149547633
MOS_MIGRATED_REVID=149547633
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
All spawn strategies already treat all normal outputs as optional. Bazel checks
at the action level whether all action outputs are created, but does not check
at the spawn level. Spawn.getOptionalOutputs is therefore unnecessary, and
removed in this change.
The only place where this was set was in StandaloneTestStrategy, which now
specifies the full set of outputs, which is now computed by TestRunnerAction.
The internal test strategy implementations are also updated in this change.
While I'm at it, also remove the use of BaseSpawn and use SimpleSpawn instead.
This may go some way towards fixing #1413 and #942.
--
PiperOrigin-RevId: 149397100
MOS_MIGRATED_REVID=149397100
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This changes command-line computation for tests:
- run the coverage collector before the run_under command
- no shell escaping: if all tools just call "$@", then this should work
Note that we still wrap the command in a sub-shell to support shell built-ins
and PATH lookup if the command does not contain a slash character '/'.
A side effect of this change is that the --run_under command now executes in
the test's runfiles directory, rather than in the exec root, if coverage is
enabled at the same time. Inside Google, it's very rare for --run_under to be
used in combination with coverage, and it seems likely to be rare externally
as well, so I don't think it warrants covering it in the release notes.
Also set TEST_BINARY to the root-relative path of the test executable for
all tests (in TestRunnerAction.java).
--
PiperOrigin-RevId: 149275688
MOS_MIGRATED_REVID=149275688
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 149110466
MOS_MIGRATED_REVID=149110466
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Move all local resource acquisition to where local execution actually happens.
Don't attempt to acquire resources per action, but only for individual spawns.
This significantly simplifies the code.
The downside is that we don't account for action-level work anymore. In
general, actions should not perform any process execution themselves, but
always delegate such work to a SpawnStrategy implementation.
This change makes sure that every Spawn has local resources set in a way that
is consistent with the previous state.
However, there are two actions - Fileset and FileWrite -, which are not spawns,
and so we now don't limit their concurrent execution anymore. For Fileset, all
work is done in a custom Fileset-specific thread pool, so this shouldn't be a
problem. I'm not sure about FileWriteAction.
--
PiperOrigin-RevId: 149012600
MOS_MIGRATED_REVID=149012600
|
|
|
|
|
|
|
|
|
|
| |
TestStrategy sometimes computed the runfiles directory separately from
the existing code in Runfiles. Now we always use the passed through value.
AFAICT, the two code paths were identical.
--
PiperOrigin-RevId: 148866489
MOS_MIGRATED_REVID=148866489
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Breaks TEST_BINARY env setting if --run_under is used.
*** Original change description ***
Rewrite TestStrategy.getArgs
This changes command-line computation for tests:
- run the coverage collector before the run_under command
- no shell escaping: if all tools just call "$@", then this should work
Note that we still wrap the command in a sub-shell to support shell built-ins
and PATH lookup if the command does not contain a slash character '/'.
A side effect of this change is that the --run_under command now executes in
the test's runfiles directory, rather than in the exec root, if...
***
--
PiperOrigin-RevId: 148003525
MOS_MIGRATED_REVID=148003525
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Broke tests on CI: http://ci.bazel.io/job/bazel-tests/570/
*** Original change description ***
Roll forward execroot change
RELNOTES[INC]: Previously, an external repository would be symlinked into the
execution root at execroot/local_repo/external/remote_repo. This changes it to
be at execroot/remote_repo. This may break genrules/Skylark actions that
hardcode execution root paths. If this causes breakages for you, ensure that
genrules are using $(location :target) to access files and Skylark rules are
using http://bazel.io/docs/skylark/lib/File.html's path, dirname, etc.
functions. Cust...
--
PiperOrigin-RevId: 147833177
MOS_MIGRATED_REVID=147833177
|