| Commit message (Collapse) | Author | Age |
|
|
|
|
|
|
|
|
|
|
| |
Using /dev/null was causing it to create symlinks to /dev/null, which
breaks Python programs. I didn't catch this earlier, because my machine had
an old linux kernel installed that didn't support sandboxing. There's an
existing integration test, which just broke in our CI.
--
PiperOrigin-RevId: 150750032
MOS_MIGRATED_REVID=150750032
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This new class is a combination of SpawnHelper and our internal code; the
plan is to migrate all spawn strategies to the new class. The strict flag
should be enabled by default, but that's a breaking change, so we need to do
it later.
- Use it in SandboxStrategy.
- Add ActionInput.getExecPath to return a PathFragment; this avoids lots of
back and forth between path fragments and strings.
This is a step towards #1593.
The previous attempt was missing a one-line patch in StandaloneTestStrategy,
which broke all tests with sandboxing. StandaloneTestStrategy was fixed in a
separate change, so this should be safe now.
--
PiperOrigin-RevId: 150733457
MOS_MIGRATED_REVID=150733457
|
|
|
|
|
|
|
|
|
|
| |
Add preconditions to enforce this and remove some now unnecessary code.
A small step towards #1593.
--
PiperOrigin-RevId: 150625693
MOS_MIGRATED_REVID=150625693
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Break bazel-tests and many other jobs on CI.
http://ci.bazel.io/job/bazel-tests/BAZEL_VERSION=HEAD,PLATFORM_NAME=linux-x86_64/651/console
*** Original change description ***
Add SpawnInputExpander helper class to arrange runfiles for spawn strategies
This new class is a combination of SpawnHelper and our internal code; the
plan is to migrate all spawn strategies to the new class. The strict flag
should be enabled by default, but that's a breaking change, so we need to do
it later.
- Use it in SandboxStrategy.
- Add ActionInput.getExecPath to return a PathFragment; this avoids lots of
back and forth between path fragments and strings.
This is a step towards #159...
***
--
PiperOrigin-RevId: 150610616
MOS_MIGRATED_REVID=150610616
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This new class is a combination of SpawnHelper and our internal code; the
plan is to migrate all spawn strategies to the new class. The strict flag
should be enabled by default, but that's a breaking change, so we need to do
it later.
- Use it in SandboxStrategy.
- Add ActionInput.getExecPath to return a PathFragment; this avoids lots of
back and forth between path fragments and strings.
This is a step towards #1593.
--
PiperOrigin-RevId: 150427021
MOS_MIGRATED_REVID=150427021
|
|
|
|
|
|
|
|
|
| |
- use SimpleSpawn in SpawnGccStrategy
- set PWD in CppCompileAction for consistency
--
PiperOrigin-RevId: 149745059
MOS_MIGRATED_REVID=149745059
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
All spawn strategies already treat all normal outputs as optional. Bazel checks
at the action level whether all action outputs are created, but does not check
at the spawn level. Spawn.getOptionalOutputs is therefore unnecessary, and
removed in this change.
The only place where this was set was in StandaloneTestStrategy, which now
specifies the full set of outputs, which is now computed by TestRunnerAction.
The internal test strategy implementations are also updated in this change.
While I'm at it, also remove the use of BaseSpawn and use SimpleSpawn instead.
This may go some way towards fixing #1413 and #942.
--
PiperOrigin-RevId: 149397100
MOS_MIGRATED_REVID=149397100
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 149110466
MOS_MIGRATED_REVID=149110466
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Move all local resource acquisition to where local execution actually happens.
Don't attempt to acquire resources per action, but only for individual spawns.
This significantly simplifies the code.
The downside is that we don't account for action-level work anymore. In
general, actions should not perform any process execution themselves, but
always delegate such work to a SpawnStrategy implementation.
This change makes sure that every Spawn has local resources set in a way that
is consistent with the previous state.
However, there are two actions - Fileset and FileWrite -, which are not spawns,
and so we now don't limit their concurrent execution anymore. For Fileset, all
work is done in a custom Fileset-specific thread pool, so this shouldn't be a
problem. I'm not sure about FileWriteAction.
--
PiperOrigin-RevId: 149012600
MOS_MIGRATED_REVID=149012600
|
|
|
|
|
|
|
|
| |
Fixes #2597
--
PiperOrigin-RevId: 148999160
MOS_MIGRATED_REVID=148999160
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 148749485
MOS_MIGRATED_REVID=148749485
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds a test based on our Python rules that makes sure that this actually
fixes the issue. Thanks to @duggelz for the suggestion. It seems like
our Python rules are the only place that actually provides an
EmptyFilesSupplier to Runfiles, so there's probably no simpler way to
test this behavior in an integration test.
Fix #1458. Fix #2394.
--
PiperOrigin-RevId: 148656193
MOS_MIGRATED_REVID=148656193
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Broke tests on CI: http://ci.bazel.io/job/bazel-tests/570/
*** Original change description ***
Roll forward execroot change
RELNOTES[INC]: Previously, an external repository would be symlinked into the
execution root at execroot/local_repo/external/remote_repo. This changes it to
be at execroot/remote_repo. This may break genrules/Skylark actions that
hardcode execution root paths. If this causes breakages for you, ensure that
genrules are using $(location :target) to access files and Skylark rules are
using http://bazel.io/docs/skylark/lib/File.html's path, dirname, etc.
functions. Cust...
--
PiperOrigin-RevId: 147833177
MOS_MIGRATED_REVID=147833177
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
RELNOTES[INC]: Previously, an external repository would be symlinked into the
execution root at execroot/local_repo/external/remote_repo. This changes it to
be at execroot/remote_repo. This may break genrules/Skylark actions that
hardcode execution root paths. If this causes breakages for you, ensure that
genrules are using $(location :target) to access files and Skylark rules are
using http://bazel.io/docs/skylark/lib/File.html's path, dirname, etc.
functions. Custom crosstools that hardcode external/<repo> paths will have to
be updated.
Issue #1262.
--
PiperOrigin-RevId: 147726370
MOS_MIGRATED_REVID=147726370
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The behavior of the Linux sandbox was changed to not hide the local hostname by default.
It is now only hidden when the --sandbox_fake_hostname flag is specified.
Also, instead of using the hostname "sandbox" in this case, it now uses "localhost", which fixes the issue of sandboxed processes not being able to resolve their local hostname.
RELNOTES: For increased compatibility with environments where UTS namespaces are not available, the Linux sandbox no longer hides the hostname of the local machine by default. Use --sandbox_fake_hostname to re-enable this feature.
--
PiperOrigin-RevId: 146244268
MOS_MIGRATED_REVID=146244268
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Drop the Spawn.getOwner() method, which was duplicating functionality
already available through Spawn.getResourceOwner(), in favor of the
latter. In order to do that, change Executor.reportSubcommand to take
a Spawn instance instead, which in turn requires updating all call
sites.
This is part of establishing Spawn as an abstraction for lower-level
local and remote execution in all cases instead of passing args, env,
input/output files and possibly other metadata as individual method
parameters to the underlying implementations.
In order for that to be maintainable, Spawn should be light-weight and
easy to construct correctly, and not contain too much unnecessary data.
--
PiperOrigin-RevId: 146224914
MOS_MIGRATED_REVID=146224914
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Remove getSpawnInfo and asShellCommand, which are never really
overridden in a useful way. asShellCommand moves to the Spawns class,
and getSpawnInfo is only ever called by SpawnAction, and the
implementation moves there.
I'm considering using Spawn as the general lower-level abstraction for both
local and remote execution. It sort of is that already, except it's not
used consistently - we often pass a tuple of (args, env) plus possibly
input and output files through parameter-heavy method call hierarchies
instead of using this existing abstraction.
However, I'm concerned about the amount of baggage it's carrying as well
as the number of implementations for what is supposed to be a simple
interface (or possibly even a simple value class), and this is an attempt
to slim it down a bit.
This should have no visible effects on builds.
--
PiperOrigin-RevId: 146109838
MOS_MIGRATED_REVID=146109838
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
ActionSpawn/SpawnAction now deal exclusively in RunfilesSuppliers, manifests
maps are no more.
There is some lingering awkwardness, in particular:
- Manifests still need to be tracked in some places, we can work out if this is
still necessary on a case by case basis.
- Skylark makes actions' runfiles available via 'resolve_command' where they are
consumed by 'action'. I've updated the documentation, though the name isn't
entirely accurate anymore. That being said these interfaces _are_ marked as
experimental, so we _should_ be able to be flexible here.
Overall, I think the benefits consolidating runfiles into suppliers, from both
code cleanliness and performance perspectives (no longer needing to parse
manifests), outweights the awkwardnesses.
RELNOTES: resolve_command/action's input_manifest return/parameter is now list
--
PiperOrigin-RevId: 145817429
MOS_MIGRATED_REVID=145817429
|
|
|
|
|
|
|
|
| |
Fixes #2372.
--
PiperOrigin-RevId: 145539067
MOS_MIGRATED_REVID=145539067
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
1. no flag: only the direct reason is given (command execution termination status), and also the instruction to use "--verbose_failures"
2. flag "--verbose_failures": gives failed execution command and the instruction to use "--sandbox_debug --strategy"
3. flag "--verbose_failures --sandbox_debug": gives failed execution command, debugging message from sandboxing, and the instruction to use "--strategy"
Also removes "cd <sandbox_path>" in given failed command, since debugging is only necessary with flag "--verbose_failures --sandbox_debug" and the path is already given in sandboxing debugging message.
Addresses #2174.
Fixes bazel_sandboxing_test.sh and bazel_test_test.sh for the new error message.
Fixes timeout.
--
PiperOrigin-RevId: 144285435
MOS_MIGRATED_REVID=144285435
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Still breaks //src/test/shell/bazel:bazel_test_test
http://ci.bazel.io/job/bazel-tests/BAZEL_VERSION=HEAD,PLATFORM_NAME=linux-x86_64/lastCompletedBuild/testReport/(root)/(empty)/test_test_timeout/
--
PiperOrigin-RevId: 144107285
MOS_MIGRATED_REVID=144107285
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
1. no flag: only the direct reason is given (command execution termination status), and also the instruction to use "--verbose_failures"
2. flag "--verbose_failures": gives failed execution command and the instruction to use "--sandbox_debug --strategy"
3. flag "--verbose_failures --sandbox_debug": gives failed execution command, debugging message from sandboxing, and the instruction to use "--strategy"
Also removes "cd <sandbox_path>" in given failed command, since debugging is only necessary with flag "--verbose_failures --sandbox_debug" and the path is already given in sandboxing debugging message.
Addresses #2174.
Fixes bazel_sandboxing_test.sh and bazel_test_test.sh for the new error message.
--
PiperOrigin-RevId: 144058351
MOS_MIGRATED_REVID=144058351
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Breaks //src/test/shell/bazel:bazel_sandboxing_test:
http://ci.bazel.io/job/bazel-tests/lastCompletedBuild/BAZEL_VERSION=latest-jdk7,PLATFORM_NAME=linux-x86_64/testReport/(root)/(empty)/test_failing_action_with_ioexception_while_copying_outputs_throws_correct_exception/
*** Original change description ***
Gives 3 levels of sandbox error message under different flags.
1. no flag: only the direct reason is given (command execution termination status), and also the instruction to use "--verbose_failures"
2. flag "--verbose_failures": gives failed execution command and the instruction to use "--sandbox_debug --strategy"
3. flag "--verbose_failures --sandbox_debug": gives failed execution command, debugging message from sandboxing, and the instruction to use "--strategy"
Also removes "cd <sandbox_...
***
ROLLBACK_OF=143937589
--
PiperOrigin-RevId: 143951901
MOS_MIGRATED_REVID=143951901
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
1. no flag: only the direct reason is given (command execution termination status), and also the instruction to use "--verbose_failures"
2. flag "--verbose_failures": gives failed execution command and the instruction to use "--sandbox_debug --strategy"
3. flag "--verbose_failures --sandbox_debug": gives failed execution command, debugging message from sandboxing, and the instruction to use "--strategy"
Also removes "cd <sandbox_path>" in given failed command, since debugging is only necessary with flag "--verbose_failures --sandbox_debug" and the path is already given in sandboxing debugging message.
Addresses #2174.
--
PiperOrigin-RevId: 143937589
MOS_MIGRATED_REVID=143937589
|
|
|
|
|
|
|
|
|
|
|
|
| |
This change is part of the mu-bazel effort, which aims to build a minimally
useful Bazel binary with most extraneous functionality removed. As part of
that, we want to enforce layering of packages. In particular, lib.actions must
not depend on lib.rules or lib.exec. lib.rules must not depend on lib.exec.
Moving these classes is a necessary step to enforce that layering.
--
PiperOrigin-RevId: 142668172
MOS_MIGRATED_REVID=142668172
|
|
|
|
|
|
|
|
|
|
| |
RELNOTES: New flag --sandbox_add_mount_pair to specify customized source:target path pairs to bind mount inside the sandbox.
--
Change-Id: Ifbacfc0e16bbaedcf5b6d3937799710f2cfa3d58
Reviewed-on: https://cr.bazel.build/7150
PiperOrigin-RevId: 142542381
MOS_MIGRATED_REVID=142542381
|
|
|
|
|
|
|
|
| |
Fixes #2231.
--
PiperOrigin-RevId: 142266332
MOS_MIGRATED_REVID=142266332
|
|
|
|
|
|
|
|
|
|
| |
Instead they mount the individual files. Currently, non-empty tree artifacts result in ERR_DIRECTORY_NOT_EMPTY because we attempt to mount both the directory and the files inside it.
Fixes https://github.com/bazelbuild/bazel/issues/1745.
--
PiperOrigin-RevId: 141599030
MOS_MIGRATED_REVID=141599030
|
|
|
|
|
|
|
|
| |
Fixes https://github.com/bazelbuild/bazel/issues/1928.
--
PiperOrigin-RevId: 141077737
MOS_MIGRATED_REVID=141077737
|
|
|
|
|
|
|
| |
This significantly simplifies several of our modules.
--
MOS_MIGRATED_REVID=137713119
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=137500893
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=137370890
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=137012950
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=136818366
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Roll-forward with fix.
Tested with `bazel build src:srcs //src/test/...` using a bootstrapped bazel.
Fixes #1923.
RELNOTES[INC]: Non test-only targets can no longer depends on test-only targets.
*** Original change description ***
Automated [] rollback of commit a9f20b0d6459d395444c45cf5e94a899f3443633.
*** Reason for rollback ***
Broke Bazel CI:
ERROR: /home/ci/workspace/Bazel/JAVA_VERSION/1.8/PLATFORM_NAME/linux-x86_64/src/BUILD:284:1: in filegroup rule //src:srcs: non-test target '//src:srcs' depends on testonly target '//src/java_tools/junitrunner/java/com/google/testing/junit/junit4:srcs' and doesn't have testonly attribute set.
ERROR: Analysis of target '//scripts/packages:packages' failed; build aborted.
http://ci.bazel.io/view/Bazel%20b...
***
--
MOS_MIGRATED_REVID=136589937
|
|
|
|
|
|
|
| |
RELNOTES: Darwin sandboxing is default.
--
MOS_MIGRATED_REVID=136013826
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=136011723
|
|
|
|
|
|
|
|
|
| |
symlinks.
This bug caused rare NotASymlinkException exceptions when using sandboxed workers.
--
MOS_MIGRATED_REVID=136010526
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Make darwin sandboxing default again.
*** Original change description ***
Fix #1849: Sandboxing on OS X should be turned off by default for 0.3.2.
This restructures the way we set the default Spawn strategy so that each BlazeModule supplying a SpawnActionContext has an ActionContextConsumer that sets its own SpawnActionContext as the default, with the BazelRulesModule being put as the last module loaded in BazelMain, so that it can override that decision - it only does, if the user explicitly specifies a --spawn_strategy flag. IMHO this is a much saner approach than...
***
ROLLBACK_OF=134770427
RELNOTES: Darwin sandboxing is default.
--
MOS_MIGRATED_REVID=135905657
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
"block-network" tag.
To block network access, you can set the "block-network" tag on a target like this:
genrule(
name = "no_access_to_network",
cmd = "curl http://www.bazel.io/this_will_fail",
tags = [ "block-network" ],
)
This is needed to fix a performance issue due to a bug in the Linux kernel: https://lkml.org/lkml/2014/8/28/656
RELNOTES[INC]: Sandboxed actions can access the network by default, unless their target has a "block-network" tag.
--
MOS_MIGRATED_REVID=135470811
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=135360041
|
|
|
|
|
|
|
|
|
|
|
|
| |
This restructures the way we set the default Spawn strategy so that each BlazeModule supplying a SpawnActionContext has an ActionContextConsumer that sets its own SpawnActionContext as the default, with the BazelRulesModule being put as the last module loaded in BazelMain, so that it can override that decision - it only does, if the user explicitly specifies a --spawn_strategy flag. IMHO this is a much saner approach than the older one.
So the flow is essentially this:
- StandaloneActionContextConsumer sets the default strategy to "standalone".
- SandboxActionContextConsumer sets the default strategy to "sandboxed", but only on Linux
- BazelRulesModule sets the default strategy to the value of the --spawn_strategy flag, if it is set.
--
MOS_MIGRATED_REVID=134770427
|
|
|
|
|
|
|
|
|
| |
simple synchronous cleanup.
Tested with bazel building itself that this does not result in a performance degradation.
--
MOS_MIGRATED_REVID=134766597
|
|
|
|
|
|
|
|
|
| |
empty, writable directory at a specified path when running actions. (Supported on Linux only for now.)
RELNOTES: Added a new flag --sandbox_tmpfs_path, which asks the sandbox to mount an empty, writable directory at a specified path when running actions. (Supported on Linux only for now.)
--
MOS_MIGRATED_REVID=134526345
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=134523222
|
|
|
|
|
|
|
| |
sandbox directories could not be cleaned up.
--
MOS_MIGRATED_REVID=134286101
|
|
|
|
|
|
|
| |
SandboxExecRoot#copyOutputs could hide an earlier ExecException from SandboxRunner#run in the Darwin and Linux sandbox strategies.
--
MOS_MIGRATED_REVID=134273806
|
|
|
|
|
|
|
| |
sandboxed execution strategies.
--
MOS_MIGRATED_REVID=134054610
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Breaks TensorFlow and other Bazel jobs on ci.bazel.io
*** Original change description ***
Change execution root for external repositories to be ../repo
Some of the important aspect of this change:
* Remote repos in the execution root are under output_base/execroot/repo_name, so the prefix is ../repo_name (to escape the local workspace name).
* Package roots for external repos were previously "output_base/", they are now output_base/external/repo_name (which means source artifacts always have a relative path from their repository).
* Outputs are under bazel-bin/external/repo_name/ (or similarly under genfiles). Note that this is a bit of a change from how this was implemented in the previous cl.
Fixes #1262.
RELNOTES[INC]: Previously, an external repository would be symlinked into the
execution root at execroot/local_repo/external/remote_repo. This changes it to
be at execroot/remote_repo. This may break genrules/Skylark actions that
hardcode execution root paths. If this causes breakages for you, ensure that
genrules are using $(location :target) to access files and Skylark rules are
using http://bazel.io/docs/skylark/lib/File.html's path, dirname, etc.
functions.
Roll forward of bdfd58a.
--
MOS_MIGRATED_REVID=133709658
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=133697962
|