| Commit message (Collapse) | Author | Age |
|
|
|
|
|
|
| |
Part of #2219.
Change-Id: I339009c13639144ca756eb07c520df7d430a64e3
PiperOrigin-RevId: 161826487
|
|
|
|
|
|
|
| |
Part of #2219.
Change-Id: I7293fd13bd8e0931f92afd051e18a9e7ce63762d
PiperOrigin-RevId: 161721445
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is a legacy dependency on the configuration transition table, which is
only needed for static configurations. Dynamic configurations didn't actually
use anything in that table: this was just a convenience interface that could
have equally been defined somewhere else. So this cl defines it somewhere else.
There's still one last dependency: Transitions.configurationHook. We'll tackle
that in a followup cl.
PiperOrigin-RevId: 161141650
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Before this change, top-level configs were still built statically,
then cloned into dynamic configs to run the actual build.
After this change, no static configuration should ever be created in
a dynamically configured build.
We're still keeping support for --experimental_dynamic_configs=off a
bit longer to provide easy reversion on any bad surprises. But barring
that we'll quickly move toward making --experimental_dynamic_configs=off
a no-op, then ripping out Bazel's static configuration logic.
PiperOrigin-RevId: 161005150
|
|
|
|
|
| |
Change-Id: I008c16d747c5d06f433af14cd0793536cff2723e
PiperOrigin-RevId: 158975286
|
|
|
|
|
|
|
| |
configurable. Add a test for the current behavior of treating an unreadable BUILD file as a package loading error.
RELNOTES: None
PiperOrigin-RevId: 158314187
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(Automated g4 rollback of commit de92f9d8ea093416fae999073bbfcf3cf501ab55).
*** Reason for rollback ***
The problems that forced commit de92f9d8ea093416fae999073bbfcf3cf501ab55 were fixed in commit e6392cd380fce14d719890c78d5eb2657e8a6cfc .
*** Original change description being rolled forward ***
Implement dynamically configured LIPO builds.
Quick overview:
- provide a dynamic interface for getting the artifact owner
configuration
- provide a (dynamic) RuleTransitionFactory LIPO_ON_DEMAND to replace
the (static) RuleClass.Configurator LIPO_ON_DEMAND. Eventually
we'll remove the rule class configurator interface entirely....
***
ROLLBACK_OF=156180015
PiperOrigin-RevId: 157865224
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Causes crash bug in certain circumstances.
*** Original change description ***
Implement dynamically configured LIPO builds.
Quick overview:
- provide a dynamic interface for getting the artifact owner
configuration
- provide a (dynamic) RuleTransitionFactory LIPO_ON_DEMAND to replace
the (static) RuleClass.Configurator LIPO_ON_DEMAND. Eventually
we'll remove the rule class configurator interface entirely.
This doesn't actually turn dynamic LIPO on. So the direct effect of
this change should be a no-op. The flip will come in a followup
change. For now, dynamic...
PiperOrigin-RevId: 156180015
|
|
|
|
|
|
|
|
| |
This is useful for dealing with all the existing implementations in the face of
interface changes that are irrelevant.
RELNOTES: None
PiperOrigin-RevId: 155525021
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Quick overview:
- provide a dynamic interface for getting the artifact owner
configuration
- provide a (dynamic) RuleTransitionFactory LIPO_ON_DEMAND to replace
the (static) RuleClass.Configurator LIPO_ON_DEMAND. Eventually
we'll remove the rule class configurator interface entirely.
This doesn't actually turn dynamic LIPO on. So the direct effect of
this change should be a no-op. The flip will come in a followup
change. For now, dynamic LIPO can be triggered with
--experimental_dynamic_configs=notrim.
PiperOrigin-RevId: 155096056
|
|
|
|
|
|
| |
wrapper objects: for OwnedArtifacts, which are the most numerous during builds, and for Labels for TransitiveTraversalValues, which are the most numerous during queries.
PiperOrigin-RevId: 154989520
|
|
|
|
|
|
|
|
|
| |
This is the first of two CLs for making command line options able to affect the Skylark interpreter. It introduces SkylarkSemanticsOptions, and stores it as a precomputed (injected) value in Skyframe. The next CL will read these options from Skyframe when constructing the Skylark environment.
This CL affects the dataflow from command/test initialization to Skyframe. Some code paths, like those used for testing, use the default SkylarkSemanticsOptions and therefore won't be able to use (for example) --incompatible_* flags. The call sites to update were found by searching for uses of defaultVisibility and working upward from there.
RELNOTES: None
PiperOrigin-RevId: 154432058
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
The go rules have tons of transitive dependencies that are not declared in their "local" WORKSPACE files, so this broke lots of projects on ci.bazel.build
I'll fix up the go rules, update all of _their_ reverse-dep projects, and resubmit.
*** Original change description ***
Repositories can only be accessed in projects that define them in their WORKSPACE file
This is prep for #1943 - hierarchical workspace loading.
RELNOTES[INC]: Remote repositories must define any remote repositories they
themselves use (e.g., if @x//:foo depends on @y//:bar, @y must be defined
in @x's WORKSPACE file).
PiperOrigin-RevId: 154321845
|
|
|
|
|
|
|
|
|
|
|
|
| |
WORKSPACE file
This is prep for #1943 - hierarchical workspace loading.
RELNOTES[INC]: Remote repositories must define any remote repositories they
themselves use (e.g., if @x//:foo depends on @y//:bar, @y must be defined
in @x's WORKSPACE file).
PiperOrigin-RevId: 154295762
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We need to lookup repositories as part of converting exec paths to artifacts,
which in turn is needed for action cache lookups. These lookups should not
cause a Skyframe exit, so we must not throw an exception here, unless the
error makes it impossible to continue. Instead, we need to leave the decision
whether to error out or not to the caller.
Note that we may unnecessarily fetch a remote repository in order to do the
action cache lookup, even if the action no longer depends on the input file,
although this should only be possible for C++ compile actions. It's possible
that there's another bug in the C++ compile action key computation that also
contributes.
This change also makes it so that the post-resolution action cache code
ignores any errors wrt. repository lookup rather than throwing. If any of the
paths could not be found, then the action cache lookup fails and we re-execute
the corresponding action, which is exactly what should happen.
Fixes #2759.
PiperOrigin-RevId: 153696243
|
|
|
|
|
|
|
|
|
|
|
| |
RELNOTES: Adds a --override_repository option that takes a repository
name and path. This forces Bazel to use the directory at that path
for the repository. Example usage:
`--override_repository=foo=/home/user/gitroot/foo`.
Fixes #1266
PiperOrigin-RevId: 153599291
|
|
|
|
| |
PiperOrigin-RevId: 153473961
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 152882585
|
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 152804937
|
|
|
|
|
|
|
|
| |
keep references to actions indefinitely. Instead, once an action is finished executing, keep just some metadata about it. This allows actions to be unconditionally dropped when running with --batch, --discard_analysis_cache, and --keep_going, even if profiling is enabled.
The additional fields here add between 8 and 12 bytes per component, and we have one component per action. This additional penalty is only incurred when we are already saving memory, so I think it's ok. The full penalty will be realized only towards the end of the build, when every action has started executing at least once. Users can still specify --noexperimental_enable_critical_path_profiling if they want to squeeze even more memory out.
PiperOrigin-RevId: 152328870
|
|
|
|
|
|
|
|
|
| |
This is useful for IDEs and other tools utilizing command-line aspects
for reflection over build graph.
RELNOTES: None.
PiperOrigin-RevId: 152038248
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
--noexperimental_enable_critical_path_profiling flags are all specified, then Bazel will delete Actions from ActionLookupValues as they are executed in order to save memory.
Because an already-run action may output an artifact that is only requested later in the build, we need to maintain a way for the artifact to look up the action. But in most cases we don't need to keep the action itself, just its output metadata.
Some actions unfortunately are needed post-execution, and so we special-case them.
Also includes dependency change with description:
Move action out of key. This keeps action references from polluting the graph -- actions are just stored in one SkyValue, instead of being present in SkyKeys.
This does mean additional memory used: we have a separate ActionLookupData object per Action executed. That may reach ~24M for million-action builds.
PiperOrigin-RevId: 151756383
|
|
|
|
|
|
| |
doesn't work because the PackageFactory keeps a reference to it.
PiperOrigin-RevId: 151632473
|
|
|
|
|
|
|
|
| |
ConfiguredTargetValues. Also clear transitive packages for both, even for top-level targets.
This is not expected to save significant memory, but is expected to reduce the number of references to Packages, allowing them to be dropped more easily when discarding analysis cache and running in batch mode.
PiperOrigin-RevId: 151508877
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 151129669
MOS_MIGRATED_REVID=151129669
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Move all local resource acquisition to where local execution actually happens.
Don't attempt to acquire resources per action, but only for individual spawns.
This significantly simplifies the code.
The downside is that we don't account for action-level work anymore. In
general, actions should not perform any process execution themselves, but
always delegate such work to a SpawnStrategy implementation.
This change makes sure that every Spawn has local resources set in a way that
is consistent with the previous state.
However, there are two actions - Fileset and FileWrite -, which are not spawns,
and so we now don't limit their concurrent execution anymore. For Fileset, all
work is done in a custom Fileset-specific thread pool, so this shouldn't be a
problem. I'm not sure about FileWriteAction.
--
PiperOrigin-RevId: 149012600
MOS_MIGRATED_REVID=149012600
|
|
|
|
|
|
|
|
| |
...now that the ExtendedEventHandler can serve the same purpose.
--
PiperOrigin-RevId: 148640618
MOS_MIGRATED_REVID=148640618
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
With more specific information to be reported by Skyfunctions, e.g.,
to inform the build-event protocol on missing files, the EventHandler
interface is no longer enough. Therefore, provide an enriched context
for reporting events.
--
Change-Id: I2d06166fe4d5b9054e24ad8c752fafc039e3f9f8
Reviewed-on: https://cr.bazel.build/8794
PiperOrigin-RevId: 148463437
MOS_MIGRATED_REVID=148463437
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This combines both previous changes and extends them to work both with and
without kchodorow@'s rollout of the exec root rearrangement. Unfortunately,
each of these changes individually breaks something somewhere, so they must
all go into a single commit.
Change 1:
CppCompileAction must return false from inputsKnown for .d pruning
This is necessary (but not sufficient) for the action cache to work
correctly. Consider the following sequence of events:
1. action is executed
2. .d pruning is performed
3. action cache writes entry with post-pruning inputs list
4. action gets regenerated (e.g., due to server restart)
5. the action cache calls inputsKnown(), which returns true
6. action cache checks entry from step 3 against pre-pruning inputs list,
which results in a cache miss
The action cache needs to know that the current list is not the final list,
so inputsKnown() in step 5 must return false if .d pruning is enabled.
Change 2:
Fix artifact root discovery for external artifacts
The SkyframeExecutor was assuming that all exec paths were coming from the
main repository. Instead, rely on external exec paths to start with "../".
Additional change 3:
In addition, update the PackageRootResolverWithEnvironment and the
HeaderDiscovery to use the single unified repository name guessing
implementation. Previously, the PackageRootResolverWithEnvironment was
poisoning the source artifact cache, which then caused subsequent lookups
to return a bad artifact.
Add a precondition to double-check that artifacts looked up by exec path
have the correct source root.
For compatibility with kchodorow@'s upcoming exec root refactor, if the exec
path starts with "external", then assume it's coming from an external
repository. This must be removed when that change is successfully rolled out,
or it will break if anyone creates a package called 'external'.
Additional change 4:
On top of all of that, PackageRootResolverWithEnvironment and SkyframeExecutor
must use the same source root computation as the Package class itself. I
extracted the corresponding code to Root, and added comments both there and
in Package to clearly indicate that these methods have to always be modified
in sync.
Fixes #2490.
--
PiperOrigin-RevId: 148439309
MOS_MIGRATED_REVID=148439309
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
*** Reason for rollback ***
Broke tests on CI: http://ci.bazel.io/job/bazel-tests/570/
*** Original change description ***
Roll forward execroot change
RELNOTES[INC]: Previously, an external repository would be symlinked into the
execution root at execroot/local_repo/external/remote_repo. This changes it to
be at execroot/remote_repo. This may break genrules/Skylark actions that
hardcode execution root paths. If this causes breakages for you, ensure that
genrules are using $(location :target) to access files and Skylark rules are
using http://bazel.io/docs/skylark/lib/File.html's path, dirname, etc.
functions. Cust...
--
PiperOrigin-RevId: 147833177
MOS_MIGRATED_REVID=147833177
|
|
|
|
|
|
|
|
| |
magic artifact.
--
PiperOrigin-RevId: 147830857
MOS_MIGRATED_REVID=147830857
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
RELNOTES[INC]: Previously, an external repository would be symlinked into the
execution root at execroot/local_repo/external/remote_repo. This changes it to
be at execroot/remote_repo. This may break genrules/Skylark actions that
hardcode execution root paths. If this causes breakages for you, ensure that
genrules are using $(location :target) to access files and Skylark rules are
using http://bazel.io/docs/skylark/lib/File.html's path, dirname, etc.
functions. Custom crosstools that hardcode external/<repo> paths will have to
be updated.
Issue #1262.
--
PiperOrigin-RevId: 147726370
MOS_MIGRATED_REVID=147726370
|
|
|
|
|
|
|
|
| |
providers.
--
PiperOrigin-RevId: 147526961
MOS_MIGRATED_REVID=147526961
|
|
|
|
|
|
|
|
|
|
|
|
| |
(i) Log [the first 100] external [non external-repo] file paths when we encounter them in skyframe [when externalFileAction is not ASSUME_NON_EXISTENT_AND_IMMUTABLE_FOR_EXTERNAL_PATHS]. These caveats are to prevent log spam.
(ii) Fix grammar when logging the results of handleDiffsWithMissingDiffInformation in the case when all package paths have known diff information and we're merely checking external/output/external-repo files.
(iii) Log the purpose of each skyframe graph scan we do (see (ii)).
--
PiperOrigin-RevId: 147508404
MOS_MIGRATED_REVID=147508404
|
|
|
|
|
|
| |
--
PiperOrigin-RevId: 147018240
MOS_MIGRATED_REVID=147018240
|
|
|
|
|
|
|
|
|
|
|
|
| |
workspace name is needed for package loading, and so splitting out this computation into a separate skyframe node that can be change-pruned gives us better incrementality; previously we'd need to reload all packages on a WORKSPACE file change.
N.B.
(i) This CL doesn't solve all the other performance issues with //external in Bazel/Blaze since it's still inefficiently used for resolving labels like @foo//bar:baz.
(ii) This CL doesn't address the wasteful invalidation + change pruning of all the packages.
--
PiperOrigin-RevId: 146925369
MOS_MIGRATED_REVID=146925369
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
ActionEnvironmentFunction returns the list of environment
variable with the one overwritten by --action_env being
replaced. This let other Skyframe function declares
dependency to any value of the environment and being
influenced by the --action_env flag.
This will be used to declare dependency of remote repositories
on environment variables (step 3 of
https://bazel.build/designs/2016/10/18/repository-invalidation.html)
--
Change-Id: I1ed3fb6f48e8e17d4d64c903fccecb6ed7596350
Reviewed-on: https://cr.bazel.build/7974
PiperOrigin-RevId: 146918603
MOS_MIGRATED_REVID=146918603
|
|
|
|
|
|
|
|
|
|
| |
Two of the functions are never used, several of the parameter are not
used or can be simplified (require callers to convert from Target ->
Label).
--
PiperOrigin-RevId: 146787380
MOS_MIGRATED_REVID=146787380
|
|
|
|
|
|
|
|
|
|
|
|
| |
Using a precomputed values prevent us from being able to depends on any
environment variable (or we would invalidate the whole graph each time
the environment change, that is each time we run a command).
--
Change-Id: If322df4240271d04b9278e1222f936156531580c
Reviewed-on: https://cr.bazel.build/8400
PiperOrigin-RevId: 146298592
MOS_MIGRATED_REVID=146298592
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
If an Aspect registered an action that an extra-action is
shadowing, its name is used when creating the extra-action's ID and
name.
Since recently, an aspect can see other aspects applied to the same
target. This CL record the names of other aspects applied to the target
as well, disambiguating the action owners.
--
PiperOrigin-RevId: 142264153
MOS_MIGRATED_REVID=142264153
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Short story: Dependency -> BuildConfiguration maps can have multiple values
because of split transitions.
This is unfortunately one of those instances where the test logic is forked from
production. So this doesn't actually demonstrate bugs in production code. We
already have tests in ConfigurationsForTargetsTest that directly check the
production logic. So this cl's primary value is to unbreak tests that depend on
the forked logic.
--
PiperOrigin-RevId: 141094830
MOS_MIGRATED_REVID=141094830
|
|
|
|
|
|
|
|
|
|
|
| |
In particular, correctly report the expansion of a test
suite even if only requested to build (but not test).
--
Change-Id: Ia25305a4f5ed53118cbb1970a8055d156e8e50be
Reviewed-on: https://cr.bazel.build/7535
PiperOrigin-RevId: 141027655
MOS_MIGRATED_REVID=141027655
|
|
|
|
|
|
|
|
| |
Without this, many Bazel tests fail with --experimental_dynamic_configs=notrim_partial.
--
PiperOrigin-RevId: 140873563
MOS_MIGRATED_REVID=140873563
|
|
|
|
|
|
|
|
| |
during execution.
--
PiperOrigin-RevId: 140864295
MOS_MIGRATED_REVID=140864295
|
|
|
|
|
|
|
|
| |
This is in preparation for extracting some common reporting code, and also a
step towards interleaving target pattern eval and config creation.
--
MOS_MIGRATED_REVID=139890205
|
|
|
|
|
|
|
|
|
| |
avoid memory blow-up intra-configured-target analysis, use a semaphore to ensure that CPU-bound work only occurs on #CPU-many threads.
RELNOTES: Use --loading_phase_threads to control the number of threads used during the loading/analysis phase.
--
MOS_MIGRATED_REVID=139477645
|
|
|
|
|
|
|
|
|
| |
SkyQueryEnvironment#beforeEvaluateQuery if possible to save on latency for small queries.
This assumes that if the graph is up to date, then the data in SkyQueryEnvironment is also up to date. It also assumes that a ForkJoinPool remains usable until #shutdownNow is called.
--
MOS_MIGRATED_REVID=139386363
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=139357569
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
test infrastructure.
The small picture story is that SkyframeExecutor.getDynamicConfigOptions
(which gets dynamic BuildOptions for tests) wasn't updated with dynamic
split support when that was added to
ConfiguredTargetFunction.getDynamicTransitionOptions (which does the same
thing for production builds). This change plugs that gap. See
373e3e28274cca5b87f48abe33884edb84016dd3 for the original change.
The bigger picture story is that Bazel's configured target creation
logic is forked: test code goes down a similar but sadly not-quite-the-same
path that results in tons of duplicated logic, spaghetti code mess,
and risk of bugs like this one. We'd like to ultimately undo that fork.
But unfortunately it's an involved effort that won't happen overnight. In
the meantime, this change takes one small step by merging the two methods
that caused this bug.
--
MOS_MIGRATED_REVID=139342710
|
|
|
|
|
| |
--
MOS_MIGRATED_REVID=139209942
|