aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/skyframe/SkyframeExecutor.java
Commit message (Collapse)AuthorAge
* Add a SkyFunction to perform toolchain resolution.Gravatar John Cater2017-07-14
| | | | | | | Part of #2219. Change-Id: I339009c13639144ca756eb07c520df7d430a64e3 PiperOrigin-RevId: 161826487
* Add skyfunction to return all registered toolchain labels.Gravatar John Cater2017-07-13
| | | | | | | Part of #2219. Change-Id: I7293fd13bd8e0931f92afd051e18a9e7ce63762d PiperOrigin-RevId: 161721445
* Factor out BuildConfigurationCollection.Transitions.getDynamicTransition.Gravatar gregce2017-07-07
| | | | | | | | | | | | This is a legacy dependency on the configuration transition table, which is only needed for static configurations. Dynamic configurations didn't actually use anything in that table: this was just a convenience interface that could have equally been defined somewhere else. So this cl defines it somewhere else. There's still one last dependency: Transitions.configurationHook. We'll tackle that in a followup cl. PiperOrigin-RevId: 161141650
* Create top-level configs dynamically: period.Gravatar gregce2017-07-06
| | | | | | | | | | | | | | | Before this change, top-level configs were still built statically, then cloned into dynamic configs to run the actual build. After this change, no static configuration should ever be created in a dynamically configured build. We're still keeping support for --experimental_dynamic_configs=off a bit longer to provide easy reversion on any bad surprises. But barring that we'll quickly move toward making --experimental_dynamic_configs=off a no-op, then ripping out Bazel's static configuration logic. PiperOrigin-RevId: 161005150
* Remove unused interface PreprocessorGravatar Klaus Aehlig2017-06-16
| | | | | Change-Id: I008c16d747c5d06f433af14cd0793536cff2723e PiperOrigin-RevId: 158975286
* Make PackageFunction's strategy for handling unreadable BUILD files ↵Gravatar nharmata2017-06-08
| | | | | | | configurable. Add a test for the current behavior of treating an unreadable BUILD file as a package loading error. RELNOTES: None PiperOrigin-RevId: 158314187
* Automated g4 rollforward of commit b71e99b1f3746103e5d6802eebc24096b3494959.Gravatar gregce2017-06-05
| | | | | | | | | | | | | | | | | | | | | | | | (Automated g4 rollback of commit de92f9d8ea093416fae999073bbfcf3cf501ab55). *** Reason for rollback *** The problems that forced commit de92f9d8ea093416fae999073bbfcf3cf501ab55 were fixed in commit e6392cd380fce14d719890c78d5eb2657e8a6cfc . *** Original change description being rolled forward *** Implement dynamically configured LIPO builds. Quick overview: - provide a dynamic interface for getting the artifact owner configuration - provide a (dynamic) RuleTransitionFactory LIPO_ON_DEMAND to replace the (static) RuleClass.Configurator LIPO_ON_DEMAND. Eventually we'll remove the rule class configurator interface entirely.... *** ROLLBACK_OF=156180015 PiperOrigin-RevId: 157865224
* Automated g4 rollback of commit b71e99b1f3746103e5d6802eebc24096b3494959.Gravatar mstaib2017-05-17
| | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Causes crash bug in certain circumstances. *** Original change description *** Implement dynamically configured LIPO builds. Quick overview: - provide a dynamic interface for getting the artifact owner configuration - provide a (dynamic) RuleTransitionFactory LIPO_ON_DEMAND to replace the (static) RuleClass.Configurator LIPO_ON_DEMAND. Eventually we'll remove the rule class configurator interface entirely. This doesn't actually turn dynamic LIPO on. So the direct effect of this change should be a no-op. The flip will come in a followup change. For now, dynamic... PiperOrigin-RevId: 156180015
* Introduce an EvaluationProgressReceiver implementation that does nothing.Gravatar nharmata2017-05-10
| | | | | | | | This is useful for dealing with all the existing implementations in the face of interface changes that are irrelevant. RELNOTES: None PiperOrigin-RevId: 155525021
* Implement dynamically configured LIPO builds.Gravatar gregce2017-05-04
| | | | | | | | | | | | | | | Quick overview: - provide a dynamic interface for getting the artifact owner configuration - provide a (dynamic) RuleTransitionFactory LIPO_ON_DEMAND to replace the (static) RuleClass.Configurator LIPO_ON_DEMAND. Eventually we'll remove the rule class configurator interface entirely. This doesn't actually turn dynamic LIPO on. So the direct effect of this change should be a no-op. The flip will come in a followup change. For now, dynamic LIPO can be triggered with --experimental_dynamic_configs=notrim. PiperOrigin-RevId: 155096056
* Make SkyKey an interface, and start the migration of not creating SkyKey ↵Gravatar janakr2017-05-04
| | | | | | wrapper objects: for OwnedArtifacts, which are the most numerous during builds, and for Labels for TransitiveTraversalValues, which are the most numerous during queries. PiperOrigin-RevId: 154989520
* Store Skylark command-line flags in SkyframeGravatar brandjon2017-04-28
| | | | | | | | | This is the first of two CLs for making command line options able to affect the Skylark interpreter. It introduces SkylarkSemanticsOptions, and stores it as a precomputed (injected) value in Skyframe. The next CL will read these options from Skyframe when constructing the Skylark environment. This CL affects the dataflow from command/test initialization to Skyframe. Some code paths, like those used for testing, use the default SkylarkSemanticsOptions and therefore won't be able to use (for example) --incompatible_* flags. The call sites to update were found by searching for uses of defaultVisibility and working upward from there. RELNOTES: None PiperOrigin-RevId: 154432058
* Automated g4 rollback of commit d5ee3b5397135eebd4b5d5b6bd4a4444093c4df8.Gravatar kchodorow2017-04-26
| | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** The go rules have tons of transitive dependencies that are not declared in their "local" WORKSPACE files, so this broke lots of projects on ci.bazel.build I'll fix up the go rules, update all of _their_ reverse-dep projects, and resubmit. *** Original change description *** Repositories can only be accessed in projects that define them in their WORKSPACE file This is prep for #1943 - hierarchical workspace loading. RELNOTES[INC]: Remote repositories must define any remote repositories they themselves use (e.g., if @x//:foo depends on @y//:bar, @y must be defined in @x's WORKSPACE file). PiperOrigin-RevId: 154321845
* Repositories can only be accessed in projects that define them in their ↵Gravatar kchodorow2017-04-26
| | | | | | | | | | | | WORKSPACE file This is prep for #1943 - hierarchical workspace loading. RELNOTES[INC]: Remote repositories must define any remote repositories they themselves use (e.g., if @x//:foo depends on @y//:bar, @y must be defined in @x's WORKSPACE file). PiperOrigin-RevId: 154295762
* Rewrite repository lookup to return a failed value rather than throwGravatar ulfjack2017-04-20
| | | | | | | | | | | | | | | | | | | | | | | We need to lookup repositories as part of converting exec paths to artifacts, which in turn is needed for action cache lookups. These lookups should not cause a Skyframe exit, so we must not throw an exception here, unless the error makes it impossible to continue. Instead, we need to leave the decision whether to error out or not to the caller. Note that we may unnecessarily fetch a remote repository in order to do the action cache lookup, even if the action no longer depends on the input file, although this should only be possible for C++ compile actions. It's possible that there's another bug in the C++ compile action key computation that also contributes. This change also makes it so that the post-resolution action cache code ignores any errors wrt. repository lookup rather than throwing. If any of the paths could not be found, then the action cache lookup fails and we re-execute the corresponding action, which is exactly what should happen. Fixes #2759. PiperOrigin-RevId: 153696243
* Add repository override optionGravatar kchodorow2017-04-20
| | | | | | | | | | | RELNOTES: Adds a --override_repository option that takes a repository name and path. This forces Bazel to use the directory at that path for the repository. Example usage: `--override_repository=foo=/home/user/gitroot/foo`. Fixes #1266 PiperOrigin-RevId: 153599291
* Fix crash on graph cycle in configuration loadingGravatar gregce2017-04-19
| | | | PiperOrigin-RevId: 153473961
* Delete Preprocessor.Factory.SupplierGravatar laurentlb2017-04-12
| | | | | RELNOTES: None. PiperOrigin-RevId: 152882585
* Remove Preprocessing support from PackageFactoryGravatar laurentlb2017-04-12
| | | | | RELNOTES: None. PiperOrigin-RevId: 152804937
* When tracking the critical path, if not keeping incremental state, don't ↵Gravatar janakr2017-04-06
| | | | | | | | keep references to actions indefinitely. Instead, once an action is finished executing, keep just some metadata about it. This allows actions to be unconditionally dropped when running with --batch, --discard_analysis_cache, and --keep_going, even if profiling is enabled. The additional fields here add between 8 and 12 bytes per component, and we have one component per action. This additional penalty is only incurred when we are already saving memory, so I think it's ok. The full penalty will be realized only towards the end of the build, when every action has started executing at least once. Users can still specify --noexperimental_enable_critical_path_profiling if they want to squeeze even more memory out. PiperOrigin-RevId: 152328870
* Add "--build_manual_tests" options that forces manual tests to be built.Gravatar dslomov2017-04-04
| | | | | | | | | This is useful for IDEs and other tools utilizing command-line aspects for reflection over build graph. RELNOTES: None. PiperOrigin-RevId: 152038248
* If --batch, --keep_going, --discard_analysis_cache, and the new ↵Gravatar janakr2017-03-31
| | | | | | | | | | | | | | | --noexperimental_enable_critical_path_profiling flags are all specified, then Bazel will delete Actions from ActionLookupValues as they are executed in order to save memory. Because an already-run action may output an artifact that is only requested later in the build, we need to maintain a way for the artifact to look up the action. But in most cases we don't need to keep the action itself, just its output metadata. Some actions unfortunately are needed post-execution, and so we special-case them. Also includes dependency change with description: Move action out of key. This keeps action references from polluting the graph -- actions are just stored in one SkyValue, instead of being present in SkyKeys. This does mean additional memory used: we have a separate ActionLookupData object per Action executed. That may reach ~24M for million-action builds. PiperOrigin-RevId: 151756383
* Actually clear the per-build syscall cache after loading. Setting it to null ↵Gravatar janakr2017-03-31
| | | | | | doesn't work because the PackageFactory keeps a reference to it. PiperOrigin-RevId: 151632473
* Clear AspectValues when discarding analysis cache, along with ↵Gravatar janakr2017-03-29
| | | | | | | | ConfiguredTargetValues. Also clear transitive packages for both, even for top-level targets. This is not expected to save significant memory, but is expected to reduce the number of references to Packages, allowing them to be dropped more easily when discarding analysis cache and running in batch mode. PiperOrigin-RevId: 151508877
* Unit tests respect top-level rule-class transitions.Gravatar Cal Peyser2017-03-27
| | | | | | -- PiperOrigin-RevId: 151129669 MOS_MIGRATED_REVID=151129669
* Rationalize local resource acquisitionGravatar Ulf Adams2017-03-03
| | | | | | | | | | | | | | | | | | | | | | Move all local resource acquisition to where local execution actually happens. Don't attempt to acquire resources per action, but only for individual spawns. This significantly simplifies the code. The downside is that we don't account for action-level work anymore. In general, actions should not perform any process execution themselves, but always delegate such work to a SpawnStrategy implementation. This change makes sure that every Spawn has local resources set in a way that is consistent with the previous state. However, there are two actions - Fileset and FileWrite -, which are not spawns, and so we now don't limit their concurrent execution anymore. For Fileset, all work is done in a custom Fileset-specific thread pool, so this shouldn't be a problem. I'm not sure about FileWriteAction. -- PiperOrigin-RevId: 149012600 MOS_MIGRATED_REVID=149012600
* Clean up event busGravatar Klaus Aehlig2017-02-27
| | | | | | | | ...now that the ExtendedEventHandler can serve the same purpose. -- PiperOrigin-RevId: 148640618 MOS_MIGRATED_REVID=148640618
* Provide more reporting options to SkyFunctions Gravatar Klaus Aehlig2017-02-27
| | | | | | | | | | | | | With more specific information to be reported by Skyfunctions, e.g., to inform the build-event protocol on missing files, the EventHandler interface is no longer enough. Therefore, provide an enriched context for reporting events. -- Change-Id: I2d06166fe4d5b9054e24ad8c752fafc039e3f9f8 Reviewed-on: https://cr.bazel.build/8794 PiperOrigin-RevId: 148463437 MOS_MIGRATED_REVID=148463437
* Fix Cpp action cachingGravatar Ulf Adams2017-02-24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This combines both previous changes and extends them to work both with and without kchodorow@'s rollout of the exec root rearrangement. Unfortunately, each of these changes individually breaks something somewhere, so they must all go into a single commit. Change 1: CppCompileAction must return false from inputsKnown for .d pruning This is necessary (but not sufficient) for the action cache to work correctly. Consider the following sequence of events: 1. action is executed 2. .d pruning is performed 3. action cache writes entry with post-pruning inputs list 4. action gets regenerated (e.g., due to server restart) 5. the action cache calls inputsKnown(), which returns true 6. action cache checks entry from step 3 against pre-pruning inputs list, which results in a cache miss The action cache needs to know that the current list is not the final list, so inputsKnown() in step 5 must return false if .d pruning is enabled. Change 2: Fix artifact root discovery for external artifacts The SkyframeExecutor was assuming that all exec paths were coming from the main repository. Instead, rely on external exec paths to start with "../". Additional change 3: In addition, update the PackageRootResolverWithEnvironment and the HeaderDiscovery to use the single unified repository name guessing implementation. Previously, the PackageRootResolverWithEnvironment was poisoning the source artifact cache, which then caused subsequent lookups to return a bad artifact. Add a precondition to double-check that artifacts looked up by exec path have the correct source root. For compatibility with kchodorow@'s upcoming exec root refactor, if the exec path starts with "external", then assume it's coming from an external repository. This must be removed when that change is successfully rolled out, or it will break if anyone creates a package called 'external'. Additional change 4: On top of all of that, PackageRootResolverWithEnvironment and SkyframeExecutor must use the same source root computation as the Package class itself. I extracted the corresponding code to Root, and added comments both there and in Package to clearly indicate that these methods have to always be modified in sync. Fixes #2490. -- PiperOrigin-RevId: 148439309 MOS_MIGRATED_REVID=148439309
* Rollback of commit 4b73e972d909bcd533f2f9940f95a00b9b73bdde.Gravatar Dmitry Lomov2017-02-17
| | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Broke tests on CI: http://ci.bazel.io/job/bazel-tests/570/ *** Original change description *** Roll forward execroot change RELNOTES[INC]: Previously, an external repository would be symlinked into the execution root at execroot/local_repo/external/remote_repo. This changes it to be at execroot/remote_repo. This may break genrules/Skylark actions that hardcode execution root paths. If this causes breakages for you, ensure that genrules are using $(location :target) to access files and Skylark rules are using http://bazel.io/docs/skylark/lib/File.html's path, dirname, etc. functions. Cust... -- PiperOrigin-RevId: 147833177 MOS_MIGRATED_REVID=147833177
* Make the interface dynamic library builder a regular label instead of a ↵Gravatar Lukacs Berki2017-02-17
| | | | | | | | magic artifact. -- PiperOrigin-RevId: 147830857 MOS_MIGRATED_REVID=147830857
* Roll forward execroot changeGravatar Kristina Chodorow2017-02-16
| | | | | | | | | | | | | | | | | RELNOTES[INC]: Previously, an external repository would be symlinked into the execution root at execroot/local_repo/external/remote_repo. This changes it to be at execroot/remote_repo. This may break genrules/Skylark actions that hardcode execution root paths. If this causes breakages for you, ensure that genrules are using $(location :target) to access files and Skylark rules are using http://bazel.io/docs/skylark/lib/File.html's path, dirname, etc. functions. Custom crosstools that hardcode external/<repo> paths will have to be updated. Issue #1262. -- PiperOrigin-RevId: 147726370 MOS_MIGRATED_REVID=147726370
* Restrict aspects visible to other aspects according to their advertised ↵Gravatar Dmitry Lomov2017-02-15
| | | | | | | | providers. -- PiperOrigin-RevId: 147526961 MOS_MIGRATED_REVID=147526961
* Some minor logging improvements related to diff awareness:Gravatar Nathan Harmata2017-02-15
| | | | | | | | | | | | (i) Log [the first 100] external [non external-repo] file paths when we encounter them in skyframe [when externalFileAction is not ASSUME_NON_EXISTENT_AND_IMMUTABLE_FOR_EXTERNAL_PATHS]. These caveats are to prevent log spam. (ii) Fix grammar when logging the results of handleDiffsWithMissingDiffInformation in the case when all package paths have known diff information and we're merely checking external/output/external-repo files. (iii) Log the purpose of each skyframe graph scan we do (see (ii)). -- PiperOrigin-RevId: 147508404 MOS_MIGRATED_REVID=147508404
* Remove dead code - unused method in PackageManager interfaceGravatar Ulf Adams2017-02-09
| | | | | | -- PiperOrigin-RevId: 147018240 MOS_MIGRATED_REVID=147018240
* Introduce a new SkyValue that merely contains the workspace name. The ↵Gravatar Nathan Harmata2017-02-09
| | | | | | | | | | | | workspace name is needed for package loading, and so splitting out this computation into a separate skyframe node that can be change-pruned gives us better incrementality; previously we'd need to reload all packages on a WORKSPACE file change. N.B. (i) This CL doesn't solve all the other performance issues with //external in Bazel/Blaze since it's still inefficiently used for resolving labels like @foo//bar:baz. (ii) This CL doesn't address the wasteful invalidation + change pruning of all the packages. -- PiperOrigin-RevId: 146925369 MOS_MIGRATED_REVID=146925369
* Introduce a Skyframe function: ActionEnvironmentFunction Gravatar Damien Martin-Guillerez2017-02-08
| | | | | | | | | | | | | | | | | | ActionEnvironmentFunction returns the list of environment variable with the one overwritten by --action_env being replaced. This let other Skyframe function declares dependency to any value of the environment and being influenced by the --action_env flag. This will be used to declare dependency of remote repositories on environment variables (step 3 of https://bazel.build/designs/2016/10/18/repository-invalidation.html) -- Change-Id: I1ed3fb6f48e8e17d4d64c903fccecb6ed7596350 Reviewed-on: https://cr.bazel.build/7974 PiperOrigin-RevId: 146918603 MOS_MIGRATED_REVID=146918603
* Remove dead code in SkyframeLabelVisitorGravatar Ulf Adams2017-02-07
| | | | | | | | | | Two of the functions are never used, several of the parameter are not used or can be simplified (require callers to convert from Target -> Label). -- PiperOrigin-RevId: 146787380 MOS_MIGRATED_REVID=146787380
* Inject the changes in the client environment for the ClientEnvironmentFunction Gravatar Damien Martin-Guillerez2017-02-02
| | | | | | | | | | | | Using a precomputed values prevent us from being able to depends on any environment variable (or we would invalidate the whole graph each time the environment change, that is each time we run a command). -- Change-Id: If322df4240271d04b9278e1222f936156531580c Reviewed-on: https://cr.bazel.build/8400 PiperOrigin-RevId: 146298592 MOS_MIGRATED_REVID=146298592
* Names of extra-action protos now take into account all aspect names.Gravatar Dmitry Lomov2016-12-16
| | | | | | | | | | | | | If an Aspect registered an action that an extra-action is shadowing, its name is used when creating the extra-action's ID and name. Since recently, an aspect can see other aspects applied to the same target. This CL record the names of other aspects applied to the target as well, disambiguating the action owners. -- PiperOrigin-RevId: 142264153 MOS_MIGRATED_REVID=142264153
* Fix test configured target creation code that was dropping split deps.Gravatar Greg Estren2016-12-06
| | | | | | | | | | | | | | | Short story: Dependency -> BuildConfiguration maps can have multiple values because of split transitions. This is unfortunately one of those instances where the test logic is forked from production. So this doesn't actually demonstrate bugs in production code. We already have tests in ConfigurationsForTargetsTest that directly check the production logic. So this cl's primary value is to unbreak tests that depend on the forked logic. -- PiperOrigin-RevId: 141094830 MOS_MIGRATED_REVID=141094830
* Make build events correctly report target expansionGravatar Klaus Aehlig2016-12-05
| | | | | | | | | | | In particular, correctly report the expansion of a test suite even if only requested to build (but not test). -- Change-Id: Ia25305a4f5ed53118cbb1970a8055d156e8e50be Reviewed-on: https://cr.bazel.build/7535 PiperOrigin-RevId: 141027655 MOS_MIGRATED_REVID=141027655
* Fix bug in configuration creation logic that wrongly trims notrim configs.Gravatar Greg Estren2016-12-05
| | | | | | | | Without this, many Bazel tests fail with --experimental_dynamic_configs=notrim_partial. -- PiperOrigin-RevId: 140873563 MOS_MIGRATED_REVID=140873563
* Null out perBuildSyscallCache after analysis is finished to save memory ↵Gravatar Janak Ramakrishnan2016-12-02
| | | | | | | | during execution. -- PiperOrigin-RevId: 140864295 MOS_MIGRATED_REVID=140864295
* Refactor LegacyLoadingPhaseRunner to be more like SkyframeLoadingPhaseRunner.Gravatar Ulf Adams2016-11-22
| | | | | | | | This is in preparation for extracting some common reporting code, and also a step towards interleaving target pattern eval and config creation. -- MOS_MIGRATED_REVID=139890205
* Run the analysis phase with as many threads as the user wants. In order to ↵Gravatar Janak Ramakrishnan2016-11-18
| | | | | | | | | avoid memory blow-up intra-configured-target analysis, use a semaphore to ensure that CPU-bound work only occurs on #CPU-many threads. RELNOTES: Use --loading_phase_threads to control the number of threads used during the loading/analysis phase. -- MOS_MIGRATED_REVID=139477645
* Cache all previously computed values in ↵Gravatar Janak Ramakrishnan2016-11-17
| | | | | | | | | SkyQueryEnvironment#beforeEvaluateQuery if possible to save on latency for small queries. This assumes that if the graph is up to date, then the data in SkyQueryEnvironment is also up to date. It also assumes that a ForkJoinPool remains usable until #shutdownNow is called. -- MOS_MIGRATED_REVID=139386363
* Re-use the precomputed universe key in WalkableGraphFactory#prepareAndGet.Gravatar Janak Ramakrishnan2016-11-17
| | | | | -- MOS_MIGRATED_REVID=139357569
* Fixes incomplete support for dynamic split transitions in Bazel'sGravatar Greg Estren2016-11-17
| | | | | | | | | | | | | | | | | | | | | | test infrastructure. The small picture story is that SkyframeExecutor.getDynamicConfigOptions (which gets dynamic BuildOptions for tests) wasn't updated with dynamic split support when that was added to ConfiguredTargetFunction.getDynamicTransitionOptions (which does the same thing for production builds). This change plugs that gap. See 373e3e28274cca5b87f48abe33884edb84016dd3 for the original change. The bigger picture story is that Bazel's configured target creation logic is forked: test code goes down a similar but sadly not-quite-the-same path that results in tons of duplicated logic, spaghetti code mess, and risk of bugs like this one. We'd like to ultimately undo that fork. But unfortunately it's an involved effort that won't happen overnight. In the meantime, this change takes one small step by merging the two methods that caused this bug. -- MOS_MIGRATED_REVID=139342710
* Code cleanup for interleaving target pattern eval and config creation.Gravatar Ulf Adams2016-11-16
| | | | | -- MOS_MIGRATED_REVID=139209942
generated by cgit v1.2.3 (git 2.39.1) at 2024-08-19 02:49:55 +0000