aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/skyframe/SkyframeBuildView.java
Commit message (Collapse)AuthorAge
* Introduce an EvaluationProgressReceiver implementation that does nothing.Gravatar nharmata2017-05-10
| | | | | | | | This is useful for dealing with all the existing implementations in the face of interface changes that are irrelevant. RELNOTES: None PiperOrigin-RevId: 155525021
* If --batch, --keep_going, --discard_analysis_cache, and the new ↵Gravatar janakr2017-03-31
| | | | | | | | | | | | | | | --noexperimental_enable_critical_path_profiling flags are all specified, then Bazel will delete Actions from ActionLookupValues as they are executed in order to save memory. Because an already-run action may output an artifact that is only requested later in the build, we need to maintain a way for the artifact to look up the action. But in most cases we don't need to keep the action itself, just its output metadata. Some actions unfortunately are needed post-execution, and so we special-case them. Also includes dependency change with description: Move action out of key. This keeps action references from polluting the graph -- actions are just stored in one SkyValue, instead of being present in SkyKeys. This does mean additional memory used: we have a separate ActionLookupData object per Action executed. That may reach ~24M for million-action builds. PiperOrigin-RevId: 151756383
* Clear AspectValues when discarding analysis cache, along with ↵Gravatar janakr2017-03-29
| | | | | | | | ConfiguredTargetValues. Also clear transitive packages for both, even for top-level targets. This is not expected to save significant memory, but is expected to reduce the number of references to Packages, allowing them to be dropped more easily when discarding analysis cache and running in batch mode. PiperOrigin-RevId: 151508877
* Provide more reporting options to SkyFunctions Gravatar Klaus Aehlig2017-02-27
| | | | | | | | | | | | | With more specific information to be reported by Skyfunctions, e.g., to inform the build-event protocol on missing files, the EventHandler interface is no longer enough. Therefore, provide an enriched context for reporting events. -- Change-Id: I2d06166fe4d5b9054e24ad8c752fafc039e3f9f8 Reviewed-on: https://cr.bazel.build/8794 PiperOrigin-RevId: 148463437 MOS_MIGRATED_REVID=148463437
* Make the interface dynamic library builder a regular label instead of a ↵Gravatar Lukacs Berki2017-02-17
| | | | | | | | magic artifact. -- PiperOrigin-RevId: 147830857 MOS_MIGRATED_REVID=147830857
* Change TargetContext.findDirectPrerequisite from config1 == config2 to ↵Gravatar Greg Estren2017-01-18
| | | | | | | | | | config1.equals(c2). This fixes an obscure bug between dynamic configurations, host configuration caching, and Skyframe skyKey interning that makes Bazel crash under certain sequences of builds. See changes for details. -- PiperOrigin-RevId: 144766296 MOS_MIGRATED_REVID=144766296
* Run the analysis phase with as many threads as the user wants. In order to ↵Gravatar Janak Ramakrishnan2016-11-18
| | | | | | | | | avoid memory blow-up intra-configured-target analysis, use a semaphore to ensure that CPU-bound work only occurs on #CPU-many threads. RELNOTES: Use --loading_phase_threads to control the number of threads used during the loading/analysis phase. -- MOS_MIGRATED_REVID=139477645
* Aspects-on-aspects implementation.Gravatar Dmitry Lomov2016-11-15
| | | | | -- MOS_MIGRATED_REVID=139189444
* Don't inherit the host configuration's fragments from the target config in ↵Gravatar Greg Estren2016-10-05
| | | | | | | --experimental_dynamic_configs=notrim mode. -- MOS_MIGRATED_REVID=135126724
* Fixed EvaluationProgressReceiver object namesGravatar Googler2016-09-27
| | | | | | | | | | EvaluationProgressReceiver objects have two common naming schemes currently, and calling them invalidationReceiver is misleading, so to make the naming convention standard, all object names are based on "progressReceiver." -- MOS_MIGRATED_REVID=134411011
* Remove ArtifactFactory dependency on incorrect exec rootGravatar Kristina Chodorow2016-08-25
| | | | | | | | | | | Somewhat trickily, this changes the execRoot field from referring to [output_base]/execroot/wsname (not really the exec root) to [output_base]/execroot (actually the execroot). Progress on #1681. -- MOS_MIGRATED_REVID=131286181
* Allow Skyframe graph lookups and value retrievals to throw InterruptedException.Gravatar Janak Ramakrishnan2016-08-16
| | | | | | | The only place we now don't handle InterruptedException is in the action graph created after analysis, since I'm not sure that will be around for that much longer. -- MOS_MIGRATED_REVID=130327770
* Changes DependencyResolver <Attribute, Dep> map from a ListMultimap to new classGravatar Greg Estren2016-08-12
| | | | | | | | | | | | | | | | | | | | | | OrderedSetMultimap. This maintains insertion order while eliminating duplicates. Certain rules, in particular, otherwise break this invariant: https://github.com/bazelbuild/bazel/blo[]e3e28274cca5b87f48abe33884edb84016dd3/src/main/java/com/google/devtools/build/lib/skyframe/ConfiguredTargetFunction.java#L403 There's no reason (to my knowledge) to need multiple instances of the same <Attribute, Dependency> pair. More context from Google code review: (Michael Staib): > There are many things which pass around a dependentNodeMap or help construct one or modify one. We want an interface which has the right guarantees. > ListMultimap is not the right interface because it has no guarantee of unique elements, which we want - we don't want the problem that this CL ran into, and there's no reason (that we know of, to be confirmed) that anyone would want multiple identical Dependencies. > SetMultimap is not the right interface because it has no guarantee of deterministic iteration order or efficient iteration, which we want - dependency order sometimes matters (e.g., Java classpath or C++ link order). > We agreed that the best way to get what we want is to define our own interface with its own simultaneous uniqueness and iterability guarantees. Unspoken in the discussion was why we wouldn't just use LinkedHashMultimap as the thing we pass around. IMO the reason for that is that we don't care that it be a LinkedHashMultimap specifically; if tomorrow Guava comes out with a faster cooler map that has deterministic and efficient iteration and guarantees element uniqueness, we want it. > In this case we're going to make the "interface" be a (final?) class: OrderedSetMultimap, an extension of ForwardingSetMultimap which delegates to LinkedHashMultimap, an implementation which does support both of those guarantees. > I had mentioned in the conversation that none of the Multimap implementations make guarantees about key iteration order, but this is not true - LinkedHashMultimap preserves key insertion order. We should perhaps declare this as part of the OrderedSetMultimap contract as well. -- MOS_MIGRATED_REVID=130037643
* Refactor cycle detection logic to handle dynamic configurations.Gravatar Greg Estren2016-06-09
| | | | | | | | | | | | | | | | | | | | | | | Currently, analysis-time cycle detection expects all cycles to come from ConfiguredTargetFunction. With dynamic configurations, ConfiguredTargetFunction calls out to TransitiveTargetFunction to figure out which configuration fragments its deps need. If there's a cycle between the current target and a dep, the dep's TransitiveTargetFunction fails, which the current cycle detection code can't handle. But even if it could handle it, since the failure occurs in the dep we'd get error messages like: "in cc_library rule //the:dep: cycle in dependency graph" instead of the expected: "in cc_library rule //the:top_level_rule: cycle in dependency graph" This used to not be a problem because loading-phase cycle detection caught the cycle before all this triggered. But interleaved loading and analysis removes that gate. Tested: BuildViewTest cycle detection tests with dynamic configurations turned on -- MOS_MIGRATED_REVID=124391277
* Add an "alias" rule.Gravatar Lukacs Berki2016-05-10
| | | | | | | This will be used to replace RedirectChaser so that we don't need to load packages during configuration creation anymore. -- MOS_MIGRATED_REVID=121935989
* Split ActionMetadata into ActionAnalysisMetadata and ActionExecutionMetadata.Gravatar Rumou Duan2016-04-26
| | | | | | | Except in action execution logic (ActionExecutionFunction, SkyframeActionExecutor, etc.), switch Action interface references to either ActionAnalysisMetadata if possible or ActionExecutionMetadata. -- MOS_MIGRATED_REVID=120723431
* Remove some dead code.Gravatar Ulf Adams2016-04-12
| | | | | | | | The cast to Label can never succeed for a configured target node, so this code could never have been executed (or we'd see crashes at this location). -- MOS_MIGRATED_REVID=119615022
* Pass the source path -> Artifact mapping into FdoSupport from a provider ↵Gravatar Lukacs Berki2016-02-28
| | | | | | | | | instead of special-casing it. This removes the need to deserialize artifacts in FdoSupport, which in turn removes the need to support artifact deserialization at all, which makes our lives simpler and Thoreauvian programming is good. -- MOS_MIGRATED_REVID=115660698
* Tiny refactoring: extract shared code between -k and -nok.Gravatar Ulf Adams2016-02-08
| | | | | -- MOS_MIGRATED_REVID=114086842
* Correctly flag loading errors in the interleaved case.Gravatar Ulf Adams2016-02-03
| | | | | | | | | | | | | | | In the interleaved case, loading errors can now also be discovered during the analysis phase. Add a boolean flag to the SkyframeAnalysisResult to indicate that such an error happened, and pass it through in BuildView. Also refactor BuildView to simplify the code a bit - simply pass the SkyframeAnalysisResult to the createResult method. There is already an integration test for this - I'm adding a faster unit test in BuildViewTest, as this is part of the BuildView contract. -- MOS_MIGRATED_REVID=113716870
* Implement proper error handling for interleaved loading and analysis.Gravatar Ulf Adams2016-02-02
| | | | | | | | Add test coverage by re-running BuildViewTest with the new Skyframe loading phase runner. -- MOS_MIGRATED_REVID=113517509
* Small refactoring of SkyframeBuildView to reduce the diff for the next change.Gravatar Ulf Adams2016-01-29
| | | | | -- MOS_MIGRATED_REVID=113338481
* Refactor DependencyResolver to collect and return loading errors.Gravatar Ulf Adams2016-01-28
| | | | | | | | | This should never be triggered in production, where we always run a loading phase first and only analyze targets that load successfully. I.e., this is just plumbing which will be hooked up in a subsequent change. -- MOS_MIGRATED_REVID=113258593
* Fix: return an error from the analysis phase with keep_going in error cases.Gravatar Ulf Adams2016-01-27
| | | | | | | | | | | - if there are failed top-level aspects - if there are conflicting actions I ended up rewriting how errors are signaled from the SkyframeBuildView. I think this is safe, but please review carefully. -- MOS_MIGRATED_REVID=113150100
* Move analysis root cause tracking to the ConfiguredTargetFunction.Gravatar Ulf Adams2016-01-25
| | | | | | | | | | | | | | | | | | | | | | | | | The main remaining problem with interleaved loading and analysis is error handling. When interleaving, we don't run a real loading phase anymore, and loading errors can occur during the analysis phase, and need to be handled there. The plan is to have ConfiguredTargetFunction throw a ConfiguredValueCreationException with a list of all loading root causes, which requires that we also catch ConfiguredValueCreationException here, which in turn breaks analysis root cause handling, as that is currently relying on Skyframe root cause tracking. Moving analysis root cause handling into CTFunction makes it possible to subsequently also implement loading root cause handling here. This is also necessary if we want to have complete root cause handling in the general case: a target may have any number (and combination) of loading and analysis root causes at the same time. For now, we only pass a single analysis root cause, which mirrors the current Skyframe-based implementation. -- MOS_MIGRATED_REVID=112930871
* Fix a couple of bugs related to error handling for top-level aspects.Gravatar Ulf Adams2016-01-20
| | | | | -- MOS_MIGRATED_REVID=112561390
* General cleanup for the configured target / aspect creation code.Gravatar Ulf Adams2016-01-19
| | | | | | | | | | | | | | - update some comments - add some comments to make it easier to follow - delete some dead code, in particular the SkyframeDependencyResolver can never be null; remove an non-applicable @Nullable annotation I'm trying to figure out how the error handling code works, in order to add support for interleaved loading+analysis, which requires handling loading errors in this code path. -- MOS_MIGRATED_REVID=112456674
* Use Bazel Preconditions variant which avoids varargs array creationGravatar Mark Schaller2015-12-10
| | | | | | | Reduces garbage. -- MOS_MIGRATED_REVID=109914243
* Inline SkyframeBuildView.createAspect into its only call-site, to reduce ↵Gravatar Carmi Grushko2015-11-13
| | | | | | | stack depth. -- MOS_MIGRATED_REVID=107688035
* Aspect terminology update.Gravatar Dmitry Lomov2015-11-10
| | | | | | | | Aspect => ConfiguredAspect AspectWithParameters => Aspect -- MOS_MIGRATED_REVID=107375211
* Add a TODO for a known dynamic configuration problem: hostGravatar Greg Estren2015-11-06
| | | | | | | | | | | | | | configurations get trimmed to the same fragments as target configurations. This isn't correct, because null fragments in the target configuration (which in practice don't exist) shouldn't necessarily be excluded from the host config. In the worst case then can crash fragment loaders, which expect the fragment options to exist even if the fragment ends up being null. -- MOS_MIGRATED_REVID=107173093
* Pass AspectWithParameters instead of separately passing AspectParameters and ↵Gravatar Carmi Grushko2015-11-04
| | | | | | | | | aspect-attributes. The former contains both. -- MOS_MIGRATED_REVID=106961145
* When creating RuleContext, explicitly pass the set of attributes an attached ↵Gravatar Carmi Grushko2015-11-02
| | | | | | | Aspect provides. -- MOS_MIGRATED_REVID=106882046
* Implement propagation along dependencies for Skylark aspects.Gravatar Dmitry Lomov2015-11-02
| | | | | -- MOS_MIGRATED_REVID=106694515
* Improve error diagnostics for Skylark aspects.Gravatar Dmitry Lomov2015-10-20
| | | | | -- MOS_MIGRATED_REVID=105851371
* Implement aspect(...) Skylark function.Gravatar Dmitry Lomov2015-10-20
| | | | | -- MOS_MIGRATED_REVID=105844221
* Rollback of commit ac7195025b073948785db8c6975a53fc305c5087.Gravatar Peter Schmitt2015-10-20
| | | | | | | | | | | | | *** Reason for rollback *** Uses tests that don't run on Bazel *** Original change description *** Implement aspect(...) Skylark function. -- MOS_MIGRATED_REVID=105808850
* Implement aspect(...) Skylark function.Gravatar Dmitry Lomov2015-10-16
| | | | | -- MOS_MIGRATED_REVID=105596479
* Move the remaining state from BuildView to SkyframeBuildView.Gravatar Ulf Adams2015-09-28
| | | | | -- MOS_MIGRATED_REVID=104009600
* Rationalize copyright headersGravatar Damien Martin-Guillerez2015-09-25
| | | | | | | | | | | The headers were modified with `find . -type f -exec 'sed' '-Ei' 's|Copyright 201([45]) Google|Copyright 201\1 The Bazel Authors|' '{}' ';'` And manual edit for not Google owned copyright. Because of the nature of ijar, I did not modified the header of file owned by Alan Donovan. The list of authors were extracted from the git log. It is missing older Google contributors that can be added on-demand. -- MOS_MIGRATED_REVID=103938715
* Remove more uses of errorEventHandler (previously of Reporter).Gravatar Ulf Adams2015-09-24
| | | | | | | Removes mutable global state. -- MOS_MIGRATED_REVID=103837106
* Remove all direct uses of SkyframeExecutor.reporter.Gravatar Ulf Adams2015-09-24
| | | | | | | | | Instead, pass an appropriate EventHandler instance in. This is in preparation for creating a per-command EventHandler, in preparation for allowing multiple commands to run in parallel. This is removal of shared global state. -- MOS_MIGRATED_REVID=103828963
* Move ownership of SkyframeBuildView to SkyframeExecutor.Gravatar Ulf Adams2015-09-23
| | | | | | | | Also move ownership of ArtifactFactory to SkyframeBuildView; simplify the code. -- MOS_MIGRATED_REVID=103722228
* Move Label from the lib.syntax to the lib.cmdline package so that:Gravatar Lukacs Berki2015-09-21
| | | | | | | | | | - Label parsing can be simplified - lib.syntax is only contains the code for Skylark and is reasonably independent from the problem domain of building things This change is mostly only changes to imports declarations. The rest is reversing the dependency between :cmdline and :syntax and moving a tiny amount of code between Printer and FilesetEntry and the addition of SkylarkPrintableValue that I couldn't be bothered to separate out into its own change. -- MOS_MIGRATED_REVID=103527877
* Add experimental flag, that partially disables loading phase (pattern ↵Gravatar Marian Lobur2015-09-16
| | | | | | | evaluation, test_suite expansion and configuration creation is still there). Also remove some unused code. -- MOS_MIGRATED_REVID=103177839
* Add a method to EvaluationProgressReceiver for getting information on ↵Gravatar Nathan Harmata2015-09-03
| | | | | | | SkyFunction#compute calls. -- MOS_MIGRATED_REVID=102268773
* Aspects can get information from their base rule.Gravatar Marian Lobur2015-09-02
| | | | | -- MOS_MIGRATED_REVID=102126786
* Only print the label (and not the configuration hash key) for error messages ↵Gravatar Lukacs Berki2015-08-31
| | | | | | | with --keep_going. -- MOS_MIGRATED_REVID=101921704
* Implement the core structure for dynamic configurations.Gravatar Greg Estren2015-08-26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a big change, so let me walk you through the key pieces: 1) This cl provides an alternative mechanism for creating configurations and doing configuration transitions that's "dynamic" in that the configurations can be created on the fly and the transitions are arbitrary mappings from BuildOptions --> BuildOptions that can also be created on the fly. It also integrates this with ConfiguredTargetFunction, so the configured target graph automatically uses this framework. 2) It does *not* replace old-style (which we'll call "static") configurations. For a number of important reasons: It's not yet at feature parity (particularly: no LIPO). It's not remotely tested across real projects enough to have confidence that it's battle-ready. It doesn't yet handle certain "special" functions like BuildConfiguration.prepareToBuild() and BuildConfiguration.getRoots(). It still relies on the old static transition logic to determine what transitions to apply (eventually we'll distribute that logic out, but that's too much for a single cl). We need the option to toggle it on and off until we have enough confidence in it. So with this cl, builds can be done in either mode. 3) The new flag --experimental_dynamic_configs toggles use of dynamic configurations. 4) Dynamic configurations are created with the Skyframe function BuildConfigurationFunction (this was already created in an earlier change). This consumes a BuildOptions and a set of configuration fragments to produce a BuildConfiguration. 5) Dynamic transitions are instances of the new class PatchTransition, which simply maps an input BuildOptions to an output BuildOptions. 6) Since static and dynamic configurations have to co-exist (for now), this cl tries hard to keep today's transition logic defined in a single place (vs. forking a dedicated copy for each configuration style). This is done via the new interface BuildConfiguration.TransitionApplier. BuildConfiguration.evaluateTransition is modified to feed its transition logic into TransitionApplier's common API. Both dynamic and static configurations have their own implementations that "do the right thing" with the results. 7) The transition applier for dynamic configurations basically stores the Transition, then allows DependencyResolver (which calls BuildConfiguration.evaluateTransition) to return Dependency instances containing that Transition (vs. a BuildConfiguration, which they traditionally contain). 7.5) An earlier variation of the dynamic transition applier retained BuildOptions (e.g. when it got a Transition it immediately applied it to get its output BuildOptions, then stored that). This had the advantage of making composing of transitions easier, especially within BuildConfiguration.evaluateTransition (which can theoretically apply multiple transitions to the input configuration). But it turns out that applying transitions has a cost, and it's simply more performant to pass them through until they're really needed. 8) In dynamic configuration mode, after ConfiguredTargetFunction gets its deps (e.g. an <Attribute, Dependency> multimap), it "trims" the configurations for its dependencies by a) only including config fragments required by the deps' subtrees and b) applying the transitions that came from 7). This all happens in the new method ConfiguredTargetFunction.trimConfigurations. 9) trimConfigurations is heavily performance-optimized based on a lot of experience running this through a large project within Google. As it turns out, the cost of host transitions can be atrocious (because there are a lot of them). Also, BuildOptions.clone() is expensive. And just creating BuildConfiguration SkyKeys also has a cost (largely because of BuildOptions cloning), so that shouldn't be done except when really necessary. My experience with this convinced me it's worth making this method complicated for the sake of making it fast. Since it basically visits every edge in the configured target graph (at least), it really needs to be slick. 10) Since host transitions in particular are problematic w.r.t. speed, I compute the host *once* in ConfigurationCollectionFunction.getHostConfiguration() and expose that reference to ConfiguredTargetFunction and other Skyframe functions. This limits recomputation to just when the fragments are trimmed. 11) Since options cloning is expensive, I'm doing something scary: exposing a BuildConfiguration.getOptions() method that returns a direct reference. Since BuildOptions is mutable, this is dangerous in the wrong hands. I can experiment with going back to cloning (with the caching of host transitions it may not matter as much), but we may ultimately have to put up with this risk for the sake of performant analysis time. What would be *really* awesome would be to make BuildOptions immutable. But that's not going to happen in this cl. So in short, the key abstractions in this cl are: - PatchTransition - BuildConfiguration.TransitionApplier - ConfiguredTargetFunction.trimConfigurations The current implementation imposes no analysis time penalty -- MOS_MIGRATED_REVID=101474620
* Convert evaluated tracking to take a lazy SkyValueGravatar Mark Schaller2015-08-21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The EvaluationProgressReceiver#evaluated method took a SkyValue, but that parameter was only used by some of its implementations, and only under some conditions. Outside of tests, the main users are SkyframeBuildView's ConfiguredTargetValueInvalidationReceiver and SkyframeBuilder's ExecutionProgressReceiver. The former builds up a set of built ConfiguredTarget keys when the SkyValue is non-null and the EvaluationState is BUILT, and so its nullity check can live behind those two conditions. The latter cares about builting up a set of ConfiguredTargets, and raising events on the eventBus when a TARGET_COMPLETION or ASPECT_COMPLETION value is evaluated and is non-null. The extraction of these values can live behind the conditions that check the type of the SkyKey. By making the SkyValue parameter lazy, this change enforces that it's only accessed under these conditions. This CL introduces a semantic change that should be small in effect. The SkyframeBuildView keeps track of a set, dirtiedConfiguredTargetKeys, and ConfiguredTarget keys evaluated as CLEAN were removed from it if they had a non-null value. With this change, ConfiguredTarget keys evaluated as CLEAN get removed regardless of whether their values are null or non-null. The set is used to determine whether artifact conflict search has to be rerun, and the extra removals that result from this CL could cause fewer artifact conflict searches to run, but the only affected searches would be those that were caused by dirtied configured target values in error all of which were subsequently marked as clean, which is probably rare. -- MOS_MIGRATED_REVID=101144655
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-07 06:53:21 +0000