aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/skyframe/SkyframeExecutor.java
Commit message (Collapse)AuthorAge
...
* Stop throwing an exception if a Package was successfully created but ↵Gravatar Janak Ramakrishnan2015-09-17
| | | | | | | | | | | contains errors. Instead, require callers to process the package and throw if they need to. This allows us to avoid embedding a Package in an exception, which is icky. This also allows us to remove Package#containsTemporaryErrors. Most callers' changes are fairly straightforward. The exception is EnvironmentBackedRecursivePackageProvider, which cannot throw an exception of its own in case of a package with errors (because it doesn't do that in keep_going mode), but whose request for a package with errors *should* shut down the build in case of nokeep_going mode. To do this in Skyframe, we have a new PackageErrorFunction which is to be called only in this situation, and will unconditionally throw. EnvironmentBackedRecursivePackageProvider can then catch this exception and continue on as usual, except that the exception will shut down the thread pool in a nokeep_going build. -- MOS_MIGRATED_REVID=103247761
* Add experimental flag, that partially disables loading phase (pattern ↵Gravatar Marian Lobur2015-09-16
| | | | | | | evaluation, test_suite expansion and configuration creation is still there). Also remove some unused code. -- MOS_MIGRATED_REVID=103177839
* Use AutoProfiler in the Bazel codebase.Gravatar Nathan Harmata2015-09-08
| | | | | -- MOS_MIGRATED_REVID=102584924
* Use another cache to store preprocessing results instead of redoing the work.Gravatar Ulf Adams2015-09-04
| | | | | | | This should have no effect. -- MOS_MIGRATED_REVID=102342697
* Remove unused code.Gravatar Marian Lobur2015-09-04
| | | | | -- MOS_MIGRATED_REVID=102339394
* Fix some warnings.Gravatar Ulf Adams2015-09-04
| | | | | -- MOS_MIGRATED_REVID=102332437
* Add a method to EvaluationProgressReceiver for getting information on ↵Gravatar Nathan Harmata2015-09-03
| | | | | | | SkyFunction#compute calls. -- MOS_MIGRATED_REVID=102268773
* Aspects can get information from their base rule.Gravatar Marian Lobur2015-09-02
| | | | | -- MOS_MIGRATED_REVID=102126786
* Allow for customization of the per-build system call caches used in Skyframe.Gravatar Eric Fellheimer2015-09-01
| | | | | -- MOS_MIGRATED_REVID=101984361
* Move PackageIdentifier to cmdlineGravatar Kristina Chodorow2015-08-31
| | | | | | | | This is necessary to have TargetResolver depend on it without making it depend on the packages target. First step of #389. -- MOS_MIGRATED_REVID=101790345
* Be resilient to a non-decreasing clock in ↵Gravatar Nathan Harmata2015-08-28
| | | | | | | | | BlazeRuntime#recordLastExecutionTime. Also add @Nullable annotations as appropriate. Fixes #394. -- MOS_MIGRATED_REVID=101685096
* Implement the core structure for dynamic configurations.Gravatar Greg Estren2015-08-26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a big change, so let me walk you through the key pieces: 1) This cl provides an alternative mechanism for creating configurations and doing configuration transitions that's "dynamic" in that the configurations can be created on the fly and the transitions are arbitrary mappings from BuildOptions --> BuildOptions that can also be created on the fly. It also integrates this with ConfiguredTargetFunction, so the configured target graph automatically uses this framework. 2) It does *not* replace old-style (which we'll call "static") configurations. For a number of important reasons: It's not yet at feature parity (particularly: no LIPO). It's not remotely tested across real projects enough to have confidence that it's battle-ready. It doesn't yet handle certain "special" functions like BuildConfiguration.prepareToBuild() and BuildConfiguration.getRoots(). It still relies on the old static transition logic to determine what transitions to apply (eventually we'll distribute that logic out, but that's too much for a single cl). We need the option to toggle it on and off until we have enough confidence in it. So with this cl, builds can be done in either mode. 3) The new flag --experimental_dynamic_configs toggles use of dynamic configurations. 4) Dynamic configurations are created with the Skyframe function BuildConfigurationFunction (this was already created in an earlier change). This consumes a BuildOptions and a set of configuration fragments to produce a BuildConfiguration. 5) Dynamic transitions are instances of the new class PatchTransition, which simply maps an input BuildOptions to an output BuildOptions. 6) Since static and dynamic configurations have to co-exist (for now), this cl tries hard to keep today's transition logic defined in a single place (vs. forking a dedicated copy for each configuration style). This is done via the new interface BuildConfiguration.TransitionApplier. BuildConfiguration.evaluateTransition is modified to feed its transition logic into TransitionApplier's common API. Both dynamic and static configurations have their own implementations that "do the right thing" with the results. 7) The transition applier for dynamic configurations basically stores the Transition, then allows DependencyResolver (which calls BuildConfiguration.evaluateTransition) to return Dependency instances containing that Transition (vs. a BuildConfiguration, which they traditionally contain). 7.5) An earlier variation of the dynamic transition applier retained BuildOptions (e.g. when it got a Transition it immediately applied it to get its output BuildOptions, then stored that). This had the advantage of making composing of transitions easier, especially within BuildConfiguration.evaluateTransition (which can theoretically apply multiple transitions to the input configuration). But it turns out that applying transitions has a cost, and it's simply more performant to pass them through until they're really needed. 8) In dynamic configuration mode, after ConfiguredTargetFunction gets its deps (e.g. an <Attribute, Dependency> multimap), it "trims" the configurations for its dependencies by a) only including config fragments required by the deps' subtrees and b) applying the transitions that came from 7). This all happens in the new method ConfiguredTargetFunction.trimConfigurations. 9) trimConfigurations is heavily performance-optimized based on a lot of experience running this through a large project within Google. As it turns out, the cost of host transitions can be atrocious (because there are a lot of them). Also, BuildOptions.clone() is expensive. And just creating BuildConfiguration SkyKeys also has a cost (largely because of BuildOptions cloning), so that shouldn't be done except when really necessary. My experience with this convinced me it's worth making this method complicated for the sake of making it fast. Since it basically visits every edge in the configured target graph (at least), it really needs to be slick. 10) Since host transitions in particular are problematic w.r.t. speed, I compute the host *once* in ConfigurationCollectionFunction.getHostConfiguration() and expose that reference to ConfiguredTargetFunction and other Skyframe functions. This limits recomputation to just when the fragments are trimmed. 11) Since options cloning is expensive, I'm doing something scary: exposing a BuildConfiguration.getOptions() method that returns a direct reference. Since BuildOptions is mutable, this is dangerous in the wrong hands. I can experiment with going back to cloning (with the caching of host transitions it may not matter as much), but we may ultimately have to put up with this risk for the sake of performant analysis time. What would be *really* awesome would be to make BuildOptions immutable. But that's not going to happen in this cl. So in short, the key abstractions in this cl are: - PatchTransition - BuildConfiguration.TransitionApplier - ConfiguredTargetFunction.trimConfigurations The current implementation imposes no analysis time penalty -- MOS_MIGRATED_REVID=101474620
* Rename some variables missed by a codebase-wide renaming a long time ago.Gravatar Nathan Harmata2015-08-25
| | | | | -- MOS_MIGRATED_REVID=101388809
* Preserve specified order of keys when analyzing configured targets.Gravatar Janak Ramakrishnan2015-08-24
| | | | | -- MOS_MIGRATED_REVID=101235139
* Convert evaluated tracking to take a lazy SkyValueGravatar Mark Schaller2015-08-21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The EvaluationProgressReceiver#evaluated method took a SkyValue, but that parameter was only used by some of its implementations, and only under some conditions. Outside of tests, the main users are SkyframeBuildView's ConfiguredTargetValueInvalidationReceiver and SkyframeBuilder's ExecutionProgressReceiver. The former builds up a set of built ConfiguredTarget keys when the SkyValue is non-null and the EvaluationState is BUILT, and so its nullity check can live behind those two conditions. The latter cares about builting up a set of ConfiguredTargets, and raising events on the eventBus when a TARGET_COMPLETION or ASPECT_COMPLETION value is evaluated and is non-null. The extraction of these values can live behind the conditions that check the type of the SkyKey. By making the SkyValue parameter lazy, this change enforces that it's only accessed under these conditions. This CL introduces a semantic change that should be small in effect. The SkyframeBuildView keeps track of a set, dirtiedConfiguredTargetKeys, and ConfiguredTarget keys evaluated as CLEAN were removed from it if they had a non-null value. With this change, ConfiguredTarget keys evaluated as CLEAN get removed regardless of whether their values are null or non-null. The set is used to determine whether artifact conflict search has to be rerun, and the extra removals that result from this CL could cause fewer artifact conflict searches to run, but the only affected searches would be those that were caused by dirtied configured target values in error all of which were subsequently marked as clean, which is probably rare. -- MOS_MIGRATED_REVID=101144655
* A prototype implementation of top-level aspects.Gravatar Dmitry Lomov2015-08-20
| | | | | -- MOS_MIGRATED_REVID=101033236
* Allow for metadata lookup alongside the existing #prepareAndGet() graph ↵Gravatar Eric Fellheimer2015-08-17
| | | | | | | construction. -- MOS_MIGRATED_REVID=100709648
* When we have enough information to determine that a file didn't change type, ↵Gravatar Nathan Harmata2015-08-13
| | | | | | | don't unnecessarily invalidate DirectoryListingStateValues. -- MOS_MIGRATED_REVID=100511935
* Convert invalidated tracking from per-value to per-keyGravatar Mark Schaller2015-08-11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The primary user of invalidation tracking is the SkyframeBuildView, which monitored which ConfiguredTargetValues were invalidated. It did that so the SkyframeExecutor could limit its search for artifact conflicts to when the set of configured targets in the build changed. For the newly introduced set of dirtied keys, "dirtiedConfiguredTargetKeys" in SkyframeBuildView, to be as useful as the "dirtyConfiguredTargets" set it replaced, the ConfiguredTargetValueInvalidationReceiver must only remove a key from the set if it was found to be clean when it was re-evaluated. If it was rebuilt, then the key must stay in the set, to represent that the set of configured target values has truly changed. This CL introduces a semantic change that hopefully has a small effect, if any. Previously, the informInvalidationReceiver method in InvalidatingNodeVisitor only informed the invalidationReceiver when a non-null value was invalidated. (This is usually, but not always, equivalent to a non-error value. The uncommon exception is that in keep-going builds, some nodes with errors may also have values, and the invalidator would inform the receiver when such a node was invalidated.) Now, the receiver is informed that a node is invalidated regardless of whether its value is null. Because the receiver uses this information to determine whether artifact conflict search has to be rerun, and that search is expensive, it's possible this change will negatively impact performance. However, the only way an extra search could be invoked is if the invalidated configured target nodes are all in error. That seems like it would happen rarely, if at all. Further cleanup of unused SkyValues returned by markDirty to come in a subsequent CL. -- MOS_MIGRATED_REVID=100304744
* Create a parameter to GlobFunction to determine whether it should deal with ↵Gravatar Eric Fellheimer2015-08-04
| | | | | | | | | non-pattern sub-expressions via file stat or directory listing. There is an inherent incrementality tradeoff here: A directory listing should be independent of any file edits in the directory, but may be overly conservative if an unrelated file is added or removed. The file stat is overly conservative when the file is modified, since glob() cares only about existence vs. non-existence. -- MOS_MIGRATED_REVID=99838654
* Fix ClassCastException.Gravatar Nathan Harmata2015-08-04
| | | | | -- MOS_MIGRATED_REVID=99487015
* Adds TransitiveTraversalFunction that loads transitive deps of a targetGravatar Mark Schaller2015-07-30
| | | | | | | | | Unlike TransitiveTargetFunction, it does not return nested sets of the traversed targets. Used primarily for its side effects of loading the transitive targets into the graph. -- MOS_MIGRATED_REVID=99388411
* Elegantly handle unbounded file symlink resolutions, e.g. 'a' -> 'b' -> ↵Gravatar Nathan Harmata2015-07-29
| | | | | | | 'a/nope'. -- MOS_MIGRATED_REVID=99337668
* Some additional timing logging for discarding analysis cache, multi group ↵Gravatar Eric Fellheimer2015-07-28
| | | | | | | accounting, and artifact conflicts. -- MOS_MIGRATED_REVID=99217433
* Make globs work in remote repositories.Gravatar Lukacs Berki2015-07-08
| | | | | | | | | | | This involved quite a few changes, mainly changing a bunch of places where we refer to packages by a PathFragment to PackageIdentifier. The only wart is the code in PathPackageLocator: ideally, it would just call into PackageLookupFunction. Unfortunately, it is (through globbing and Parser.include) called from within a Skyframe function, and we don't want to have two eval() calls going on at the same time, so we cannot use that. There is a potential correctness issue there: PathPackageLocator now assumes where external repositories are put and assumes that they are there when it gets control, but my understanding is that the associated RepositoryValue is always evaluated before, so it works out okay. -- MOS_MIGRATED_REVID=97751539
* Activate interleaved package and transitive target loadingGravatar Mark Schaller2015-07-08
| | | | | | | | Hooks up the recently introduced interleaved loading functions to normal graph loading. -- MOS_MIGRATED_REVID=97679451
* Rollback of accidentally submitted change.Gravatar Lukacs Berki2015-07-07
| | | | | -- MOS_MIGRATED_REVID=97648982
* Make globs work in remote repositories.Gravatar Lukacs Berki2015-07-07
| | | | | | | | | | | This involved quite a few changes, mainly changing a bunch of places where we refer to packages by a PathFragment to PackageIdentifier. The only wart is the code in PathPackageLocator: ideally, it would just call into PackageLookupFunction. Unfortunately, it is (through globbing and Parser.include) called from within a Skyframe function, and we don't want to have two eval() calls going on at the same time, so we cannot use that. There is a potential correctness issue there: PathPackageLocator now assumes where external repositories are put and assumes that they are there when it gets control, but my understanding is that the associated RepositoryValue is always evaluated before, so it works out okay. -- MOS_MIGRATED_REVID=97647787
* Introduce interleaved package and transitive target loadingGravatar Mark Schaller2015-07-01
| | | | | | | | | | Adds SkyFunctions and assorted values that implement interleaved loading of packages and their targets' transitive dependencies. They are not hooked up to any graph loading components, yet. -- MOS_MIGRATED_REVID=97278368
* Add some utilities and relax some visibility restrictions to make ↵Gravatar Janak Ramakrishnan2015-06-19
| | | | | | | alternative include scanning implementations possible. -- MOS_MIGRATED_REVID=96337469
* Print an error message when an @foo dep isn't foundGravatar Kristina Chodorow2015-06-17
| | | | | | | | | TransitiveTargetFunction only prints an error message if the package names match, so it was just exiting with "loading failed" when there was an error with external dependencies. -- MOS_MIGRATED_REVID=96204337
* Get jdk.WORKSPACE into the default WORKSPACE file by reading it as a Java ↵Gravatar Lukacs Berki2015-06-15
| | | | | | | | | | | resource, then passing it to the parser as a string instead of putting it into embedded_binaries then passing a Path to it to the parser. This makes the upcoming default WORKSPACE rules for Android much more palatable. In particular, Android rules won't need to be special cased when building the Bazel binary because the contents are self-contained in BazelRuleClassProvider (and the jdk.WORKSPACE file, which is a simple Java resource) Even better would be not to use a string, but some kind of structured data, but that's probably more effort than it's worth. -- MOS_MIGRATED_REVID=95983199
* Handle exceptions encountered resolving packages during the execution phaseGravatar Michajlo Matijkiw2015-06-10
| | | | | | | | | | | | | | Currently we may do lookups of not-already-cached packages during the execution phase for actions that discover inputs. Exceptions encountered during this would go unhandled and result in a crash. Here we introduce PackageRootResolutionException which wraps these exceptions and triggers an ActionExecutionException which is cleanly handled in the exec phase. As part of this change SkyframeActionExecutor#getArtifactRoots(...) will fail properly on errors getting package roots. -- MOS_MIGRATED_REVID=95578891
* Make the PackageFunction cache an actual Cache.Gravatar Eric Fellheimer2015-05-27
| | | | | -- MOS_MIGRATED_REVID=94569621
* Add an entry to TransitiveTargetValue that tracks theGravatar Greg Estren2015-05-27
| | | | | | | | | | | | | | | | configuration fragments needed by a rule's transitive closure. Also add a Skyframe BuildConfiguration node. Memory and performance profiling shows no noticeable performance hit in loading or analysis and a 0.35% memory increase for moderately sized (by Google standards) build graphs when these are depended upon in ConfiguredTargetFunction. -- MOS_MIGRATED_REVID=94517099
* Minor refactoring to SkyframeExecutor progress receiver.Gravatar Eric Fellheimer2015-05-27
| | | | | -- MOS_MIGRATED_REVID=94506006
* Add a hidden flag which allows us to set the size of the legacy globbing ↵Gravatar Eric Fellheimer2015-05-22
| | | | | | | thread pool. -- MOS_MIGRATED_REVID=94236393
* Deleted little-used PackageLoadedEvent.Gravatar Eric Fellheimer2015-05-18
| | | | | -- MOS_MIGRATED_REVID=93881974
* Add a SingleCycleReporter for analysis phase cycles, which can (only) occur ↵Gravatar Nathan Harmata2015-05-15
| | | | | | | | | due to configurations. RELNOTES: -- MOS_MIGRATED_REVID=93543318
* Add fetch commandGravatar Kristina Chodorow2015-04-28
| | | | | | | | | | | | | | This command takes a one or more targets and fetches any external repository prerequisites that will be needed to build them. Example usage: bazel fetch //foo:bar If //foo:bar depends on, say, a maven_jar, it'll be downloaded. -- MOS_MIGRATED_REVID=92279626
* Clean up analysis error reporting a bit:Gravatar Janak Ramakrishnan2015-04-28
| | | | | | | | | 1. Make --analysis_warnings_as_errors a no-op. 2. Stop doing a sanity check that we didn't succeed in analysis even with errors emitted. 3. As a result, emit an error about shared actions to the proper listener. -- MOS_MIGRATED_REVID=92262247
* Fix SkyframeExecutor to not assume only one node for the configuration(s).Gravatar Ulf Adams2015-04-27
| | | | | | | | | | | | | As far as I can tell, the dependency on the BuildConfigurationCollection node is not required anymore. The BuildConfiguration is part of the SkyKey, and it uses object identity. This shouldn't affect garbage collection due to the explicit BuildView.dropConfiguredTargets. If we change that in the future, we will have to be (even more) careful. -- MOS_MIGRATED_REVID=92134490
* Simplify the createConfiguration method in SkyframeExecutor.Gravatar Ulf Adams2015-04-23
| | | | | | | | | Instead of passing BuildConfigurationKey instances around, just pass in the little data we actually need. This allows removing the BuildConfigurationKey class. -- MOS_MIGRATED_REVID=91865340
* Pass in the test environment using ↵Gravatar Lukacs Berki2015-04-20
| | | | | | | | | | | BuildConfiguration.Options.testEnvironment instead of special-casing it in a large number of classes. The variables in the client environment are read in BlazeRuntime#beforeCommand() now. Note that this entails a slight loss of caching: before, "--test_env=a=A,b=B" and "--test_env=b=B,a=A" were equivalent, now they are not, since instead of comparing Map<String, String>, List<Map.Entry<String,String>> instances are compared. -- MOS_MIGRATED_REVID=91570828
* Remove the full client environment from BuildConfiguration in favor of ↵Gravatar Lukacs Berki2015-04-17
| | | | | | | computing the test environment as early as possible, and passing that along. -- MOS_MIGRATED_REVID=91388451
* Tweak the default Skyframe thread count during testing. Since test cases are ↵Gravatar Eric Fellheimer2015-04-16
| | | | | | | typically small, the current default is too high. -- MOS_MIGRATED_REVID=91225981
* Clear dirty output tree counters so that the next build does not get the old ↵Gravatar Miguel Alcon Pinto2015-04-10
| | | | | | | counters. -- MOS_MIGRATED_REVID=90749273
* optionally error on files outside of known rootsGravatar Michajlo Matijkiw2015-04-01
| | | | | | | | | | | | The goal of this change is to provide a means for disallowing files outside of declared, known locations in order to better enforce the hermeticy and in turn reproducability of builds. Previously when encountering these files (typically via external symlinks) we would add a dependency on build_id, now we can optionally fail the build. -- MOS_MIGRATED_REVID=90015665
* Add PrepareDepsOfPatternsFunction, use before queryGravatar Mark Schaller2015-03-31
| | | | | | | | | | | | | This introduces a single SkyFunction that has the desired side effect of loading matching targets and their transitive dependencies in the graph. It replaces the two calls to buildDriver.evaluate that made sure the graph loaded the necessary values before query evaluation with just one call. -- MOS_MIGRATED_REVID=89864338
* Proper support for the external immutable directory mechanism for symlink ↵Gravatar Nathan Harmata2015-03-25
| | | | | | | chains. -- MOS_MIGRATED_REVID=89511018
generated by cgit v1.2.3 (git 2.39.1) at 2024-08-16 11:05:55 +0000