aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/skyframe/SkyframeExecutor.java
Commit message (Collapse)AuthorAge
* Fix SkyQuery bug where we weren't respecting the package blacklist. We do ↵Gravatar Nathan Harmata2016-04-18
| | | | | | | | | this by changing both the relevant Skyframe and the SkyQuery code to propagate (minimal!) blacklist information in the SkyKeys themselves. There are other approaches to solving this problem, but I like how this solution doesn't involve duplication of logic. Also, it has the following nice benefit: previously, RecursiveDirectoryTraversalFunction would declare a dep on the blacklist for every directory traversed which adds an edge for each directory traversed. -- MOS_MIGRATED_REVID=120049635
* Inject the BlazeDirectories in SkyframeExecutor.Gravatar Ulf Adams2016-04-15
| | | | | | | | | The BlazeDirectories are also needed for loading the WORKSPACE file, so inject them as part of preparePackageLoading rather than in createConfigurations, which is too late. -- MOS_MIGRATED_REVID=119931633
* Make preprocessor modules mostly stateless; pass in the directories instead.Gravatar Ulf Adams2016-04-12
| | | | | -- MOS_MIGRATED_REVID=119631623
* SkyframeExecutor: Raise an event at the beginning of the loading phaseGravatar Klaus Aehlig2016-04-07
| | | | | | | | | | | The main purpose of this event is to provide the LoadingProgressReceiver to any interested listener. Note that this event will only be raised if the option --experimental_skyframe_target_pattern_evaluator is given. -- Change-Id: Ic675c9e1255b7d60de28bdad1ec1838655abd389 Reviewed-on: https://bazel-review.googlesource.com/#/c/3269 MOS_MIGRATED_REVID=119258070
* SkyframeExecutor: use a LoadingProgressReceiver during loading phaseGravatar Klaus Aehlig2016-04-07
| | | | | | | | | | In this way, information about the progress of the loading phase is collected. -- Change-Id: I80ff0c91e27f0b55780c18c05fbb4fdda93263d3 Reviewed-on: https://bazel-review.googlesource.com/#/c/3268 MOS_MIGRATED_REVID=119255431
* Remove AspectClass.getDefinitionGravatar Dmitry Lomov2016-04-07
| | | | | | | | | | Aspect becomes a triple (AspectClass, AspectDefinition, AspectParameters) and loses its equals() method. After this CL, SkylarkAspectClass.getDefintion still exists and is deprecated. -- MOS_MIGRATED_REVID=119159653
* Move TimestampGranularityMonitor to CommandEnvironment.Gravatar Ulf Adams2016-03-23
| | | | | | | | This is one of the last pieces of state in BlazeRuntime that isn't safe to share across command invocations. -- MOS_MIGRATED_REVID=117910631
* Remove the WalkableGraphFactory#afterUse hook, which was only called in ↵Gravatar Nathan Harmata2016-03-21
| | | | | | | SkyQueryEnvironment. QueryEnvironmentFactory, recently introduced by unknown commit, is a much more general purpose mechanism. -- MOS_MIGRATED_REVID=117590252
* Allow PackageRootResolver to take in directories for findPackageRoots().Gravatar Alex Humesky2016-03-18
| | | | | -- MOS_MIGRATED_REVID=117512506
* Don't keep packages in the default repository around after loading.Gravatar Brian Silverman2016-03-17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, this would get thrown when referring to the same package from both the main and default repositories: java.lang.IllegalArgumentException: Multiple entries with same key: tools/cpp=/home/brian/971-Robot-Code and tools/cpp=/home/brian/971-Robot-Code at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:136) at com.google.common.collect.RegularImmutableMap.checkNoConflictInKeyBucket(RegularImmutableMap.java:98) at com.google.common.collect.RegularImmutableMap.fromEntryArray(RegularImmutableMap.java:84) at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:295) at com.google.devtools.build.lib.buildtool.BuildTool.transformPackageRoots(BuildTool.java:301) at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:209) at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:334) at com.google.devtools.build.lib.runtime.commands.TestCommand.doTest(TestCommand.java:119) at com.google.devtools.build.lib.runtime.commands.TestCommand.exec(TestCommand.java:104) at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:371) at com.google.devtools.build.lib.runtime.BlazeRuntime$3.exec(BlazeRuntime.java:1016) at com.google.devtools.build.lib.server.RPCService.executeRequest(RPCService.java:65) at com.google.devtools.build.lib.server.RPCServer.executeRequest(RPCServer.java:434) at com.google.devtools.build.lib.server.RPCServer.serve(RPCServer.java:229) at com.google.devtools.build.lib.runtime.BlazeRuntime.serverMain(BlazeRuntime.java:975) at com.google.devtools.build.lib.runtime.BlazeRuntime.main(BlazeRuntime.java:772) at com.google.devtools.build.lib.bazel.BazelMain.main(BazelMain.java:55) And this would get thrown for any packages in the main repository loaded from other repositories: java.lang.RuntimeException: Unrecoverable error while evaluating node 'PACKAGE:@//tools/build_rules/go/toolchain' (requested by nodes ) at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:982) at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:499) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalArgumentException: Invalid BUILD file name for package '@//tools/build_rules/go/toolchain': /home/brian/bazel/tools/build_rules/go/toolchain/BUILD at com.google.devtools.build.lib.packages.Package.finishInit(Package.java:299) at com.google.devtools.build.lib.packages.Package$Builder.finishBuild(Package.java:1308) at com.google.devtools.build.lib.skyframe.PackageFunction.compute(PackageFunction.java:501) at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:933) ... 4 more Sponsor's comment: note the abundance of new Label.resolveRepositoryRelative() calls. They are ugly, but it's only making existing ugliness explicit. Yes, we should fix it, especially in the implementation of configurable attributes. Refs #940 -- Change-Id: I8bd7f7b00bec58a7157507595421bc50c81b404c Reviewed-on: https://bazel-review.googlesource.com/#/c/2591 MOS_MIGRATED_REVID=117429733
* Use two configurations for AspectKeys.Gravatar Michael Staib2016-03-01
| | | | | | | | | | | | In order for Aspects to support dynamic configuration, they need to have two configurations: one to instantiate the Aspect with, containing all the fragment dependencies of the Aspect itself, and one to create the ConfiguredTargetValue.key with, containing only the dependencies of the Rule. This expands AspectKey to have a second configuration, although it currently does not populate that key with anything different. -- MOS_MIGRATED_REVID=115997454
* Pass the source path -> Artifact mapping into FdoSupport from a provider ↵Gravatar Lukacs Berki2016-02-28
| | | | | | | | | instead of special-casing it. This removes the need to deserialize artifacts in FdoSupport, which in turn removes the need to support artifact deserialization at all, which makes our lives simpler and Thoreauvian programming is good. -- MOS_MIGRATED_REVID=115660698
* Fix the --loading_phase_threads to work during "blaze build". Previously and ↵Gravatar Eric Fellheimer2016-02-23
| | | | | | | confusingly it only worked on "blaze query". -- MOS_MIGRATED_REVID=115338436
* Add an experimental flag to enable the Skyframe-based loading phase runner.Gravatar Ulf Adams2016-02-12
| | | | | | | | | | | | | | Refactor the CommandEnvironment to no longer create a LoadingPhaseRunner eagerly. Unfortunately, that means we can't reuse the TargetPatternEvaluator either. Fortunately, the SkyframeTargetPatternEvaluator is a very lightweight class, which only contains the offset as state, so we just create a new instance whenever we need one. On the plus side, the LoadingPhaseRunner API is now stateless, and the QueryCommand no longer creates a unnecessary LoadingPhaseRunner instance. -- MOS_MIGRATED_REVID=114519731
* Fix SkyframeLoadingPhaseRunner posting of EventBus events.Gravatar Ulf Adams2016-02-10
| | | | | | | | The TargetParsingCompleteEvent was posting the post-expansion targets, and the LoadingPhaseCompleteEvent was missing the test-suite targets. -- MOS_MIGRATED_REVID=114312273
* Fix callback handling in the Skyframe-based target pattern eval.Gravatar Ulf Adams2016-02-10
| | | | | | | | Add a unit test - there is test coverage for this in RunCommandTest, but this runs much faster. -- MOS_MIGRATED_REVID=114302062
* Add an intermediate SkyFunction for resolving the external packageGravatar Damien Martin-Guillerez2016-02-07
| | | | | | | | | | | | | | The WORKSPACE file parsing needs to be separated into several parts to enable load of labels in the WORKSPACE file. This change adds an intermediate SkyFunction, ExternalPackageFunction, that requires all the WORKSPACE file part to be parsed to resolve //external: labels. Issue #824 Step 1. -- MOS_MIGRATED_REVID=113984026
* Use a clever hybrid approach for evaluating globs during package loading: ↵Gravatar Nathan Harmata2016-02-05
| | | | | | | | | | | | | | | | | first try to get a skyframe cache-hit; otherwise, fall back to legacy globbing. This gives us the best of both worlds: no extra skyframe restarts on glob-dep-misses, and much better incremental performance in the common case that a package's globs haven't changed. See the class-comment for PackageFunction.SkyframeHybridGlobber for a detailed description and explanation. This CL has no impact on semantics and is a strict performance win. Bazel users: Here's an example benchmark (does an incremental loading phase of a target T but forces all packages to be reloaded): nharmata@nharmata:~/bazel$ N=10; B="output/bazel"; T=//src/main/java/com/google/devtools/build/lib:bazel/BazelServer_deploy.jar; CMD="build --noanalyze $T"; $B clean &> /dev/null; P=base_workspace/tools/build_rules/prelude_bazel; rm $P; touch base_workspace/tools/build_rules/BUILD; $B $CMD &> /dev/null; time for i in $(seq 1 $N); do echo "#hi" >> $P; $B $CMD &> /dev/null; done For a very large internal Google target, this CL improves the benchmark performance by ~6%. A more targeted benchmark would be for loading a single package that has lots of expensive globs. For example, the time to incrementally load a single pathological Google-internal package was reduced by ~36%. Alternatives considered: Introduce skyframe native globbing, gated by flags for both globbing during preprocessing and globbing during regular BUILD file evaluation. The approach in this CL is superior performance-wise. -- MOS_MIGRATED_REVID=113899687
* Trim the BuildOptions input of ConfigurationFragment.key to only thoseGravatar Greg Estren2016-02-02
| | | | | | | | | | | | | | | | | | | | | | | | | | options actually needed by the fragment. This protects against, e.g., unnecessarily duplicating CppConfiguration instances when only Java flags change. This is a recommit of ca1b21ac6d8a58041db822725b42de151b163dee which was rolled back because it broke LIPO. This change is particularly important for dynamic configurations, which may mix and match fragments arbitrarily throughout a build. This not only has performance implications, but also correctness implications: code that expects two configured targets to have the same fragment (value) shouldn't break just because the second CT's configuration is a trimmed version of the first's. The original change breaks FDO/LIPO because CppConfiguration can't be shared across configurations. That's because it mutates state when prepareHook() is called, and each configuration calls prepareHook. We should ultimately solve this by refactoring the FDO/LIPO implementation but don't want to block dynamic configuration progress on that. So this change only enables trimming for dynamic configurations. -- MOS_MIGRATED_REVID=113570250
* Move tests from SkyframeLoadingAndAnalysisTest to LoadingPhaseRunnerTest.Gravatar Ulf Adams2016-02-02
| | | | | | | | This exposes an issue with the new Skyframe-based implementation, which is fixed here as well. -- MOS_MIGRATED_REVID=113556169
* Implement proper error handling for interleaved loading and analysis.Gravatar Ulf Adams2016-02-02
| | | | | | | | Add test coverage by re-running BuildViewTest with the new Skyframe loading phase runner. -- MOS_MIGRATED_REVID=113517509
* Store data about aspect configurations on Dependencies.Gravatar Michael Staib2016-01-27
| | | | | | | | | | | | | | | | | | | | Dependencies are the data structure which needs to propagate the configuration for each aspect as created by trimConfigurations down to the point where it's actually used. We need this to store different configurations for different aspects in a world where aspects have their own configurations, which may have more fragments than the target they're attached to. That world is on its way. Also in this CL: * Refactor Dependency to be an abstract parent class with separate implementations for Attribute.Transitions and BuildConfigurations, as well as null configurations, to avoid having to check nullness in various places. Users of the API will not see this, but get factory methods instead of constructors. As a consequence of this, refactor Dependency to be its own top-level class instead of a nested class in DependencyResolver. -- MOS_MIGRATED_REVID=113109615
* Extract parsing of the WORKSPACE file in two partsGravatar Damien Martin-Guillerez2016-01-22
| | | | | | | | | | | The WORKSPACE file AST is now parsed as a separate SkyFunction and this will be used to have multiple SkyValue for the same WORKSPACE file, splitting the execution of the AST after load statements to enable load statement of external dependencies in the WORKSPACE file. -- MOS_MIGRATED_REVID=112768897
* Perform package loading in parallel with transitive target visitation. This ↵Gravatar Janak Ramakrishnan2016-01-21
| | | | | | | is a partial rollback of commit f1e257d because it turns out that loading sequentially can be a bottleneck. -- MOS_MIGRATED_REVID=112628616
* Allow opaque OptionsClassProvider to be passed around in BuildDriver#meta et ↵Gravatar Nathan Harmata2016-01-20
| | | | | | | all so that custom implementations can have custom options passed around. -- MOS_MIGRATED_REVID=112502778
* Use AutoProfiler in SkyQueryEnvironment#init. Also augment the ↵Gravatar Nathan Harmata2016-01-14
| | | | | | | WalkableGraphFactory interface with a hook to inform the factory when the WalkableGraph is done being used. -- MOS_MIGRATED_REVID=112074284
* Add the ability to customize the bazel client's exit code used when the ↵Gravatar Nathan Harmata2016-01-08
| | | | | | | bazel server exits abruptly. -- MOS_MIGRATED_REVID=111641619
* Split PrepareDepsOfTargetsUnderDirectory into two parts, one which does the ↵Gravatar Janak Ramakrishnan2016-01-07
| | | | | | | directory traversal and package loading, and the other which requests deps on all the transitive targets. We need values from the first half, but the second half can fail to evaluate because of a target cycle. By splitting them, we ensure that there will be values in the graph, so we can get the targets below a directory even if there are cycles present. -- MOS_MIGRATED_REVID=111609889
* In SkyQueryEnvironment, don't silently give up when there's a cycle in the ↵Gravatar Janak Ramakrishnan2016-01-07
| | | | | | | graph. We can compute the universe target patterns outside of skyframe, which is the only reason we need the value we were requesting. Giving up was preventing us from evaluating "..." patterns even if the "..." pattern didn't contain any cycles itself. -- MOS_MIGRATED_REVID=111605976
* Use Bazel Preconditions variant which avoids varargs array creationGravatar Mark Schaller2015-12-10
| | | | | | | Reduces garbage. -- MOS_MIGRATED_REVID=109914243
* Rollback of commit ca1b21ac6d8a58041db822725b42de151b163dee.Gravatar Ulf Adams2015-12-08
| | | | | | | | | | | | | | | | | *** Reason for rollback *** Broke some alipo builds. I had to leave in the change to TransitiveTargetFunctionTest, which is relied upon by follow-up CLs. *** Original change description *** Pre-trim build options for all remaining calls to ConfigurationFragment.key. Move the trimming logic into key() itself to: a) eliminate code redundancy b) guarantee all future calls to key() also do this. -- MOS_MIGRATED_REVID=109683849
* Don't treat external files as immutableGravatar Kristina Chodorow2015-12-08
| | | | | | | | | Fixes #352. RELNOTES: Files in external repositories are now treated as mutable, which will make the correctness guarantees of using external repositories stronger (existent), but may cause performance penalties. -- MOS_MIGRATED_REVID=109676408
* Do not recurse into the convenience symlinks when evaluating the "//..." ↵Gravatar Lukacs Berki2015-11-20
| | | | | | | | | target pattern. Apart from the tests, I also tested this manually running "bazel query //..." in a tree with convenience symlinks. -- MOS_MIGRATED_REVID=108325454
* Allow for a set of known modified files to be passed into the ↵Gravatar Eric Fellheimer2015-11-18
| | | | | | | FileSystemValueChecker when checking for dirty actions. -- MOS_MIGRATED_REVID=108046467
* Track the entire OutputService instead of just the BatchStatter.Gravatar Eric Fellheimer2015-11-16
| | | | | -- MOS_MIGRATED_REVID=107800790
* Use Labels, rather than PathFragments, to represent Skylark loads ↵Gravatar John Field2015-11-13
| | | | | | | | | | | | | | | | | | | | | internally. The load location for a Skylark Aspect is specified via a PathFragment, for consistency with current non-Aspect Skylark loads. This should be a semantics-preserving change for users. In a subsequent CL, I'll change the Skylark syntax to allow load statements to use labels as well as paths, with the goal of eventually deprecating the latter. Also: - Removed the hack for handling relative loads in the prelude file. - Refactored some redundant functionality in PackageFunction and SkylarkImportLookupFunction for handling loads. - Removed the ability to put the BUILD file for the package containing a Skylark file under a different package root than the Skylark file itself. This functionality isn't currently used and is inconsistent with Blaze's handling of the package path elsewhere. - Added BUILD files to a number of tests that load Skylark files; this is consistent with the requirement that all Skylark files need to be part of some package. - Changed the constants used to set the location of the prelude file from paths to labels. -- MOS_MIGRATED_REVID=107741568
* When getting package roots for exec paths, don't start with the file, which ↵Gravatar Janak Ramakrishnan2015-11-13
| | | | | | | is guaranteed not to be a package. Instead, start with its parent. This will be faster and take less memory. -- MOS_MIGRATED_REVID=107725767
* Pre-trim build options for all remaining calls to ConfigurationFragment.key. ↵Gravatar Greg Estren2015-11-13
| | | | | | | | | | | | Move the trimming logic into key() itself to: a) eliminate code redundancy b) guarantee all future calls to key() also do this. -- MOS_MIGRATED_REVID=107713353
* Allow package blacklisting to be done via a file checked into the depot. By ↵Gravatar Eric Fellheimer2015-11-12
| | | | | | | default this is disabled. -- MOS_MIGRATED_REVID=107644420
* Allow FilesystemValueChecker to operate on a WalkableGraph and add TODOs for ↵Gravatar Nathan Harmata2015-11-10
| | | | | | | | | replacing MemoizingEvaluator#getValues et al with WalkableGraph usage. I initially attempted to do this but punted once I realized how much work it would be. Also make DirectoryListingStateValue and FileStateValue public for use in outside callers of FilesystemValueChecker. -- MOS_MIGRATED_REVID=107447425
* Add a Skyframe injected value representing a Blacklist of package patterns. ↵Gravatar Eric Fellheimer2015-11-10
| | | | | | | | | | | | | Any Package name starting with a blacklisted prefix is treated as a deleted package. Note the relation to prior art in Blaze: 1. Deleted packages indicates to Blaze to pretend certain packages do not exist. The deleted packages are passed in as fully resolved package names. This is important because the invalidation logic when these values changes references the exact package names, as opposed to prefix patterns. 2. Negative patterns in target parsing. (eg, "foo/..., -foo/bar/..."). Note that if //foo:biz depends on //foo/bar:bang, we still visit that dependency. In other words, the negative patterns just control the target pattern parsing, but not the later evaluation of those targets. Blacklisting acts like both (1) and (2), but in the form of a Precomputed injected value (so modifying the value would be more expensive than modifying deleted_packages). -- MOS_MIGRATED_REVID=107431993
* Aspect terminology update.Gravatar Dmitry Lomov2015-11-10
| | | | | | | | Aspect => ConfiguredAspect AspectWithParameters => Aspect -- MOS_MIGRATED_REVID=107375211
* Refactor the loading phase runner creation and remove the test duplication.Gravatar Ulf Adams2015-11-06
| | | | | | | | | The loading phase runner is now always created by the SkyframeExecutor, and the duplicate test is dropped in favor of subclassing and overriding one method. -- MOS_MIGRATED_REVID=107188756
* Implement the offset for the new skyframe-based LoadingPhaseRunner.Gravatar Ulf Adams2015-11-05
| | | | | -- MOS_MIGRATED_REVID=107071400
* Increase the concurrency level of the filesystem call caches used in legacy ↵Gravatar Nathan Harmata2015-11-05
| | | | | | | package loading from the LoadingCache default of 4 to value of --legacy_globbing_threads. -- MOS_MIGRATED_REVID=107056166
* Post preliminary events so it doesn't crash outright.Gravatar Ulf Adams2015-11-04
| | | | | | | | This makes the new loading phase runner work for basic builds, but there are still a lot of broken tests if enabled by default. -- MOS_MIGRATED_REVID=106952054
* Add a skyframe-based loading phase runner hidden behind a compile-time const.Gravatar Ulf Adams2015-11-03
| | | | | | | More testing required before we even add a command-line option. -- MOS_MIGRATED_REVID=106922350
* Parametrize aspect definition with AspectParameters.Gravatar Dmitry Lomov2015-11-02
| | | | | -- MOS_MIGRATED_REVID=106848269
* Implement propagation along dependencies for Skylark aspects.Gravatar Dmitry Lomov2015-11-02
| | | | | -- MOS_MIGRATED_REVID=106694515
* Introduce Path#isSpecialFile, FileSystem#isSpecialFile, and ↵Gravatar Nathan Harmata2015-10-21
| | | | | | | FileStatus#isSpecialFile to help disambiguate between a regular file and a special file, since the file size of a special file cannot be trusted. -- MOS_MIGRATED_REVID=105903622
generated by cgit v1.2.3 (git 2.39.1) at 2024-06-30 03:13:33 +0000