aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/skyframe/SequencedSkyframeExecutor.java
Commit message (Collapse)AuthorAge
* Get rid of the last reference to getAttributeMapper by keeping packages whichGravatar mjhalupka2018-02-13
| | | | | | are referenced by TopLevelTargets when we discard the analysis cache. PiperOrigin-RevId: 185574670
* Remove references to getTarget(). Requires threading through anGravatar mjhalupka2018-02-08
| | | | | | | ExtendedEventHandler so we can get the target via the package manager during rule dumps. PiperOrigin-RevId: 185042522
* Add option to optionally wipe state at the end of a build.Gravatar ccalvarin2018-01-22
| | | | | | | This will serve as an alternative to --batch, leaving behind a server without state from the previous build. RELNOTES: Introduces --[no]keep_state_after_build PiperOrigin-RevId: 182778500
* Introduce Root class.Gravatar tomlu2018-01-17
| | | | | | | | | | | This class represents a root (such as a package path or an output root) used for file lookups and artifacts. It is meant to be as opaque as possible in order to hide the user's environment from sky keys and sky functions. Roots are used by RootedPaths and ArtifactRoots. This CL attempts to make the minimum number of modifications necessary to change RootedPath and ArtifactRoot to use these fields. Deprecated methods and invasive accessors are permitted to minimise the risk of any observable changes. RELNOTES: None PiperOrigin-RevId: 182271759
* Convert ActionLookupKey implementations to directly implement SkyKey, ↵Gravatar janakr2018-01-11
| | | | | | removing the layer of indirection of getting SkyKey out of ActionLookupKey, which uses more memory for no reason. PiperOrigin-RevId: 181658615
* Bump up the max number of changed files we INFO-log to 10.Gravatar nharmata2017-12-22
| | | | | RELNOTES: None PiperOrigin-RevId: 179944277
* Rename --keep_incrementality_data to --track_incremental_state.Gravatar ccalvarin2017-12-14
| | | | | | | | | New name clears the namespace a 2nd flag that will wipe the build graph after the build. The old name would be confusing as it could easily apply to that, and so needs to be more specifically just about tracking state in the first place. The new flag can be clearly separate and about keeping state after the build. Partial roll forward of https://github.com/bazelbuild/bazel/commit/9321316b34767b06c3071b2cf2a4de189874fcce, with fixes to documentation that are still relevant. RELNOTES: Rename --keep_incrementality_data to --track_incremental_state PiperOrigin-RevId: 179078292
* Automated rollback of commit 9321316b34767b06c3071b2cf2a4de189874fcce.Gravatar ccalvarin2017-12-11
| | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Design change, 2 boolean flags instead of 1 enum flag *** Original change description *** Add --incremental_state_retention_strategy This option is intended to replace some of the uses of --batch. It lets users specify that builds should not be incremental, and how eagerly to discard the state that is kept around for incrementality. Note that for both values discard_eargerly and keep_for_life_of_build, the build graph is kept around until the next build. This may change. Will add tests for keep_for_life_of_build in a later change, for now it will warn that that feature is experimen... *** ROLLBACK_OF=178661777 RELNOTES: None. PiperOrigin-RevId: 178681472
* Add --incremental_state_retention_strategyGravatar ccalvarin2017-12-11
| | | | | | | | | This option is intended to replace some of the uses of --batch. It lets users specify that builds should not be incremental, and how eagerly to discard the state that is kept around for incrementality. Note that for both values discard_eargerly and keep_for_life_of_build, the build graph is kept around until the next build. This may change. Will add tests for keep_for_life_of_build in a later change, for now it will warn that that feature is experimental. RELNOTES: --[no]keep_incrementality_data is gone, replaced by the enum-valued --incremental_state_retention_strategy PiperOrigin-RevId: 178661777
* Automated rollback of commit d0f06a6e4b61adc39bf5e1cfbae39501a89dc8e3.Gravatar nharmata2017-12-05
| | | | | RELNOTES: None PiperOrigin-RevId: 177965330
* Fix getBuildFiles to not assume BUILD is the name of the build file.Gravatar John Cater2017-12-04
| | | | | | | Fixes #4056. Change-Id: Ia7425c2146f15e9293605ee3da53007805e82275 PiperOrigin-RevId: 177813070
* Add ActionKeyContext to Action#getKey.Gravatar tomlu2017-11-29
| | | | | | | This key context can be used by actions to share partial key computations, for instance when computing MD5s for nested sets. RELNOTES: None PiperOrigin-RevId: 177359607
* Remove the concept of a mandatory input which is allowed to be missing.Gravatar Googler2017-11-29
| | | | | RELNOTES: None. PiperOrigin-RevId: 177332323
* Update PathPackageLocator to take a list of potential build file names,Gravatar John Cater2017-11-28
| | | | | | | | | | | | | | instead of assuming BUILD. - Default the list to the same value as PackageLookupFunction: BUILD.bazel, BUILD. - Move BuildFileNames to the packages package, so it is more generally available. Part of #4056. Change-Id: Ie12512b492cd7d47a9e56ec3bc209f829feaf4b5 PiperOrigin-RevId: 177261295
* Automated rollback of commit b64119807b014d9f3b99fb8a02e22daf1a8299b6.Gravatar nharmata2017-11-28
| | | | | | | | | | | | | *** Reason for rollback *** Tickles the shell integration test version of b/35042288 *** Original change description *** Change BlacklistedPackagesPrefixesFunction to take a pair of a hardcoded set of directories and a file path containing more directories to blacklist. The current usage of PrecomputedValue#BLACKLISTED_PACKAGE_PREFIXES_FILE is overly general and is only meaningfully used by unit tests; in practice, the blacklist file path can never change over the lifetime of the Bazel server. Perform a minor simplifying refactor as a result of this. RELNOTES: None. PiperOrigin-RevId: 177176068
* Change BlacklistedPackagesPrefixesFunction to take a pair of a hardcoded set ↵Gravatar nharmata2017-11-28
| | | | | | | of directories and a file path containing more directories to blacklist. The current usage of PrecomputedValue#BLACKLISTED_PACKAGE_PREFIXES_FILE is overly general and is only meaningfully used by unit tests; in practice, the blacklist file path can never change over the lifetime of the Bazel server. Perform a minor simplifying refactor as a result of this. RELNOTES: None. PiperOrigin-RevId: 177164057
* RELNOTES: --keep_incrementality_data flag allows Bazel servers to be run in ↵Gravatar janakr2017-11-12
| | | | | | | | | | | | | | memory-saving non-incremental mode independent of --batch and --discard_analysis_cache. A command run with --nokeep_incrementality_data will discard data that would be needed for incremental builds. Subsequent commands can be sent to the same server, but they will not get the benefit of incrementality from this command. However, if --keep_incrementality_data is specified on a subsequent command, the commands after that will get the benefits of incrementality. There are two benefits to not being dependent on --batch. First, this allows Bazel servers to be run in extreme memory-saving mode without the startup penalties (JVM startup, JITting) that --batch execution imposes. Second, this allows Bazel developers to inspect the state of a Bazel server after an extreme memory-saving build. In order to avoid discarding data unnecessarily (for instance, on a "bazel info used-heap-size-after-gc" or "bazel dump --skyframe=summary") the actual resetting of the graph is done lazily, right before its use in SequencedSkyframeExecutor#sync. This is morally a partial rollback of https://github.com/bazelbuild/bazel/commit/98cd82cbdcac7c48164a611c5a9aa8fc2f1720ef. For now, our tests specify all of the flags. After this change sticks, I plan to get rid of the --batch flag from these tests, which should allow for some clean-ups. Eventually --batch and --discard_analysis_cache may not imply that we don't keep incremental state: we can require that it be specified explicitly. PiperOrigin-RevId: 175335075
* Replace all usages of Blaze's Preconditions class with guava.Gravatar tomlu2017-11-09
| | | | | | | | Blaze had its own class to avoid GC from varargs array creation for the precondition happy path. Guava now (mostly) implements these, making it unnecessary to maintain our own. This change was almost entirely automated by search-and-replace. A few BUILD files needed fixing up since I removed an export of preconditions from lib:util, which was all done by add_deps. There was one incorrect usage of Preconditions that was caught by error prone (which checks Guava's version of Preconditions) that I had to change manually. PiperOrigin-RevId: 175033526
* Don't require --keep_going to discard graph edges. It's unnecessary.Gravatar janakr2017-11-02
| | | | PiperOrigin-RevId: 174202685
* Thread FileSystem through to a single Path#delete call site.Gravatar tomlu2017-10-30
| | | | | | This requires a fairly large amount of changes to fundamental objects like BlazeRuntime, Executor, and so on, as well as changing a lot of test code to thread the file system through. I expect future CLs to be much smaller. PiperOrigin-RevId: 173678144
* Reset evaluator eagerly when incremental state changes. And add a regression ↵Gravatar janakr2017-10-24
| | | | | | test to make sure we are using the expected type of node entries when discarding/keeping graph edges. PiperOrigin-RevId: 173131307
* Add memory profiler.Gravatar tomlu2017-10-18
| | | | | | | | | | | This adds two dump command, bazel dump --rules and bazel dump --skylark_memory. dump --rules outputs a summary of the count, action count, and memory consumption of each rule and aspect class. dump --skylark_memory outputs a pprof-compatible file with all Skylark analysis allocations. Users can then use pprof as per normal to analyse their builds. RELNOTES: Add memory profiler. PiperOrigin-RevId: 172558600
* Pull clearing of legacy configured target data up to SkyframeExecutor and ↵Gravatar janakr2017-10-10
| | | | | | rename method to better reflect what it is primarily doing. PiperOrigin-RevId: 171621356
* Allow NodeEntry implementations to keep just deps, as opposed to all edges ↵Gravatar janakr2017-10-10
| | | | | | or no edges. Also add option to disable checks in MemoizingEvaluatorTest that don't make sense for implementations that don't keep track of dirty nodes. Also extract RecordingDifferencer to an interface. And add a test for the situation that a node changes during a build that it's not requested, and which fails, necessitating cleanup. PiperOrigin-RevId: 171616817
* Move SkylarkSemanticsOptions to the packages/ directory, alongside ↵Gravatar brandjon2017-10-07
| | | | | | | | | SkylarkSemanticsCodec Note that the syntax package and its test package still depend indirectly on the options parser via other Bazel-specific packages. RELNOTES: None PiperOrigin-RevId: 171342823
* More narrowly specify SkyframeExecutor#prepareExecution as ↵Gravatar janakr2017-09-28
| | | | | | #detectModifiedOutputFiles and move implementation to SequencedSkyframeExecutor. PiperOrigin-RevId: 170230031
* Stop injecting embedded artifacts. They're never used by anyone in Skyframe, ↵Gravatar janakr2017-09-26
| | | | | | and error-checking for their existence is already done by the client. PiperOrigin-RevId: 169966701
* Remove unused "extraPrecomputedValues" from WorkspaceBuilder, and therefore ↵Gravatar janakr2017-09-25
| | | | | | | | remove all "precomputed values for analysis" from SkyframeExecutor. We use SkyframeExecutor#injectExtraPrecomputedValues these days. This simplifies logic around when to inject precomputed values, and makes the graph more consistent. PiperOrigin-RevId: 169733304
* Remove product name precomputed value.Gravatar janakr2017-09-25
| | | | PiperOrigin-RevId: 169723724
* Rename all logger instances to "logger" (instead "LOG" or "log").Gravatar lberki2017-09-05
| | | | | RELNOTES: None. PiperOrigin-RevId: 167505493
* Reduce to one SequencedSkyframeExecutor#create method. The others were minor ↵Gravatar janakr2017-07-20
| | | | | | convenience methods that didn't (IMO) carry their weight. PiperOrigin-RevId: 162528716
* Make PackageFunction's strategy for handling unreadable BUILD files ↵Gravatar nharmata2017-06-08
| | | | | | | configurable. Add a test for the current behavior of treating an unreadable BUILD file as a package loading error. RELNOTES: None PiperOrigin-RevId: 158314187
* Factor out Bazel-specific ctor constants in SequencedSkyframeExecutor into a ↵Gravatar nharmata2017-06-06
| | | | | | | separate class. They're currently passed in around various callsites in the codebase and tests. RELNOTES: None PiperOrigin-RevId: 158139074
* Replace constants (static final CONSTANT_CASE) declaration type which use ↵Gravatar Googler2017-05-08
| | | | | | | | | | | | the general collection interface (e.g. List) with an immutable type (e.g. ImmutableList). For constant field declarations, you should use the immutable type (such as ImmutableList) instead of the general collection interface type (such as List). This communicates to your callers important semantic guarantees ([] For more info, see:[] Cleanup change automatically generated by error-prone refactoring //third_party/java_src/error_prone/project/core/src/main/java/com/google/errorprone/bugpatterns:MutableConstantField_refactoring on targets //third_party/bazel/... PiperOrigin-RevId: 155305768
* Make SkyKey an interface, and start the migration of not creating SkyKey ↵Gravatar janakr2017-05-04
| | | | | | wrapper objects: for OwnedArtifacts, which are the most numerous during builds, and for Labels for TransitiveTraversalValues, which are the most numerous during queries. PiperOrigin-RevId: 154989520
* Store Skylark command-line flags in SkyframeGravatar brandjon2017-04-28
| | | | | | | | | This is the first of two CLs for making command line options able to affect the Skylark interpreter. It introduces SkylarkSemanticsOptions, and stores it as a precomputed (injected) value in Skyframe. The next CL will read these options from Skyframe when constructing the Skylark environment. This CL affects the dataflow from command/test initialization to Skyframe. Some code paths, like those used for testing, use the default SkylarkSemanticsOptions and therefore won't be able to use (for example) --incompatible_* flags. The call sites to update were found by searching for uses of defaultVisibility and working upward from there. RELNOTES: None PiperOrigin-RevId: 154432058
* Delete Preprocessor.Factory.SupplierGravatar laurentlb2017-04-12
| | | | | RELNOTES: None. PiperOrigin-RevId: 152882585
* When tracking the critical path, if not keeping incremental state, don't ↵Gravatar janakr2017-04-06
| | | | | | | | keep references to actions indefinitely. Instead, once an action is finished executing, keep just some metadata about it. This allows actions to be unconditionally dropped when running with --batch, --discard_analysis_cache, and --keep_going, even if profiling is enabled. The additional fields here add between 8 and 12 bytes per component, and we have one component per action. This additional penalty is only incurred when we are already saving memory, so I think it's ok. The full penalty will be realized only towards the end of the build, when every action has started executing at least once. Users can still specify --noexperimental_enable_critical_path_profiling if they want to squeeze even more memory out. PiperOrigin-RevId: 152328870
* If --batch, --keep_going, --discard_analysis_cache, and the new ↵Gravatar janakr2017-03-31
| | | | | | | | | | | | | | | --noexperimental_enable_critical_path_profiling flags are all specified, then Bazel will delete Actions from ActionLookupValues as they are executed in order to save memory. Because an already-run action may output an artifact that is only requested later in the build, we need to maintain a way for the artifact to look up the action. But in most cases we don't need to keep the action itself, just its output metadata. Some actions unfortunately are needed post-execution, and so we special-case them. Also includes dependency change with description: Move action out of key. This keeps action references from polluting the graph -- actions are just stored in one SkyValue, instead of being present in SkyKeys. This does mean additional memory used: we have a separate ActionLookupData object per Action executed. That may reach ~24M for million-action builds. PiperOrigin-RevId: 151756383
* Drop loading-phase values if --discard_analysis_cache is true and we're not ↵Gravatar janakr2017-03-31
| | | | | | keeping incremental state. PiperOrigin-RevId: 151639711
* Clear AspectValues when discarding analysis cache, along with ↵Gravatar janakr2017-03-29
| | | | | | | | ConfiguredTargetValues. Also clear transitive packages for both, even for top-level targets. This is not expected to save significant memory, but is expected to reduce the number of references to Packages, allowing them to be dropped more easily when discarding analysis cache and running in batch mode. PiperOrigin-RevId: 151508877
* Provide more reporting options to SkyFunctions Gravatar Klaus Aehlig2017-02-27
| | | | | | | | | | | | | With more specific information to be reported by Skyfunctions, e.g., to inform the build-event protocol on missing files, the EventHandler interface is no longer enough. Therefore, provide an enriched context for reporting events. -- Change-Id: I2d06166fe4d5b9054e24ad8c752fafc039e3f9f8 Reviewed-on: https://cr.bazel.build/8794 PiperOrigin-RevId: 148463437 MOS_MIGRATED_REVID=148463437
* Fix a hypothetical issue with 'buildfiles' and 'loadfiles' with duplicate ↵Gravatar Nathan Harmata2017-02-16
| | | | | | | | | | | | | | | | | | targets. These two functions don't use a Uniquifier internally so if the same bzl file foo.bzl is loaded multiple ways in 'e' in 'loadfiles(e)' then 'f' in 'f(loadfiles(e))' will observe foo.bzl multiple times. This isn't observable in practice (i.e. I cannot write a black-box query test that would fail without this CL) because: (1) BlazeQueryEnvironment#eval(QueryExpression, VariableContext<Target>, Callback<Target>) uses an intermediate aggregating callback and thus doesn't use the streaming query evaluation model. This means that the internal deduping in BlazeQueryEnvironment#getBuildFiles is sufficient. (2) SkyQueryEnvironment uses an outer BatchStreamedCallback which internally uses a Uniquifier. Still, this CL is useful for SkyQuery because without we're doing wasted work (e.g. in my example above, 'f' will wastefully do duplicate work with the .bzl files from the inner loadfiles(e)) because the deduping happens on the final query result, not on the intermediate result from 'buildfiles'/'loadfiles'. For 'buildfiles', there's an additional wart with (1) above because FakeSubincludeTarget doesn't override Object#equals/hashCode. -- PiperOrigin-RevId: 147656183 MOS_MIGRATED_REVID=147656183
* Some minor logging improvements related to diff awareness:Gravatar Nathan Harmata2017-02-15
| | | | | | | | | | | | (i) Log [the first 100] external [non external-repo] file paths when we encounter them in skyframe [when externalFileAction is not ASSUME_NON_EXISTENT_AND_IMMUTABLE_FOR_EXTERNAL_PATHS]. These caveats are to prevent log spam. (ii) Fix grammar when logging the results of handleDiffsWithMissingDiffInformation in the case when all package paths have known diff information and we're merely checking external/output/external-repo files. (iii) Log the purpose of each skyframe graph scan we do (see (ii)). -- PiperOrigin-RevId: 147508404 MOS_MIGRATED_REVID=147508404
* Inject the changes in the client environment for the ClientEnvironmentFunction Gravatar Damien Martin-Guillerez2017-02-02
| | | | | | | | | | | | Using a precomputed values prevent us from being able to depends on any environment variable (or we would invalidate the whole graph each time the environment change, that is each time we run a command). -- Change-Id: If322df4240271d04b9278e1222f936156531580c Reviewed-on: https://cr.bazel.build/8400 PiperOrigin-RevId: 146298592 MOS_MIGRATED_REVID=146298592
* Cache all previously computed values in ↵Gravatar Janak Ramakrishnan2016-11-17
| | | | | | | | | SkyQueryEnvironment#beforeEvaluateQuery if possible to save on latency for small queries. This assumes that if the graph is up to date, then the data in SkyQueryEnvironment is also up to date. It also assumes that a ForkJoinPool remains usable until #shutdownNow is called. -- MOS_MIGRATED_REVID=139386363
* Cache PerBuildSyscallCache if concurrency level is unchanged.Gravatar Janak Ramakrishnan2016-11-14
| | | | | -- MOS_MIGRATED_REVID=138893829
* Update package lookup to check for files named BUILD.bazel before files namedGravatar John Cater2016-11-11
| | | | | | | | | | | BUILD. Fixes #552. RELNOTES[NEW]: Packages are defined in BUILD.bazel as well as BUILD files. -- MOS_MIGRATED_REVID=138828981
* Rollback of commit b776d6c12e952eb358c1a036cc9d93d8944c4c77.Gravatar Nathan Harmata2016-10-18
| | | | | | | *** Reason for rollback *** -- MOS_MIGRATED_REVID=136448178
* Rollback of commit 0c7a42a09d85ddffd9b860bcb31e4c43a00632c1.Gravatar Nathan Harmata2016-10-17
| | | | | | | | | | | | | | | | | | *** Reason for rollback *** [] *** Original change description *** Slight refactor of ExternalFilesHelper: -Make FileType and ExternalFileAction public. -Have producers use ExternalFileAction, rather than a boolean, to specify the desired behavior. And a big change in semantics (doesn't affect Bazel): -Replace ExternalFileAction.ERROR_OUT with ExternalFileAction.ASSUME_NON_EXISTENT_AND_IMMUTABLE, which does what it sounds like. This new action, like the old ERROR_OUT, is _not_ used in Bazel. -- MOS_MIGRATED_REVID=136206810
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-08 10:36:47 +0000