aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/skyframe/SequencedSkyframeExecutor.java
Commit message (Collapse)AuthorAge
* Reduce to one SequencedSkyframeExecutor#create method. The others were minor ↵Gravatar janakr2017-07-20
| | | | | | convenience methods that didn't (IMO) carry their weight. PiperOrigin-RevId: 162528716
* Make PackageFunction's strategy for handling unreadable BUILD files ↵Gravatar nharmata2017-06-08
| | | | | | | configurable. Add a test for the current behavior of treating an unreadable BUILD file as a package loading error. RELNOTES: None PiperOrigin-RevId: 158314187
* Factor out Bazel-specific ctor constants in SequencedSkyframeExecutor into a ↵Gravatar nharmata2017-06-06
| | | | | | | separate class. They're currently passed in around various callsites in the codebase and tests. RELNOTES: None PiperOrigin-RevId: 158139074
* Replace constants (static final CONSTANT_CASE) declaration type which use ↵Gravatar Googler2017-05-08
| | | | | | | | | | | | the general collection interface (e.g. List) with an immutable type (e.g. ImmutableList). For constant field declarations, you should use the immutable type (such as ImmutableList) instead of the general collection interface type (such as List). This communicates to your callers important semantic guarantees ([] For more info, see:[] Cleanup change automatically generated by error-prone refactoring //third_party/java_src/error_prone/project/core/src/main/java/com/google/errorprone/bugpatterns:MutableConstantField_refactoring on targets //third_party/bazel/... PiperOrigin-RevId: 155305768
* Make SkyKey an interface, and start the migration of not creating SkyKey ↵Gravatar janakr2017-05-04
| | | | | | wrapper objects: for OwnedArtifacts, which are the most numerous during builds, and for Labels for TransitiveTraversalValues, which are the most numerous during queries. PiperOrigin-RevId: 154989520
* Store Skylark command-line flags in SkyframeGravatar brandjon2017-04-28
| | | | | | | | | This is the first of two CLs for making command line options able to affect the Skylark interpreter. It introduces SkylarkSemanticsOptions, and stores it as a precomputed (injected) value in Skyframe. The next CL will read these options from Skyframe when constructing the Skylark environment. This CL affects the dataflow from command/test initialization to Skyframe. Some code paths, like those used for testing, use the default SkylarkSemanticsOptions and therefore won't be able to use (for example) --incompatible_* flags. The call sites to update were found by searching for uses of defaultVisibility and working upward from there. RELNOTES: None PiperOrigin-RevId: 154432058
* Delete Preprocessor.Factory.SupplierGravatar laurentlb2017-04-12
| | | | | RELNOTES: None. PiperOrigin-RevId: 152882585
* When tracking the critical path, if not keeping incremental state, don't ↵Gravatar janakr2017-04-06
| | | | | | | | keep references to actions indefinitely. Instead, once an action is finished executing, keep just some metadata about it. This allows actions to be unconditionally dropped when running with --batch, --discard_analysis_cache, and --keep_going, even if profiling is enabled. The additional fields here add between 8 and 12 bytes per component, and we have one component per action. This additional penalty is only incurred when we are already saving memory, so I think it's ok. The full penalty will be realized only towards the end of the build, when every action has started executing at least once. Users can still specify --noexperimental_enable_critical_path_profiling if they want to squeeze even more memory out. PiperOrigin-RevId: 152328870
* If --batch, --keep_going, --discard_analysis_cache, and the new ↵Gravatar janakr2017-03-31
| | | | | | | | | | | | | | | --noexperimental_enable_critical_path_profiling flags are all specified, then Bazel will delete Actions from ActionLookupValues as they are executed in order to save memory. Because an already-run action may output an artifact that is only requested later in the build, we need to maintain a way for the artifact to look up the action. But in most cases we don't need to keep the action itself, just its output metadata. Some actions unfortunately are needed post-execution, and so we special-case them. Also includes dependency change with description: Move action out of key. This keeps action references from polluting the graph -- actions are just stored in one SkyValue, instead of being present in SkyKeys. This does mean additional memory used: we have a separate ActionLookupData object per Action executed. That may reach ~24M for million-action builds. PiperOrigin-RevId: 151756383
* Drop loading-phase values if --discard_analysis_cache is true and we're not ↵Gravatar janakr2017-03-31
| | | | | | keeping incremental state. PiperOrigin-RevId: 151639711
* Clear AspectValues when discarding analysis cache, along with ↵Gravatar janakr2017-03-29
| | | | | | | | ConfiguredTargetValues. Also clear transitive packages for both, even for top-level targets. This is not expected to save significant memory, but is expected to reduce the number of references to Packages, allowing them to be dropped more easily when discarding analysis cache and running in batch mode. PiperOrigin-RevId: 151508877
* Provide more reporting options to SkyFunctions Gravatar Klaus Aehlig2017-02-27
| | | | | | | | | | | | | With more specific information to be reported by Skyfunctions, e.g., to inform the build-event protocol on missing files, the EventHandler interface is no longer enough. Therefore, provide an enriched context for reporting events. -- Change-Id: I2d06166fe4d5b9054e24ad8c752fafc039e3f9f8 Reviewed-on: https://cr.bazel.build/8794 PiperOrigin-RevId: 148463437 MOS_MIGRATED_REVID=148463437
* Fix a hypothetical issue with 'buildfiles' and 'loadfiles' with duplicate ↵Gravatar Nathan Harmata2017-02-16
| | | | | | | | | | | | | | | | | | targets. These two functions don't use a Uniquifier internally so if the same bzl file foo.bzl is loaded multiple ways in 'e' in 'loadfiles(e)' then 'f' in 'f(loadfiles(e))' will observe foo.bzl multiple times. This isn't observable in practice (i.e. I cannot write a black-box query test that would fail without this CL) because: (1) BlazeQueryEnvironment#eval(QueryExpression, VariableContext<Target>, Callback<Target>) uses an intermediate aggregating callback and thus doesn't use the streaming query evaluation model. This means that the internal deduping in BlazeQueryEnvironment#getBuildFiles is sufficient. (2) SkyQueryEnvironment uses an outer BatchStreamedCallback which internally uses a Uniquifier. Still, this CL is useful for SkyQuery because without we're doing wasted work (e.g. in my example above, 'f' will wastefully do duplicate work with the .bzl files from the inner loadfiles(e)) because the deduping happens on the final query result, not on the intermediate result from 'buildfiles'/'loadfiles'. For 'buildfiles', there's an additional wart with (1) above because FakeSubincludeTarget doesn't override Object#equals/hashCode. -- PiperOrigin-RevId: 147656183 MOS_MIGRATED_REVID=147656183
* Some minor logging improvements related to diff awareness:Gravatar Nathan Harmata2017-02-15
| | | | | | | | | | | | (i) Log [the first 100] external [non external-repo] file paths when we encounter them in skyframe [when externalFileAction is not ASSUME_NON_EXISTENT_AND_IMMUTABLE_FOR_EXTERNAL_PATHS]. These caveats are to prevent log spam. (ii) Fix grammar when logging the results of handleDiffsWithMissingDiffInformation in the case when all package paths have known diff information and we're merely checking external/output/external-repo files. (iii) Log the purpose of each skyframe graph scan we do (see (ii)). -- PiperOrigin-RevId: 147508404 MOS_MIGRATED_REVID=147508404
* Inject the changes in the client environment for the ClientEnvironmentFunction Gravatar Damien Martin-Guillerez2017-02-02
| | | | | | | | | | | | Using a precomputed values prevent us from being able to depends on any environment variable (or we would invalidate the whole graph each time the environment change, that is each time we run a command). -- Change-Id: If322df4240271d04b9278e1222f936156531580c Reviewed-on: https://cr.bazel.build/8400 PiperOrigin-RevId: 146298592 MOS_MIGRATED_REVID=146298592
* Cache all previously computed values in ↵Gravatar Janak Ramakrishnan2016-11-17
| | | | | | | | | SkyQueryEnvironment#beforeEvaluateQuery if possible to save on latency for small queries. This assumes that if the graph is up to date, then the data in SkyQueryEnvironment is also up to date. It also assumes that a ForkJoinPool remains usable until #shutdownNow is called. -- MOS_MIGRATED_REVID=139386363
* Cache PerBuildSyscallCache if concurrency level is unchanged.Gravatar Janak Ramakrishnan2016-11-14
| | | | | -- MOS_MIGRATED_REVID=138893829
* Update package lookup to check for files named BUILD.bazel before files namedGravatar John Cater2016-11-11
| | | | | | | | | | | BUILD. Fixes #552. RELNOTES[NEW]: Packages are defined in BUILD.bazel as well as BUILD files. -- MOS_MIGRATED_REVID=138828981
* Rollback of commit b776d6c12e952eb358c1a036cc9d93d8944c4c77.Gravatar Nathan Harmata2016-10-18
| | | | | | | *** Reason for rollback *** -- MOS_MIGRATED_REVID=136448178
* Rollback of commit 0c7a42a09d85ddffd9b860bcb31e4c43a00632c1.Gravatar Nathan Harmata2016-10-17
| | | | | | | | | | | | | | | | | | *** Reason for rollback *** [] *** Original change description *** Slight refactor of ExternalFilesHelper: -Make FileType and ExternalFileAction public. -Have producers use ExternalFileAction, rather than a boolean, to specify the desired behavior. And a big change in semantics (doesn't affect Bazel): -Replace ExternalFileAction.ERROR_OUT with ExternalFileAction.ASSUME_NON_EXISTENT_AND_IMMUTABLE, which does what it sounds like. This new action, like the old ERROR_OUT, is _not_ used in Bazel. -- MOS_MIGRATED_REVID=136206810
* Make --watchfs a common command option.Gravatar Ulf Adams2016-10-14
| | | | | | | | | | | | | | | | | | | | | | | | | Adding an options parameter to DiffAwareness#getCurrentView seems like the simplest way to achieve that. Alternatives considered: 1. Making the diff awareness modules stateful. However, I did not want to do so as I've also been working on improving the module API to reduce state, or at least to have a proper lifecycle management for any necessary state. 2. Making the watchFs flag a constructor parameter. However, that would also invalidate any implementations that don't use the flag (of which we have several). 3. Only passing in a single boolean flag instead of an options class provider; however, this is a more principled, futureproof API, which allows other modules / awareness implementations to use their own options. RELNOTES: --watchfs is now a command option; the startup option of the same name is deprecated. I.e., use bazel build --watchfs, not blaze --watchfs build. -- MOS_MIGRATED_REVID=136154395
* Rollback of commit 2891ec527eed27d0c6460d66f51cb66a43373b6a.Gravatar Damien Martin-Guillerez2016-10-14
| | | | | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Causes our integration tests on Darwin to time out *** Original change description *** Make --watchfs a common command option. Adding an options parameter to DiffAwareness#getCurrentView seems like the simplest way to achieve that. Alternatives considered: 1. Making the diff awareness modules stateful. However, I did not want to do so as I've also been working on improving the module API to reduce state, or at least to have a proper lifecycle management for any necessary state. 2. Making the watchFs flag a constructor parameter. However, that would also invalidate any implement... *** -- MOS_MIGRATED_REVID=136070807
* Slight refactor of ExternalFilesHelper:Gravatar Nathan Harmata2016-10-14
| | | | | | | | | | | -Make FileType and ExternalFileAction public. -Have producers use ExternalFileAction, rather than a boolean, to specify the desired behavior. And a big change in semantics (doesn't affect Bazel): -Replace ExternalFileAction.ERROR_OUT with ExternalFileAction.ASSUME_NON_EXISTENT_AND_IMMUTABLE, which does what it sounds like. This new action, like the old ERROR_OUT, is _not_ used in Bazel. -- MOS_MIGRATED_REVID=136063159
* Make --watchfs a common command option.Gravatar Ulf Adams2016-10-13
| | | | | | | | | | | | | | | | | | | | | | | | | Adding an options parameter to DiffAwareness#getCurrentView seems like the simplest way to achieve that. Alternatives considered: 1. Making the diff awareness modules stateful. However, I did not want to do so as I've also been working on improving the module API to reduce state, or at least to have a proper lifecycle management for any necessary state. 2. Making the watchFs flag a constructor parameter. However, that would also invalidate any implementations that don't use the flag (of which we have several). 3. Only passing in a single boolean flag instead of an options class provider; however, this is a more principled, futureproof API, which allows other modules / awareness implementations to use their own options. RELNOTES: --watchfs is now a command option; the startup option of the same name is deprecated. I.e., use bazel build --watchfs, not blaze --watchfs build. -- MOS_MIGRATED_REVID=136026835
* Adds an enum to describe how PackageLookupFunction should handle package labelsGravatar John Cater2016-10-13
| | | | | | | which cross into a sub-repository. Part of #1592. -- MOS_MIGRATED_REVID=135931868
* Track client environment in SkyframeGravatar Klaus Aehlig2016-09-14
| | | | | | | | | | | | | ...to determine which actions have to be recomputed based on changes to the client environment. Note that this change does it the simple way and reconsideres all actions on a changed client environment, while still only reexecuting those, where the part that was inherited from the environment actually did change. -- Change-Id: Ie1116d094642165e5e959447a6fcf49d19b37d6e Reviewed-on: https://bazel-review.googlesource.com/#/c/5431 MOS_MIGRATED_REVID=133010705
* Make --experimental_interleave_loading_and_analysis a no-op.Gravatar Ulf Adams2016-09-06
| | | | | | | | | | | | This is now enabled by default, and this change removes the code path where it's disabled. Remove a few tests that were testing the removed code, and rewrite some others that still seem useful. We still drop configured targets on configuration changes, so we use that to check that things are dropped from Skyframe. -- MOS_MIGRATED_REVID=132226208
* Replace the occurrences of Constants.PRODUCT_NAME for a call toGravatar Luis Fernando Pino Duque2016-05-23
| | | | | | | | | BlazeRuntime#getProductName() or a reference to TestConstants.PRODUCT_NAME for tests. This CL prepares the codebase in order to delete the constant. -- MOS_MIGRATED_REVID=122993568
* Fix SkyQuery bug where we weren't respecting the package blacklist. We do ↵Gravatar Nathan Harmata2016-04-18
| | | | | | | | | this by changing both the relevant Skyframe and the SkyQuery code to propagate (minimal!) blacklist information in the SkyKeys themselves. There are other approaches to solving this problem, but I like how this solution doesn't involve duplication of logic. Also, it has the following nice benefit: previously, RecursiveDirectoryTraversalFunction would declare a dep on the blacklist for every directory traversed which adds an edge for each directory traversed. -- MOS_MIGRATED_REVID=120049635
* Fix compiler warning due to lack of generic parameter.Gravatar Nathan Harmata2016-04-15
| | | | | -- MOS_MIGRATED_REVID=119920313
* Respect --noexperimental_check_output_files in FileSystemValueChecker. ↵Gravatar Nathan Harmata2016-03-25
| | | | | | | | | FileStateValues for output files can make their way into the Skyframe graph if a source file is symlink to an output file. Also fix a bug where ExternalFilesHelper#isExternalFileSeen would always return true after returning true once in the past. This meant if an external file ever made its way into the Skyframe graph, we would always do a full graph scan at the beginning of each build (iow, we would always waste some CPU time doing nothing interesting). -- MOS_MIGRATED_REVID=118190190
* Move TimestampGranularityMonitor to CommandEnvironment.Gravatar Ulf Adams2016-03-23
| | | | | | | | This is one of the last pieces of state in BlazeRuntime that isn't safe to share across command invocations. -- MOS_MIGRATED_REVID=117910631
* Add an intermediate SkyFunction for resolving the external packageGravatar Damien Martin-Guillerez2016-02-07
| | | | | | | | | | | | | | The WORKSPACE file parsing needs to be separated into several parts to enable load of labels in the WORKSPACE file. This change adds an intermediate SkyFunction, ExternalPackageFunction, that requires all the WORKSPACE file part to be parsed to resolve //external: labels. Issue #824 Step 1. -- MOS_MIGRATED_REVID=113984026
* Use Bazel Preconditions variant which avoids varargs array creationGravatar Mark Schaller2015-12-10
| | | | | | | Reduces garbage. -- MOS_MIGRATED_REVID=109914243
* Follow-up on f9fdc8dfced8b2b14561720623126a91e04b22cb -- reinstate ↵Gravatar Janak Ramakrishnan2015-12-08
| | | | | | | short-circuit check when all package paths have good diff information and no external files have been seen. -- MOS_MIGRATED_REVID=109703164
* Don't treat external files as immutableGravatar Kristina Chodorow2015-12-08
| | | | | | | | | Fixes #352. RELNOTES: Files in external repositories are now treated as mutable, which will make the correctness guarantees of using external repositories stronger (existent), but may cause performance penalties. -- MOS_MIGRATED_REVID=109676408
* Allow FilesystemValueChecker to operate on a WalkableGraph and add TODOs for ↵Gravatar Nathan Harmata2015-11-10
| | | | | | | | | replacing MemoizingEvaluator#getValues et al with WalkableGraph usage. I initially attempted to do this but punted once I realized how much work it would be. Also make DirectoryListingStateValue and FileStateValue public for use in outside callers of FilesystemValueChecker. -- MOS_MIGRATED_REVID=107447425
* Increase the concurrency level of the filesystem call caches used in legacy ↵Gravatar Nathan Harmata2015-11-05
| | | | | | | package loading from the LoadingCache default of 4 to value of --legacy_globbing_threads. -- MOS_MIGRATED_REVID=107056166
* Set the SkyframeActionExecutor.reporter in prepareForExecution.Gravatar Ulf Adams2015-09-30
| | | | | | | Remove a lot of now-unused Reporter parameters. -- MOS_MIGRATED_REVID=104100061
* Rationalize copyright headersGravatar Damien Martin-Guillerez2015-09-25
| | | | | | | | | | | The headers were modified with `find . -type f -exec 'sed' '-Ei' 's|Copyright 201([45]) Google|Copyright 201\1 The Bazel Authors|' '{}' ';'` And manual edit for not Google owned copyright. Because of the nature of ijar, I did not modified the header of file owned by Alan Donovan. The list of authors were extracted from the git log. It is missing older Google contributors that can be added on-demand. -- MOS_MIGRATED_REVID=103938715
* Remove all direct uses of SkyframeExecutor.reporter.Gravatar Ulf Adams2015-09-24
| | | | | | | | | Instead, pass an appropriate EventHandler instance in. This is in preparation for creating a per-command EventHandler, in preparation for allowing multiple commands to run in parallel. This is removal of shared global state. -- MOS_MIGRATED_REVID=103828963
* Move callUninterruptibly to a helper class in lib.concurrent.Gravatar Ulf Adams2015-09-24
| | | | | -- MOS_MIGRATED_REVID=103747062
* Move ownership of SkyframeBuildView to SkyframeExecutor.Gravatar Ulf Adams2015-09-23
| | | | | | | | Also move ownership of ArtifactFactory to SkyframeBuildView; simplify the code. -- MOS_MIGRATED_REVID=103722228
* Use AutoProfiler in the Bazel codebase.Gravatar Nathan Harmata2015-09-08
| | | | | -- MOS_MIGRATED_REVID=102584924
* Allow for customization of the per-build system call caches used in Skyframe.Gravatar Eric Fellheimer2015-09-01
| | | | | -- MOS_MIGRATED_REVID=101984361
* Move PackageIdentifier to cmdlineGravatar Kristina Chodorow2015-08-31
| | | | | | | | This is necessary to have TargetResolver depend on it without making it depend on the packages target. First step of #389. -- MOS_MIGRATED_REVID=101790345
* Rename some variables missed by a codebase-wide renaming a long time ago.Gravatar Nathan Harmata2015-08-25
| | | | | -- MOS_MIGRATED_REVID=101388809
* When we have enough information to determine that a file didn't change type, ↵Gravatar Nathan Harmata2015-08-13
| | | | | | | don't unnecessarily invalidate DirectoryListingStateValues. -- MOS_MIGRATED_REVID=100511935
* Remove stale comment.Gravatar Nathan Harmata2015-08-04
| | | | | | | RELNOTES: -- MOS_MIGRATED_REVID=99589366
* Allow modules to specify additional nodes in the graph to be invalidated and ↵Gravatar Janak Ramakrishnan2015-07-29
| | | | | | | | | | | use this functionality to properly invalidate http download nodes if the downloaded zip file doesn't match the specified hash. This still means that the actual files downloaded may not match, but checking all such files may be too expensive. This helps with #336 but before that issue can be closed all remote repositories (git, etc.), should have similar functionality added. -- MOS_MIGRATED_REVID=99317085
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-07 22:17:55 +0000