aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/skyframe
Commit message (Collapse)AuthorAge
* Rollback of commit 82d43279f93d95e4c41b4bc598a3cc05ddd1ae1a.Gravatar Laszlo Csomor2016-09-21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Breaks TensorFlow and other Bazel jobs on ci.bazel.io *** Original change description *** Change execution root for external repositories to be ../repo Some of the important aspect of this change: * Remote repos in the execution root are under output_base/execroot/repo_name, so the prefix is ../repo_name (to escape the local workspace name). * Package roots for external repos were previously "output_base/", they are now output_base/external/repo_name (which means source artifacts always have a relative path from their repository). * Outputs are under bazel-bin/external/repo_name/ (or similarly under genfiles). Note that this is a bit of a change from how this was implemented in the previous cl. Fixes #1262. RELNOTES[INC]: Previously, an external repository would be symlinked into the execution root at execroot/local_repo/external/remote_repo. This changes it to be at execroot/remote_repo. This may break genrules/Skylark actions that hardcode execution root paths. If this causes breakages for you, ensure that genrules are using $(location :target) to access files and Skylark rules are using http://bazel.io/docs/skylark/lib/File.html's path, dirname, etc. functions. Roll forward of bdfd58a. -- MOS_MIGRATED_REVID=133709658
* Change execution root for external repositories to be ../repoGravatar Kristina Chodorow2016-09-20
| | | | | | | | | | | | | | | | | | | | | | | Some of the important aspect of this change: * Remote repos in the execution root are under output_base/execroot/repo_name, so the prefix is ../repo_name (to escape the local workspace name). * Package roots for external repos were previously "output_base/", they are now output_base/external/repo_name (which means source artifacts always have a relative path from their repository). * Outputs are under bazel-bin/external/repo_name/ (or similarly under genfiles). Note that this is a bit of a change from how this was implemented in the previous cl. Fixes #1262. RELNOTES[INC]: Previously, an external repository would be symlinked into the execution root at execroot/local_repo/external/remote_repo. This changes it to be at execroot/remote_repo. This may break genrules/Skylark actions that hardcode execution root paths. If this causes breakages for you, ensure that genrules are using $(location :target) to access files and Skylark rules are using http://bazel.io/docs/skylark/lib/File.html's path, dirname, etc. functions. Roll forward of bdfd58a. -- MOS_MIGRATED_REVID=133606309
* Enable aspect invocations to be matched by output filters.Gravatar Michael Staib2016-09-19
| | | | | | | | | | | | | Currently aspects have no tag, i.e., they don't get checked against output filters at all. This changes aspects to have a tag of the same form as the rule they are attached to (i.e., the label of the target). Since the default output filters match on ^//package[:/], this should cover most uses of the output filter, while still allowing for finer control for those who crave it. -- MOS_MIGRATED_REVID=133425215
* Make environment dependency of actions factor through individual valuesGravatar Klaus Aehlig2016-09-15
| | | | | | | | | | | | | | With actions depending on the (white-listed part) of the environment as a whole, even though they are only re-executed if the used parts of the environment change, each action has to be reconsidered on any change of the environment. For large dependency graphs, this can be a considerable amount of effort; therefore add intermediate values for the individual variables and make actions only depend on those actually used. -- Change-Id: I283d289da3e0782dc4f9ac084a41425166cfede0 Reviewed-on: https://bazel-review.googlesource.com/#/c/5494 MOS_MIGRATED_REVID=133255911
* Track client environment in SkyframeGravatar Klaus Aehlig2016-09-14
| | | | | | | | | | | | | ...to determine which actions have to be recomputed based on changes to the client environment. Note that this change does it the simple way and reconsideres all actions on a changed client environment, while still only reexecuting those, where the part that was inherited from the environment actually did change. -- Change-Id: Ie1116d094642165e5e959447a6fcf49d19b37d6e Reviewed-on: https://bazel-review.googlesource.com/#/c/5431 MOS_MIGRATED_REVID=133010705
* Make the code that is using execRoot paths use execRoot pathsGravatar Kristina Chodorow2016-09-09
| | | | | | | | | | | Chipping away at making the big CL for #1262 smaller. Only runfiles paths are different right now, so this makes getPathUnderExecRoot and getSourceRoot return the same thing. Also corrected a couple places where Label.EXTERNAL_PATH_PREFIX and Label.EXTERNAL_PACKAGE_NAME were being used incorrectly. -- MOS_MIGRATED_REVID=132671081
* Add an "is main repo" field to RootsGravatar Kristina Chodorow2016-09-09
| | | | | | | This is part of prepping for #1262. -- MOS_MIGRATED_REVID=132553178
* Make --experimental_interleave_loading_and_analysis a no-op.Gravatar Ulf Adams2016-09-06
| | | | | | | | | | | | This is now enabled by default, and this change removes the code path where it's disabled. Remove a few tests that were testing the removed code, and rewrite some others that still seem useful. We still drop configured targets on configuration changes, so we use that to check that things are dropped from Skyframe. -- MOS_MIGRATED_REVID=132226208
* Do not load the transitive closure of the Android SDK on every build.Gravatar Lukacs Berki2016-08-31
| | | | | | | This makes the sanity check dependent on the configuration fragments returning proper roots, but it's not that bad because it already depends on them returning the proper set of implicit labels and #getImplicitLabels() will go away soon anyway. -- MOS_MIGRATED_REVID=131705535
* Make aspects attached to aliases work when their actual= attribute is a select.Gravatar Lukacs Berki2016-08-31
| | | | | | | Fixes https://github.com/bazelbuild/e4b/issues/6 . -- MOS_MIGRATED_REVID=131698950
* Implement a better #toString() for ConfigurationFragmentKey.Gravatar Lukacs Berki2016-08-29
| | | | | -- MOS_MIGRATED_REVID=131590706
* Remove all references to (incorrect) BlazeDirectories.buildDataDirectory rootGravatar Kristina Chodorow2016-08-26
| | | | | | | | | The buildDataDirectory is calculated off of the incorrect execroot. More progress towards #1681. -- MOS_MIGRATED_REVID=131407798
* Add client environment to ActionExecutionContextGravatar Klaus Aehlig2016-08-26
| | | | | | | | | | | As the execution of an action now also depends on the client environment, make the latter part of the ActionExecutionContext, so that enough context is provided to actually execute an action. -- Change-Id: Ida7bf407ef0c0375728faba92494bfd47dcbaeb8 Reviewed-on: https://bazel-review.googlesource.com/#/c/5391 MOS_MIGRATED_REVID=131377490
* Remove ArtifactFactory dependency on incorrect exec rootGravatar Kristina Chodorow2016-08-25
| | | | | | | | | | | Somewhat trickily, this changes the execRoot field from referring to [output_base]/execroot/wsname (not really the exec root) to [output_base]/execroot (actually the execroot). Progress on #1681. -- MOS_MIGRATED_REVID=131286181
* Action#discoverInputs returns an Iterable<Artifact> instead of aGravatar Cal Peyser2016-08-25
| | | | | | | Collection<Artifact> -- MOS_MIGRATED_REVID=131285541
* Also display the object identity of the configuration in ConfiguredTargetKey ↵Gravatar Lukacs Berki2016-08-25
| | | | | | | in its #toString(). -- MOS_MIGRATED_REVID=131267480
* Add more debugging info in case of action conflicts. This will tell us if ↵Gravatar Janak Ramakrishnan2016-08-25
| | | | | | | we're unwittingly creating duplicate ConfiguredTargetKey objects, as well as duplicate artifacts. -- MOS_MIGRATED_REVID=131220032
* Automated [] rollback of commit 846a5ab98fc26d72024890fdb79a5d3bc6a5a1ba + ↵Gravatar Nathan Harmata2016-08-23
| | | | | | | | | | | | | | | manual rollback of [] *** Reason for rollback *** Depot has been fixed / is in the process of being fixed. See the work tracked on [] *** Original change description *** Automated [] rollback of commit bb5d5efb4b50710241b5b374eb67084f4bf08278. -- MOS_MIGRATED_REVID=131095905
* Make Digest (renamed Md5Digest) a little more multi-purpose.Gravatar Shreya Bhattarai2016-08-23
| | | | | -- MOS_MIGRATED_REVID=130986194
* Rollback of commit bb5d5efb4b50710241b5b374eb67084f4bf08278.Gravatar Tobias Werth2016-08-23
| | | | | -- MOS_MIGRATED_REVID=130941264
* Refactor ActionExecutionFunction to avoid unnecessary unwrapping of SkyKeys ↵Gravatar Janak Ramakrishnan2016-08-22
| | | | | | | to Artifacts. Also avoid overhead of creating a set when we only need an iterable, and avoid an unnecessary iteration over the iterable as a contains check. In future, we may not want to pay the cost of constructing a true map in SkyFunction.Environment implementations. Part of this is a preliminary refactor to allow that. -- MOS_MIGRATED_REVID=130765995
* Refactor GlobFunction to avoid random-access lookup of the returned map of a ↵Gravatar Janak Ramakrishnan2016-08-18
| | | | | | | SkyFunction.Environment#getValues call. In future, we may not want to pay the cost of constructing a true map in SkyFunction.Environment implementations. This is a preliminary refactor to allow that. -- MOS_MIGRATED_REVID=130649663
* Use array comparison and not reference comparison between digests.Gravatar Shreya Bhattarai2016-08-18
| | | | | -- MOS_MIGRATED_REVID=130576075
* RELNOTES: The string list returned by the skylark 'glob' function is now ↵Gravatar Nathan Harmata2016-08-18
| | | | | | | | | | | | | | sorted. Previously, it would return a list formed by concatenating the sorted results of each pattern in the 'includes' list. A bunch of cleanups and one bug fix: -Remove the unused-except-for tests GlobCache#globsUpToDate. This code has been dead for a very very long time, ever since we switched to using Skyframe. -Change the semantics of the 'glob' function as described above. -Change UnixGlob to return unsorted results. Document this in UnixGlob and GlobCache. -Change LegacyGlobber to conditionally return sorted results. Have users other than PackageFunction get sorted results (as described above). Have PackageFunction's use case get completely unsorted results, and have PackageFunction do the sorting itself. -Have PackageFunction's HybridGlobber unconditionally sort the glob result list. This ensure deterministic glob results, fixing a bug where the order of the elements of the result depended on the contents of the Skyframe graph, which of course depends on the sequence of incremental Blaze commands. -- MOS_MIGRATED_REVID=130540152
* Allow SkyframePackageRootResolver's methods to throw interrupts, now that ↵Gravatar Janak Ramakrishnan2016-08-18
| | | | | | | the interface allows it. -- MOS_MIGRATED_REVID=130530262
* Simplify SkyframeExecutor#evaluateSkyKeyForCodeMigration and allow it to ↵Gravatar Janak Ramakrishnan2016-08-17
| | | | | | | throw InterruptedException. -- MOS_MIGRATED_REVID=130424634
* Allow Skyframe graph lookups and value retrievals to throw InterruptedException.Gravatar Janak Ramakrishnan2016-08-16
| | | | | | | The only place we now don't handle InterruptedException is in the action graph created after analysis, since I'm not sure that will be around for that much longer. -- MOS_MIGRATED_REVID=130327770
* Changes DependencyResolver <Attribute, Dep> map from a ListMultimap to new classGravatar Greg Estren2016-08-12
| | | | | | | | | | | | | | | | | | | | | | OrderedSetMultimap. This maintains insertion order while eliminating duplicates. Certain rules, in particular, otherwise break this invariant: https://github.com/bazelbuild/bazel/blo[]e3e28274cca5b87f48abe33884edb84016dd3/src/main/java/com/google/devtools/build/lib/skyframe/ConfiguredTargetFunction.java#L403 There's no reason (to my knowledge) to need multiple instances of the same <Attribute, Dependency> pair. More context from Google code review: (Michael Staib): > There are many things which pass around a dependentNodeMap or help construct one or modify one. We want an interface which has the right guarantees. > ListMultimap is not the right interface because it has no guarantee of unique elements, which we want - we don't want the problem that this CL ran into, and there's no reason (that we know of, to be confirmed) that anyone would want multiple identical Dependencies. > SetMultimap is not the right interface because it has no guarantee of deterministic iteration order or efficient iteration, which we want - dependency order sometimes matters (e.g., Java classpath or C++ link order). > We agreed that the best way to get what we want is to define our own interface with its own simultaneous uniqueness and iterability guarantees. Unspoken in the discussion was why we wouldn't just use LinkedHashMultimap as the thing we pass around. IMO the reason for that is that we don't care that it be a LinkedHashMultimap specifically; if tomorrow Guava comes out with a faster cooler map that has deterministic and efficient iteration and guarantees element uniqueness, we want it. > In this case we're going to make the "interface" be a (final?) class: OrderedSetMultimap, an extension of ForwardingSetMultimap which delegates to LinkedHashMultimap, an implementation which does support both of those guarantees. > I had mentioned in the conversation that none of the Multimap implementations make guarantees about key iteration order, but this is not true - LinkedHashMultimap preserves key insertion order. We should perhaps declare this as part of the OrderedSetMultimap contract as well. -- MOS_MIGRATED_REVID=130037643
* Implements dynamic split transitions on latebound attributes.Gravatar Greg Estren2016-08-10
| | | | | | | | | With this change, dynamic configs are at full feature parity for split transitions (minus some small differences in composed transitions from BuildConfigurationCollection.configurationHook). -- MOS_MIGRATED_REVID=129802414
* Implements dynamic split transitions (minus latebound attribute splits).Gravatar Greg Estren2016-08-08
| | | | | | | | | | With the prereq work behind this, this is surprisingly straightforward. The main change is to eliminate BuildConfiguration.SplittableTransitionApplier, make both DynamicTransitionApplier and StaticTransitionApplier split-aware, and add awareness of this to ConfiguredTargetFunction.trimConfigurations. Latebound splits will follow next. -- MOS_MIGRATED_REVID=129480309
* Make SkylarkClassObject "Bazel-specific".Gravatar Dmitry Lomov2016-08-05
| | | | | | | This in preparation to DeclaredProviders implementation. -- MOS_MIGRATED_REVID=129420617
* Remove static checks from the parser.Gravatar Laurent Le Brun2016-08-04
| | | | | -- MOS_MIGRATED_REVID=129313959
* Late-bound split attribute configs weren't being properly propagated to deps ↵Gravatar Greg Estren2016-08-04
| | | | | | | | | | | | | | | | with dynamic configs. The old code incorrectly applied a no-op instead of hard-setting a specific config. Testing: This is a prereq piece for the main change adding dynamic split transitions. Once we have that change, standard Bazel tests over implemented late-bound split attributes (e.g. AppleBinaryRule: ":cc_toolchain") will provide proper coverage. There's no easy way to test this directly since the affected code won't really work until the dynamic split change is in. -- MOS_MIGRATED_REVID=129278253
* Start removing the parsePython booleanGravatar Laurent Le Brun2016-08-03
| | | | | | | It's still used in one place and should be removed completely in October. -- MOS_MIGRATED_REVID=129207133
* Add an enum representing the specific build file name (WORKSPACE, BUILD) to ↵Gravatar John Cater2016-08-03
| | | | | | | the PackageLookupValue to reduce the number of references to the filename "BUILD". -- MOS_MIGRATED_REVID=129203257
* Provides a clearer message when target analysis fails because its dynamicGravatar Greg Estren2016-08-03
| | | | | | | | | | | | | | | config is missing required fragments. Before this change, Bazel crashes with the mysterious error: "Fragment foo can't load missing options BarOptions" with no details on which target or dep needed Foo. So figuring out the source of the error is painful. With this change, we instead get: //foo:foo: dependency //bar:bar from attribute "deps" is missing required config fragments: JavaConfiguration -- MOS_MIGRATED_REVID=129143764
* Dynamic configs: also include fragments needed to resolve the rule's ↵Gravatar Greg Estren2016-08-03
| | | | | | | | | | | | | | late-bound attributes To be clear: late-bound attributes introduce two new classes of fragments. First is those needed by Attribute.LateBoundDefault to figure out what the attribute's value is. Second is whatever's needed by the actual dep (for label-typed attributes) and its deps. This change only adds the first. The second is harder since we can't determine it until we're already in the analysis phase. -- MOS_MIGRATED_REVID=129117755
* Handle some previously unhandled scenarios of cycle reporting for ↵Gravatar Googler2016-08-03
| | | | | | | | | TEST_COMPLETION and TARGET_COMPLETION nodes. Fixes #1323. -- MOS_MIGRATED_REVID=129106695
* Avoid unnecessarily nesting FilesetTraversalParams if the nesting adds no ↵Gravatar Janak Ramakrishnan2016-08-02
| | | | | | | information. -- MOS_MIGRATED_REVID=129012839
* Refactor FileArtifactValue and ArtifactValue now that presence of mtime and ↵Gravatar Janak Ramakrishnan2016-08-01
| | | | | | | digest are mutually exclusive. -- MOS_MIGRATED_REVID=128843642
* Remove FileArtifactValue#createWithDigest method now that empty files are ↵Gravatar Janak Ramakrishnan2016-07-29
| | | | | | | always digested. -- MOS_MIGRATED_REVID=128768429
* Fix leftover "size" check when deciding whether to use digest or mtime. ↵Gravatar Janak Ramakrishnan2016-07-29
| | | | | | | | | Since we no longer stored mtime for empty files, this bug meant that we always compared empty files equal (which is good). But we shouldn't be using Metadata based on mtime for them. A follow-up change will do a refactor to make this impossible. -- MOS_MIGRATED_REVID=128742054
* Avoid intermediate Map/Collections in ActionExecutionFunctionGravatar Michajlo Matijkiw2016-07-29
| | | | | -- MOS_MIGRATED_REVID=128731657
* Avoid copying map when adding discovered inputs to action metadata.Gravatar Janak Ramakrishnan2016-07-28
| | | | | -- MOS_MIGRATED_REVID=128641967
* Rename some PackageId and RepositoryName fields/methods in prep for deep ↵Gravatar Kristina Chodorow2016-07-28
| | | | | | | | | | | | | execroot change This is in prep for making the execution root path for external repositories ../repo_name (instead of external/repo_name). Right now, the getRunfilesPath() returns that path, so that is renamed getExecRoot() (since the runfiles are really just a reflection of the execRoot structure). getSourceRoot() replaces getPathFragment, which has always been a confusing name (it's not clear from the name what the difference is between it and getPackageFragment()). It returns the relative path to source files for external repositories (external/repo_name). Also renamed/moved to more sensible class a few static RepositoryName fields. -- MOS_MIGRATED_REVID=128594419
* Clean up DependencyResolver's interface for the dynamic config migration and ↵Gravatar Greg Estren2016-07-28
| | | | | | | | | | | | | | | | | | | | for general readability. Major changes: - Remove the intermediate Attribute -> LabelAndConfiguration multimap (computed in resolveAttributes). Instead, feed discovered values directly into the final Attribute -> Dependency map via a new RuleResolver interface. - Remove all references to LabelAndConfiguration. The configuration is always the owning rule's configuration except for two special cases: late-bound attributes with splits and late-bound attributes with LateBoundDefault.useHostConfiguration. The original interface made this very unclear and required a lot of awkward and sometimes incorrect logic. The new interface only involves configurations for the cases that actually need them. - Remove an ugly hack caused by BuildConfiguration.evaluateTransition mixing poorly with LateBoundDefault.useHostConfiguration (https://github.com/bazelbuild/bazel/blo[]e172693c27f3efc95ed163e43a9f0a7a6fb4017/src/main/java/com/google/devtools/build/lib/analysis/DependencyResolver.java#L488). - Remove a hack that applies split transitions twice because of BuildConfiguration.evaluateTransition mixing poorly with late-bound split attributes (https://github.com/bazelbuild/bazel/blo[]e172693c27f3efc95ed163e43a9f0a7a6fb4017/src/main/java/com/google/devtools/build/lib/analysis/DependencyResolver.java#L319). This happens to be innocent now but won't be when nested splits are possible. - Solidifies the API contract for Attribute.LateBoundDefault.useHostConfiguration. - Applies clearer naming and more consistent ordering to method parameters. - Better documentation. This is all also prep work for dynamic split transitions. tl;dr: late-bound attributes are legitimately special. Treat them that way to make the rest of DependencyResolver cleaner and hack-free. -- MOS_MIGRATED_REVID=128582618
* Description redacted.Gravatar Googler2016-07-22
| | | | | -- MOS_MIGRATED_REVID=128199284
* Create a symlink with the right workspace name under the execrootGravatar Kristina Chodorow2016-07-22
| | | | | | | | | | | | | | | | | | The execution root currently uses the basename of the workspace directory for the workspace name, not the name in the WORKSPACE file. (For example, if our sources were in /path/to/foo and our WORKSPACE file had workspace(name = "bar"), our execution root would look like execroot/foo.) This creates a symlink bar -> foo, so that accessing ../repo_name actually works for the main repository. RELNOTES[INC]: The main repository's execution root is under the main repository's workspace name, not the source directory's basename. This shouldn't have any effect on most builds, but it's possible it could break someone doing weird things with paths in actions. -- MOS_MIGRATED_REVID=128175455
* Use byte[] rather than ByteString for file digests.Gravatar Googler2016-07-20
| | | | | | | | | | | | | | | | | ActionInputFileCache: Change getDigest() to return the underlying byte[16] owned by each FileArtifactValue. Remove throws clause from getInputFromDigest(); this should be an in-memory operation, and no implementation actually throws. PerActionFileCache: Invert mapping from artifact to digest only if needed. Remove interner, as it was used only for the reverse map keys, not the returned values. This should be a significant cpu savings as eagerly constructing the reverse maps was a noticeable hotspot. -- MOS_MIGRATED_REVID=127972359
* Sort the results returned by the HybridGlobber if we are using results from ↵Gravatar Janak Ramakrishnan2016-07-19
| | | | | | | | | Skyframe globbing. This adds a log(n) factor to uses of globs, but getting globs to be returned in a reasonable order that can be emulated by legacy globbing is hard and bug-prone right now, and we must sort anyway if we are merging legacy and Skyframe globs. Note that this log(n) factor is already present on clean builds with legacy globbing. If we end up seeing performance issues on incremental loading, we can investigate making GlobFunction efficiently return elements in sorted order. (We would still need to sort if merging legacy and Skyframe globs, but that should be a relatively rare occurrence, and can be dealt with by a more efficient merge sort if necessary.) -- MOS_MIGRATED_REVID=127752554
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-05 12:09:07 +0000