aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/skyframe
Commit message (Collapse)AuthorAge
* Re-enable aliasing of CodedInputStream during deserialization, removed as ↵Gravatar janakr2018-03-23
| | | | | | | | part of primary codepath somewhere around https://github.com/bazelbuild/bazel/commit/bde43ec8a96a62b8fbf67ad60d5154cf121647d9 (and killed entirely in unknown commit. Force copying of a ByteString read from CodedInputStream in NestedSetCodec and persisted, since otherwise we might hang onto the entire buffer indefinitely. PiperOrigin-RevId: 190256337
* Adds hook to AspectFunction to allow for inlining SkylarkImportLookupFunction.Gravatar cpeyser2018-03-23
| | | | | | If inlining is off, fixes a bug where null return values resulted in an error. PiperOrigin-RevId: 190255818
* Fix test flakiness due to the filesystem mutations done during the 'fetch' ↵Gravatar nharmata2018-03-23
| | | | | | | | | work done by BazelPackageBuilderHelperForTesting#sanityCheckBazelPackageLoader. We fix this flakiness by _not_ having RepositoryDelegatorFunction unconditionally 'fetch' local repos. We then have to make a change to BazelPackageLoader test to reinstate these mutations, since it was only working before due to the unconditional filesystem mutations done here :p RELNOTES: None PiperOrigin-RevId: 190222830
* Implement TargetProviderForQueryEnvironment.Gravatar dbabkin2018-03-23
| | | | | | | DirectTargetProvider works with WalkableGraph directly. If it fails to find target, then last try to use delegate TargetProvider which supposed to be PackageManager from SkyframeExecutor. RELNOTES:none PiperOrigin-RevId: 190204106
* Victory lap: Remove all code that used to support the three-argument form of ↵Gravatar lberki2018-03-23
| | | | | | | vardef(). RELNOTES: None. PiperOrigin-RevId: 190196933
* Add a DefaultBuildOptions specifying default build flags that may differ ↵Gravatar mjhalupka2018-03-22
| | | | | | from the provided defaults in Options classes. These are used to create BuildOptionsDiffForReconstruction, which lets us store only the diffs in our BuildConfigurationValue.Keys. PiperOrigin-RevId: 190117455
* Allow NestedSetCodec to share members across multiple deserializations.Gravatar cpeyser2018-03-22
| | | | | | This avoids redundancy in memory when multiple NestedSets share a member PiperOrigin-RevId: 190085907
* Clean up unnecessary "additional data" from memoizing deserialization. Since ↵Gravatar janakr2018-03-21
| | | | | | memoization is now a simple on-off switch, change semantics to have at most one memoizing frame: starting memoization is now an idempotent operation. PiperOrigin-RevId: 189993914
* Memoize non-rule configured targets. This is probably a good idea in ↵Gravatar janakr2018-03-21
| | | | | | general, but especially necessary because OutputFileConfiguredTarget (which is not a RuleConfiguredTarget) contains a reference to its generating configured target, so all the optimizations needed for RuleConfiguredTarget apply to OutputFileConfiguredTarget. PiperOrigin-RevId: 189942424
* Add behavior to NestedSetCodec to prevent it from running during testing.Gravatar cpeyser2018-03-21
| | | | PiperOrigin-RevId: 189906038
* Hand-rolls EnumRuntimeCodecGravatar shahan2018-03-20
| | | | PiperOrigin-RevId: 189830623
* Add EnumMapCodec. Some hackery necessary to get Class for Enum when map is ↵Gravatar janakr2018-03-20
| | | | | | | | | | empty. Since EnumMap.equals() ignores the key type when the maps are empty [1], we could just return a canonical empty EnumMap, but that seems to depend on implementation details in a hacky and potentially subtly broken way. The Unsafe method will hopefully break spectacularly if it goes wrong. [1] http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/classes/java/util/EnumMap.java#l685 PiperOrigin-RevId: 189803280
* Add ability to shallow-freeze individual objectsGravatar brandjon2018-03-20
| | | | | | | | | Clarify that the IMMUTABLE Mutability should only be used for deeply immutable things, not tuples. Created a new SHALLOW_IMMUTABLE Mutability for them. Note that the new shallow-freezing functionality marks things as deeply IMMUTABLE without traversing its contents. I.e., it lies, and it is up to the caller to ensure this doesn't cause problems. RELNOTES: NONE PiperOrigin-RevId: 189767422
* @Autocodec a few leftover classes. Also, add @AutoCodec builder support for ↵Gravatar cpeyser2018-03-20
| | | | | | boolean getters in "isFoo" form. PiperOrigin-RevId: 189753768
* Automated rollback of commit 72d28f3efc2842510a34cacd930c0204143f7412.Gravatar shreyax2018-03-19
| | | | | | | | | | | | | | | | | | | | | | | | Fix skylark caching to properly include transitive dependencies when there is a diamond-like dependency in the loaded bzl files. Also add guards to make sure we're not attempting to cache skylark files that transitively request a dependency that is in error. *** Reason for rollback *** Looking for source of non-determinism *** Original change description *** Automated rollback of commit 7ba939dfd5df48903929e9c14ebd0449656403e4. *** Reason for rollback *** Likely cause for non-determinism in skyframe *** Original change description *** Cache SkylarkLookupImportValues in memory so that we don't recompute them multiple times. PiperOrigin-RevId: 189686604
* Automated rollback of commit e6af9f9178e731a45e552b1d249ded90124945ba.Gravatar janakr2018-03-19
| | | | | | | | | | | | *** Reason for rollback *** See linked bug. *** Original change description *** Add behavior to NestedSetCodec to prevent it from running during testing. PiperOrigin-RevId: 189663863
* Add some testing methods to get ConfiguredTargetAndData more easily, and ↵Gravatar janakr2018-03-19
| | | | | | rename some methods that were still called ConfiguredTargetAndTarget. Move some tests over to using ConfiguredTargetAndData instead of calling ConfiguredTarget#getConfiguration() directly. PiperOrigin-RevId: 189642264
* Add behavior to NestedSetCodec to prevent it from running during testing.Gravatar cpeyser2018-03-19
| | | | PiperOrigin-RevId: 189602622
* Add unit tests for ↵Gravatar janakr2018-03-16
| | | | | | https://github.com/bazelbuild/bazel/commit/f304d2be2e8c26bd85434f1d6ca036f380162fb9 (oops). Also don't use concrete build() methods: there must be an abstract one, and crash explicitly if we don't have a Builder class, rather than with an NPE down the line. And remove non-functional partial handling of iterables. PiperOrigin-RevId: 189422625
* Adds codecs for more list variants.Gravatar shahan2018-03-16
| | | | PiperOrigin-RevId: 189417853
* Add @AutoCodec strategy for @AutoValue.Builder.Gravatar janakr2018-03-16
| | | | | | Still missing a lot of features (most notably the ability to construct an iterable element by element), but this is ok for now. PiperOrigin-RevId: 189410898
* Deletes list marshaller and makes map entry marshaller into a runtime codec.Gravatar shahan2018-03-16
| | | | PiperOrigin-RevId: 189390024
* Refactor DependencyResolver#dependentNodeMap to take in a set of toolchain ↵Gravatar juliexxia2018-03-16
| | | | | | labels instead of the entire toolchain context since that's all the dependency resolver really needs. This will help make cquery transitions outputter easier to implement. PiperOrigin-RevId: 189342812
* Makes leaf level AutoCodec marshallers into runtime codecs.Gravatar shahan2018-03-15
| | | | | | Next change will convert containers. PiperOrigin-RevId: 189261293
* Optimize GC churn due to PackageFunction#getContainingDirectory. While I'm ↵Gravatar nharmata2018-03-15
| | | | | | | | | here, also slightly restructure the code in #handleLabelsCrossingSubpackagesAndPropagateInconsistentFilesystemExceptions to make it more readable (and defer PackageIdentifier allocations). Alternatives considered: Add a PathFragment#getParentDirectoryOfRelative(String other) instance method, and use it in PackageFunction#getContainingDirectory. I thought the approach in this CL would be preferable to adding a specialized method like that to PathFragment. RELNOTES: None PiperOrigin-RevId: 189197855
* Simplify ClientEnvironmentValue invalidation.Gravatar Benjamin Peterson2018-03-15
| | | | | | | Unconditionally inject the new values, and let skyframe change pruning avoid over-invalidation. Change-Id: I3f478ea756121bde1078e9e79ddcfbeb54951cbb PiperOrigin-RevId: 189156698
* Splits the iterableMarshaller into runtime codecs.Gravatar shahan2018-03-14
| | | | | | Makes NestedSetCodec into a runtime codec instead of a Marshaller. PiperOrigin-RevId: 189110883
* Allow BazelPackageLoader to load external repositories.Gravatar carmi2018-03-13
| | | | | | | | | Also, disallow BazelPackageLoader from fetching missing external repos. Integration tests for BazelPackageLoader wrt external repos will be left for a follow-up CL. RELNOTES: None. PiperOrigin-RevId: 188967694
* Tags some more non-serializable classes (for serialization).Gravatar shahan2018-03-13
| | | | PiperOrigin-RevId: 188930446
* Integrate memoization into standard serialization. This involves a number of ↵Gravatar janakr2018-03-13
| | | | | | | | | | | | | | | | large changes: 1. SerializationContext and DeserializationContext become the owners of the Memoizer if requested. They produce new versions of themselves on demand that are memoization-aware. Because of intricacies of Skylark that I do not fully understand, we inject a Mutability object when starting memoization, and so to be conservative, that injection starts up a new memoization frame, nested if we were already memoizing, just like before. It would be nice to decouple this injection from memoization in the future. 2. MemoizingCodec is deleted, but really ObjectCodec becomes MemoizingCodec, so it lives on. BaseCodec is deleted since it now has only one implementation. 3. The simplified model of registering MemoizingCodecs is adopted for ObjectCodecs: all codecs are registered based on their #getEncodedClass and #additionalEncodedSubclasses. This also allows us to register codecs that are defined using tricky parameter types, since we're no longer trying to reflectively examine them. This required a clean-up of such codecs, and the addition of ArrayListCodec to stop NullableListCodec from making lists unmodifiable when they shouldn't be. 4. @AutoCodec is extended to allow users to specify that memoization should start with this codec. To ensure bit-equivalence, SkyKeySerializer disables memoization. PiperOrigin-RevId: 188918251
* Fix toolchain and execution platform registration to use patterns.Gravatar John Cater2018-03-12
| | | | | | | | This allows more flexibility in registering toolchains and execution platforms, both in the WORKSPACE and from the command-line. Change-Id: I6fe75507d1a74de74085b7c927fdf093c152b894 PiperOrigin-RevId: 188813688
* Automated rollback of commit 7ba939dfd5df48903929e9c14ebd0449656403e4.Gravatar shreyax2018-03-12
| | | | | | | | | | | | *** Reason for rollback *** Likely cause for non-determinism in skyframe *** Original change description *** Cache SkylarkLookupImportValues in memory so that we don't recompute them multiple times. PiperOrigin-RevId: 188729929
* Get rid of non-class-based lookups for ObjectCodecs. There were basically no ↵Gravatar janakr2018-03-10
| | | | | | | | remaining non-test users (ErrorInfoEncoder was it). Should help to simplify our codec registration. I want to converge MemoizingCodec and ObjectCodec, and the unnecessary complexity around ObjectCodec registration is annoying. PiperOrigin-RevId: 188620988
* Open-source ImmutableMapCodec and make it able to handle arbitrary ↵Gravatar janakr2018-03-10
| | | | | | | | | | keys/values without injecting the codecs. Also allow it to handle ImmutableSortedMap, since we were always silently degrading to ImmutableMap for objects that weren't declared as ImmutableSortedMap, and there's no good way to handle non-natural comparators. This will lead to runtime failures if a class actually needs an ImmutableSortedMap with a different comparator than the natural one. This changes the semantics of ImmutableMap serialization. Previously, we went off the declared type. In the case of a declared ImmutableSortedMap, we ignored the comparator, potentially leading to incorrect serialization (new test added in AutoCodecProcessorTest that would have caught that). Moreover, declared ImmutableMaps were deserialized as ImmutableMaps even if they were actually ImmutableSortedMaps. Now, we preserve the ordering unconditionally, and preserve the type if possible. I think this is a better state to be in. This is needed to kill off MemoizingCodec, since MemoizingCodec has an ImmutableMapMemoizingCodec, which I want to get rid of in favor of this. PiperOrigin-RevId: 188619637
* Make memoizing serialize and deserialize methods that take a codec ↵Gravatar janakr2018-03-10
| | | | | | package-private. PiperOrigin-RevId: 188610579
* Explicitly don't serialize/deserialize Package objects in ↵Gravatar janakr2018-03-10
| | | | | | ConfiguredTargetValue. The nested set of packages isn't usually set when serializing/deserializing, but it's good to be explicit, especially in tests. PiperOrigin-RevId: 188605043
* Add BuildConfiguration to ConfiguredTargetAndTarget and rename it to ↵Gravatar janakr2018-03-10
| | | | | | ConfiguredTargetAndData. We want to get BuildConfiguration out of ConfiguredTarget because it uses >800K when serialized. PiperOrigin-RevId: 188600002
* ActionOwner references only BuildEvent instead of full BuildConfigurationGravatar shahan2018-03-10
| | | | | | This reduces the size of its serialized representation. PiperOrigin-RevId: 188597127
* Allow ObjectCodecRegistry to handle MemoizingCodecs. Initially this is just ↵Gravatar janakr2018-03-09
| | | | | | | | | | | | | | | | Skylark memoization, but we may extend in future to handle more than just Skylark this way. In fact, we probably want most of our ObjectCodecs to be MemoizingCodecs that can efficiently fall back to ObjectCodec if not using memoization (maybe coming in a follow-up). At a high level, this CL merges the functionality of MemoizingCodecMap into ObjectCodecRegistry, adds auto-registration of MemoizingCodec, and uses that to get rid of a lot of codecs that were just delegating. The big one to get rid of there is SkylarkValueCodec: all of its delegation duties are implicitly now in ObjectCodecRegistry and friends. One danger with this CL is that many of the features of Skylark serialization are only being tested in unit tests, which had to be reworked as part of this change. I don't think we've lost any coverage, but I could be wrong. SkylarkValueCodec had a bunch of methods that were effectively test-only, which made it easier to remove. The plan is to provide a Memoizer.Serializer inside the SerializationContext. At the top level, it will be a DUMMY_SERIALIZER that does no memoization, but a MemoizingCodec can do context = context.ensureMemoizing() which will recreate the context with a true memoizing serializer. Then all references to the Serializer in codec code can be cleaned up, and the MemoizingCodec and ObjectCodec signatures will be the same. At that point, we can make all ObjectCodecs compatible with memoization by default (with strategy MEMOIZE_AFTER), and add a "memoize" boolean to @AutoCodec. That should allow us to have full interoperability between all codecs. This CL also makes CodecScanner deterministic in the order of classes that it processes (there was a lurking bug here where constants must be deterministically ordered but that wasn't enforced at all). PiperOrigin-RevId: 188559983
* Remove cycle error reporting from query results since they're not strictly ↵Gravatar juliexxia2018-03-09
| | | | | | relevant and only trigger when the implicit or explicit max depth > 20 which is confusing. PiperOrigin-RevId: 188559702
* Intern CollectTargetsInPackagKey. Bug from ↵Gravatar janakr2018-03-08
| | | | | | https://github.com/bazelbuild/bazel/commit/5fb2a487e53cc3d80e3654d5b63d062f7f70588b. PiperOrigin-RevId: 188348546
* Fix PackageFunction's call to Package.Builder.Helper#onLoadingComplete to passGravatar nharmata2018-03-07
| | | | | | | | | | along the wall time of the load, even when the package in question was in PackageFunction's internal cache (e.g. the current #compute call is a PackageFunction Skyframe restart). Also clarify the intent of the 'loadTimeMs' param in #onLoadingComplete. RELNOTES: None PiperOrigin-RevId: 188253198
* Add option to dump command lines of actions in dump.Gravatar twerth2018-03-07
| | | | | RELNOTES: None PiperOrigin-RevId: 188164754
* Don't use computeIfAbsent in action graph dumping.Gravatar twerth2018-03-07
| | | | | | | | | | | | | | | computeIfAbsent may throw (on best effort basis) a ConcurrentModificationException if you change the underlying Map in the computeIfAbsent call, for example when you call computeIfAbsent recursively in the mappingFunction. NestedSets are an inherent recursive structure, so we need to call computeIfAbsent recursively in the action graph dump. Since computeIfAbsent doesn't throw the exception all the time, it's hard to come up with a reliable test case. RELNOTES: None PiperOrigin-RevId: 188151283
* Post PatternExpandingError from the skyframe target pattern evaluator as needed.Gravatar Benjamin Peterson2018-03-07
| | | | | | | | | This fixes https://github.com/bazelbuild/bazel/issues/4731. Also, add test coverage for posting of ParsingFailedEvent and handling of cycles in LoadingPhaseRunnerTest. Change-Id: I88c9d33417b9c3c7a06c92a6137d58f37b991b0c PiperOrigin-RevId: 188138972
* Expose an actions provider on RuleConfiguredTarget instances.Gravatar cparsons2018-03-06
| | | | | | | | | Given a target (for example from a skylark aspect), one will be able to access a list of actions that the target generated using "target.actions". This is without additional memory footprint. Actions themselves are not fully exposed to skylark (and thus there isn't much meaning to gather from them in skylark yet). Access methods will follow soon. RELNOTES: None. PiperOrigin-RevId: 188098079
* @AutoCodec File{State}Value.Gravatar janakr2018-03-06
| | | | PiperOrigin-RevId: 188085895
* @AutoCodec TransitiveTraversalValue, DirectoryListingStateValue.Gravatar janakr2018-03-06
| | | | PiperOrigin-RevId: 188077900
* Convert a bunch of non-AutoCodec classes to use @AutoCodec: ↵Gravatar janakr2018-03-06
| | | | | | BuildConfiguration, BuildOptions, CollectTargetsUnderDirectoryValue. PiperOrigin-RevId: 188059815
* Add wildcard support to AutoCodec.Gravatar mjhalupka2018-03-06
| | | | PiperOrigin-RevId: 188034513
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-04 15:30:13 +0000