aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/test/java/com/google/devtools/build/lib/skyframe
Commit message (Collapse)AuthorAge
* Add EnumMapCodec. Some hackery necessary to get Class for Enum when map is ↵Gravatar janakr2018-03-20
| | | | | | | | | | empty. Since EnumMap.equals() ignores the key type when the maps are empty [1], we could just return a canonical empty EnumMap, but that seems to depend on implementation details in a hacky and potentially subtly broken way. The Unsafe method will hopefully break spectacularly if it goes wrong. [1] http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/classes/java/util/EnumMap.java#l685 PiperOrigin-RevId: 189803280
* Makes leaf level AutoCodec marshallers into runtime codecs.Gravatar shahan2018-03-15
| | | | | | Next change will convert containers. PiperOrigin-RevId: 189261293
* Allow BazelPackageLoader to load external repositories.Gravatar carmi2018-03-13
| | | | | | | | | Also, disallow BazelPackageLoader from fetching missing external repos. Integration tests for BazelPackageLoader wrt external repos will be left for a follow-up CL. RELNOTES: None. PiperOrigin-RevId: 188967694
* Integrate memoization into standard serialization. This involves a number of ↵Gravatar janakr2018-03-13
| | | | | | | | | | | | | | | | large changes: 1. SerializationContext and DeserializationContext become the owners of the Memoizer if requested. They produce new versions of themselves on demand that are memoization-aware. Because of intricacies of Skylark that I do not fully understand, we inject a Mutability object when starting memoization, and so to be conservative, that injection starts up a new memoization frame, nested if we were already memoizing, just like before. It would be nice to decouple this injection from memoization in the future. 2. MemoizingCodec is deleted, but really ObjectCodec becomes MemoizingCodec, so it lives on. BaseCodec is deleted since it now has only one implementation. 3. The simplified model of registering MemoizingCodecs is adopted for ObjectCodecs: all codecs are registered based on their #getEncodedClass and #additionalEncodedSubclasses. This also allows us to register codecs that are defined using tricky parameter types, since we're no longer trying to reflectively examine them. This required a clean-up of such codecs, and the addition of ArrayListCodec to stop NullableListCodec from making lists unmodifiable when they shouldn't be. 4. @AutoCodec is extended to allow users to specify that memoization should start with this codec. To ensure bit-equivalence, SkyKeySerializer disables memoization. PiperOrigin-RevId: 188918251
* tests: add TimestampGranularityUtilsGravatar laszlocsomor2018-03-13
| | | | | | | | | | | | | | Add a new test utility class TimestampGranularityUtils which offers methods to wait for the filesystem timestamp granularity to pass. Such waiting is necessary if a test asserts changes in a file's mtime or ctime. This is a follow-up to comments on https://github.com/bazelbuild/bazel/commit/a31e035fb42aa8db9fd248ef2ebd665a411053b8. PiperOrigin-RevId: 188847729
* Fix toolchain and execution platform registration to use patterns.Gravatar John Cater2018-03-12
| | | | | | | | This allows more flexibility in registering toolchains and execution platforms, both in the WORKSPACE and from the command-line. Change-Id: I6fe75507d1a74de74085b7c927fdf093c152b894 PiperOrigin-RevId: 188813688
* Get rid of non-class-based lookups for ObjectCodecs. There were basically no ↵Gravatar janakr2018-03-10
| | | | | | | | remaining non-test users (ErrorInfoEncoder was it). Should help to simplify our codec registration. I want to converge MemoizingCodec and ObjectCodec, and the unnecessary complexity around ObjectCodec registration is annoying. PiperOrigin-RevId: 188620988
* Open-source ImmutableMapCodec and make it able to handle arbitrary ↵Gravatar janakr2018-03-10
| | | | | | | | | | keys/values without injecting the codecs. Also allow it to handle ImmutableSortedMap, since we were always silently degrading to ImmutableMap for objects that weren't declared as ImmutableSortedMap, and there's no good way to handle non-natural comparators. This will lead to runtime failures if a class actually needs an ImmutableSortedMap with a different comparator than the natural one. This changes the semantics of ImmutableMap serialization. Previously, we went off the declared type. In the case of a declared ImmutableSortedMap, we ignored the comparator, potentially leading to incorrect serialization (new test added in AutoCodecProcessorTest that would have caught that). Moreover, declared ImmutableMaps were deserialized as ImmutableMaps even if they were actually ImmutableSortedMaps. Now, we preserve the ordering unconditionally, and preserve the type if possible. I think this is a better state to be in. This is needed to kill off MemoizingCodec, since MemoizingCodec has an ImmutableMapMemoizingCodec, which I want to get rid of in favor of this. PiperOrigin-RevId: 188619637
* Add BuildConfiguration to ConfiguredTargetAndTarget and rename it to ↵Gravatar janakr2018-03-10
| | | | | | ConfiguredTargetAndData. We want to get BuildConfiguration out of ConfiguredTarget because it uses >800K when serialized. PiperOrigin-RevId: 188600002
* Allow ObjectCodecRegistry to handle MemoizingCodecs. Initially this is just ↵Gravatar janakr2018-03-09
| | | | | | | | | | | | | | | | Skylark memoization, but we may extend in future to handle more than just Skylark this way. In fact, we probably want most of our ObjectCodecs to be MemoizingCodecs that can efficiently fall back to ObjectCodec if not using memoization (maybe coming in a follow-up). At a high level, this CL merges the functionality of MemoizingCodecMap into ObjectCodecRegistry, adds auto-registration of MemoizingCodec, and uses that to get rid of a lot of codecs that were just delegating. The big one to get rid of there is SkylarkValueCodec: all of its delegation duties are implicitly now in ObjectCodecRegistry and friends. One danger with this CL is that many of the features of Skylark serialization are only being tested in unit tests, which had to be reworked as part of this change. I don't think we've lost any coverage, but I could be wrong. SkylarkValueCodec had a bunch of methods that were effectively test-only, which made it easier to remove. The plan is to provide a Memoizer.Serializer inside the SerializationContext. At the top level, it will be a DUMMY_SERIALIZER that does no memoization, but a MemoizingCodec can do context = context.ensureMemoizing() which will recreate the context with a true memoizing serializer. Then all references to the Serializer in codec code can be cleaned up, and the MemoizingCodec and ObjectCodec signatures will be the same. At that point, we can make all ObjectCodecs compatible with memoization by default (with strategy MEMOIZE_AFTER), and add a "memoize" boolean to @AutoCodec. That should allow us to have full interoperability between all codecs. This CL also makes CodecScanner deterministic in the order of classes that it processes (there was a lurking bug here where constants must be deterministically ordered but that wasn't enforced at all). PiperOrigin-RevId: 188559983
* tests: fix flaky RecursiveFSTraversalFunctionTestGravatar Laszlo Csomor2018-03-09
| | | | | | | | | | | | | | | | | | Fix the testTraversalOfGeneratedDirectory method in RecursiveFilesystemTraversalFunctionTest that was flaky on Windows. The fix is to wait so that changes to files in the InMemoryFileSystem will have observable effects on the file ctimes. Depends on https://github.com/bazelbuild/bazel/pull/4787 Fixes https://github.com/bazelbuild/bazel/issues/4755 Closes #4788. PiperOrigin-RevId: 188470080
* tests: fix flaky FilesystemValueCheckerTestGravatar Laszlo Csomor2018-03-09
| | | | | | | | | | | | | Fix FilesystemValueCheckerTest.testExplicitFiles() by ensuring that the filesystem timestamp granularity has elapsed before attempting to update the files. Fixes https://github.com/bazelbuild/bazel/issues/4755 Closes #4786. PiperOrigin-RevId: 188467381
* tests: fix flaky SkyframeAwareActionTestGravatar Laszlo Csomor2018-03-09
| | | | | | | | | | | | | Fix testCacheBypassingActionWithMtimeChangingInput in SkyframeAwareActionTest by ensuring that enough time elapses between file updates so their effects are observable on the file's ctime. Fixes https://github.com/bazelbuild/bazel/issues/4755 Closes #4787. PiperOrigin-RevId: 188458542
* Expose an actions provider on RuleConfiguredTarget instances.Gravatar cparsons2018-03-06
| | | | | | | | | Given a target (for example from a skylark aspect), one will be able to access a list of actions that the target generated using "target.actions". This is without additional memory footprint. Actions themselves are not fully exposed to skylark (and thus there isn't much meaning to gather from them in skylark yet). Access methods will follow soon. RELNOTES: None. PiperOrigin-RevId: 188098079
* @AutoCodec File{State}Value.Gravatar janakr2018-03-06
| | | | PiperOrigin-RevId: 188085895
* @AutoCodec RepositoryName.Gravatar janakr2018-03-05
| | | | PiperOrigin-RevId: 187956593
* Serialize Path using AutoCodec.Gravatar janakr2018-03-05
| | | | PiperOrigin-RevId: 187945746
* Replace LegacySkyKey by AbstractSkyKey or custom SkyKeys. AbstractSkyKey ↵Gravatar janakr2018-03-02
| | | | | | | | doesn't save memory in the 32-bit case, but makes it easier for people to see how many SkyKeys we have. There's some unnecessary interning in tests, but it was easier to copypasta and doesn't harm anything, I think. PiperOrigin-RevId: 187694309
* Deletes CODEC fields now that they are no longer needed.Gravatar shahan2018-02-28
| | | | PiperOrigin-RevId: 187397314
* Refactor the AbstractAction computeKey method to be easier to add invariantsGravatar jcater2018-02-28
| | | | | | for every type of action. PiperOrigin-RevId: 187368369
* Create a runtime codec for HashCode.Gravatar mjhalupka2018-02-28
| | | | | | Remove the HashCode marshaller. PiperOrigin-RevId: 187350917
* Deletes AutoCodec.Strategy.SINGLETON now that we have @AutoCodec field tags.Gravatar shahan2018-02-28
| | | | | | This also gets rid of some static initialization cycles which we should try very hard to avoid in the future. PiperOrigin-RevId: 187334087
* Update ToolchainUtil to properly load and use the available executionGravatar John Cater2018-02-28
| | | | | | | | | platforms, and correctly merge together the results from TRF. Part of #4442. Change-Id: I31d83fa73a93d39a0e18d05a43a1c8666ac5a2d2 PiperOrigin-RevId: 187324257
* Expose --output_user_base to the Bazel server processGravatar Klaus Aehlig2018-02-27
| | | | | | | | | | | ...so that it can use that path to compute other directories in the output user base, in particular the default location for caches. The first cache we will add is the hash-index cache for downloads of external archives, but a spawn cache might be added later in the output user base as well. Change-Id: I24b1c33235c8f76ec008ecb1789163de2b2a45be PiperOrigin-RevId: 187164275
* Fail gracefully on conflicting actions generated by an aspect. These can ↵Gravatar janakr2018-02-26
| | | | | | come from Skylark, so we shouldn't crash. As a safety measure, subclasses of ActionLookupValue are now responsible for detecting action conflicts themselves. PiperOrigin-RevId: 187095271
* expectThrows has been renamed to assertThrowsGravatar cushon2018-02-24
| | | | | | More information: https://github.com/junit-team/junit5/issues/531 PiperOrigin-RevId: 186900384
* Integrates CodecScanner into SkyValueEncoder.Gravatar shahan2018-02-23
| | | | | | | | * Deletes a number of CODEC references, now superceded by registry functionality. * Minimal SkyKeySerializer modifications for correctness, as certain codec changes require codecs to be in available in the registry. PiperOrigin-RevId: 186834520
* A couple quality-of-life improvements for Bazel devs, in response to an ↵Gravatar nharmata2018-02-23
| | | | | | | | | | email from philwo@. (i) Only have TimestampGranularityMonitor log the first file of relevance. This reduces log spam, especially in tests, while still maintaining useful information in the logs. (ii) Don't have ExternalFilesHelper log the fact that it encountered an external file when we're in a unit test or an integration test. Tests, especially bazel tests that use external repositories, tend to involve lots of "external" files. RELNOTES: None PiperOrigin-RevId: 186799176
* Remove some default cc toolchain definitions.Gravatar John Cater2018-02-23
| | | | | Change-Id: Idb0c9fe770cd73c2760bee92997063c4c9b2e94e PiperOrigin-RevId: 186768436
* Automated rollback of commit 7fe59b98eefc96a6310f0b0221d4e0f18e2a9000.Gravatar shreyax2018-02-23
| | | | | | | | | | | | *** Reason for rollback *** Fixed bug due to TransitiveTargetFunction requesting multiple Package dependencies when computing its aspect deps by only applying the optimization to TransitiveTraversalFunction. *** Original change description *** Automated rollback of commit cce164aed44aba1de244f0d764cd33a5cc6980b2. PiperOrigin-RevId: 186766812
* Re-enable loading the PlatformInfo providers inGravatar John Cater2018-02-22
| | | | | | | | RegisteredExecutionPlatformsFunction, so that errors can be handled as early as possible. Change-Id: I2c73a9202d6bb02e04a32c18c4986e1e204aa9d5 PiperOrigin-RevId: 186694303
* Deletes POLYMORPHIC strategy. ObjectCodec now uses runtime type information ↵Gravatar shahan2018-02-20
| | | | | | to select a codec. PiperOrigin-RevId: 186378153
* Adds ObjectCodecRegistry to {Des|S}erializationContext.Gravatar shahan2018-02-20
| | | | | | | | | | | | * AutoCodec now delegates to the registry. * Adds getSuperclass logic for resolving a codec. * Small cleanups for classes that break the registry. TODO after this change: * Explicit CODEC definitions are no longer needed and existing ones should be cleaned up. * POLYMORPHIC is no longer be needed and should be cleaned up. PiperOrigin-RevId: 186351845
* Automated rollback of commit cce164aed44aba1de244f0d764cd33a5cc6980b2.Gravatar fwe2018-02-19
| | | | PiperOrigin-RevId: 186211672
* Re-use previously computed deps for TransitiveBaseTraversalFunction#compute ↵Gravatar shreyax2018-02-16
| | | | | | if we have them instead of re-computing them each time on a skyframe restart. PiperOrigin-RevId: 186017079
* Remove BuildConfiguration and PlatformInfo from theGravatar jcater2018-02-15
| | | | | | RegisteredExecutionPlatformsFunction. PiperOrigin-RevId: 185947540
* Stop storing ActionTemplate in a SkyKey: it's too heavyweight. Use the same ↵Gravatar janakr2018-02-15
| | | | | | mechanism as for normal actions, have the ActionTemplateExpansionFunction look the template up when needed. PiperOrigin-RevId: 185861672
* Get PlatformInfo out of ToolchainResolutionKey, replace with the ↵Gravatar janakr2018-02-14
| | | | | | ConfiguredTargetKeys that own the PlatformInfo. PiperOrigin-RevId: 185770105
* Start slimming down ToolchainResolutionKey: get rid of BuildConfiguration. ↵Gravatar janakr2018-02-14
| | | | | | The real blocker is PlatformInfo, which is coming. PiperOrigin-RevId: 185742130
* Make ActionLookupData a SkyKey. Should save some memory.Gravatar janakr2018-02-14
| | | | PiperOrigin-RevId: 185694489
* Thread {Des,S}erializationContext through Package serialization interfaceGravatar michajlo2018-02-13
| | | | PiperOrigin-RevId: 185609826
* Replaces InjectingObjectCodec with dependencies threaded through ↵Gravatar shahan2018-02-13
| | | | | | (Des|S)erializationContext. PiperOrigin-RevId: 185547740
* Add context argument to ObjectCodec.{serialize,deserialize}Gravatar michajlo2018-02-11
| | | | | | | | | | Context implementations are currently empty, just doing the plumbing in this change. Once this is in we can start passing along the ObjectCodecRegistry, which will allow runtime codec resolution for classes not known at compile time. We'll also inevitably add some memoization helpers, allowing us to optimize the serialization process further. PiperOrigin-RevId: 185305674
* Replace path implementation.Gravatar tomlu2018-02-08
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Path and PathFragment have been replaced with String-based implementations. They are pretty similar, but each method is dissimilar enough that I did not feel sharing code was appropriate. A summary of changes: PATH ==== * Subsumes LocalPath (deleted, its tests repurposed) * Use a simple string to back Path * Path instances are no longer interned; Reference equality will no longer work * Always normalized (same as before) * Some operations will now be slower, like instance compares (which were previously just a reference check) * Multiple identical paths will now consume more memory since they are not interned PATH FRAGMENT ============= * Use a simple string to back PathFragment * No more segment arrays with interned strings * Always normalized * Remove isNormalized * Replace some isNormalizied uses with containsUpLevelReferences() to check if path fragments try to escape their scope * To check if user input is normalized, supply static methods on PathFragment to validate the string before constructing a PathFragment * Because PathFragments are always normalized, we have to replace checks for literal "." from PathFragment#getPathString to PathFragment#getSafePathString. The latter returns "." for the empty string. * The previous implementation supported efficient segment semantics (segment count, iterating over segments). This is now expensive since we do longer have a segment array. ARTIFACT ======== * Remove Path instance. It is instead dynamically constructed on request. This is necessary to avoid this CL becoming a memory regression. RELNOTES: None PiperOrigin-RevId: 185062932
* Extract registry functionality from ObjectCodecsGravatar michajlo2018-02-07
| | | | | | | | | | This partially solves the age old problem of how to find a codec for a value we don't know the type of at compile time, and allows us to represent such values on the wire more compactly. @AutoCodec's injecting codec should be able to make use of this right away - we'll need to make an API change to the ObjectCodec interface to allow the existing system to make use. PiperOrigin-RevId: 184918173
* Refactor to get rid of some getConfiguredTarget() calls in tests and replaceGravatar mjhalupka2018-02-07
| | | | | | | them with getConfiguredTargetAndTarget() so we can get rid of ConfiguredTarget.getTarget() callers. This should be a test only change. PiperOrigin-RevId: 184877255
* Update ToolchainResolutionFunction to consider multiple availableGravatar John Cater2018-02-06
| | | | | | | | | execution platforms. Part of #4442. Change-Id: I6678d57f4aaadcb19032bf58820606242ba66a25 PiperOrigin-RevId: 184707708
* Fix test compilation failure.Gravatar felly2018-02-05
| | | | PiperOrigin-RevId: 184587712
* Fix Fileset incrementality bug when Fileset consumes a generated file. The ↵Gravatar felly2018-02-05
| | | | | | | | | | | | | | | | native skyframe implementation was actually quite incorrect in this case: It was adding a skyframe dependency on a FileValue for the output file. Without a transitive dependency on the source files and actions that determine the output file's state, this could never work (and explains why the incremental build would fail). Instead, we now depend on the Artifact corresponding to the output file instead. This change updates the business logic RecursiveFilesystemTraversalFunction. This approach keeps the business logic of Fileset filesystem traversal centralized in RFTF. To avoid making weird recursive Skyframe nodes in the output tree, we inline Skyframe dependencies and do direct filesystem operations over the output tree. There are now three states we can be in when looking up a file: 1. Source file: As before, make a skyframe dep on the corresponding file 2. Top-level file of an output tree: Make a dep on the corresponding Artifact 3. Recursive file under an output directory: Do direct filesystem operations. It doesn't make sense to make Skyframe nodes corresponding to these files. In the future, I think we should consider failing fast on this case. RELNOTES: None PiperOrigin-RevId: 184556044
* Refactor ArtifactSkyKey to get rid of an unnecessary wrapper class: actually ↵Gravatar janakr2018-02-05
| | | | | | | | essentially promote OwnedArtifact to ArtifactSkyKey and rename it to ArtifactSkyKey. The king is dead... Also add some other execution-phase codecs. PiperOrigin-RevId: 184552706
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-03 12:31:45 +0000