aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/collect
Commit message (Collapse)AuthorAge
* Allow deserialization futures as NestedSet contents, with unrolling blocking ↵Gravatar cpeyser2018-06-13
| | | | | | | | on that future. This allows NestedSet deserialization not to block on storage reads - in-progress deserializations are simply made a member of the NestedSets they produce. PiperOrigin-RevId: 200440131
* Save nested set size after initial expand to avoid GC from resizing list ↵Gravatar tomlu2018-06-13
| | | | | | | | | | | when replaying. We stash the size in the same field as the order using bit packing. The small additional cost of masking out the int should be less than the resizing when replaying large nested sets. The upper size bound on nested sets will be 512M objects as a result. If Java had unsigned ints we could push it to 1G. RELNOTES: None PiperOrigin-RevId: 200417105
* Make Artifact#equals take the owner into account for derived artifacts.Gravatar janakr2018-06-08
| | | | | | | | Derived artifacts' owners are important because they are used to determine the artifact's generating action. Source artifacts' owners are not used in this way, so I left them alone. This allows us to get rid of most uses of ArtifactSkyKey. We may be able to delete it entirely in a follow-up. PiperOrigin-RevId: 199836436
* Add cache to NestedSetCodecWithStore to merge NestedSets that should be ↵Gravatar janakr2018-06-04
| | | | | | | | | | reference-equal on deserialization. We cannot just intern NestedSets because NestedSets with the same underlying children may still not be equal, so we wrap them in an object that does consider their children when calculating equality. We wish to preserve the invariant that if NestedSets inside two different objects are reference-equal, they will continue to be reference-equal after deserialization. Not doing that causes bugs. Unfortunately, because Artifact#equals does not take ArtifactOwner into account, this introduces a new bug (exposed via a disabled test here) where unequal singleton NestedSets may be considered equal. I will clean this up in the future by fixing Artifact#equals. PiperOrigin-RevId: 199208045
* Serialize NestedSetStore$NestedSetSize directly as an ordinal instead ofGravatar cpeyser2018-05-22
| | | | | | through SerializationContext, to save a byte. PiperOrigin-RevId: 197597779
* Implement a limit for named_set_of_file eventsGravatar ulfjack2018-05-15
| | | | | | | | | | | This adds a command-line option which can be used to force Bazel to split up very large events, e.g., events with hundreds of thousands of entries. This is rare, but happens occasionally. It would be better to control the maximum event size directly, but that is significantly more difficult to do here. PiperOrigin-RevId: 196690600
* Use com.google.common.cache instead of MapMaker.Gravatar Googler2018-05-03
| | | | PiperOrigin-RevId: 195316047
* Serialization of NestedSet<NestedSet> with NestedSetCodecWithStore waits onGravatar cpeyser2018-05-03
| | | | | | backend writes for inner NestedSet serialization. PiperOrigin-RevId: 195294676
* Clean up code that directly imports nested classes like Builder, Entry, etc.Gravatar jcater2018-05-01
| | | | PiperOrigin-RevId: 195040539
* Wire up future to wait for in NestedSetCodecWithStore.Gravatar janakr2018-04-30
| | | | PiperOrigin-RevId: 194895469
* Refactor NestedSet to permit alternate NestedSetStore implementations.Gravatar cpeyser2018-04-30
| | | | PiperOrigin-RevId: 194787067
* Serialize NestedSet sizes that get special treatement (empty set, singleton)Gravatar cpeyser2018-04-25
| | | | | | instead of one boolean each to save a byte in the encoded stream. PiperOrigin-RevId: 194280646
* Serialize singleton NestedSets directly in NestedSetCodecWithStore.Gravatar cpeyser2018-04-25
| | | | PiperOrigin-RevId: 194232982
* Introduce NestedSetStore, which persists information used for NestedSet ↵Gravatar cpeyser2018-04-19
| | | | | | serialization. PiperOrigin-RevId: 193536486
* PiperOrigin-RevId: 193371485Gravatar carmi2018-04-18
|
* Clean up IterablesChain.Gravatar tomlu2018-04-18
| | | | | | | | | | | * Change builder return type to Iterable<T> instead of IterableChain<T>. It is over-specified and unnecessary to state the return type so precisely. * Optimize builder for cases where we add 0 or 1 iterables to the chain. In this case, we can simply return the underlying iterables without adding wrappers. * Extract DedupingIterable, it doesn't have anything to do with IterablesChain and is only used in one place RELNOTES: None PiperOrigin-RevId: 193363048
* Automated rollback of commit 3643afad04be41caa4b247d327230e8a1e32bb6a.Gravatar carmi2018-04-17
| | | | | | | | | | | *** Reason for rollback *** Breaks //third_party/java_src/copybara/java/com/google/copybara/config:parser: [] *** Original change description *** PiperOrigin-RevId: 193292991
* PiperOrigin-RevId: 193274137Gravatar carmi2018-04-17
|
* Introduce NestedSetCodecWithStore, which serializes NestedSet using a ↵Gravatar cpeyser2018-04-16
| | | | | | process-global bimap of fingerprints to NestedSet contents. PiperOrigin-RevId: 193063717
* Make some objects frequently encountered during serialization into ↵Gravatar janakr2018-04-10
| | | | | | | | constants. A lot of care is needed here because we're using reference equality. I plan to add value-equality constants in a follow-up. Add ImmutableSortedSet marshaller because I think it might have been needed, and hey, why not. PiperOrigin-RevId: 192326359
* Use @AutoCodec for NestedSet serialization.Gravatar janakr2018-04-05
| | | | PiperOrigin-RevId: 191797413
* Don't do serialization of empty/singleton nested sets into a child ↵Gravatar janakr2018-04-03
| | | | | | | | | | CodedOutputStream. It creates immense amounts of garbage and we don't ever use the result: it's only used for Object[] children anyway. We can consider removing the child CodedOutputStream entirely and relying on normal serialization memoization, but for now, let's just do the simple thing. Also fix a weird code-only bug that had been there since NestedSetCodec was written (I think): NestedSet.EMPTY_CHILDREN is an Object[], and therefore we never took the fast path of just writing 0 and moving on. While the code as written was misleading, the bits written to the output stream were the same, until this change, when there was a divergence. PiperOrigin-RevId: 191520712
* Disable nested set sharing across multiple nested set deserialization ↵Gravatar janakr2018-04-03
| | | | | | | | sessions. This is incorrect in the presence of memoization: a single element may be serialized as just a pair of integers (type + memoization index). Lots of different nested sets may contain elements that are serialized this way, so they will have the same digests. We could consider doing a parallel hash computation, but for now just disable. This is not a full rollback of https://github.com/bazelbuild/bazel/commit/39cef6d6a4a9e3ae80b11a9ccc0f35325852777c since there was a refactoring in it that it doesn't seem worth it to roll back. PiperOrigin-RevId: 191509089
* During NestedSet deserialization, maintain a list of known children outside ofGravatar cpeyser2018-03-26
| | | | | | | the global digestToChild map. Since digestToChild contains weak references, this is required to ensure the children are not GCed. PiperOrigin-RevId: 190476243
* Re-enable aliasing of CodedInputStream during deserialization, removed as ↵Gravatar janakr2018-03-23
| | | | | | | | part of primary codepath somewhere around https://github.com/bazelbuild/bazel/commit/bde43ec8a96a62b8fbf67ad60d5154cf121647d9 (and killed entirely in unknown commit. Force copying of a ByteString read from CodedInputStream in NestedSetCodec and persisted, since otherwise we might hang onto the entire buffer indefinitely. PiperOrigin-RevId: 190256337
* Optimistically check the Set within NestedSetVisitor.VisitedState using ↵Gravatar Googler2018-03-22
| | | | | | | | | contains before invoking the heavier add. This reduces cpu-work and contention. When blaze is invoked on a large fileset and the build is already up-to-date, this makes things about 1.9s (33%) faster. PiperOrigin-RevId: 190125803
* Allow NestedSetCodec to share members across multiple deserializations.Gravatar cpeyser2018-03-22
| | | | | | This avoids redundancy in memory when multiple NestedSets share a member PiperOrigin-RevId: 190085907
* Add behavior to NestedSetCodec to prevent it from running during testing.Gravatar cpeyser2018-03-21
| | | | PiperOrigin-RevId: 189906038
* Automated rollback of commit e6af9f9178e731a45e552b1d249ded90124945ba.Gravatar janakr2018-03-19
| | | | | | | | | | | | *** Reason for rollback *** See linked bug. *** Original change description *** Add behavior to NestedSetCodec to prevent it from running during testing. PiperOrigin-RevId: 189663863
* Add behavior to NestedSetCodec to prevent it from running during testing.Gravatar cpeyser2018-03-19
| | | | PiperOrigin-RevId: 189602622
* @AutoCodec WriteZipControlFileAction (and ImmutableIterable).Gravatar janakr2018-03-16
| | | | | | Since autocodec library is now a dependency of lib/collect, properly annotate ImmutableSharedKeyMap to boot. PiperOrigin-RevId: 189432552
* Splits the iterableMarshaller into runtime codecs.Gravatar shahan2018-03-14
| | | | | | Makes NestedSetCodec into a runtime codec instead of a Marshaller. PiperOrigin-RevId: 189110883
* Introduce an Extrema aggregator.Gravatar nharmata2018-02-28
| | | | | RELNOTES: None PiperOrigin-RevId: 187370833
* Better @AutoCodec for TransitiveInfoProviderMapImpl: avoid having to ↵Gravatar janakr2018-02-23
| | | | | | serialize offset table at the cost of some overhead reconstructing the table. Also fewer code changes, although there is a serialization-only method added as a hack. PiperOrigin-RevId: 186808832
* @AutoCodec TransitiveInfoProviderMapImpl and JavaInfo.Gravatar janakr2018-02-23
| | | | PiperOrigin-RevId: 186789569
* Adds ObjectCodecRegistry to {Des|S}erializationContext.Gravatar shahan2018-02-20
| | | | | | | | | | | | * AutoCodec now delegates to the registry. * Adds getSuperclass logic for resolving a codec. * Small cleanups for classes that break the registry. TODO after this change: * Explicit CODEC definitions are no longer needed and existing ones should be cleaned up. * POLYMORPHIC is no longer be needed and should be cleaned up. PiperOrigin-RevId: 186351845
* Move CommandLine, CommandLineItem, and ParamFileInfo from ↵Gravatar tomlu2018-02-15
| | | | | | | | | lib.analysis.actions -> lib.actions. These are fundamental types that want to sit alongside types like Spawn. RELNOTES: None PiperOrigin-RevId: 185887971
* Add a CODEC for CppCompileAction.Gravatar cpeyser2018-02-14
| | | | PiperOrigin-RevId: 185733313
* Replaces InjectingObjectCodec with dependencies threaded through ↵Gravatar shahan2018-02-13
| | | | | | (Des|S)erializationContext. PiperOrigin-RevId: 185547740
* Add context argument to ObjectCodec.{serialize,deserialize}Gravatar michajlo2018-02-11
| | | | | | | | | | Context implementations are currently empty, just doing the plumbing in this change. Once this is in we can start passing along the ObjectCodecRegistry, which will allow runtime codec resolution for classes not known at compile time. We'll also inevitably add some memoization helpers, allowing us to optimize the serialization process further. PiperOrigin-RevId: 185305674
* Add NestedSet support in AutoCodec for type parameters which have an ↵Gravatar cpeyser2018-02-05
| | | | | | InjectingObjectCodec. PiperOrigin-RevId: 184566571
* Allow CustomCommandLine's mapFn to expand each object into multiple items.Gravatar tomlu2018-02-01
| | | | | | This is needed to migrate JavaCompileAction away from CustomMultiArgv. PiperOrigin-RevId: 184136486
* Improve safety of NestedSetFingerprintCache by detecting multiple instances ↵Gravatar tomlu2018-01-31
| | | | | | | | | | | | of the same mapFn class. This code tries to add protection against the user creating new mapFn instances per-rule. This would cause the nested set cache to be computed per-rule instead of shared across rule instances, causing memory bloat and slowdowns. Since this can only happen in native code, we can get away with detecting this and crashing blaze. I think this is a better choice than silently allowing it / falling back to slow computations. The user can override this behaviour by inheriting from CommandLineItem.CapturingMapFn, in which case the user is explicitly saying they assume responsibility for the number of instances of the mapFn the application will use. PiperOrigin-RevId: 184061642
* Add memory-efficient map for storing nested set -> digest.Gravatar tomlu2018-01-31
| | | | | | | | | Instead of using ConcurrentHashMap, we use a dead-simple open addressed hash hable with a giant byte array with 16-byte slots. We then read or write fingerprints straight into and out of the array, obviating the need to generate intermediate garbage. Locking mechanism is a read-write lock. This should be faster than full synchronisation for read-heavy loads. RELNOTES: None PiperOrigin-RevId: 184019301
* Use nested set cache in key computation for CustomCommandLine.Gravatar tomlu2018-01-29
| | | | | RELNOTES: None PiperOrigin-RevId: 183727976
* Serializer implementation for NestedSetGravatar shahan2018-01-25
| | | | | | | | | | | | | Adds some logging to test helpers for size of serialized data. Jan 25, 2018 7:16:25 AM com.google.devtools.build.lib.skyframe.serialization.testutils.SerializerTester testSerializeDeserialize INFO: total serialized bytes = 70 Jan 25, 2018 7:16:25 AM com.google.devtools.build.lib.skyframe.serialization.testutils.ObjectCodecTester testSerializeDeserialize INFO: total serialized bytes = 208 Kryo output is significantly smaller. PiperOrigin-RevId: 183300353
* Add CommandLineItem interface.Gravatar tomlu2018-01-25
| | | | | | | | This interface makes it clearer in the type system exactly how items that go into a CustomCommandLine are turned into strings. It is a preparatory change to allow command line fingerprints to be more cheaply calculated, but it is valuable in itself from a code quality standpoint. PiperOrigin-RevId: 183274022
* Avoid expansion of NestedSets in MiddlemanFactory.Gravatar Benjamin Peterson2017-12-11
| | | | | | | | | | | | MiddlemanFactory wants to check if the middleman inputs set is empty or singleton. A NestedSet can answer both queries efficiently if the right APIs are used. My ultimate goal here is to avoid the expansion of runfiles artifacts nested sets until execution. Change-Id: I29a269df757ef41b1410bbb492cf24c926df6114 PiperOrigin-RevId: 178600943
* Add ActionKeyContext to Action#getKey.Gravatar tomlu2017-11-29
| | | | | | | This key context can be used by actions to share partial key computations, for instance when computing MD5s for nested sets. RELNOTES: None PiperOrigin-RevId: 177359607
* Support nested set serialization.Gravatar tomlu2017-11-27
| | | | PiperOrigin-RevId: 177032673
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-03 07:37:46 +0000