aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/main/java/com/google/devtools/build/lib/collect
Commit message (Collapse)AuthorAge
* Reduce contention in the digest map.Gravatar tomlu2018-08-09
| | | | | | | | | During reads and writes a read lock is taken. If we need to resize the table, a write lock is taken to flush all the readers and writers before the table is resized. The read lock should be significantly cheaper than full synchronisation since multiple threads can grab a read lock at once. RELNOTES: None PiperOrigin-RevId: 208059810
* Automated rollback of commit f309ad3be36363070e87eef0ee04b12f4956d601.Gravatar janakr2018-07-24
| | | | | | | | | | *** Reason for rollback *** Fixed duplicate derived inputs bug. Test is in diffbase. RELNOTES[INC]: If the same artifact is generated by two distinct but identical actions, and a downstream action has both those actions' outputs in its inputs, the artifact will now appear twice in the downstream action's inputs. If this causes problems in Skylark actions, you can use the uniquify=True argument in Args.add_args. PiperOrigin-RevId: 205863806
* Automated rollback of commit d491bf10f42e213292382c98a1dc439537f00f43.Gravatar janakr2018-07-17
| | | | | | | | | | | | | | | | | | *** Reason for rollback *** Still bugs lurking. See linked bug. *** Original change description *** Automated rollback of commit eb587075b0d6ffab1cf9e69ede1b7e547905e547. *** Reason for rollback *** Depot has been fixed. RELNOTES[INC]: If the same artifact is generated by two distinct but identical actions, and a downstream action has both those actions' outputs in its inputs, the artifact will now appear twice in the downstream action's inputs. If this causes problems in Skylark actions, you can use the uniquify=True argument in Args.add_args. PiperOrigin-RevId: 204997569
* Automated rollback of commit eb587075b0d6ffab1cf9e69ede1b7e547905e547.Gravatar janakr2018-07-16
| | | | | | | | | | *** Reason for rollback *** Depot has been fixed. RELNOTES[INC]: If the same artifact is generated by two distinct but identical actions, and a downstream action has both those actions' outputs in its inputs, the artifact will now appear twice in the downstream action's inputs. If this causes problems in Skylark actions, you can use the uniquify=True argument in Args.add_args. PiperOrigin-RevId: 204827477
* Automated rollback of commit 45b308a62f42c2c0bcfe79dcd4046c4025a31059.Gravatar janakr2018-06-20
| | | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Breaks Javascript compilation. There's currently no way to iterate over a depset of Artifacts and deduplicate by identical paths in Skylark. This means that actions that want to do something once per Artifact in a depset (add a flag to the command line with the path of the Artifact for instance) will have duplicate entries if there are multiple Artifacts with the same path. This is not a true automated rollback, since I tried to make this as minimal as possible, to avoid merge conflicts and keep some of the benefits of the rolled back CL. In particular, the tests that were changed in the original CL to give artifacts their correct owners did not need to be changed back, since the owners are still correct. Moreover, this effectively contains rollbacks of unknown commit and https://github.com/bazelbuild/bazel/commit/39d6f89644107a8f7c080c4f4aec8527c1a73954, but keeps test coverage from those CLs as well. Comments added to CL thread where not a clean rollback (there are plenty of files not changed at all: ActionArtifactCycleReporter is the main wart, since it can still handle SkyKeys with Artifact as the argument, but Artifact is no longer the argument to any SkyKey). RELNOTES: None *** Original change description *** Make Artifact#equals take the owner into account for derived artifacts. Derived artifacts' owners are important because they are used to determine the artifact's generating action. Source artifacts' owners are not used in this way, so I left them alone. This allows us to get rid of most uses of ArtifactSkyKey. We may be able to delete it entirely in a follow-up. PiperOrigin-RevId: 201464780
* Delete switch for nested set serialization. It's fast enough to be on by ↵Gravatar janakr2018-06-19
| | | | | | default. PiperOrigin-RevId: 201218341
* Fix bug in NestedSetStore where racing deserializations could create ↵Gravatar janakr2018-06-16
| | | | | | multiple futures for the same fingerprint and add a test showing that racing serializations can result in duplicate writes. PiperOrigin-RevId: 200860099
* Allow deserialization futures as NestedSet contents, with unrolling blocking ↵Gravatar cpeyser2018-06-13
| | | | | | | | on that future. This allows NestedSet deserialization not to block on storage reads - in-progress deserializations are simply made a member of the NestedSets they produce. PiperOrigin-RevId: 200440131
* Save nested set size after initial expand to avoid GC from resizing list ↵Gravatar tomlu2018-06-13
| | | | | | | | | | | when replaying. We stash the size in the same field as the order using bit packing. The small additional cost of masking out the int should be less than the resizing when replaying large nested sets. The upper size bound on nested sets will be 512M objects as a result. If Java had unsigned ints we could push it to 1G. RELNOTES: None PiperOrigin-RevId: 200417105
* Make Artifact#equals take the owner into account for derived artifacts.Gravatar janakr2018-06-08
| | | | | | | | Derived artifacts' owners are important because they are used to determine the artifact's generating action. Source artifacts' owners are not used in this way, so I left them alone. This allows us to get rid of most uses of ArtifactSkyKey. We may be able to delete it entirely in a follow-up. PiperOrigin-RevId: 199836436
* Add cache to NestedSetCodecWithStore to merge NestedSets that should be ↵Gravatar janakr2018-06-04
| | | | | | | | | | reference-equal on deserialization. We cannot just intern NestedSets because NestedSets with the same underlying children may still not be equal, so we wrap them in an object that does consider their children when calculating equality. We wish to preserve the invariant that if NestedSets inside two different objects are reference-equal, they will continue to be reference-equal after deserialization. Not doing that causes bugs. Unfortunately, because Artifact#equals does not take ArtifactOwner into account, this introduces a new bug (exposed via a disabled test here) where unequal singleton NestedSets may be considered equal. I will clean this up in the future by fixing Artifact#equals. PiperOrigin-RevId: 199208045
* Serialize NestedSetStore$NestedSetSize directly as an ordinal instead ofGravatar cpeyser2018-05-22
| | | | | | through SerializationContext, to save a byte. PiperOrigin-RevId: 197597779
* Implement a limit for named_set_of_file eventsGravatar ulfjack2018-05-15
| | | | | | | | | | | This adds a command-line option which can be used to force Bazel to split up very large events, e.g., events with hundreds of thousands of entries. This is rare, but happens occasionally. It would be better to control the maximum event size directly, but that is significantly more difficult to do here. PiperOrigin-RevId: 196690600
* Use com.google.common.cache instead of MapMaker.Gravatar Googler2018-05-03
| | | | PiperOrigin-RevId: 195316047
* Serialization of NestedSet<NestedSet> with NestedSetCodecWithStore waits onGravatar cpeyser2018-05-03
| | | | | | backend writes for inner NestedSet serialization. PiperOrigin-RevId: 195294676
* Clean up code that directly imports nested classes like Builder, Entry, etc.Gravatar jcater2018-05-01
| | | | PiperOrigin-RevId: 195040539
* Wire up future to wait for in NestedSetCodecWithStore.Gravatar janakr2018-04-30
| | | | PiperOrigin-RevId: 194895469
* Refactor NestedSet to permit alternate NestedSetStore implementations.Gravatar cpeyser2018-04-30
| | | | PiperOrigin-RevId: 194787067
* Serialize NestedSet sizes that get special treatement (empty set, singleton)Gravatar cpeyser2018-04-25
| | | | | | instead of one boolean each to save a byte in the encoded stream. PiperOrigin-RevId: 194280646
* Serialize singleton NestedSets directly in NestedSetCodecWithStore.Gravatar cpeyser2018-04-25
| | | | PiperOrigin-RevId: 194232982
* Introduce NestedSetStore, which persists information used for NestedSet ↵Gravatar cpeyser2018-04-19
| | | | | | serialization. PiperOrigin-RevId: 193536486
* PiperOrigin-RevId: 193371485Gravatar carmi2018-04-18
|
* Clean up IterablesChain.Gravatar tomlu2018-04-18
| | | | | | | | | | | * Change builder return type to Iterable<T> instead of IterableChain<T>. It is over-specified and unnecessary to state the return type so precisely. * Optimize builder for cases where we add 0 or 1 iterables to the chain. In this case, we can simply return the underlying iterables without adding wrappers. * Extract DedupingIterable, it doesn't have anything to do with IterablesChain and is only used in one place RELNOTES: None PiperOrigin-RevId: 193363048
* Automated rollback of commit 3643afad04be41caa4b247d327230e8a1e32bb6a.Gravatar carmi2018-04-17
| | | | | | | | | | | *** Reason for rollback *** Breaks //third_party/java_src/copybara/java/com/google/copybara/config:parser: [] *** Original change description *** PiperOrigin-RevId: 193292991
* PiperOrigin-RevId: 193274137Gravatar carmi2018-04-17
|
* Introduce NestedSetCodecWithStore, which serializes NestedSet using a ↵Gravatar cpeyser2018-04-16
| | | | | | process-global bimap of fingerprints to NestedSet contents. PiperOrigin-RevId: 193063717
* Make some objects frequently encountered during serialization into ↵Gravatar janakr2018-04-10
| | | | | | | | constants. A lot of care is needed here because we're using reference equality. I plan to add value-equality constants in a follow-up. Add ImmutableSortedSet marshaller because I think it might have been needed, and hey, why not. PiperOrigin-RevId: 192326359
* Use @AutoCodec for NestedSet serialization.Gravatar janakr2018-04-05
| | | | PiperOrigin-RevId: 191797413
* Don't do serialization of empty/singleton nested sets into a child ↵Gravatar janakr2018-04-03
| | | | | | | | | | CodedOutputStream. It creates immense amounts of garbage and we don't ever use the result: it's only used for Object[] children anyway. We can consider removing the child CodedOutputStream entirely and relying on normal serialization memoization, but for now, let's just do the simple thing. Also fix a weird code-only bug that had been there since NestedSetCodec was written (I think): NestedSet.EMPTY_CHILDREN is an Object[], and therefore we never took the fast path of just writing 0 and moving on. While the code as written was misleading, the bits written to the output stream were the same, until this change, when there was a divergence. PiperOrigin-RevId: 191520712
* Disable nested set sharing across multiple nested set deserialization ↵Gravatar janakr2018-04-03
| | | | | | | | sessions. This is incorrect in the presence of memoization: a single element may be serialized as just a pair of integers (type + memoization index). Lots of different nested sets may contain elements that are serialized this way, so they will have the same digests. We could consider doing a parallel hash computation, but for now just disable. This is not a full rollback of https://github.com/bazelbuild/bazel/commit/39cef6d6a4a9e3ae80b11a9ccc0f35325852777c since there was a refactoring in it that it doesn't seem worth it to roll back. PiperOrigin-RevId: 191509089
* During NestedSet deserialization, maintain a list of known children outside ofGravatar cpeyser2018-03-26
| | | | | | | the global digestToChild map. Since digestToChild contains weak references, this is required to ensure the children are not GCed. PiperOrigin-RevId: 190476243
* Re-enable aliasing of CodedInputStream during deserialization, removed as ↵Gravatar janakr2018-03-23
| | | | | | | | part of primary codepath somewhere around https://github.com/bazelbuild/bazel/commit/bde43ec8a96a62b8fbf67ad60d5154cf121647d9 (and killed entirely in unknown commit. Force copying of a ByteString read from CodedInputStream in NestedSetCodec and persisted, since otherwise we might hang onto the entire buffer indefinitely. PiperOrigin-RevId: 190256337
* Optimistically check the Set within NestedSetVisitor.VisitedState using ↵Gravatar Googler2018-03-22
| | | | | | | | | contains before invoking the heavier add. This reduces cpu-work and contention. When blaze is invoked on a large fileset and the build is already up-to-date, this makes things about 1.9s (33%) faster. PiperOrigin-RevId: 190125803
* Allow NestedSetCodec to share members across multiple deserializations.Gravatar cpeyser2018-03-22
| | | | | | This avoids redundancy in memory when multiple NestedSets share a member PiperOrigin-RevId: 190085907
* Add behavior to NestedSetCodec to prevent it from running during testing.Gravatar cpeyser2018-03-21
| | | | PiperOrigin-RevId: 189906038
* Automated rollback of commit e6af9f9178e731a45e552b1d249ded90124945ba.Gravatar janakr2018-03-19
| | | | | | | | | | | | *** Reason for rollback *** See linked bug. *** Original change description *** Add behavior to NestedSetCodec to prevent it from running during testing. PiperOrigin-RevId: 189663863
* Add behavior to NestedSetCodec to prevent it from running during testing.Gravatar cpeyser2018-03-19
| | | | PiperOrigin-RevId: 189602622
* @AutoCodec WriteZipControlFileAction (and ImmutableIterable).Gravatar janakr2018-03-16
| | | | | | Since autocodec library is now a dependency of lib/collect, properly annotate ImmutableSharedKeyMap to boot. PiperOrigin-RevId: 189432552
* Splits the iterableMarshaller into runtime codecs.Gravatar shahan2018-03-14
| | | | | | Makes NestedSetCodec into a runtime codec instead of a Marshaller. PiperOrigin-RevId: 189110883
* Introduce an Extrema aggregator.Gravatar nharmata2018-02-28
| | | | | RELNOTES: None PiperOrigin-RevId: 187370833
* Better @AutoCodec for TransitiveInfoProviderMapImpl: avoid having to ↵Gravatar janakr2018-02-23
| | | | | | serialize offset table at the cost of some overhead reconstructing the table. Also fewer code changes, although there is a serialization-only method added as a hack. PiperOrigin-RevId: 186808832
* @AutoCodec TransitiveInfoProviderMapImpl and JavaInfo.Gravatar janakr2018-02-23
| | | | PiperOrigin-RevId: 186789569
* Adds ObjectCodecRegistry to {Des|S}erializationContext.Gravatar shahan2018-02-20
| | | | | | | | | | | | * AutoCodec now delegates to the registry. * Adds getSuperclass logic for resolving a codec. * Small cleanups for classes that break the registry. TODO after this change: * Explicit CODEC definitions are no longer needed and existing ones should be cleaned up. * POLYMORPHIC is no longer be needed and should be cleaned up. PiperOrigin-RevId: 186351845
* Move CommandLine, CommandLineItem, and ParamFileInfo from ↵Gravatar tomlu2018-02-15
| | | | | | | | | lib.analysis.actions -> lib.actions. These are fundamental types that want to sit alongside types like Spawn. RELNOTES: None PiperOrigin-RevId: 185887971
* Add a CODEC for CppCompileAction.Gravatar cpeyser2018-02-14
| | | | PiperOrigin-RevId: 185733313
* Replaces InjectingObjectCodec with dependencies threaded through ↵Gravatar shahan2018-02-13
| | | | | | (Des|S)erializationContext. PiperOrigin-RevId: 185547740
* Add context argument to ObjectCodec.{serialize,deserialize}Gravatar michajlo2018-02-11
| | | | | | | | | | Context implementations are currently empty, just doing the plumbing in this change. Once this is in we can start passing along the ObjectCodecRegistry, which will allow runtime codec resolution for classes not known at compile time. We'll also inevitably add some memoization helpers, allowing us to optimize the serialization process further. PiperOrigin-RevId: 185305674
* Add NestedSet support in AutoCodec for type parameters which have an ↵Gravatar cpeyser2018-02-05
| | | | | | InjectingObjectCodec. PiperOrigin-RevId: 184566571
* Allow CustomCommandLine's mapFn to expand each object into multiple items.Gravatar tomlu2018-02-01
| | | | | | This is needed to migrate JavaCompileAction away from CustomMultiArgv. PiperOrigin-RevId: 184136486
* Improve safety of NestedSetFingerprintCache by detecting multiple instances ↵Gravatar tomlu2018-01-31
| | | | | | | | | | | | of the same mapFn class. This code tries to add protection against the user creating new mapFn instances per-rule. This would cause the nested set cache to be computed per-rule instead of shared across rule instances, causing memory bloat and slowdowns. Since this can only happen in native code, we can get away with detecting this and crashing blaze. I think this is a better choice than silently allowing it / falling back to slow computations. The user can override this behaviour by inheriting from CommandLineItem.CapturingMapFn, in which case the user is explicitly saying they assume responsibility for the number of instances of the mapFn the application will use. PiperOrigin-RevId: 184061642
generated by cgit v1.2.3 (git 2.39.1) at 2024-06-16 20:51:07 +0000