aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/test/java/com/google/devtools/build/lib/skyframe
Commit message (Collapse)AuthorAge
* Check if ParentFileValue is a directory when evaluating a FileFunction node.Gravatar neerajen2018-07-19
| | | | | RELNOTES: None. PiperOrigin-RevId: 205288166
* Introduce --strict_fileset_output which treats all output Artifacts ↵Gravatar felly2018-07-18
| | | | | | encountered in a Fileset as a regular file. PiperOrigin-RevId: 205152271
* Automated rollback of commit d491bf10f42e213292382c98a1dc439537f00f43.Gravatar janakr2018-07-17
| | | | | | | | | | | | | | | | | | *** Reason for rollback *** Still bugs lurking. See linked bug. *** Original change description *** Automated rollback of commit eb587075b0d6ffab1cf9e69ede1b7e547905e547. *** Reason for rollback *** Depot has been fixed. RELNOTES[INC]: If the same artifact is generated by two distinct but identical actions, and a downstream action has both those actions' outputs in its inputs, the artifact will now appear twice in the downstream action's inputs. If this causes problems in Skylark actions, you can use the uniquify=True argument in Args.add_args. PiperOrigin-RevId: 204997569
* Move constraint value lookup to a new utility function.Gravatar John Cater2018-07-17
| | | | | Change-Id: I13645199a21991b1458264f9d9ac2db2736066b2 PiperOrigin-RevId: 204963740
* Move platform lookup to new dedicated helper class.Gravatar John Cater2018-07-17
| | | | | Change-Id: I2195a3f7d60e9835bc00d6f151ea91271fdccf6f PiperOrigin-RevId: 204954233
* Automated rollback of commit eb587075b0d6ffab1cf9e69ede1b7e547905e547.Gravatar janakr2018-07-16
| | | | | | | | | | *** Reason for rollback *** Depot has been fixed. RELNOTES[INC]: If the same artifact is generated by two distinct but identical actions, and a downstream action has both those actions' outputs in its inputs, the artifact will now appear twice in the downstream action's inputs. If this causes problems in Skylark actions, you can use the uniquify=True argument in Args.add_args. PiperOrigin-RevId: 204827477
* Add implicit mapping from "@mainrepo" to "@". This fixes the issue where ↵Gravatar Danna Kelmer2018-07-16
| | | | | | | | | | referencing the main repository using its name caused bazel to treat it as a separate external repository. Closes #5586. Fixes #3115. RELNOTES: None PiperOrigin-RevId: 204752150
* Code Cleanup: Simplify the ResolvedFile interface by removing unnecessary ↵Gravatar kush2018-07-13
| | | | | | | test hooks. RELNOTES: None PiperOrigin-RevId: 204468647
* Set the version of a computed node to the max of its child versions rather ↵Gravatar janakr2018-07-11
| | | | | | | | | | | | | | | | | than the graph version when that is feasible. * It's not feasible when the computation accesses outside state, i.e. is non-hermetic, so see below. * It's also more complicated (and not worth the trouble) when the computation is taking place just for the error status. Have SkyFunctionName declare whether the function it corresponds to is hermetic or non-hermetic. Only non-hermetically-generated SkyValues can be directly marked changed, and non-hermetic SkyFunctions have their values saved at the graph version, not the max of the child versions. All SkyFunctions are hermetic except for the ones that can be explicitly dirtied. A marked-hermetic SkyFunction that has a transient error due to filesystem access can be re-evaluated and get the correct version: if it throws an IOException at version 1 and then, when re-evaluated at version 2 with unchanged dependencies, has a value, the version will be version 1. All Skyframe unit tests that were doing non-hermetic things to nodes need to declare that those nodes are non-hermetic. I tried to make the minimal set of changes there, so that we had good incidental coverage of hermetic+non-hermetic nodes. Also did some drive-by clean-ups around that code. Artifacts are a weird case, since they're doing untracked filesystem access (for source directories). Using max(child versions) for them gives rise to the following correctness bug: 1. do a build at v1 that creates a FileStateValue for dir/ at v1. Then at v2, add a file to dir/ and do a build that consumes dir/ as a source artifact. Now the artifact for dir/ will (incorrectly) have v1. Then at v1, do that build again. We'll consume the "artifact from the future". However, this can only have an effect when using the local action cache, since the incorrect value of the artifact (the mtime) is only consumed by the action cache. Bazel is already broken in this way (incremental builds don't invalidate directories), so this change doesn't make things worse. PiperOrigin-RevId: 204210719
* Make the FileArtifactvalue of generated targets available via the ↵Gravatar kush2018-07-11
| | | | | | | | | | | | | | FilesetOutputSymlink whenever available. In this change I'm simply plumbing the FileArtifactValue we requested within RecursiveFilesystemTraversalFunction to the FilesetOutputSymlink. This does not work when the targets are output directories (or symlink to output dirs). The main scenarios this happens is when: 1. Fileset depends on the output dir created by a genrule. 2. Fileset depends on a GoAppengineBinary which creates an output dir. 3. Fileset depends on another Fileset in the non-recommended way (Fileset.entry.files = [<another_fileset>]) instead of the recommended way (FilesetEntry.srcdir = <another_fileset>). RELNOTES: None PiperOrigin-RevId: 204209612
* Remove ToolchainContextException and make exception handling more robust.Gravatar John Cater2018-07-10
| | | | | Change-Id: Iefeab0de85541b963c6cc262339e73b79f630f8b PiperOrigin-RevId: 203952823
* Pull up calls to ToolchainContext.resolvedToolchainLabels and finishGravatar John Cater2018-07-09
| | | | | | | ToolchainContext configuration earlier. Change-Id: I034f56a8a181aac19ecb38e51558ccdc6c0c7a6a PiperOrigin-RevId: 203835978
* Bazel server: ensure OutputStreams are closedGravatar laszlocsomor2018-07-05
| | | | | | | | | | | | | | | | | | | Use try-with-resources to ensure OutputStreams that we open via FileSystem.OutputStream(path) are closed. Eagerly closing OutputStreams avoids hanging on to file handles until the garbage collector finalizes the OutputStream, meaning Bazel on Windows (and other processes) can delete or mutate these files. Hopefully this avoids intermittent file deletion errors that sometimes occur on Windows. See https://github.com/bazelbuild/bazel/issues/5512 RELNOTES: none PiperOrigin-RevId: 203342889
* Automated refactor of Label.parseAbsolute() to always pass a repository mappingGravatar dannark2018-06-27
| | | | | RELNOTES: None PiperOrigin-RevId: 202360925
* Instead of depending on mutable inputs of upstream CppCompileAction instances,Gravatar shahan2018-06-26
| | | | | | | | | CppCompileAction.discoverInputsStage2 retrieves values of discovered modules from ActionExecutionValue. This addresses a possible a correctness issue. PiperOrigin-RevId: 202162180
* ToolchainContext directly implements SkylarkValue and SkylarkIndexable.Gravatar John Cater2018-06-22
| | | | | Change-Id: I355b138e143771fd826ab03951df821ea7d58ac5 PiperOrigin-RevId: 201740564
* Move HashFunction out of FileSystem, and turn it into a class, instead of an ↵Gravatar ccalvarin2018-06-21
| | | | | | | | | enum. Now that we aren't using enum names for the hash functions, we also accept the standard names, such as SHA-256. RELNOTES: None. PiperOrigin-RevId: 201624286
* Automated rollback of commit 45b308a62f42c2c0bcfe79dcd4046c4025a31059.Gravatar janakr2018-06-20
| | | | | | | | | | | | | | | | | | | | | | | | *** Reason for rollback *** Breaks Javascript compilation. There's currently no way to iterate over a depset of Artifacts and deduplicate by identical paths in Skylark. This means that actions that want to do something once per Artifact in a depset (add a flag to the command line with the path of the Artifact for instance) will have duplicate entries if there are multiple Artifacts with the same path. This is not a true automated rollback, since I tried to make this as minimal as possible, to avoid merge conflicts and keep some of the benefits of the rolled back CL. In particular, the tests that were changed in the original CL to give artifacts their correct owners did not need to be changed back, since the owners are still correct. Moreover, this effectively contains rollbacks of unknown commit and https://github.com/bazelbuild/bazel/commit/39d6f89644107a8f7c080c4f4aec8527c1a73954, but keeps test coverage from those CLs as well. Comments added to CL thread where not a clean rollback (there are plenty of files not changed at all: ActionArtifactCycleReporter is the main wart, since it can still handle SkyKeys with Artifact as the argument, but Artifact is no longer the argument to any SkyKey). RELNOTES: None *** Original change description *** Make Artifact#equals take the owner into account for derived artifacts. Derived artifacts' owners are important because they are used to determine the artifact's generating action. Source artifacts' owners are not used in this way, so I left them alone. This allows us to get rid of most uses of ArtifactSkyKey. We may be able to delete it entirely in a follow-up. PiperOrigin-RevId: 201464780
* Add new BuildMetrics event to BEP.Gravatar tomlu2018-06-19
| | | | | | | To start we add just a single metric, the number of actions constructed in the current build. RELNOTES: None PiperOrigin-RevId: 201248490
* Expand the abilities of codecs. Let ImmutableMultimapCodec actually handle ↵Gravatar janakr2018-06-19
| | | | | | | | LinkedHashMultimap as well. Let ImmutableSetRuntimeCodec handle some other sets. We squash these extra types into the same codec because we want them to become immutable anyway, and they can be dealt with easily by the existing codecs. Add some more non-Bazel classes that should be DynamicCodec'ed. Also, when a disallowed value is serialized, throw a SerializationException instead of crashing. This makes it easier to recover in tests. PiperOrigin-RevId: 201237631
* Put a trail into every SerializationException, not just NoCodecException, ↵Gravatar janakr2018-06-19
| | | | | | and have DynamicCodec add to it. It's very useful to have this trail available when debugging or whitelisting. PiperOrigin-RevId: 201205884
* Remove support for --discard_actions_after_execution.Gravatar tomlu2018-06-18
| | | | | | | The memory savings from this flag are not worth the complexity, and it interferes with action restarting. RELNOTES: Remove support for --discard_actions_after_execution. PiperOrigin-RevId: 201077905
* Refactoring: uses OutputService for ActionFileSystem injectionGravatar shahan2018-06-18
| | | | | | Moves more Action-oriented from skyframe package, which has become very unwieldy, to action package. This is needed to avoid circular dependencies caused when build-base is needed for skyframe. PiperOrigin-RevId: 200996982
* ActionFS supports fast digests. Add test coverage for this and some file ↵Gravatar felly2018-06-15
| | | | | | | metadata operations. RELNOTES: None PiperOrigin-RevId: 200765925
* Support ActionFS opening files with append. Also, fix ActionFS notifying its ↵Gravatar felly2018-06-15
| | | | | | | metadata consumer multiple times per inline output. RELNOTES: None PiperOrigin-RevId: 200730252
* Create an ImmutableTableCodec.Gravatar mjhalupka2018-06-14
| | | | PiperOrigin-RevId: 200561008
* Support unconditional fetching of repositoriesGravatar Klaus Aehlig2018-06-14
| | | | | | | | | | | Make all external repositories depend on an additional SkyValue controllable via commands, so support unconditional fetching of all external repositories, as it is needed by the the `sync` command. Improves on #5175, provides a work around for #4907. Change-Id: I30033614c1a2fad3f1363b85ff69cf92f697c255 PiperOrigin-RevId: 200543985
* Serialize lambdas when they are cast to Serializable.Gravatar janakr2018-06-13
| | | | | | We could conceivably do some monkey-patching at server startup to make all lambdas act Serializable, but one step at a time. PiperOrigin-RevId: 200509321
* Add functionality to make certain SkyValues unshareable, meaning they are ↵Gravatar janakr2018-06-13
| | | | | | not serialized. Tag TestCompletionValue and any ActionExecutionValue coming from a NotifyOnActionCacheHit (i.e., tests) like that. To make such values really not shared, request the ActionExecutionValue from TestCompletionFunction as opposed to the ArtifactValue (propagating the unshareable bit up seemed like too much fuss, and I have a dream of getting rid of ArtifactValue anyway). PiperOrigin-RevId: 200504358
* Fix a bug with flush() not flushing on local writes in ActionFS, and add ↵Gravatar felly2018-06-12
| | | | | | | | | unit test coverage for this. Note that ActionFS is not generic enough to make use of FileSystemTest. RELNOTES: None PiperOrigin-RevId: 200304871
* Add more data to the exception thrown when putOnlyEntry's verification fails.Gravatar mstaib2018-06-12
| | | | | | | | | If this exception is thrown, performance no longer matters - we're detonating the place and riding the explosion out to stderr. So we might as well just dump everything we know. RELNOTES: None. PiperOrigin-RevId: 200290439
* Always run toolchain resolution, even when no toolchain types are requested, ↵Gravatar John Cater2018-06-12
| | | | | | | in order to properly choose the execution platform from the available execution platforms. Change-Id: I05dc84403e0db765865e9b91c4222894fa867cd9 PiperOrigin-RevId: 200211635
* Allow delegation to the underlying filesystem only for the source tree.Gravatar felly2018-06-11
| | | | | | | The on-disk execRoot/blaze-out is now off limits to ActionFS. RELNOTES: None PiperOrigin-RevId: 200080287
* Make Artifact#equals take the owner into account for derived artifacts.Gravatar janakr2018-06-08
| | | | | | | | Derived artifacts' owners are important because they are used to determine the artifact's generating action. Source artifacts' owners are not used in this way, so I left them alone. This allows us to get rid of most uses of ArtifactSkyKey. We may be able to delete it entirely in a follow-up. PiperOrigin-RevId: 199836436
* Deletes ActionInputFileCache.Gravatar shahan2018-06-07
| | | | PiperOrigin-RevId: 199732415
* Refactoring: makes the code simpler by deleting Metadata and InputArtifactData.Gravatar shahan2018-06-06
| | | | | | (minor) ActionFS now implements MetadataProvider.getInput PiperOrigin-RevId: 199575194
* Disable `repo_mapping` by default via a flag.Gravatar dannark2018-06-05
| | | | | RELNOTES: None PiperOrigin-RevId: 199374368
* Allow rules and targets to specify extra execution platform constraints.Gravatar John Cater2018-06-05
| | | | | | | | | | | | | | | | | | | | | | RuleClass.Builder now allows authors to specify whether a rule's targets can add additional constraints on the execution platform, and to declare additional constraints for all targets of that rule. Targets which support this now have an attribute, "exec_compatible_with", which supports specifying additional constraints that the execution platform used must match. This attribute is non-configurable. It will only affect execution platforms used during toolchain resolution. Part of #5217. Change-Id: Id2400dbf869a00aa2be3e3d2f085c2850cd6dc00 Closes #5227. Change-Id: If7d55f08f7f44bc7d7f6dfec86a3e6bcd68574b9 PiperOrigin-RevId: 199326255
* Add functionality to SerializationContext and @AutoCodec to check that a ↵Gravatar janakr2018-06-05
| | | | | | | | class is allowed to be serialized in the current context. A codec can now add an explicitly allowed class that can be serialized underneath it (via SerializationContext#addExplicitlyAllowedClass), and that class's codec can check that it is explicitly allowed (via SerializationContext#checkClassExplicitlyAllowed). It is a runtime crash if a codec checks that it was explicitly allowed and finds that it wasn't. Thus, if PackageCodec is invoked without it having been explicitly allowed, we will crash, preventing Package from sneaking into a value it shouldn't be in. This is only enabled if the codec is memoizing. PiperOrigin-RevId: 199317936
* Process 'repo_mapping' attribute from WORKSPACE rules. 'repo_mapping' is a ↵Gravatar dannark2018-06-04
| | | | | | | | | | | | | | | | way to remap references to repositories within an external repository by another name. This CL only adds the mappings to the Package object, but it does not actually enable the reassignments. Example usage (in a WORKSPACE file): local_repository( name = ?a?, path = ?../a?, repo_mapping = {?@x? : ?@y?} ) This change also creates a new SkyKey which represents the mappings. This is to prevent all packages from depending on the external package, and instead depending just on the mappings. i.e. a change to the WORKSPACE file that does not touch the mappings shouldn't cause a reload of the package. RELNOTES:None PiperOrigin-RevId: 199187963
* Remove ConfiguredTarget from TargetCompletionValue, since it is no longer ↵Gravatar janakr2018-06-04
| | | | | | needed: we can get the ConfiguredTargetKey directly from the TargetCompletionKey. Since that was the only use of the actual value in EvaluationProgressReceiver#evaluated, remove it, instead just provide a boolean enum that gives whether or not evaluation succeeded. PiperOrigin-RevId: 199178047
* Failing targets are identified by comparing ConfiguredTarget.Key instead of ↵Gravatar cpeyser2018-06-04
| | | | | | ConfiguredTarget. Since ConfiguredTarget does not implement equals(), reference equality has been used until now, and this is not valid for the case where the ConfiguredTarget has been deserialized. PiperOrigin-RevId: 199133926
* Carry tree artifacts' self data through aggregating middlemen.Gravatar Benjamin Peterson2018-06-03
| | | | | | | | | | | | | | | | | | | | | | | | This CL is a followup to c868c47. It is intended to fix #5296. The immediate problem in that issue is that remote caching finds a tree artifact in an input runfiles tree without metadata in the cache. The metadata isn't there because the runfiles artifacts are now all behind a middleman action, and ArtifactFunction didn't propagate tree artifact values through middlemen. This change fixes the problem by adding a tree artifact's self data value to the values carried through to dependent actions. Remote caching with tree artifact inputs seems to have been working mostly by accident. (Evidence: it wasn't tested, and remote execution of actions with tree artifact inputs results a Java traceback before and after c868c47.) This CL is only really a bandaid to return to the old "working" state. If remote execution wants to start supporting tree artifacts properly in the future, we'll likely need to carry the tree artifact child data through middleman actions, too. Closes #5299. Change-Id: I2e07d4fca0f6d2d34d97b7edb27f9d0063776225 PiperOrigin-RevId: 199079382
* Allow multiple trimming transition factories to be added.Gravatar mstaib2018-06-01
| | | | | | | | If multiple trimming transition factories are added, they are composed in the order they were added. RELNOTES: None. PiperOrigin-RevId: 198934666
* Refactor root cause reporting in ConfiguredTargetFunctionGravatar ulfjack2018-05-29
| | | | | | We now track Causes instead of plain Labels, which will allow us to do better reporting in the future. Add basic tests. PiperOrigin-RevId: 198380468
* Simplify split transition semantics: noops now return the input build options.Gravatar gregce2018-05-23
| | | | | | This results in less special logic in the implementation and a simpler API. PiperOrigin-RevId: 197772283
* Rename PatchTransition.apply to PatchTransiton.patch.Gravatar gregce2018-05-23
| | | | | | | This unclashes with the incoming ConfigurationTransition.apply method described in https://docs.google.com/document/d/1_UJKmAQ9EE8i3Pl0il3YLTYr-Q9EKYYyLatt2zohfyM/edit#heading=h.96gongkwg852. PiperOrigin-RevId: 197769784
* Remove string prefix optimization. With new UnsafeJdk9StringCodec, it's no ↵Gravatar janakr2018-05-21
| | | | | | longer worth it. PiperOrigin-RevId: 197477783
* Remove ValueConstants. They're not pulling their weight in CPU overhead.Gravatar janakr2018-05-21
| | | | PiperOrigin-RevId: 197465288
* When using JDK9, replace naive StringCodec with an optimized codec that uses ↵Gravatar janakr2018-05-21
| | | | | | the String's underlying raw bytes. This avoids byte copying and UTF interpretation. Experiments show it is approximately 15(!) times faster than the naive StringCodec for serialization and 2 times faster for deserialization (10 times faster for non-ASCII strings). PiperOrigin-RevId: 197441758
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-08 07:35:59 +0000