| Commit message (Collapse) | Author | Age |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Path and PathFragment have been replaced with String-based implementations. They are pretty similar, but each method is dissimilar enough that I did not feel sharing code was appropriate.
A summary of changes:
PATH
====
* Subsumes LocalPath (deleted, its tests repurposed)
* Use a simple string to back Path
* Path instances are no longer interned; Reference equality will no longer work
* Always normalized (same as before)
* Some operations will now be slower, like instance compares (which were previously just a reference check)
* Multiple identical paths will now consume more memory since they are not interned
PATH FRAGMENT
=============
* Use a simple string to back PathFragment
* No more segment arrays with interned strings
* Always normalized
* Remove isNormalized
* Replace some isNormalizied uses with containsUpLevelReferences() to check if path fragments try to escape their scope
* To check if user input is normalized, supply static methods on PathFragment to validate the string before constructing a PathFragment
* Because PathFragments are always normalized, we have to replace checks for literal "." from PathFragment#getPathString to PathFragment#getSafePathString. The latter returns "." for the empty string.
* The previous implementation supported efficient segment semantics (segment count, iterating over segments). This is now expensive since we do longer have a segment array.
ARTIFACT
========
* Remove Path instance. It is instead dynamically constructed on request. This is necessary to avoid this CL becoming a memory regression.
RELNOTES: None
PiperOrigin-RevId: 185062932
|
| |
|
|
|
|
| |
SpecialArtifact.
PiperOrigin-RevId: 184539696
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Local execution has an inherent race condition: if a user modifies a file while an action is executed, then it is impossible for Bazel to tell which version of the file was actually read during action execution. The file may have been modified before or after the tool has read it, or, in the worst case, the tool may have read both the original and the modified version. In addition, the file may be changed back to the original state before Bazel can check the file, so computing the digest before / after may not be sufficient.
This is a concern for both local and remote caches, although the cost of poisoning a shared remote cache is significantly higher, and is what has triggered this work.
Fixes #3360.
We solve this by keeping a reference to the FileContentsProxy, and using that to check for modificaitons before storing the cache entry. We output a warning if this check fails.
This change does not increase memory consumption; Java objects are always allocated in multiples of 8 bytes, we use compressed oops, and the FileArtifactValue currently has 12 bytes worth of fields (excl. object overhead), so adding another pointer is effectively free.
As a possible performance optimization on purely local builds, we could also consider not computing digests at all, and only use the FileContentsProxy for caching.
PiperOrigin-RevId: 182510358
|
| |
|
|
|
|
| |
This is slightly more descriptive, and we will potentially want to use the name Root for a broader concept shared between ArtifactRoot and RootedPath.
PiperOrigin-RevId: 182082367
|
| |
|
|
|
|
| |
removing the layer of indirection of getting SkyKey out of ActionLookupKey, which uses more memory for no reason.
PiperOrigin-RevId: 181658615
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fixes #4322, #4306.
*** Reason for rollback ***
Introduces a deadlock (see https://github.com/bazelbuild/bazel/issues/4322)
*** Original change description ***
Make FileSystem operate on LocalPath instead of Path.
PiperOrigin-RevId: 179549866
|
| |
|
|
| |
PiperOrigin-RevId: 179082062
|
| |
|
|
|
|
|
| |
This key context can be used by actions to share partial key computations, for instance when computing MD5s for nested sets.
RELNOTES: None
PiperOrigin-RevId: 177359607
|
| |
|
|
|
|
| |
or no edges. Also add option to disable checks in MemoizingEvaluatorTest that don't make sense for implementations that don't keep track of dirty nodes. Also extract RecordingDifferencer to an interface. And add a test for the situation that a node changes during a build that it's not requested, and which fails, necessitating cleanup.
PiperOrigin-RevId: 171616817
|
| |
|
|
|
|
|
|
|
|
|
|
| |
precomputed value. Instead, manually check if the value has changed, and if it has, invalidate its consuming WorkspaceStatusValue node, forcing its re-evaluation, where it will pick up the new value.
This seems more awkward than the original code, but it is more correct in spirit: injecting a precomputed value which can change even while the source state does not is a smell. Long-term, the key for the WorkspaceStatusValue should incorporate a hash of the action, and that hash should be in the configuration, just as other configuration flags are. That isn't possible right now just because we don't have configuration trimming, and we drop all nodes on configuration changes, so putting workspace status options into the configuration would lose change pruning whenever we changed workspace status options.
If/when those problems are fixed, we can extend this change to have WorkspaceStatusFunction continue to request the action out-of-band, but keyed by the hash. Then we can stop invalidating stale nodes.
See also https://github.com/bazelbuild/bazel/issues/3785.
PiperOrigin-RevId: 169947071
|
| |
|
|
|
|
|
|
|
|
| |
Replace all previous uses of Metadata with FileArtifactValue (or a simple inner
class in the case of ActionCacheChecker.CONSTANT_METADATA).
Care was taken to make the equals method obey the equals contract, even in the
presence of multiple implementations.
PiperOrigin-RevId: 160115080
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Adjust the interface of Metadata to match the interface of FileArtifactValue.
These classes are almost the same, but not quite, so we currently have to
create and return new objects whenever the MetadataHandler is called; by making
the interfaces identical, we can just return a FileArtifactValue instead (in a
future change).
This is in preparation for merging the ActionInputFileCache and MetadataHandler
interfaces.
PiperOrigin-RevId: 159802834
|
| |
|
|
|
|
|
|
|
| |
the currently defined hash function for blobs. Some refactoring. Adding an option to set the hash function in the remote worker, defaulting to the current behavior (unfortunately it is a build option, have not found a clean way to specify it at runtime).
BUG=62622420
TESTED=remote worker
RELNOTES: none
PiperOrigin-RevId: 159473116
|
| |
|
|
|
|
| |
RELNOTES: None.
PiperOrigin-RevId: 157446717
|
| |
|
|
|
|
|
|
|
|
|
|
| |
'create' method.
This paves the way for changing PathFragment to e.g. an abstract class with multiple subclasses. This way we can split out the windows-specific stuff into one of these concrete classes, making the code more readable and also saving memory (since the shallow heap size of the NonWindowsPathFragment subclass will hopefully be smaller than that of the current PathFragment).
This also lets us pursue gc churn optimizations. We can now do interning in PathFragment#create and can also get rid of unnecessary intermediate PathFragment allocations.
RELNOTES: None
PiperOrigin-RevId: 152145768
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
--noexperimental_enable_critical_path_profiling flags are all specified, then Bazel will delete Actions from ActionLookupValues as they are executed in order to save memory.
Because an already-run action may output an artifact that is only requested later in the build, we need to maintain a way for the artifact to look up the action. But in most cases we don't need to keep the action itself, just its output metadata.
Some actions unfortunately are needed post-execution, and so we special-case them.
Also includes dependency change with description:
Move action out of key. This keeps action references from polluting the graph -- actions are just stored in one SkyValue, instead of being present in SkyKeys.
This does mean additional memory used: we have a separate ActionLookupData object per Action executed. That may reach ~24M for million-action builds.
PiperOrigin-RevId: 151756383
|
| |
|
|
|
| |
--
MOS_MIGRATED_REVID=130986194
|
| |
|
|
|
| |
--
MOS_MIGRATED_REVID=130576075
|
| |
|
|
|
|
|
| |
digest are mutually exclusive.
--
MOS_MIGRATED_REVID=128843642
|
| |
|
|
|
|
|
| |
always digested.
--
MOS_MIGRATED_REVID=128768429
|
| |
|
|
|
|
|
| |
Except in action execution logic (ActionExecutionFunction, SkyframeActionExecutor, etc.), switch Action interface references to either ActionAnalysisMetadata if possible or ActionExecutionMetadata.
--
MOS_MIGRATED_REVID=120723431
|
| |
|
|
|
|
|
| |
Remove ArtifactFile, which is rendered obsolete by TreeFileArtifact.
--
MOS_MIGRATED_REVID=119789154
|
| |
|
|
|
|
|
|
| |
This is one of the last pieces of state in BlazeRuntime that isn't safe to
share across command invocations.
--
MOS_MIGRATED_REVID=117910631
|
|
|
ArtifactFunction.
--
MOS_MIGRATED_REVID=114174899
|
tomlu
cpeyser
ulfjack
janakr
aehlig
olaola
lberki
nharmata
Shreya Bhattarai
Rumou Duan
Michael Thvedt