Add caching of computed file digests based on file metadata.

This change modifies DigestUtils to add a cache of (path, inode, mtime,
size) to the digest of the file for those digests that are computed by
reading the file contents.

The cache itself is optional because relying on file metadata to cache
the file digests could lead to correctness issues.  Enabling the cache
is exposed via a new (undocumented) --cache_computed_file_digests flag
that we can use post-release to tune the built-in values if they prove
to be incorrect or problematic.

For Bazel, enable this cache unconditionally because the rest of
Bazel already relies on mtimes and other file metadata to determine
changes to files.

The rationale for this change is performance: once we have lost the
in-memory file metadata (e.g. because of a flag flip), Bazel has to
redigest all of the output files to recompute action cache keys.  For
a pathological case of rebuilding a large app with 5GB of outputs and
then flipping the --[no]check_visibility flag on the command line, we
get the following numbers before this change:

____Elapsed time: 11.170s, Critical Path: 8.34s
____Elapsed time: 11.027s, Critical Path: 8.20s
____Elapsed time: 11.084s, Critical Path: 7.46s
____Elapsed time: 11.051s, Critical Path: 6.61s
____Elapsed time: 11.211s, Critical Path: 7.81s
____Elapsed time: 10.884s, Critical Path: 8.20s
____Elapsed time: 11.385s, Critical Path: 8.12s
____Elapsed time: 11.723s, Critical Path: 8.18s
____Elapsed time: 11.327s, Critical Path: 7.73s
____Elapsed time: 11.028s, Critical Path: 7.89s

And after this change:

____Elapsed time: 4.294s, Critical Path: 0.27s
____Elapsed time: 4.376s, Critical Path: 0.83s
____Elapsed time: 8.083s, Critical Path: 0.52s
____Elapsed time: 4.302s, Critical Path: 0.64s
____Elapsed time: 4.282s, Critical Path: 0.37s
____Elapsed time: 4.219s, Critical Path: 0.61s
____Elapsed time: 4.214s, Critical Path: 0.97s
____Elapsed time: 4.185s, Critical Path: 0.71s
____Elapsed time: 7.962s, Critical Path: 4.30s
____Elapsed time: 4.149s, Critical Path: 1.03s

--
PiperOrigin-RevId: 150351444
MOS_MIGRATED_REVID=150351444

1 files changed, 13 insertions, 0 deletions

diff --git a/src/main/java/com/google/devtools/build/lib/exec/ExecutionOptions.java b/src/main/java/com/google/devtools/build/lib/exec/ExecutionOptions.java
index b8b2ac2509..7766b6c7a0 100644
--- a/src/main/java/com/google/devtools/build/lib/exec/ExecutionOptions.java
+++ b/src/main/java/com/google/devtools/build/lib/exec/ExecutionOptions.java
@@ -205,4 +205,17 @@ public class ExecutionOptions extends OptionsBase {
     help = "Print the contents of the SpawnActionContext and ContextProviders maps."
   )
   public boolean debugPrintActionContexts;
+
+  @Option(
+    name = "cache_computed_file_digests",
+    defaultValue = "50000",
+    category = "undocumented",
+    help =
+        "If greater than 0, configures Blaze to cache file digests in memory based on their "
+            + "metadata instead of recomputing the digests from disk every time they are needed. "
+            + "Setting this to 0 ensures correctness because not all file changes can be noted "
+            + "from file metadata. When not 0, the number indicates the size of the cache as the "
+            + "number of file digests to be cached."
+  )
+  public long cacheSizeForComputedFileDigests;
 }