Prune modules when building modules themselves to reduce build times shorten

critical paths.

When the inputs of a module M are reduced to a set S, then that same set S also
needs to be supplied to compile something that uses M. To do this, input
discovery is divided into two stages. For each CppCompileAction, the first
stage discovers the necessary modules (M above). These are then added as inputs
to ensure that they are built. The second stage then finds all the modules (S
above) that are required to use those and also adds them as inputs.

For now, the new behavior is guarded by a new flag
--experimental_prune_more_modules.

This is currently implemented by reading the .d files of used modules add
adding all their module dependencies. There are two noteworthy alternatives:
1. Hack up input discovery to understand modules, e.g. if a modular header is
   hit, continue scanning from all it's headers. However, this seems very
   brittle and a lot of additional information would have to be passed to the
   input discovery.
2. Directly pass the results from input discovery of one CppCompileAction to
   another one. However, this seems to tightly couple the execution of
   different CppCompileActions and might lead to a mess of different states,
   more memory consumption, etc.
With the current implementation, there is a bit of runtime overhead of
reading the .d files (many times). This could potentially be improved by
caching the results. However, even without this caching, the runtime overhead
is limited (<10%) for all builds I have tried (I have tried with builds where
all the compile results are already in the executor's cache.

--
MOS_MIGRATED_REVID=140793217

1 files changed, 13 insertions, 0 deletions

diff --git a/src/main/java/com/google/devtools/build/lib/actions/Action.java b/src/main/java/com/google/devtools/build/lib/actions/Action.java
index ecf8b66182..12316b7264 100644
--- a/src/main/java/com/google/devtools/build/lib/actions/Action.java
+++ b/src/main/java/com/google/devtools/build/lib/actions/Action.java
@@ -19,6 +19,7 @@ import com.google.devtools.build.lib.concurrent.ThreadSafety.ConditionallyThread
 import com.google.devtools.build.lib.profiler.Describable;
 import com.google.devtools.build.lib.vfs.Path;
 import com.google.devtools.build.lib.vfs.PathFragment;
+import com.google.devtools.build.skyframe.SkyFunction;
 import java.io.IOException;
 import java.util.Collection;
 import javax.annotation.Nullable;
@@ -152,6 +153,18 @@ public interface Action extends ActionExecutionMetadata, Describable {
       throws ActionExecutionException, InterruptedException;
 
   /**
+   * Used in combination with {@link #discoverInputs} if inputs need to be found before execution in
+   * multiple steps. Returns null if two-stage input discovery isn't necessary.
+   *
+   * <p>Any deps requested here must not change unless one of the action's inputs changes.
+   * Otherwise, changes to nodes that should cause re-execution of actions might be prevented by the
+   * action cache.
+   */
+  @Nullable
+  Iterable<Artifact> discoverInputsStage2(SkyFunction.Environment env)
+      throws ActionExecutionException, InterruptedException;
+
+  /**
    * If an action does not know the exact set of inputs ahead of time, it will usually either do:
    * <ul>
    * <li> Execution time pruning: The action provides a superset set of inputs at action creation