workers: A multitude of bug fixes and improved logging.

I know this should have been split up, but I was a bit on fire today and did it all in one go ^^;

Fixes spurious "Stream closed: Stream closed" errors, by noticing dead workers and retrying with a fresh one. (Configurable with the --worker_max_retries flag.)

Fixes an "IllegalArgumentException" when a non-worker compatible Spawn is given to the strategy. We fall back to StandaloneSpawnStrategy now.

Redirect the stderr of worker processes to separate log files in a common sub-directory and print a message that tells you about the location on worker start-up for easier debugging. The log can be found in <output_base>/worker-logs/*.log.

Adds the mnemonic of the Spawn to log messages and the log filename.

Adds verbose messages on worker start-up and shutdown. (Enable it with --worker_verbose!)

Shuts down the worker pool after a build finished by default, until we sort out one last remaining correctness issue. This also conserves resources, though makes incremental builds a bit slower. Want the maximum performance anyway? Try --experimental_workers_keep_running.

Adds stack traces to errors that are caused by buggy workers to aid development.

Fixes weird dupli..tripli..quadruple error messages ("Compiling failed: Stream closed: Stream closed: Stream closed: Stream closed.").

--
MOS_MIGRATED_REVID=102983853

1 files changed, 13 insertions, 2 deletions

diff --git a/src/main/java/com/google/devtools/build/lib/worker/WorkerKey.java b/src/main/java/com/google/devtools/build/lib/worker/WorkerKey.java
index e0ce5cf0bf..b9247ce686 100644
--- a/src/main/java/com/google/devtools/build/lib/worker/WorkerKey.java
+++ b/src/main/java/com/google/devtools/build/lib/worker/WorkerKey.java
@@ -30,11 +30,13 @@ final class WorkerKey {
   private final ImmutableList<String> args;
   private final ImmutableMap<String, String> env;
   private final Path workDir;
+  private final String mnemonic;
 
-  WorkerKey(List<String> args, Map<String, String> env, Path workDir) {
+  WorkerKey(List<String> args, Map<String, String> env, Path workDir, String mnemonic) {
     this.args = ImmutableList.copyOf(Preconditions.checkNotNull(args));
     this.env = ImmutableMap.copyOf(Preconditions.checkNotNull(env));
     this.workDir = Preconditions.checkNotNull(workDir);
+    this.mnemonic = Preconditions.checkNotNull(mnemonic);
   }
 
   public ImmutableList<String> getArgs() {
@@ -49,6 +51,10 @@ final class WorkerKey {
     return workDir;
   }
 
+  public String getMnemonic() {
+    return mnemonic;
+  }
+
   @Override
   public boolean equals(Object o) {
     if (this == o) {
@@ -66,7 +72,11 @@ final class WorkerKey {
     if (!env.equals(workerKey.env)) {
       return false;
     }
-    return workDir.equals(workerKey.workDir);
+    if (!workDir.equals(workerKey.workDir)) {
+      return false;
+    }
+    return mnemonic.equals(workerKey.mnemonic);
+
   }
 
   @Override
@@ -74,6 +84,7 @@ final class WorkerKey {
     int result = args.hashCode();
     result = 31 * result + env.hashCode();
     result = 31 * result + workDir.hashCode();
+    result = 31 * result + mnemonic.hashCode();
     return result;
   }