Be more lenient with slow-to-react servers.

Increase the ping timeout on connect from five to ten seconds. This gives
servers which may be suffering from gc pressure or other ailments extra time to
respond.

On the other end, wait for orphaned servers to really die before proceeding.
This prevents race conditions around the delivery of SIGKILL and the starting
of the new server.

This may make us fail slower when the server is having hard times, however it
should give us better determinism, and these conditions should be rare.

--
MOS_MIGRATED_REVID=130118918

1 files changed, 11 insertions, 0 deletions

diff --git a/src/main/cpp/blaze_util_linux.cc b/src/main/cpp/blaze_util_linux.cc
index 27ab344159..82142d5b06 100644
--- a/src/main/cpp/blaze_util_linux.cc
+++ b/src/main/cpp/blaze_util_linux.cc
@@ -250,7 +250,18 @@ bool KillServerProcess(
     return false;
   }
 
+  // Kill the process and make sure it's dead before proceeding.
   killpg(pid, SIGKILL);
+  int check_killed_retries = 10;
+  while (killpg(pid, 0) == 0) {
+    if (check_killed_retries-- > 0) {
+      sleep(1);
+    } else {
+      die(blaze_exit_code::LOCAL_ENVIRONMENTAL_ERROR,
+          "Attempted to kill stale blaze server process (pid=%d) using "
+          "SIGKILL, but it did not die in a timely fashion.", pid);
+    }
+  }
   return true;
 }