Ensure that shutdown commands end the server process before completion

This change ensures that the server process is terminated before the
client process terminates, when evaluating a command that shuts down
the server.

When completing such a command, the server communicates to the client
that the server will terminate itself by setting a termination_expected
bit in the final RunResponse message. The client then waits up to 60s
for the server process to actually terminate. If it does not, then the
client SIGKILLs the server.

Also makes the gRPC server stop accepting new commands before the
shutdown command completes.

Drive-by fix to comments on Search{Un,Null}aryOption.

RELNOTES: Commands that shut down the server (like "shutdown") now ensure that the server process has terminated before the client process terminates.
PiperOrigin-RevId: 161537480

1 files changed, 5 insertions, 5 deletions

diff --git a/src/main/java/com/google/devtools/build/lib/server/GrpcServerImpl.java b/src/main/java/com/google/devtools/build/lib/server/GrpcServerImpl.java
index 032da198d3..6abef0e4ac 100644
--- a/src/main/java/com/google/devtools/build/lib/server/GrpcServerImpl.java
+++ b/src/main/java/com/google/devtools/build/lib/server/GrpcServerImpl.java
@@ -860,12 +860,17 @@ public class GrpcServerImpl implements RPCServer {
     // the cancel request won't find the thread to interrupt)
     Thread.interrupted();
 
+    boolean shutdown = commandExecutor.shutdown();
+    if (shutdown) {
+      server.shutdown();
+    }
     RunResponse response =
         RunResponse.newBuilder()
             .setCookie(responseCookie)
             .setCommandId(commandId)
             .setFinished(true)
             .setExitCode(exitCode)
+            .setTerminationExpected(shutdown)
             .build();
 
     try {
@@ -876,11 +881,6 @@ public class GrpcServerImpl implements RPCServer {
       log.info(String.format("Client cancelled command %s just right before its end: %s",
           commandId, e.getMessage()));
     }
-
-    if (commandExecutor.shutdown()) {
-      pidFileWatcherThread.signalShutdown();
-      server.shutdown();
-    }
   }
 
   private final CommandServerGrpc.CommandServerImplBase commandServer =