retry sequentially if multiprocessing do_bad_build_check detects fail… (#5578)

* retry sequentially if multiprocessing do_bad_build_check detects failures

https://github.com/google/oss-fuzz/issues/5441

The error seen in the build log is:

Whoops, the target binary crashed suddenly, before receiving any input
from the fuzzer!

suggesting that the fuzzer crashed before it got to do anything.
Debugging locally what I tend to see is that

a) in src/afl-forkserver.c afl_fsrv_start the read_s32_timed call
returns 0 and that triggers kill(fsrv->fsrv_pid, fsrv->kill_signal);
(SIGKILL)
b) read_s32_timed returns 0 because *stop_soon_p is non-zero at
restart_read:
c) *stop_soon_p becomes non-zero in handle_stop_sig of
src/afl-fuzz-init.c due to receiving SIGINT
d) that SIGINT is sent by the timeout script used in bad_build_check so
it is that "outer" timeout process which is sending SIGINT which then
triggers afl-forkserver's internal SIGKILL to kill the process

I get improved results if I retry the killed off fuzzers sequentially

* Remove unneeded semicolons to fix presubmit

Co-authored-by: Abhishek Arya <inferno@chromium.org>

1 files changed, 15 insertions, 0 deletions

diff --git a/infra/base-images/base-runner/test_all.py b/infra/base-images/base-runner/test_all.py
index 925ebde6..70414688 100755
--- a/infra/base-images/base-runner/test_all.py
+++ b/infra/base-images/base-runner/test_all.py
@@ -172,11 +172,26 @@ def test_all(out, fuzzing_language, allowed_broken_targets_percentage):
 
   pool = multiprocessing.Pool()
   bad_build_results = pool.map(do_bad_build_check, fuzz_targets)
+  pool.close()
+  pool.join()
   broken_targets = get_broken_fuzz_targets(bad_build_results, fuzz_targets)
   broken_targets_count = len(broken_targets)
   if not broken_targets_count:
     return True
 
+  print('Retrying failed fuzz targets sequentially', broken_targets_count)
+  pool = multiprocessing.Pool(1)
+  retry_targets = []
+  for broken_target, result in broken_targets:
+    retry_targets.append(broken_target)
+  bad_build_results = pool.map(do_bad_build_check, retry_targets)
+  pool.close()
+  pool.join()
+  broken_targets = get_broken_fuzz_targets(bad_build_results, broken_targets)
+  broken_targets_count = len(broken_targets)
+  if not broken_targets_count:
+    return True
+
   print('Broken fuzz targets', broken_targets_count)
   total_targets_count = len(fuzz_targets)
   broken_targets_percentage = 100 * broken_targets_count / total_targets_count