PhiloxRandom: Fix race in GPU fill function (#10298)

* PhiloxRandom: Fix race in GPU fill function

The PhiloxRandom fill kernel for the GPU had race conditions that caused the
outputs to be non-deterministic. In particular, the code previously executed
with N GPU threads (# thread contexts per GPU), but it would only advance the
fill addresses by N-1 stride in each step. This incorrect stride caused the
0th and N-1st threads to write to the same memory locations, racing for which
was last to write their common locations. Make the stride equal to the number
of threads to eliminate the race.

BONUS: By fixing this race, PhiloxRandom constant-sized GPU initializers now
match CPU initializers.

* Update random_ops_test.py to find race conditions

Increasing the size of arrays in the random_ops_test.py test to manifest
the race conditions to be resolved.

1 files changed, 1 insertions, 1 deletions

diff --git a/tensorflow/core/kernels/random_op_gpu.cu.cc b/tensorflow/core/kernels/random_op_gpu.cu.cc
index 5f7d9b7dd6..7afa6974c6 100644
--- a/tensorflow/core/kernels/random_op_gpu.cu.cc
+++ b/tensorflow/core/kernels/random_op_gpu.cu.cc
@@ -141,7 +141,7 @@ struct FillPhiloxRandomKernel<Distribution, false> {
       const typename Distribution::ResultType samples = dist(&gen);
       copier(&data[offset], samples);
 
-      offset += (total_thread_count - 1) * kGroupSize;
+      offset += total_thread_count * kGroupSize;
       gen.Skip(total_thread_count - 1);
     }