diff options
| author |  Shanqing Cai <cais@google.com> | 2017-06-05 10:29:50 -0700 |
|---|---|---|
| committer |  TensorFlower Gardener <gardener@tensorflow.org> | 2017-06-05 10:33:23 -0700 |
| commit | cc2dd4ac8538045e94e3f8fe4fb1c532f67c1844 (patch) | |
| tree | 80db01fd15e27ebc0d7b88959caf3140c644bd39 /tensorflow/core/debug/grpc_session_debug_test.cc | |
| parent | a5909d64320a9dfd940b298bcb0bd758e514a04f (diff) | |
tfdbg: dump debug data from different devices in separate directories
Fixes: #7051
wherein TFDBG failed to load the data dump from a Session.run() involving multiple GPUs.
The root cause of the bug was that TFDBG previously assumed that node names are unique across all partition graphs. This is however not the case when multiple GPUs exist. The Send/Recv nodes in the partition graphs of the GPUs can have duplicate names. There will potentially be other cases like this in the future due to other reasons (e.g., distributed sessions and/or graph optimization).
This CL relaxes this assumption, by dumping the GraphDef and tensor data from different devices into different sub-directories under the dump root directory.
PiperOrigin-RevId: 158029814
Diffstat (limited to 'tensorflow/core/debug/grpc_session_debug_test.cc')
| -rw-r--r-- | tensorflow/core/debug/grpc_session_debug_test.cc | 10 |
1 files changed, 8 insertions, 2 deletions
diff --git a/tensorflow/core/debug/grpc_session_debug_test.cc b/tensorflow/core/debug/grpc_session_debug_test.cc index 6c68729410..9584d8b9f3 100644 --- a/tensorflow/core/debug/grpc_session_debug_test.cc +++ b/tensorflow/core/debug/grpc_session_debug_test.cc @@ -187,7 +187,10 @@ TEST_F(GrpcSessionDebugTest, FileDebugURL) { IsSingleFloatValue(outputs[0], 4.0); std::vector<Tensor> dumped_tensors; - LoadTensorDumps("n", &dumped_tensors); + LoadTensorDumps(io::JoinPath(DebugNodeKey::DeviceNameToDevicePath( + cluster->devices()[0].name()), + "n"), + &dumped_tensors); if (i == 0 || i == 5) { ASSERT_EQ(0, dumped_tensors.size()); @@ -267,7 +270,10 @@ TEST_F(GrpcSessionDebugTest, MultiDevices_String) { TF_CHECK_OK(session->Close()); std::vector<Tensor> dumped_tensors; - LoadTensorDumps("n", &dumped_tensors); + LoadTensorDumps( + io::JoinPath(DebugNodeKey::DeviceNameToDevicePath(a_dev.name()), + "n"), + &dumped_tensors); ASSERT_EQ(1, dumped_tensors.size()); ASSERT_EQ(TensorShape({2, 2}), dumped_tensors[0].shape()); for (size_t i = 0; i < 4; ++i) { |