Update the tensorboard graph visualizer docs with information about runtime statistics,

and update the exerpt in the summaries and tensorboard doc with the updated version of the mnist_with_summaries.py tutorial code.
Change: 123603297

2 files changed, 121 insertions, 21 deletions

diff --git a/tensorflow/g3doc/how_tos/graph_viz/index.md b/tensorflow/g3doc/how_tos/graph_viz/index.md
index ec1dc994ac..c94afd70b5 100644
--- a/tensorflow/g3doc/how_tos/graph_viz/index.md
+++ b/tensorflow/g3doc/how_tos/graph_viz/index.md
@@ -248,4 +248,73 @@ The images below show the CIFAR-10 model with tensor shape information:
   </tr>
 </table>
 
+## Runtime statistics
 
+Often it is useful to collect runtime metadata for a run, such as total memory
+usage, total compute time, and tensor shapes for nodes. The code example below
+is a snippet from the train and test section of a modification of the
+[simple MNIST tutorial](http://tensorflow.org/tutorials/mnist/beginners/index.md),
+in which we have recorded summaries and runtime statistics. See the [Summaries Tutorial](../../how_tos/summaries_and_tensorboard/index.md#serializing-the-data)
+for details on how to record summaries.
+Full source is [here](https://www.tensorflow.org/code/tensorflow/examples/tutorials/mnist/mnist_with_summaries.py).
+
+```python
+  # Train the model, and also write summaries.
+  # Every 10th step, measure test-set accuracy, and write test summaries
+  # All other steps, run train_step on training data, & add training summaries
+
+  def feed_dict(train):
+    """Make a TensorFlow feed_dict: maps data onto Tensor placeholders."""
+    if train or FLAGS.fake_data:
+      xs, ys = mnist.train.next_batch(100, fake_data=FLAGS.fake_data)
+      k = FLAGS.dropout
+    else:
+      xs, ys = mnist.test.images, mnist.test.labels
+      k = 1.0
+    return {x: xs, y_: ys, keep_prob: k}
+
+  for i in range(FLAGS.max_steps):
+    if i % 10 == 0:  # Record summaries and test-set accuracy
+      summary, acc = sess.run([merged, accuracy], feed_dict=feed_dict(False))
+      test_writer.add_summary(summary, i)
+      print('Accuracy at step %s: %s' % (i, acc))
+    else:  # Record train set summaries, and train
+      if i % 100 == 99:  # Record execution stats
+        run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
+        run_metadata = tf.RunMetadata()
+        summary, _ = sess.run([merged, train_step],
+                              feed_dict=feed_dict(True),
+                              options=run_options,
+                              run_metadata=run_metadata)
+        train_writer.add_run_metadata(run_metadata, 'step%d' % i)
+        train_writer.add_summary(summary, i)
+        print('Adding run metadata for', i)
+      else:  # Record a summary
+        summary, _ = sess.run([merged, train_step], feed_dict=feed_dict(True))
+        train_writer.add_summary(summary, i)
+```
+
+This code will emit runtime statistics for every 100th step starting at step99.
+
+When you launch tensorboard and go to the Graph tab, you will now see options
+under "Session runs" which correspond to the steps where run metadata was added.
+Selecting one of these runs will show you the snapshot of the network at that
+step, fading out unused nodes. In the controls on the left hand side, you will
+be able to color the nodes by total memory or total compute time. Additionally,
+clicking on a node will display the exact total memory, compute time, and
+tensor output sizes.
+
+
+<table width="100%;">
+  <tr style="height: 380px">
+    <td>
+      <img src="../../images/colorby_compute_time.png" alt="Color by compute time" title="Color by compute time"/>
+    </td>
+    <td>
+      <img src="../../images/run_metadata_graph.png" alt="Run metadata graph" title="Run metadata graph" />
+    </td>
+    <td>
+      <img src="../../images/run_metadata_infocard.png" alt="Run metadata info card" title="Run metadata info card" />
+    </td>
+  </tr>
+</table>
diff --git a/tensorflow/g3doc/how_tos/summaries_and_tensorboard/index.md b/tensorflow/g3doc/how_tos/summaries_and_tensorboard/index.md
index 45ee082e68..0ad46c4f3d 100644
--- a/tensorflow/g3doc/how_tos/summaries_and_tensorboard/index.md
+++ b/tensorflow/g3doc/how_tos/summaries_and_tensorboard/index.md
@@ -77,7 +77,8 @@ The code below is an excerpt; full source is [here](https://www.tensorflow.org/c
 
 ```python
 def variable_summaries(var, name):
-  with tf.name_scope("summaries"):
+  """Attach a lot of summaries to a Tensor."""
+  with tf.name_scope('summaries'):
     mean = tf.reduce_mean(var)
     tf.scalar_summary('mean/' + name, mean)
     with tf.name_scope('stddev'):
@@ -87,45 +88,47 @@ def variable_summaries(var, name):
     tf.scalar_summary('min/' + name, tf.reduce_min(var))
     tf.histogram_summary(name, var)
 
-def nn_layer(input_tensor, input_dim, output_dim, layer_name):
+def nn_layer(input_tensor, input_dim, output_dim, layer_name, act=tf.nn.relu):
   """Reusable code for making a simple neural net layer.
 
   It does a matrix multiply, bias add, and then uses relu to nonlinearize.
-  It also sets up name scoping so that the resultant graph is easy to read, and
-  adds a number of summary ops.
+  It also sets up name scoping so that the resultant graph is easy to read,
+  and adds a number of summary ops.
   """
   # Adding a name scope ensures logical grouping of the layers in the graph.
   with tf.name_scope(layer_name):
     # This Variable will hold the state of the weights for the layer
-    with tf.name_scope("weights"):
+    with tf.name_scope('weights'):
       weights = weight_variable([input_dim, output_dim])
       variable_summaries(weights, layer_name + '/weights')
-    with tf.name_scope("biases"):
+    with tf.name_scope('biases'):
       biases = bias_variable([output_dim])
       variable_summaries(biases, layer_name + '/biases')
     with tf.name_scope('Wx_plus_b'):
-      activations = tf.matmul(input_tensor, weights) + biases
-      tf.histogram_summary(layer_name + '/activations', activations)
-    relu = tf.nn.relu(activations, 'relu')
-    tf.histogram_summary(layer_name + '/activations_relu', relu)
-    return tf.nn.dropout(relu, keep_prob)
+      preactivate = tf.matmul(input_tensor, weights) + biases
+      tf.histogram_summary(layer_name + '/pre_activations', preactivate)
+    activations = act(preactivate, 'activation')
+    tf.histogram_summary(layer_name + '/activations', activations)
+    return activations
 
-layer1 = nn_layer(x, 784, 50, 'layer1')
-layer2 = nn_layer(layer1, 50, 10, 'layer2')
-y = tf.nn.softmax(layer2, 'predictions')
+hidden1 = nn_layer(x, 784, 500, 'layer1')
 
+with tf.name_scope('dropout'):
+  keep_prob = tf.placeholder(tf.float32)
+  tf.scalar_summary('dropout_keep_probability', keep_prob)
+  dropped = tf.nn.dropout(hidden1, keep_prob)
+
+y = nn_layer(dropped, 500, 10, 'layer2', act=tf.nn.softmax)
 
 with tf.name_scope('cross_entropy'):
   diff = y_ * tf.log(y)
   with tf.name_scope('total'):
-    cross_entropy = -tf.reduce_sum(diff)
-  with tf.name_scope('normalized'):
-    normalized_cross_entropy = -tf.reduce_mean(diff)
-  tf.scalar_summary('cross entropy', normalized_cross_entropy)
+    cross_entropy = -tf.reduce_mean(diff)
+  tf.scalar_summary('cross entropy', cross_entropy)
 
 with tf.name_scope('train'):
-  train_step = tf.train.AdamOptimizer(
-      FLAGS.learning_rate).minimize(cross_entropy)
+  train_step = tf.train.AdamOptimizer(FLAGS.learning_rate).minimize(
+      cross_entropy)
 
 with tf.name_scope('accuracy'):
   with tf.name_scope('correct_prediction'):
@@ -136,10 +139,38 @@ with tf.name_scope('accuracy'):
 
 # Merge all the summaries and write them out to /tmp/mnist_logs (by default)
 merged = tf.merge_all_summaries()
-train_writer = tf.train.SummaryWriter(FLAGS.summaries_dir + '/train', sess.graph)
+train_writer = tf.train.SummaryWriter(FLAGS.summaries_dir + '/train',
+                                      sess.graph)
 test_writer = tf.train.SummaryWriter(FLAGS.summaries_dir + '/test')
 tf.initialize_all_variables().run()
+```
 
+After we've initialized the `SummaryWriters`, we have to add summaries to the
+`SummaryWriters` as we train and test the model.
+
+```python
+# Train the model, and also write summaries.
+# Every 10th step, measure test-set accuracy, and write test summaries
+# All other steps, run train_step on training data, & add training summaries
+
+def feed_dict(train):
+  """Make a TensorFlow feed_dict: maps data onto Tensor placeholders."""
+  if train or FLAGS.fake_data:
+    xs, ys = mnist.train.next_batch(100, fake_data=FLAGS.fake_data)
+    k = FLAGS.dropout
+  else:
+    xs, ys = mnist.test.images, mnist.test.labels
+    k = 1.0
+  return {x: xs, y_: ys, keep_prob: k}
+
+for i in range(FLAGS.max_steps):
+  if i % 10 == 0:  # Record summaries and test-set accuracy
+    summary, acc = sess.run([merged, accuracy], feed_dict=feed_dict(False))
+    test_writer.add_summary(summary, i)
+    print('Accuracy at step %s: %s' % (i, acc))
+  else:  # Record train set summaries, and train
+    summary, _ = sess.run([merged, train_step], feed_dict=feed_dict(True))
+    train_writer.add_summary(summary, i)
 ```
 
 You're now all set to visualize this data using TensorBoard.