diff options
Diffstat (limited to 'tensorflow/g3doc/how_tos/variables/index.md')
| -rw-r--r-- | tensorflow/g3doc/how_tos/variables/index.md | 215 |
1 files changed, 215 insertions, 0 deletions
diff --git a/tensorflow/g3doc/how_tos/variables/index.md b/tensorflow/g3doc/how_tos/variables/index.md new file mode 100644 index 0000000000..4ad8f8a266 --- /dev/null +++ b/tensorflow/g3doc/how_tos/variables/index.md @@ -0,0 +1,215 @@ +# Variables: Creation, Initialization, Saving, and Loading + +When you train a model, you use [Variables](../../api_docs/python/state_ops.md) +to hold and update parameters. Variables are in-memory buffers containing +tensors. They need to be explicitly initialized and can be saved to disk during +and after training. You can later restore saved values to exercise or analyse +the model. + +This document references the following TensorFlow classes. Follow the links to +their reference manual for a complete description of their API: + +* The `Variable` class [tf.Variable](../../api_docs/python/state_ops.md#Variable). +* The `Saver` class [tf.train.Saver](../../api_docs/python/state_ops.md#Saver). + + +## Creation + +When you create a [Variable](../../api_docs/python/state_ops.md) you pass a +`Tensor` as its initial value to the `Variable()` constructor. TensorFlow +provides a collection of Ops that produce tensors often used for initialization +from [constants or random values](../../api_docs/python/constant_op.md). + +Note that all these Ops require you to specify the shape of the tensors. That +shape automatically becomes the shape of the variable. Variables generally +have a fixed shape, but TensorFlow provides advanced mechanisms to reshape +variables. + +```python +# Create two variables. +weights = tf.Variable(tf.random_normal([784, 200], stddev=0.35), + name="weights") +biases = tf.Variable(tf.zeros([200]), name="biases") +``` + +Calling `tf.Variable()` adds a few Ops to the graph: + +* A `variable` Op that holds the variable value. +* An initializer Op that sets the variable to its initial value. This is + actually a `tf.assign` Op. +* The Ops for the initial value, such as the `zeros` Op for the `biases` + variable in the example are also added to the graph. + +The value returned by `tf.Variable()` value is an instance of the Python class +`tf.Variable`. + +## Initialization + +Variable initializers must be run explicitly before other Ops in your model can +be run. The easiest way to do that is to add an Op that runs all the variable +initializers, and run that Op before using the model. + +You can alternatively restore variable values from a checkpoint file, see +below. + +Use `tf.initialize_all_variables()` to add an Op to run variable initializers. +Only run that Op after you have fully constructed your model and launched it in +a session. + +```python +# Create two variables. +weights = tf.Variable(tf.random_normal([784, 200], stddev=0.35), + name="weights") +biases = tf.Variable(tf.zeros([200]), name="biases") +... +# Add an Op to initialize the variables. +init_op = tf.initialize_all_variables() + +# Later, when launching the model +with tf.Session() as sess: + # Run the init operation. + sess.Run(init_op) + ... + # Use the model + ... +``` + +### Initialization from another Variable + +You sometimes need to initialize a variable from the initial value of another +variable. As the Op added by `tf.initialize_all_variables()` initializes all +variables in parallel you have to be careful when this is needed. + +To initialize a new variable from the value of another variable use the other +variable's `initialized_value()` property. You can use the initialized value +directly as the initial value for the new variable, or you can use it as any +other tensor to compute a value for the new variable. + + +```python +# Create a variable with a random value. +weights = tf.Variable(tf.random_normal([784, 200], stddev=0.35), + name="weights") +# Create another variable with the same value as 'weights'. +w2 = tf.Variable(weights.initialized_value(), name="w2") +# Create another variable with twice the value of 'weights' +w_twice = tf.Variable(weights.initialized_value() * 0.2, name="w_twice") +``` + +### Custom Initialization + +The convenience function `tf.initialize_all_variables()` adds an Op to +initialize *all variables* in the model. You can also pass it an explicit list +of variables to initialize. See the +[Variables Documentation](../../api_docs/python/state_op.md) for more options, +including checking if variables are initialized. + +## Saving and Restoring + +The easiest way to save and restore a model is to use a `tf.train.Saver` +object. The constructor adds `save` and `restore` Ops to the graph for all, or +a specified list, of variables. The saver object provides methods to run these +Ops, specifying paths for the checkpoint files to write to or read from. + +### Checkpoint Files + +Variables are saved in binary files that, roughly, contains a map from variable +names to tensors. + +When you create a `Saver` object, you can optionally chose names for the +variables in the checkpoint files. By default, it uses the names passed to the +`tf.Variable()` call. + +### Saving Variables + +Create a `Saver` with `tf.train.Saver()` to manage all variables in +the model. + +```python +# Create some variables. +v1 = tf.Variable(..., name="v1") +v2 = tf.Variable(..., name="v2") +... +# Add an Op to initialize the variables. +init_op = tf.initialize_all_variables() + +# Add Ops to save and restore all the variables. +saver = tf.train.Saver() + +# Later, launch the model, initialize the variables, do some work, save the +# variables to disk. +with tf.Session() as sess: + sess.Run(init_op) + # Do some work with the model. + .. + # Save the variables to disk. + save_path = saver.Save(sess, "/tmp/model.ckpt") + print "Model saved in file: ", save_path +``` + +### Restoring Variables + +The same `Saver` object is used to restore variables. Note that when you +restore variables form a file you do not have to initialize them beforehand. + +```python +# Create some variables. +v1 = tf.Variable(..., name="v1") +v2 = tf.Variable(..., name="v2") +... +# Add Ops to save and restore all the variables. +saver = tf.train.Saver() + +# Later, launch the model, use the saver to restore variables from disk, and +# do some work with the model. +with tf.Session() as sess: + # Restore variables from disk. + saver.Restore(sess, "/tmp/model.ckpt") + print "Model restored." + # Do some work with the model + ... +``` + +### Chosing which Variables to Save and Restore + +If you do not pass any argument to `tf.train.Saver()` the saver +handles all variables. Each one of them is saved under the name that was +passed when the variable was created. + +It is sometimes useful to explicitly specify names for variables in the +checkpoint files. For example, you may have trained a model with a variable +named `"weights"` whose value you want to restore in a new variable named +`"params"`. + +It is also sometimes useful to only save or restore a subset of the variables +used by a model. For example, you may have trained a neural net with 5 layers, +and you now want to train a new model with 6 layers, restoring the parameters +from the 5 layers of the previously trained model into the first 5 layers of +the new model. + +You can easily specify the names and variables to save by passing to the +`tf.train.Saver()` constructor a Python dictionary: keys are the +names to use, values are the variables to manage. + +Notes: + +* You can create as many saver objects as you want if you need to save and + restore different subsets of the model variables. The same variable can be + listed in multiple saver objects, its value is only changed when the saver + `Restore()` method is run. + +* If you only restore a subset of the model variables at the start + of a session, you have to run an initialize Op for the other variables. See + [`tf.initialize_variables()`](../../api_docs/python/state_ops.md#initialize_variables) + for more information. + +```python +# Create some variables. +v1 = tf.Variable(..., name="v1") +v2 = tf.Variable(..., name="v2") +... +# Add Ops to save and restore only 'v2' using the name "my_v2" +saver = tf.train.Saver({"my_v2": v2}) +# Use the saver object normally after that. +... +``` |