diff options
author | Vijay Vasudevan <vrv@google.com> | 2017-02-17 17:05:49 -0800 |
---|---|---|
committer | TensorFlower Gardener <gardener@tensorflow.org> | 2017-02-17 17:23:48 -0800 |
commit | 93a975e114ee1c35f01ed3bdd47170e6f7129014 (patch) | |
tree | e34255aff698fe6a4a586e7940337fd278947f58 /tensorflow/contrib/imperative/README.md | |
parent | eb9624017a0040e805fda622a5f9ec6681e24246 (diff) |
Merge changes from github.
Change: 147897309
Diffstat (limited to 'tensorflow/contrib/imperative/README.md')
-rw-r--r-- | tensorflow/contrib/imperative/README.md | 155 |
1 files changed, 155 insertions, 0 deletions
diff --git a/tensorflow/contrib/imperative/README.md b/tensorflow/contrib/imperative/README.md new file mode 100644 index 0000000000..ea643a45d1 --- /dev/null +++ b/tensorflow/contrib/imperative/README.md @@ -0,0 +1,155 @@ +## Imperative programming in TensorFlow + +In the standard TensorFlow library, the specification of the computation is done +statically in terms of a computation graph, and is separate from the execution +of the graph. This model of programming is referred to as *lazy*, *deferred*, +*dynamic*, or, *asynchronous*. This library brings imperative style programming (à +la [NumPy](http://www.numpy.org)) to TensorFlow. Using this library, you can: + +* Write code in an imperative style: the results of the computation are available + right after the execution of a line of code. +* Use TensorFlow operations on tensors, and get all the benefits of GPU + acceleration. +* Include any Python control flow statements like `while` and `if` when + specifying the computation. +* Perform automatic differentiation on your code with the + standard + [`tf.gradients`](https://www.tensorflow.org/api_docs/python/train/gradient_computation#gradients) function. + +### Getting started + +This library is a thin wrapper over the standard TensorFlow Python library. The +source code is +available +[here](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/imperative). You +can get started on Linux by installing the nightly PIP package linked off +[the main page](https://github.com/tensorflow/tensorflow). Please +consult [this](https://github.com/tensorflow/tensorflow#installation) document for other platforms and the PIP package including GPU +support. + + +### Write your first imperative TensorFlow program + +```shell +$ python +``` + +```python +>>> import tensorflow.contrib.imperative as tf +>>> x = tf.constant([[7.], [6]]) +>>> y = tf.constant([[6., 7]]) +>>> tf.matmul(x, y) +array([[ 42., 49.], + [ 36., 42.]], dtype=float32) +``` + +Note that this code is identical in terms of the programmer's mental model to +the following NumPy code: + +```python +>>> import numpy as np +>>> x = np.array([[7.], [6]]) +>>> y = np.array([[6., 7]]) +>>> x.dot(y) +array([[ 42., 49.], + [ 36., 42.]]) +``` + +The library can be imported as `import tensorflow.contrib.imperative as tf` +(contrast with importing standard TensorFlow, which is done as `import +tensorflow as tf`). This import statement makes all of standard TensorFlow +available in the `tf` symbol. However, it is not necessary to create a session +object and set it up to run and fetch tensors. + + +### Features + +The library provides the following additional features on top of standard +TensorFlow: + +* Tensors are automatically fetched when used in contexts that expect their + value. + + - Printing + + ```python + x = tf.constant(10) + y = tf.constant(32) + print(x + y) + 42 + ``` + + - Use in conditionals + + ```python + x = tf.constant(30) + if x > 4: + print('Greater than 4') + Greater than 4 + + x = tf.random_normal([3]) + y = x * 2 + while tf.global_norm([y]) < 1000: + y = y * 2 + print(y) + [ -213.2868042 -511.02456665 1026.66882324] + ``` + +* Variables are automatically initialized, no need to run the + [`tf.global_variables_initializer()`](https://www.tensorflow.org/api_docs/python/state_ops/variable_helper_functions#global_variables_initializer) operation. + + ```python + x = tf.Variable(np.random.normal(size=[2, 2]), dtype=tf.float32) + y = tf.constant([[1, 2.]]) + z = tf.matmul(y, x) + print(z) + array([[-1.231673 , 3.14744973]], dtype=float32) + ``` + +* Gradients work as expected using the standard `tf.gradients` function. + + ```python + x = tf.Variable(np.random.rand(1, 3)) + y = tf.exp(x) + dy = tf.gradients(y, x) + # dy/dx should be equal to y (= exp(x)) + print(y, dy) + (array([[ 1.79997761, 2.00581881, 2.37302414]]), [array([[ 1.79997761, 2.00581881, 2.37302414]])]) + ``` + +### Caveats + +The library is implemented on top of standard TensorFlow. It still constructs a +graph in the background and defers op execution. But when an op executes for the +first time, its results are cached and the cached value is returned for future +executions, thus providing imperative semantics. Because of this implementation +choice, this library comes with the following caveats: + +* **Use inside Python loops:** A graph is constructed and kept around in + the background, both for just executing using the standard TensorFlow runtime, + and also for allowing automatic differentiation via `tf.gradients`. This means + that the graph keeps growing when TensorFlow functions are called inside a + Python loop. This library provides a `tf.new_step` method that clears the + graph as well as the cached tensors that have been kept around for gradient + computation. `tf.new_step` can be used as a context manager around, say, a + training loop to clear the graph after each training step. + + ```python + x = tf.Variable(constant_op.constant(1.0)) + for i in range(10): + # Create a new training step + with tf.new_step() as step: + # Perform computation and variable updates + step.run(tf.assign_sub(x, 0.1)) + self.assertAllClose(tf.identity(x), 1.0 - (i + 1) * 0.1) + # The graph within this context is cleared at this point. + ``` + +* **Speed:** Redundant graph construction and caching of tensor values adds + overheads that are not present in standard TensorFlow, where typically the + graph is constructed once and executed multiple times. This library is + intended as a vehicle to prototype the imperative programming model in + TensorFlow. The runtime overheads can be alleviated with various optimizations + to the runtime that would equally benefit the deferred execution mode as + well. + |