diff options
Diffstat (limited to 'tensorflow/tools/docker/notebooks')
| -rw-r--r-- | tensorflow/tools/docker/notebooks/1_hello_tensorflow.ipynb | 742 | ||||
| -rw-r--r-- | tensorflow/tools/docker/notebooks/2_getting_started.ipynb | 844 | ||||
| -rw-r--r-- | tensorflow/tools/docker/notebooks/3_mnist_from_scratch.ipynb | 1985 | ||||
| -rw-r--r-- | tensorflow/tools/docker/notebooks/LICENSE | 13 |
4 files changed, 3584 insertions, 0 deletions
diff --git a/tensorflow/tools/docker/notebooks/1_hello_tensorflow.ipynb b/tensorflow/tools/docker/notebooks/1_hello_tensorflow.ipynb new file mode 100644 index 0000000000..201711d333 --- /dev/null +++ b/tensorflow/tools/docker/notebooks/1_hello_tensorflow.ipynb @@ -0,0 +1,742 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "a3bskVXPvchm" + }, + "source": [ + "# Hello, TensorFlow\n", + "## A beginner-level, getting started, basic introduction to TensorFlow" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "Rb5rSpcZvYbX" + }, + "source": [ + "TensorFlow is a general-purpose system for graph-based computation. A typical use is machine learning. In this notebook, we'll introduce the basic concepts of TensorFlow using some simple examples.\n", + "\n", + "TensorFlow gets its name from [tensors](https://en.wikipedia.org/wiki/Tensor), which are arrays of arbitrary dimensionality. A vector is a 1-d array and is known as a 1st-order tensor. A matrix is a 2-d array and a 2nd-order tensor. The \"flow\" part of the name refers to computation flowing through a graph. Training and inference in a neural network, for example, involves the propagation of matrix computations through many nodes in a computational graph.\n", + "\n", + "When you think of doing things in TensorFlow, you might want to think of creating tensors (like matrices), adding operations (that output other tensors), and then executing the computation (running the computational graph). In particular, it's important to realize that when you add an operation on tensors, it doesn't execute immediately. Rather, TensorFlow waits for you to define all the operations you want to perform. Then, TensorFlow optimizes the computation graph, deciding how to execute the computation, before generating the data. Because of this, a tensor in TensorFlow isn't so much holding the data as a placeholder for holding the data, waiting for the data to arrive when a computation is executed." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "E8FhiMivhcYB" + }, + "source": [ + "## Adding two vectors in TensorFlow\n", + "\n", + "Let's start with something that should be simple. Let's add two length four vectors (two 1st-order tensors):\n", + "\n", + "$\\begin{bmatrix} 1. & 1. & 1. & 1.\\end{bmatrix} + \\begin{bmatrix} 2. & 2. & 2. & 2.\\end{bmatrix} = \\begin{bmatrix} 3. & 3. & 3. & 3.\\end{bmatrix}$" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 131, + "status": "ok", + "timestamp": 1446243605678, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "7391995727249e65", + "userId": "106975671469698476657" + }, + "user_tz": 420 + }, + "id": "2iv3XQ6k3eF1", + "outputId": "e21e1144-736a-4b1f-df78-a9ceab9d4c61" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[ 3. 3. 3. 3.]\n" + ] + } + ], + "source": [ + "import tensorflow as tf\n", + "\n", + "with tf.Session():\n", + " input1 = tf.constant([1.0, 1.0, 1.0, 1.0])\n", + " input2 = tf.constant([2.0, 2.0, 2.0, 2.0])\n", + " output = tf.add(input1, input2)\n", + " result = output.eval()\n", + " print result" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "dqLV5GXT3wLy" + }, + "source": [ + "What we're doing is creating two vectors, [1.0, 1.0, 1.0, 1.0] and [2.0, 2.0, 2.0, 2.0], and then adding them. Here's equivalent code in raw Python and using numpy:" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 152, + "status": "ok", + "timestamp": 1446242020458, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "7391995727249e65", + "userId": "106975671469698476657" + }, + "user_tz": 420 + }, + "id": "7DzDJ7sW79ao", + "outputId": "cf89e613-06e5-4435-bea3-9f48a4eff943" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[3.0, 3.0, 3.0, 3.0]\n" + ] + } + ], + "source": [ + "print [x + y for x, y in zip([1.0] * 4, [2.0] * 4)]" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 97, + "status": "ok", + "timestamp": 1446242021921, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "7391995727249e65", + "userId": "106975671469698476657" + }, + "user_tz": 420 + }, + "id": "MDWJf0lHAF4E", + "outputId": "66d8c4a2-92b7-4048-b365-39dc42dff2bc" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[ 1. 1. 1. 1.] + [ 2. 2. 2. 2.] = [ 3. 3. 3. 3.]\n" + ] + } + ], + "source": [ + "import numpy as np\n", + "x, y = np.full(4, 1.0), np.full(4, 2.0)\n", + "print \"{} + {} = {}\".format(x, y, x + y)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "I52jQOyO8vAn" + }, + "source": [ + "## Details of adding two vectors in TensorFlow\n", + "\n", + "The example above of adding two vectors involves a lot more than it seems, so let's look at it in more depth.\n", + "\n", + ">`import tensorflow as tf`\n", + "\n", + "This import brings TensorFlow's public API into our IPython runtime environment.\n", + "\n", + ">`with tf.Session():`\n", + "\n", + "When you run an operation in TensorFlow, you need to do it in the context of a `Session`. A session holds the computation graph, which contains the tensors and the operations. When you create tensors and operations, they are not executed immediately, but wait for other operations and tensors to be added to the graph, only executing when finally requested to produce the results of the session. Deferring the execution like this provides additional opportunities for parallelism and optimization, as TensorFlow can decide how to combine operations and where to run them after TensorFlow knows about all the operations. \n", + "\n", + ">>`input1 = tf.constant([1.0, 1.0, 1.0, 1.0])`\n", + "\n", + ">>`input2 = tf.constant([2.0, 2.0, 2.0, 2.0])`\n", + "\n", + "The next two lines create tensors using a convenience function called `constant`, which is similar to numpy's `array` and numpy's `full`. If you look at the code for `constant`, you can see the details of what it is doing to create the tensor. In summary, it creates a tensor of the necessary shape and applies the constant operator to it to fill it with the provided values. The values to `constant` can be Python or numpy arrays. `constant` can take an optional shape paramter, which works similarly to numpy's `fill` if provided, and an optional name parameter, which can be used to put a more human-readable label on the operation in the TensorFlow operation graph.\n", + "\n", + ">>`output = tf.add(input1, input2)`\n", + "\n", + "You might think `add` just adds the two vectors now, but it doesn't quite do that. What it does is put the `add` operation into the computational graph. The results of the addition aren't available yet. They've been put in the computation graph, but the computation graph hasn't been executed yet.\n", + "\n", + ">>`result = output.eval()`\n", + "\n", + ">>`print result`\n", + "\n", + "`eval()` is also slightly more complicated than it looks. Yes, it does get the value of the vector (tensor) that results from the addition. It returns this as a numpy array, which can then be printed. But, it's important to realize it also runs the computation graph at this point, because we demanded the output from the operation node of the graph; to produce that, it had to run the computation graph. So, this is the point where the addition is actually performed, not when `add` was called, as `add` just put the addition operation into the TensorFlow computation graph." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "H_5_2YY3ySr2" + }, + "source": [ + "## Multiple operations\n", + "\n", + "To use TensorFlow, you add operations on tensors that produce tensors to the computation graph, then execute that graph to run all those operations and calculate the values of all the tensors in the graph.\n", + "\n", + "Here's a simple example with two operations:" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 101, + "status": "ok", + "timestamp": 1446242580297, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "7391995727249e65", + "userId": "106975671469698476657" + }, + "user_tz": 420 + }, + "id": "-kQmn3U_yXX8", + "outputId": "e96a6e27-665e-47d3-822e-47aeb66fc7f8" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[ 6. 6. 6. 6.]\n" + ] + } + ], + "source": [ + "import tensorflow as tf\n", + "\n", + "with tf.Session():\n", + " input1 = tf.constant(1.0, shape=[4])\n", + " input2 = tf.constant(2.0, shape=[4])\n", + " input3 = tf.constant(3.0, shape=[4])\n", + " output = tf.add(tf.add(input1, input2), input3)\n", + " result = output.eval()\n", + " print result" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "Hod0zvsly8YT" + }, + "source": [ + "This version uses `constant` in a way similar to numpy's `fill`, specifying the optional shape and having the values copied out across it.\n", + "\n", + "The `add` operator supports operator overloading, so you could try writing it inline as `input1 + input2` instead as well as experimenting with other operators." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 156, + "status": "ok", + "timestamp": 1446242664353, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "7391995727249e65", + "userId": "106975671469698476657" + }, + "user_tz": 420 + }, + "id": "yS2WElRfxz53", + "outputId": "9818bf3c-5659-4a87-8b5d-40a28f1a2677" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[ 3. 3. 3. 3.]\n" + ] + } + ], + "source": [ + "with tf.Session():\n", + " input1 = tf.constant(1.0, shape=[4])\n", + " input2 = tf.constant(2.0, shape=[4])\n", + " output = input1 + input2\n", + " print output.eval()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "zszjoYUjkUNU" + }, + "source": [ + "## Adding two matrices" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "EWNYBCB6kbri" + }, + "source": [ + "Next, let's do something very similar, adding two matrices:\n", + "\n", + "$\\begin{bmatrix}\n", + " 1. & 1. & 1. \\\\\n", + " 1. & 1. & 1. \\\\\n", + "\\end{bmatrix} + \n", + "\\begin{bmatrix}\n", + " 1. & 2. & 3. \\\\\n", + " 4. & 5. & 6. \\\\\n", + "\\end{bmatrix} = \n", + "\\begin{bmatrix}\n", + " 2. & 3. & 4. \\\\\n", + " 5. & 6. & 7. \\\\\n", + "\\end{bmatrix}$" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 1540, + "status": "ok", + "timestamp": 1446242690334, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "7391995727249e65", + "userId": "106975671469698476657" + }, + "user_tz": 420 + }, + "id": "tmWcCxSilYkg", + "outputId": "f3a2e904-790b-42e1-9ca4-2f3c54d7f4a8" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[[ 2. 3. 4.]\n", + " [ 5. 6. 7.]]\n" + ] + } + ], + "source": [ + "import tensorflow as tf\n", + "import numpy as np\n", + "\n", + "with tf.Session():\n", + " input1 = tf.constant(1.0, shape=[2, 3])\n", + " input2 = tf.constant(np.reshape(np.arange(1.0, 7.0, dtype=np.float32), (2, 3)))\n", + " output = tf.add(input1, input2)\n", + " print output.eval()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "JuU3Bmglq1vd" + }, + "source": [ + "Recall that you can pass numpy or Python arrays into `constant`.\n", + "\n", + "In this example, the matrix with values from 1 to 6 is created in numpy and passed into `constant`, but TensorFlow also has `range`, `reshape`, and `tofloat` operators. Doing this entirely within TensorFlow could be more efficient if this was a very large matrix.\n", + "\n", + "Try experimenting with this code a bit -- maybe modifying some of the values, using the numpy version, doing this using, adding another operation, or doing this using TensorFlow's `range` function." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "gnXnpnuLrflb" + }, + "source": [ + "## Multiplying matrices" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "Ho-QNSOorj0y" + }, + "source": [ + "Let's move on to matrix multiplication. This time, let's use a bit vector and some random values, which is a good step toward some of what we'll need to do for regression and neural networks." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 132, + "status": "ok", + "timestamp": 1446242872027, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "7391995727249e65", + "userId": "106975671469698476657" + }, + "user_tz": 420 + }, + "id": "uNqMaFR8sIY5", + "outputId": "fc0e29a0-306c-4709-c181-1108d5a21d88" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Input:\n", + "[[ 1. 0. 0. 1.]]\n", + "Weights:\n", + "[[-0.8187139 -0.81037313]\n", + " [-0.31439888 -2.36761999]\n", + " [-1.3127892 -0.33629459]\n", + " [-1.23475349 -1.19031894]]\n", + "Output:\n", + "[[-2.05346727 -2.00069213]]\n" + ] + } + ], + "source": [ + "#@test {\"output\": \"ignore\"}\n", + "import tensorflow as tf\n", + "import numpy as np\n", + "\n", + "with tf.Session():\n", + " input_features = tf.constant(np.reshape([1, 0, 0, 1], (1, 4)).astype(np.float32))\n", + " weights = tf.constant(np.random.randn(4, 2).astype(np.float32))\n", + " output = tf.matmul(input_features, weights)\n", + " print \"Input:\"\n", + " print input_features.eval()\n", + " print \"Weights:\"\n", + " print weights.eval()\n", + " print \"Output:\"\n", + " print output.eval()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "JDAVTPhb22AP" + }, + "source": [ + "Above, we're taking a 1 x 4 vector [1 0 0 1] and multiplying it by a 4 by 2 matrix full of random values from a normal distribution (mean 0, stdev 1). The output is a 1 x 2 matrix.\n", + "\n", + "You might try modifying this example. Running the cell multiple times will generate new random weights and a new output. Or, change the input, e.g., to \\[0 0 0 1]), and run the cell again. Or, try initializing the weights using the TensorFlow op, e.g., `random_normal`, instead of using numpy to generate the random weights.\n", + "\n", + "What we have here is the basics of a simple neural network already. If we are reading in the input features, along with some expected output, and change the weights based on the error with the output each time, that's a neural network." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "XhnBjAUILuy8" + }, + "source": [ + "## Use of variables\n", + "\n", + "Let's look at adding two small matrices in a loop, not by creating new tensors every time, but by updating the existing values and then re-running the computation graph on the new data. This happens a lot with machine learning models, where we change some parameters each time such as gradient descent on some weights and then perform the same computations over and over again." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 180, + "status": "ok", + "timestamp": 1446244201894, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "7391995727249e65", + "userId": "106975671469698476657" + }, + "user_tz": 420 + }, + "id": "vJ_AgZ8lLtRv", + "outputId": "8d3aadaa-2b34-4642-889b-e3daaf5ee693" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[[-0.41494703 0.47648168]] [[-0.41494703 0.47648168]]\n", + "[[ 0.35746408 0.99504066]] [[-0.05748296 1.47152233]]\n", + "[[-0.46462393 -0.80201006]] [[-0.52210689 0.66951227]]\n", + "[[-0.99513483 -0.42322445]] [[-1.51724172 0.24628782]]\n", + "[[ 0.13371086 -0.85545826]] [[-1.38353086 -0.60917044]]\n" + ] + } + ], + "source": [ + "#@test {\"output\": \"ignore\"}\n", + "import tensorflow as tf\n", + "import numpy as np\n", + "\n", + "with tf.Session() as sess:\n", + " # Set up two variables, total and weights, that we'll change repeatedly.\n", + " total = tf.Variable(tf.zeros([1, 2]))\n", + " weights = tf.Variable(tf.random_uniform([1,2]))\n", + " \n", + " # Initialize the variables we defined above.\n", + " tf.initialize_all_variables().run()\n", + " \n", + " # This only adds the operators to the graph right now. The assignment\n", + " # and addition operations are not performed yet.\n", + " update_weights = tf.assign(weights, tf.random_uniform([1, 2], -1.0, 1.0))\n", + " update_total = tf.assign(total, tf.add(total, weights))\n", + " \n", + " for _ in range(5):\n", + " # Actually run the operation graph, so randomly generate weights and then\n", + " # add them into the total. Order does matter here. We need to update\n", + " # the weights before updating the total.\n", + " sess.run(update_weights)\n", + " sess.run(update_total)\n", + " \n", + " print weights.eval(), total.eval()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "kSYJr89aM_n0" + }, + "source": [ + "This is more complicated. At a high level, we create two variables and add operations over them, then, in a loop, repeatedly execute those operations. Let's walk through it step by step.\n", + "\n", + "Starting off, the code creates two variables, `total` and `weights`. `total` is initialized to \\[0, 0\\] and `weights` is initialized to random values between -1 and 1.\n", + "\n", + "Next, two assignment operators are added to the graph, one that updates weights with random values from [-1, 1], the other that updates the total with the new weights. Again, the operators are not executed here. In fact, this isn't even inside the loop. We won't execute these operations until the `eval` call inside the loop.\n", + "\n", + "Finally, in the for loop, we run each of the operators. In each iteration of the loop, this executes the operators we added earlier, first putting random values into the weights, then updating the totals with the new weights. This call uses `eval` on the session; the code also could have called `eval` on the operators (e.g. `update_weights.eval`).\n", + "\n", + "It can be a little hard to wrap your head around exactly what computation is done when. The important thing to remember is that computation is only performed on demand.\n", + "\n", + "Variables can be useful in cases where you have a large amount of computation and data that you want to use over and over again with just a minor change to the input each time. That happens quite a bit with neural networks, for example, where you just want to update the weights each time you go through the batches of input data, then run the same operations over again." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "fL3WfAbKzqr5" + }, + "source": [ + "## What's next?\n", + "\n", + "This has been a gentle introduction to TensorFlow, focused on what TensorFlow is and the very basics of doing anything in TensorFlow. If you'd like more, the next tutorial in the series is Getting Started with TensorFlow, also available in the [notebooks directory](http://127.0.0.1:8888/tree)." + ] + } + ], + "metadata": { + "colabVersion": "0.3.2", + "colab_default_view": {}, + "colab_views": {}, + "kernelspec": { + "display_name": "Python 2", + "language": "python", + "name": "python2" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 2 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython2", + "version": "2.7.6" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/tensorflow/tools/docker/notebooks/2_getting_started.ipynb b/tensorflow/tools/docker/notebooks/2_getting_started.ipynb new file mode 100644 index 0000000000..01d1c521fe --- /dev/null +++ b/tensorflow/tools/docker/notebooks/2_getting_started.ipynb @@ -0,0 +1,844 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "6TuWv0Y0sY8n" + }, + "source": [ + "# Getting Started in TensorFlow\n", + "## A look at a very simple neural network in TensorFlow" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "u9J5e2mQsYsQ" + }, + "source": [ + "This is an introduction to working with TensorFlow. It works through an example of a very simple neural network, walking through the steps of setting up the input, adding operators, setting up gradient descent, and running the computation graph. \n", + "\n", + "This tutorial presumes some familiarity with the TensorFlow computational model, which is introduced in the [Hello, TensorFlow](http://127.0.0.1:8888/tree) notebook, also available in this bundle." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "Dr2Sv0vD8rT-" + }, + "source": [ + "## A simple neural network\n", + "\n", + "Let's start with code. We're going to construct a very simple neural network computing a linear regression between two variables, y and x. The function it tries to compute is the best $w_1$ and $w_2$ it can find for the function $y = w_2 x + w_1$ for the data. The data we're going to give it is toy data, linear perturbed with random noise.\n", + "\n", + "This is what the network looks like:" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "collapsed": false + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAJYAAABkCAYAAABkW8nwAAAO90lEQVR4Xu2dT5Dc1J3Hv+YQT8VJ\nZUhVdprLWs4FTSrGGv4ql9CuHBCH4GaTFCLZwnIcjOAy8l6Q/1SlU4XHcg6xJgtY2OOik2KxSGoT\nGWrXzYFC2T2MDAtWitRavmQ0e9k2SYGowom4hNRPtqA9TE+rW3/cPfPepcfup6f3fu/Tv9/T+/PV\npo8//vhjsMQsULAFNjGwCrYoKy6xAAOLgVCKBRhYpZiVFcrAYgyUYgEGVilmZYUysBgDpViAgVWK\nWVmhDCzGQCkWGEuwrly5gtf++zW887/vYOn/lnD5T5cT40x9ZQrb/nEbxDtFiHeI2LJlSylGY4X2\nt8BYgUVAvfzqy3i5/TI+vPLhmq37wpYv4AHpATxw3wMMsP4cFJ5jbMAiqA4eOYg/Lv8xMcL26e34\n+vTXk8+vbv1q8n/03TsX38EfLv4h+aRE380dmmNwFY7O2gWOBVgE1Y/2/yjxUls+vwXaY1oS7tZK\n3v94MJ8zceUvV0Dea+H4AoOrQrhGHqxuT0Xjp0P7D2HqH6Yymejyu5dx5PiRZBxGnmt+bj7TdSxT\nfgv0ASuAzglwmyE8pfbZu3VaEDkDdT+AweevzGolvPjvL+LMb84knmr+yHxmqNKyCK7ZQ7OJ5yIo\n+3m6clqx8UrNB1bso2W64FQN9cnijdcdAvNAQWGRPBcLicX3Ua8S84FVcj3PnjuLhRcWkgH63OG5\nXHc7+NTBZEBP47NvffNbucpiF/e3QCaw2g0NfNvES5c+wtQ9u2G0LCj8BLAiFEaeBU0zYJ9fxkfY\njKl7FZgtCzIHIA7QUmXov/g9LmMztt6rwLBMyFROj3TkZ0fgveXh4X96GN//zvf7t2aNHGlI7VlW\n0pYmRC+AKUwAsQu5thOuvIjQEjGBGJ7CQYptdOw6etc6VzXXzcUZwJrGseWt2P28DV2I4OgyDgQK\nFgMTYtQ1xqq10eDuR6j8Fi1NxGTkwpAfRos7h05bQscQIFgibEeHMBHCVhs4EBtY8lQQd6ulvbN7\n8e6f302mC7Z/bXsuo9NkKk1X9PZ+IUyeR0sN4GscYl8DPzOP5VuPYynQwMU+dL4O3wzRbpQQ93O1\nbvQuzgRWS0p/tQA6Nuqcilq7A5u3Px28T7qw7BB1VUHqhEKTB2+pCAIVHZVD3dPgujpE6peOBzes\nQRS5nr/+b//g24nF7JN27qkCGq/J++RknHXm5JlVeiKGr/MQPQMdV0ZkCRBbNUwEMYzQhRyZEHgH\nOv29ynPM6HXtja1Rf7B4AZ7RgZv+SuMAOj+NtrYEX3avfyqMfDi2DdcLEAQBvPOX8MGtR3Ex0MEF\nJiRxP373wWZsvaeBhixDVRrg1/jxlwEWPV3ap+xVrR57Cjgpht2xEDV4mLIFvqkiaoUwwzp4U4Hv\n9/awN7YrR+vuGcAS4ZsdtKV0VNEFVqMLrIkWJGEPPP4hKA0RgiCAc1XsdJQErGQ2Ig7hOQ5sx4Hz\n0u+wvHX2akjtMWCpNhQCiCicq+AcCx1Fh9B2IegcNN6B4Teg1z0EeknzKqPFRe7a9AeLm4ajXvzU\noJEDqUahMESrKxSqbQHbDBGLoXUNlBiuUsNOT8fFQEVsNdHmdOjStTgSGOCnLTQuBDBosLxKqnTw\nntw/glPnoHMS4E6iFVjgbBGcwUGMPAjtawP73GZf/wVkAutYtAvPezYUPoKjipBdGZ5vQOgavGte\nHbfsiXD09TZUIUbg6JD3vITlrU/iYthErPOYaQk44ZhocDF8U0HDqsEOHfQaC7/2X68lyzJVTjd0\nWiJu2XMem++7+tAxSd52+hguTe3GYtjq6V3XPyqDtbA/WLyAtqRg0rHhLceo3avCsk0kjqd7uoEL\n0FJkaC/9Hh/gS9ixS0dTCaDKHVidNhoTNN2gQP/FedAmly/t2IWm2YK2xswqDbj3antzz5oToD/9\n15/i5smbcdo8vfaDQGiC37YfEyeW4KtcMu2g1HbCrp9Dx5Fw3ZCw04ZSb0Jse6CsLH1qgZFfK0zn\nn+hpznzKHGpJRzus4YJ/AX/78G94ofUC7r777pwMxAhdE6pyAK8u78CJJZ+BtcKiIw8Wea0DTx34\nZCH5oHYwM1y0TjhnziXbaWgB+4cP/RCPPfYYtm/fjpMnT+Kmm24aDrDYhdpoQdAbaMtNSB4Da6Uh\nRx4sqnB3SCTPNbtvtu9iMoU/Wg5Kt9p0h8DTp09j3759ePrpp/H4448PB1fylOtC5jTUGVifseFY\ngJXClXou+jcN6Gk2nj7JG1Gi7TG0Hkiz7OlGP/ru6OGjq46rnnjiCSwuLibe66677hocMAZWT5uN\nDVgpXGfbZ5OtybQNZq1EE6G0NXmXtGvNwbrv+4n3uu222wYPjwys9QFW2goKjbQ4Tdth6CAFeSpK\n5J3oQMUwhynS8PjMM89AVdVs3ouBtb7Aytbrw+WiMZfnednCIwOLgTUIZml43LFjB5577rnhnx4H\nuek6yztWY6yqbb+wsJBMTwwUHquu5Ijej4GVoWMoPJ4/fz7xXkM9PWa4x3rLwsDK2KMXLlxIvBeF\nR5qe2LRpU8YrN2Y2BtaA/U7hkaYnnn322exPjwPeYz1kZ2AN2YtpeCTvdeeddw5Zyvq9jIGVo28p\nPJL3ok2NLDxeb0gGVg6w0kvT8HjixIlkHJY1lauaE8GRangwsvD/noKqt+kzsLJSkCEfzdi/8cYb\nifdaKzxWoppDmxJ5FT54NH06YZShAQVmYWAVaEwqKg2PMzMzyfTEyqfHqlRzAoOH6OqwJnXoNQeB\nSWcjq0sMrJJsferUqSQsdofHylRzYg8aLyG0QtiTOvhGhFZglyKD0Mt8DKySwEqLpfD45ptvYn5+\nHr/+z19/sukwj2pOP72vyJXBy4BNME340Pg6AiNAu8IDkQysksGi4t9++2189wffxee++DkIO4Tc\nqjlrSw504Eg81FobYetq+KOwKDgagjVOnRdtBgZW0RZdpbw0BL73/nv4yZM/6bv7tVeVxkk1h4FV\nAVgbUTWHgVUBWGUcvCVV6EP/cuiztQ9NCNsMiIshrPSIeaK3oUNIlXQqaDMDqwIjlyEV0Fv6MoQl\nbENT/FTIhWSXOF2AF5jocei8cCswsAo36WcLLEPchO7yyr+9smrt6TQ3geQmcgcd2CQbIHoIDKGy\nuSwG1joEi06oU+jj3RAWR2HQgFiiTuxqJmRgVQBWGaGQDo78/OjPe9T+qpfSeBeeqIM3JPip4k8F\n7aVbMLAqMHSlg/dr7YkcCZxWg1Jz0G5UL7/EwKoArBuhmoNEbupBvPrRDhxf8qFVLFrCwKoArFQi\n4P3o/VwTpCmgdBi3r2oOIrQbNdwfGljytZ46r2U1n4FVlmW7yn3rrbfwvX/+XrKkMyPM5FLNIS2K\nbCrSNI8loKX48G6AxhIDq2SwaIcDgWWaJn71H78qRDWnlxbF1aaQxJILj6TRjRhm0L4hYrwMrJLA\nos1+BBXtyaLty5SKVs1Zverx1RB4dhIPPe/CVioeXF2rFAOrYLDIOxFQd9xxRwLVytSt90XfFaGa\nU3ATCimOgVWIGa8WkoY9AorA6pUIrqJVcwpsRiFFMbAKMONqYS9LsWWo5mS5bxV5GFg5rExhj8ZP\ndHBitbCXo+ixv5SBNWQXpmGPvNXtt98+ZCnr9zIG1oB9O2zYG/A2Y5+dgZWxC1nYy2goNt2Q3VA0\njqIDESzsZbcZ81hr2CoNe/T56KOPZrcqy8m2zazGAAt7+X8ZzGOtsCELe/mhohLGEqwyVFpY2CsG\nqLSUsQKrDJUWFvaKBWrswCpDpYWFvXKgKiYUxh5U/huwhd8idBqYRARX4bHTldd8Le8gTSpapYWW\nX0is47qnveTdi02I6aFOejlAbSdcOT2fF8NTOEixDTqnV6Uk0CC2GpW8hYTCyFXA72yj8XoAAzoE\n+nsxgNnrZc8DtL7bU9HJlDwqLY9855FkbY8ktS3LWlGLECbPo6UG8DUOsa+Bn5nH8q3HsRRo4GIS\nL6vDN0O0e70SdoB2rfeshYBF71Juyzzu90TcF59FIC8WJvSVvgiT9nnPH5nP/K7CtOPonYWzh2aT\nF2Fu+usmvPjLF3us7cXwdR6iZ6DjyogsAWKrhokghhG6kCMTAu9Ap7+r1l0cQwoLAote4+ugwT+I\nsxO78XrQKkTkqzsEkqeily8Nk0il5cfHfowv3/xlLBxf6Pk2sNhTwEkx7I6FqMHDlC3wTRVRK4QZ\n1sGbCnxfrfxgwjBtvtHXFAZW7OsQZo7hEm7Fkxf8nm+mH6TBlau0RG00OBWcY6Gj6BDaLgSdDn46\nMPwG9Hr15/MGsdco5S0GrDiAIU7D5M/AgIo9gY6Lng4+5wi3jIOea59wieCQzgEnAe4kWoEFzhbB\nGRzEyIPQDmBWpaoxSpQMUZdCwCLh1OlmDWcCBzJsSNzDiIyL8LR8Ur1lHE2nPeZzh+d6mooENW7Z\ncx6b7zuHTlvCJB1Nnz6GS1O7sUhKxDl/LEP00Vhekh8sUjThNUyYAdxr59dCSwSvAWbg5Xq7exkq\nLfRO6TMnz/TurNAEv20/Jk4swaf2xC6U2k7Y9XPoOBIm6crYh6UoaLodABOoSU3YlpLbQ48lQT0q\nnR+sEq1RBlj0dGmfsnPVOtB51IMmfEdGLQ7RkkSYkps8VbJ01QIjDdaNCIVZwOi4DnxOgsRRXIzh\nazwakY3gmphsljLWe56RBqv6wfvg3R0HFqS6CcHxC5kQHrwGo3nFSIN1Q1RaBuinyDchSyYmDRct\nhWPLPF22G2mwuo+k55kgHUylJRtZoa1A0kI0bAdGPRnSszQuYFE90yUdepoznzKHWtLRDmsglZY8\ncHZTE7UVCGqEpmtDScZZLK20wEh7LKpst9YBKQUf1A5mhovWCefMuU9eM9JbWnEQMAIY/DQOXLr+\nmqmHXkfIdj18YpSRByuFa6+2F1f+cgXkuWb3zfZdN6Twt/DCQuKpsgmVDQIXy9vPAmMB1krPRf9e\nryot/TpsXL4fG7BSuNa7Ssu4gNOvnmMFVtqY9azS0q/DxuX7sQRrXIy7kevJwNrIvV9i2xlYJRp3\nIxfNwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3Ixf9d0NIelzdt4X5\nAAAAAElFTkSuQmCC\n", + "text/plain": [ + "<IPython.core.display.Image object>" + ] + }, + "execution_count": 1, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from IPython.display import Image\n", + "import base64\n", + "Image(data=base64.decodestring(\"iVBORw0KGgoAAAANSUhEUgAAAJYAAABkCAYAAABkW8nwAAAO90lEQVR4Xu2dT5Dc1J3Hv+YQT8VJZUhVdprLWs4FTSrGGv4ql9CuHBCH4GaTFCLZwnIcjOAy8l6Q/1SlU4XHcg6xJgtY2OOik2KxSGoTGWrXzYFC2T2MDAtWitRavmQ0e9k2SYGowom4hNRPtqA9TE+rW3/cPfPepcfup6f3fu/Tv9/T+/PVpo8//vhjsMQsULAFNjGwCrYoKy6xAAOLgVCKBRhYpZiVFcrAYgyUYgEGVilmZYUysBgDpViAgVWKWVmhDCzGQCkWGEuwrly5gtf++zW887/vYOn/lnD5T5cT40x9ZQrb/nEbxDtFiHeI2LJlSylGY4X2t8BYgUVAvfzqy3i5/TI+vPLhmq37wpYv4AHpATxw3wMMsP4cFJ5jbMAiqA4eOYg/Lv8xMcL26e34+vTXk8+vbv1q8n/03TsX38EfLv4h+aRE380dmmNwFY7O2gWOBVgE1Y/2/yjxUls+vwXaY1oS7tZK3v94MJ8zceUvV0Dea+H4AoOrQrhGHqxuT0Xjp0P7D2HqH6Yymejyu5dx5PiRZBxGnmt+bj7TdSxTfgv0ASuAzglwmyE8pfbZu3VaEDkDdT+AweevzGolvPjvL+LMb84knmr+yHxmqNKyCK7ZQ7OJ5yIo+3m6clqx8UrNB1bso2W64FQN9cnijdcdAvNAQWGRPBcLicX3Ua8S84FVcj3PnjuLhRcWkgH63OG5XHc7+NTBZEBP47NvffNbucpiF/e3QCaw2g0NfNvES5c+wtQ9u2G0LCj8BLAiFEaeBU0zYJ9fxkfYjKl7FZgtCzIHIA7QUmXov/g9LmMztt6rwLBMyFROj3TkZ0fgveXh4X96GN//zvf7t2aNHGlI7VlW0pYmRC+AKUwAsQu5thOuvIjQEjGBGJ7CQYptdOw6etc6VzXXzcUZwJrGseWt2P28DV2I4OgyDgQKFgMTYtQ1xqq10eDuR6j8Fi1NxGTkwpAfRos7h05bQscQIFgibEeHMBHCVhs4EBtY8lQQd6ulvbN78e6f302mC7Z/bXsuo9NkKk1X9PZ+IUyeR0sN4GscYl8DPzOP5VuPYynQwMU+dL4O3wzRbpQQ93O1bvQuzgRWS0p/tQA6Nuqcilq7A5u3Px28T7qw7BB1VUHqhEKTB2+pCAIVHZVD3dPgujpE6peOBzesQRS5nr/+b//g24nF7JN27qkCGq/J++RknHXm5JlVeiKGr/MQPQMdV0ZkCRBbNUwEMYzQhRyZEHgHOv29ynPM6HXtja1Rf7B4AZ7RgZv+SuMAOj+NtrYEX3avfyqMfDi2DdcLEAQBvPOX8MGtR3Ex0MEFJiRxP373wWZsvaeBhixDVRrg1/jxlwEWPV3ap+xVrR57Cjgpht2xEDV4mLIFvqkiaoUwwzp4U4Hv9/awN7YrR+vuGcAS4ZsdtKV0VNEFVqMLrIkWJGEPPP4hKA0RgiCAc1XsdJQErGQ2Ig7hOQ5sx4Hz0u+wvHX2akjtMWCpNhQCiCicq+AcCx1Fh9B2IegcNN6B4Teg1z0EeknzKqPFRe7a9AeLm4ajXvzUoJEDqUahMESrKxSqbQHbDBGLoXUNlBiuUsNOT8fFQEVsNdHmdOjStTgSGOCnLTQuBDBosLxKqnTwntw/glPnoHMS4E6iFVjgbBGcwUGMPAjtawP73GZf/wVkAutYtAvPezYUPoKjipBdGZ5vQOgavGteHbfsiXD09TZUIUbg6JD3vITlrU/iYthErPOYaQk44ZhocDF8U0HDqsEOHfQaC7/2X68lyzJVTjd0WiJu2XMem++7+tAxSd52+hguTe3GYtjq6V3XPyqDtbA/WLyAtqRg0rHhLceo3avCsk0kjqd7uoEL0FJkaC/9Hh/gS9ixS0dTCaDKHVidNhoTNN2gQP/FedAmly/t2IWm2YK2xswqDbj3antzz5oToD/915/i5smbcdo8vfaDQGiC37YfEyeW4KtcMu2g1HbCrp9Dx5Fw3ZCw04ZSb0Jse6CsLH1qgZFfK0znn+hpznzKHGpJRzus4YJ/AX/78G94ofUC7r777pwMxAhdE6pyAK8u78CJJZ+BtcKiIw8Wea0DTx34ZCH5oHYwM1y0TjhnziXbaWgB+4cP/RCPPfYYtm/fjpMnT+Kmm24aDrDYhdpoQdAbaMtNSB4Da6UhRx4sqnB3SCTPNbtvtu9iMoU/Wg5Kt9p0h8DTp09j3759ePrpp/H4448PB1fylOtC5jTUGVifseFYgJXClXou+jcN6Gk2nj7JG1Gi7TG0Hkiz7OlGP/ru6OGjq46rnnjiCSwuLibe66677hocMAZWT5uNDVgpXGfbZ5OtybQNZq1EE6G0NXmXtGvNwbrv+4n3uu222wYPjwys9QFW2goKjbQ4Tdth6CAFeSpK5J3oQMUwhynS8PjMM89AVdVs3ouBtb7Aytbrw+WiMZfnednCIwOLgTUIZml43LFjB5577rnhnx4Huek6yztWY6yqbb+wsJBMTwwUHquu5Ijej4GVoWMoPJ4/fz7xXkM9PWa4x3rLwsDK2KMXLlxIvBeFR5qe2LRpU8YrN2Y2BtaA/U7hkaYnnn322exPjwPeYz1kZ2AN2YtpeCTvdeeddw5Zyvq9jIGVo28pPJL3ok2NLDxeb0gGVg6w0kvT8HjixIlkHJY1lauaE8GRangwsvD/noKqt+kzsLJSkCEfzdi/8cYbifdaKzxWoppDmxJ5FT54NH06YZShAQVmYWAVaEwqKg2PMzMzyfTEyqfHqlRzAoOH6OqwJnXoNQeBSWcjq0sMrJJsferUqSQsdofHylRzYg8aLyG0QtiTOvhGhFZglyKD0Mt8DKySwEqLpfD45ptvYn5+Hr/+z19/sukwj2pOP72vyJXBy4BNME340Pg6AiNAu8IDkQysksGi4t9++2189wffxee++DkIO4TcqjlrSw504Eg81FobYetq+KOwKDgagjVOnRdtBgZW0RZdpbw0BL73/nv4yZM/6bv7tVeVxkk1h4FVAVgbUTWHgVUBWGUcvCVV6EP/cuiztQ9NCNsMiIshrPSIeaK3oUNIlXQqaDMDqwIjlyEV0Fv6MoQlbENT/FTIhWSXOF2AF5jocei8cCswsAo36WcLLEPchO7yyr+9smrt6TQ3geQmcgcd2CQbIHoIDKGyuSwG1joEi06oU+jj3RAWR2HQgFiiTuxqJmRgVQBWGaGQDo78/OjPe9T+qpfSeBeeqIM3JPip4k8F7aVbMLAqMHSlg/dr7YkcCZxWg1Jz0G5UL7/EwKoArBuhmoNEbupBvPrRDhxf8qFVLFrCwKoArFQi4P3o/VwTpCmgdBi3r2oOIrQbNdwfGljytZ46r2U1n4FVlmW7yn3rrbfwvX/+XrKkMyPM5FLNIS2KbCrSNI8loKX48G6AxhIDq2SwaIcDgWWaJn71H78qRDWnlxbF1aaQxJILj6TRjRhm0L4hYrwMrJLAos1+BBXtyaLty5SKVs1Zverx1RB4dhIPPe/CVioeXF2rFAOrYLDIOxFQd9xxRwLVytSt90XfFaGaU3ATCimOgVWIGa8WkoY9AorA6pUIrqJVcwpsRiFFMbAKMONqYS9LsWWo5mS5bxV5GFg5rExhj8ZPdHBitbCXo+ixv5SBNWQXpmGPvNXtt98+ZCnr9zIG1oB9O2zYG/A2Y5+dgZWxC1nYy2goNt2Q3VA0jqIDESzsZbcZ81hr2CoNe/T56KOPZrcqy8m2zazGAAt7+X8ZzGOtsCELe/mhohLGEqwyVFpY2CsGqLSUsQKrDJUWFvaKBWrswCpDpYWFvXKgKiYUxh5U/huwhd8idBqYRARX4bHTldd8Le8gTSpapYWWX0is47qnveTdi02I6aFOejlAbSdcOT2fF8NTOEixDTqnV6Uk0CC2GpW8hYTCyFXA72yj8XoAAzoE+nsxgNnrZc8DtL7bU9HJlDwqLY9855FkbY8ktS3LWlGLECbPo6UG8DUOsa+Bn5nH8q3HsRRo4GISL6vDN0O0e70SdoB2rfeshYBF71Juyzzu90TcF59FIC8WJvSVvgiT9nnPH5nP/K7CtOPonYWzh2aTF2Fu+usmvPjLF3us7cXwdR6iZ6DjyogsAWKrhokghhG6kCMTAu9Ap7+r1l0cQwoLAote4+ugwT+IsxO78XrQKkTkqzsEkqeily8Nk0il5cfHfowv3/xlLBxf6Pk2sNhTwEkx7I6FqMHDlC3wTRVRK4QZ1sGbCnxfrfxgwjBtvtHXFAZW7OsQZo7hEm7Fkxf8nm+mH6TBlau0RG00OBWcY6Gj6BDaLgSdDn46MPwG9Hr15/MGsdco5S0GrDiAIU7D5M/AgIo9gY6Lng4+5wi3jIOea59wieCQzgEnAe4kWoEFzhbBGRzEyIPQDmBWpaoxSpQMUZdCwCLh1OlmDWcCBzJsSNzDiIyL8LR8Ur1lHE2nPeZzh+d6mooENW7Zcx6b7zuHTlvCJB1Nnz6GS1O7sUhKxDl/LEP00Vhekh8sUjThNUyYAdxr59dCSwSvAWbg5Xq7exkqLfRO6TMnz/TurNAEv20/Jk4swaf2xC6U2k7Y9XPoOBIm6crYh6UoaLodABOoSU3YlpLbQ48lQT0qnR+sEq1RBlj0dGmfsnPVOtB51IMmfEdGLQ7RkkSYkps8VbJ01QIjDdaNCIVZwOi4DnxOgsRRXIzhazwakY3gmphsljLWe56RBqv6wfvg3R0HFqS6CcHxC5kQHrwGo3nFSIN1Q1RaBuinyDchSyYmDRcthWPLPF22G2mwuo+k55kgHUylJRtZoa1A0kI0bAdGPRnSszQuYFE90yUdepoznzKHWtLRDmsglZY8cHZTE7UVCGqEpmtDScZZLK20wEh7LKpst9YBKQUf1A5mhovWCefMuU9eM9JbWnEQMAIY/DQOXLr+mqmHXkfIdj18YpSRByuFa6+2F1f+cgXkuWb3zfZdN6Twt/DCQuKpsgmVDQIXy9vPAmMB1krPRf9eryot/TpsXL4fG7BSuNa7Ssu4gNOvnmMFVtqY9azS0q/DxuX7sQRrXIy7kevJwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3Ixf9d0NIelzdt4X5AAAAAElFTkSuQmCC\"), embed=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "fBQq_R8B8rRf" + }, + "source": [ + "Here is the TensorFlow code for this simple neural network and the results of running this code:" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 665, + "status": "ok", + "timestamp": 1446658971218, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "4896c353dcc58d9f", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "Dy8pFefa_Ho_", + "outputId": "5a95f8c8-0c32-411d-956d-bb81aeed8e50" + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlsAAAEPCAYAAAB1MgENAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYVNW1///36oFuZpFmUFBQQGVwQkVQgx0cUENEo5HM\nIRqvSuKAibZmuOD3l1wFjWg0XjVxilfjnGCIUVDpoAmIiiIWCCjz1NDMMz2s3x+nWsumm56q6lRV\nf17Pc56qOnXqnFVAb1bvvc/a5u6IiIiISGJkhR2AiIiISCZTsiUiIiKSQEq2RERERBJIyZaIiIhI\nAinZEhEREUkgJVsiIiIiCRSXZMvMHjGzEjP7KGZfBzObamYLzew1M2sfj2uJiCSSmXU3szfNLGJm\n88zs2uj+iWa2wMw+NLMXzaxdzGduNbPF0ffPDS96EUlF8erZegwYXm3fLcDr7n408CZwa5yuJSKS\nSOXAje7eHxgC/NTMjgGmAv3d/QRgMdE2zcz6AZcBfYHzgQfMzEKJXERSUlySLXd/G9hcbfdI4Ino\n8yeAi+JxLRGRRHL3de7+YfT5DmAB0M3dX3f3yuhhs4Du0ecXAs+4e7m7LyNIxAYlOWwRSWGJnLPV\n2d1LIGi8gM4JvJaISNyZWU/gBOCdam9dDrwSfd4NWBnz3uroPhERILkT5LUukIikDTNrA7wAXB/t\n4ara/0ugzN3/ElpwIpJWchJ47hIz6+LuJWbWFVhf00FmpiRMJEO4e0bMVTKzHIJE60l3nxyzfzRw\nATAs5vDVwGExr7tH99V0XrV3IhmgoW1dPHu2LLpVeRkYHX3+Q2By9Q9UcfeU2MaNGxd6DKkWS6rE\noVhSOw73jMshHgXmu/u9VTvM7DzgJuBCd98bc+zLwLfMrIWZHQH0BmbXduKw/54y5d9bc/0O6R5/\nJnyHxohLz5aZPQ0UAh3NbAUwDrgDeN7MLgeWE9ytIyKS0szsdOC7wDwz+4BgCsQvgd8DLYBp0ZsN\nZ7n7GHefb2bPAfOBMmCMN7ZFFpGMFJdky92/U8tbZ8fj/CIiyeLu/waya3irzwE+cztwe8KCEpG0\npgryMQoLC8MO4XOpEkuqxAGKpSapEoc0D5nw7y3dv0O6xw+Z8R0aysLu7TYz9biLZAAzwzNkgnyi\nqL0TSX+NaevUsyUiIiKSQEq2RERERBJIyZaIiIhIAinZEhEREUkgJVsiIiIiCaRkS0RERCSBlGyJ\niIiIJJCSLREREZEEUrIlIiIikkBKtkREREQSSMmWiIiISAIp2RIRERFJICVbIiJJpHWoRZofJVsi\nIkm0c2fYEYhIsinZEhFJopKSsCMQkWRTsiUikkTr14cdgYgkm5ItEZEkUrIl0vwo2RIRSSINI4o0\nPzlhByAijROJRJgybQoAI84ZQf/+/UOOSOpDPVsizY96tkTSUCQS4eY7b2Zm2Uxmls3k5jtvJhKJ\nhB2W1IOSLZHmR8mWSJqJRCL8fNzPWdpyKbmdc+lyTBdy++d+3sslqU3DiCLNj5ItkTRS1aO1onIF\n67PW88aCN9i8eXPYYWUUM+tuZm+aWcTM5pnZddH9HcxsqpktNLPXzKx9zGduNbPFZrbAzM490PnV\nsyXS/CjZEkkjU6ZNIbd/Lt3O7Ma+8n3kbMvh45kfUxYpY8Q5I8IOL1OUAze6e39gCPATMzsGuAV4\n3d2PBt4EbgUws37AZUBf4HzgATOz2k6uni2R5kfJlkia2V65nZIWJZze/3Q6retEjw09mHjTRE2Q\njxN3X+fuH0af7wAWAN2BkcAT0cOeAC6KPr8QeMbdy919GbAYGFTb+dWzJdL8KNkSSSPHDj6WpSVL\n6bStE3l78+jZrid33XaXEq0EMbOewAnALKCLu5dAkJABnaOHdQNWxnxsdXRfjbZuhfLyREQrIqlK\npR9E0sSSzUt4c9ub3PPNe1jwzgIARtykkg+JYmZtgBeA6919h5lVX0K6UUtKt2gxnqIiaNsWCgsL\nKSwsbGqoIpJAxcXFFBcXN+kc5iEvQW9mHnYMIqlu+Zbl3Df7PkafMJoBnQeEHU6NzAx3r3WuUjox\nsxxgCvBPd783um8BUOjuJWbWFZju7n3N7BbA3X1C9LhXgXHu/k4N5/Vjj3WefBKOPz5530dE4qcx\nbZ2GEUVS3Kptq7h/9v18/7jvp2yilYEeBeZXJVpRLwOjo89/CEyO2f8tM2thZkcAvYHZtZ24c2fN\n2xJpbjSMKJLC1m5fy72z7uVbA77F8V3VFZIMZnY68F1gnpl9QDBc+AtgAvCcmV0OLCe4AxF3n29m\nzwHzgTJgzIG66zt31h2JIs2Nki2RFFWyo4R7Zt3DN/t/k5MOPSnscJoNd/83kF3L22fX8pnbgdvr\nc/4uXdSzJdLcaBhRJAWV7ipl0qxJjDxmJIO61VpFQNKQhhFFmh/1bImkmI27NnL3zLu5oM8FnHbY\nafu9P2fOHJ599jXWrVuNWTZdunRl1KjhDBw4MIRopaE6d4ZFi8KOQkSSScmWSArZsmcLk2ZN4uwj\nz2Zoj6H7vT9nzhyuvnoSe/aczpIls4DzOeKIQ5k+fRIPPjhWCVca0DCiSPOjYUSRFLFt7zbunnk3\nQ3sMZdgRw2o85tlnXyM7ezS7dm0mN/c6cnMvYffuvmRnj+bZZ19LarzSOBpGFGl+1LMlEqJIJMKU\naVPY63tZ120dI04cwbm9DriOsaS5Ll10N6JIc6OeLZGQRCIRbr7zZt7a9xZ/2f0XZhfPpse+Hgf8\nzKhRw6moeJxWrTpQVvZ7yspepGXLBVRUPM6oUcOTFLk0RadOQc+WajmLNB+qIC/NQtWkciBlJpNP\nuGcCb+97m5IOJbTPa0+rda04rcVpFN1QdMDPpeoE+UyqIJ8oVe1d27awahW0bx92RCLSUI1p6zSM\nKBmvalJ5dvZogJSZTF7mZSwpX0LXFl054qAjWF9Sv4k8AwcODD12aZqqSfJKtkSaBw0jSsarmlRe\nUHAWBQVnpcRk8r3le9nSYwu5q3NpU9KG9QvXUxYpY8Q5I0KNS5JDk+RFmhf1bIkkWVlFGQ+8+wDH\nHnkso48azT9e/wcAI24aQf/+/UOOTpJBS/aINC9KtiTjjRo1nOnTJ1FaGrwOJpOPDSWW8spy/ve9\n/6VdXju+f/z3ybIsBgzQ4tLNjWptiTQvSrYk4w0cOJAHHxwbM0E+nPlaFZUVPPz+w+Rl5/GjE39E\nlmkUv7nSMKJI86JkS5qFsCeVV3olf5rzJ9ydK066QolWM9elCyxYEHYUIpIsavFFEqzSK3nsg8fY\nW7GXq06+ipws/Y7T3KlnS6R5UbIlkkDuzpNzn2Tb3m1cc/I1SrQEULIl0twkvOU3s2XAVqASKHP3\nQYm+pkgqcHeenvc0G3Zt4NpB15KbnRt2SJIitGSPSPOSjF+zK4FCd9+chGuJ1CqZVeTdneciz7Fq\n2yquH3w9eTl5CbuWpB/1bIk0LwlfrsfMlgInu/vGWt7Xcj2ScNWryFdUPB7XKvJVC0oDfO3sr7Ew\nayELNy7khsE30Cq3VVyukeq0XE/dqtq7ykrIz4cdO6BFi7CjEpGGSNXlehyYZmYVwMPu/sckXFPk\nS2KryAOUlgb74pFsVS0onds/GCZ8/oHnGfLVIdwx8o5mk2hJw2RlQUEBbNgA3bqFHY2IJFoykq3T\n3X2tmXUiSLoWuPvbsQeMHz/+8+eFhYUUFhYmISyR+JgybQq5/XPpckwXVmxdwY59O+i0qhOtW7QO\nO7SEKi4upri4OOww0lbVUKKSLZHMl/Bky93XRh83mNlfgUFArcmWSCIko4r8qm2rWL9zPb1ye5Fn\nmT9Hq/ovRrfddlt4waQhLdkj0nwkNNkys1ZAlrvvMLPWwLmAWmSJq/pMfE9EFfmqeVola0tYsm4J\ne/ftpXdub5gfrHMociBaskek+Uh0z1YX4K9m5tFrPeXuUxN8TWlGqk98nz59Uq0T3+NZRT4SiXDV\nr66ipP0mdufvojxnDxfvuJieXXpqQWmpF92RKNJ8JDTZcvelwAmJvIY0b4mc+H4gDz3+EJGyZVir\nw9nbch85nxm5B+dSdENRg86TzHIUUn9m9ggwAihx9+Oi+04B/gDkAmXAGHd/L/rercDlQDlwfX1+\nqVStLZHmQxXkRRph7txFVLQ5mPK2u+iwbwjZFUcyd+6iBp2jqlduxoxBzJgxiKuvnsScOXMSFLE0\n0GPA8Gr7JgK/cvcTgXHAnQBm1g+4DOgLnA88YGZ13hauni2R5kPJlqS1UaOGU1HxOKWlb1Ba+kZ0\n4nv1/yPj76AjD2afl5K3vAvla7fjn1TQ+7BjGnSO2F65goKzyM4e/Xkvl4Qresd09ULMa4H20ecH\nAaujzy8EnnH3cndfBiwmuBHogJRsiTQfWqhN0lpTJr43dghv7rq5tDs9m5739WPXlsMB6JRfwU9+\n8oNGfANJI7cA/zaz3wEGnBbd3w2YGXPc6ui+A9IwokjzoWRL0l5jJr43ZGJ9bHX4YwYdw1s73uI3\nX/8NG/tvjEnWrmlwDMkoRyFx9Qhwrbv/zcwuBR4FzmnoSapK3WzdCitXFgKF8YtQROIuHjUFE75c\nT50BaLmeZqshPUvxnkheVHQ7M2YMiplY/wZDh85mwoRbv3RcbHX47ZXbWVqylAdGPcB5p54Xl/gy\naYJ8pi3XY2Y9gL/HTJDf5u7tYt7f4u4HmdktgLv7hOj+V4Fx7v5ODef8vL3bswfatw8e657hJSKp\nIlWX6xHZT0N6lhpybLxVVYfP75nP0tKlHMmRzJ0590vJVlPii2c5Cok7i25VFpvZme7+LzM7i2Bu\nFsDLwFNmNolg+LA3MLuuk+fnQ+vWwZI9nTvHO3QRSSVKtiQUDSnZkIjyDg0ZwttZuZOlpUs5puAY\n9u7Ym5T4JFxm9jTB+F5HM1tBcPfhfxHcadgC2BN9jbvPN7PngPl8URKiXt31AwfCe+/BBRck4EuI\nSMpQsiXNUn0n1g88bSAPPfMQPenJ3h17KYuUqTp8M+Du36nlrVNrOf524PaGXmfwYJg1S8mWSKZT\nsiWhqG/P0pw5cygpWceyZb9m587FtG7dJ24Tyesawlu5dSVTt0zlrkvuYvG7wYhRTdXhNdFdGmvw\nYPj978OOQkQSTRPkJTR1TQ6PnQu1Y8dOSkvv5dxz+3L99ZcnfIhuzfY1TJo5iW8f+20GHlL3tTJp\nontjZdoE+USo3t5t2AB9+sCmTZClqociaUET5CWt1NWz9OWin9CmTWu6dp2d8ESmZEcJ9866l2/2\n/2a9Ei3QRHdpnE6doKAAFiwALacpkrmUbEmzF1tH67Shp/GPTf9g5DEjGdStziLgIk02ZEgwb0vJ\nlkjmUse1pKy6luKZM2cORUW3U1R0e6PXFKyqozWzbCb/2vcvfvTkj+iX04/TDjut7g+LxEHVJHkR\nyVzq2ZKUdaA7BuNVe2vKtCns7LaT0q3b2JC1jq6tO7H2w7Uq6i1JM3gwPPhg2FGISCIp2ZKUVttc\nqHjUtopEIrz8ysu8v2keNqgDeXYYK5avZnXf1XV/WCROjjsOli6FbdugXbu6jxeR9KNkS1JWIu/w\nqxo+XNFpDfsOLyP70620phcVS7uzs4tG1yV5cnPhxBPh3XfhrLPCjkZEEkH/q0hKqhomnDFjEDNm\nDOLqqyd9aV5WXfO56jJl2hSsn1Heq4zWLXqRn3MoWYvzOLzVlRQUdEnEVxKpleZtiWQ2JVuSkr5c\n9uEssrNHf97LBV/M5xo6dDZDh85u8Hytfb6PJeVL6NGlO/l79pCb3YH83O7k5RU3KGkTiQclWyKZ\nTcOIknYaM7wYiUR45P8eYcHiBfTp1YetvbaS/2k+XXK6kH9IPivnr6PwK635yU+uqbG4anMvWCqJ\nNWQIXH01uIOpLKxIxlEFeYmreCUm1e82rKh4nAcfDJbAqWn/ga4TiUQYM24Mn+R+grUzduzcQfec\n7txx0R0sXLYQgBHn7L8Mz4HiUMK1P1WQr9uB2rvDD4fp06FXryQHJSINogryEqp4lWOA2ss+FBXd\n3uC7EKdMm8KGThvI75bPnvw95Jfmk7Uoi4XLFlJ0Q9EB44jHXY8i9TF4MMycqWRLJBMp2ZK4iXdi\nEs8lcBxnu28nz/LIL8vHUAeMpJaqeVvf+17YkYhIvCnZkrQyatRwpk+fRGlp8Dq4C3HsfsfFLsHT\np2cfyueXU766nBbegj0L9tCjQw9GnDMibtcTaarBg+Hpp8OOQkQSQXO2JG6SNb+prnlhVTW0cvvn\n4u4sWbmE0044jZaftmTh4oX07dWXK35wRY1ztBpzPQlozlbdDtTe7d4dLEq9YQO0apXkwESk3hrT\n1inZkrhKhcRkwj0TmFk2k85Hd2bhxoVsXruZb7X8Fr8Y+4ukx9KcKNmqW13t3aBBcPfdcMYZSQxK\nRBpEE+QldPGcZ9UU7s7iTYspqyyjZ05Psi077JBE6jRkSDBvS8mWSGZRsiUZo2qe1rq161i4cSE5\n5TkcmXMkFfMrGHFT3fOzRMI2eDC8+GLYUYhIvGkYUTJC1TytnH45rK5YzcalG/l6x6/TrUu3Wmto\nSXxpGLFudbV3S5fCaafBmjUqbiqSqjSMKM1SJBLh5+N+zvLK5bTPaU9WhywGZA+gW4tuddbREkkl\nRxwBBx0E770Hp5wSdjQiEi9aG1HSWlWP1vJOy1l72Frmz59Pt73dNEdL0tbFF8NLL4UdhYjEk5It\nSVtVPVpLWy4lv08+la0qyc/JZ9GbiyiLlNWrjpZIqvnGN4J5W5pdIZI5lGxJWqrq0VpRuYJ1uev4\nrPQzTj7kZDpaRw7POpyJN03UPC1pNDN7xMxKzOyjavuvNbMFZjbPzO6I2X+rmS2OvnduU6590kmw\nZw/Mn9+Us4hIKlGyJWkntker5cCWVOyuIH9TPmsXrOWI3Udw1213KdGSpnoMGB67w8wKga8Dx7r7\nscBd0f19gcuAvsD5wANmjZ/ebhb0bmkoUSRzKNmStBLbo1WSU8KiXYs4qedJFKwvoMeGHurRkrhw\n97eBzdV2XwPc4e7l0WOiizgxEnjG3cvdfRmwGBjUlOt/4xvw17825Qwikkp0N6KEpjHV5qdMm0Ju\n/1w653Zm5fyVtCxvybrN6zii3RFKtCTRjgKGmtn/ALuBn7v7+0A3YGbMcauj+xrt9NNh1aqgFMQR\nRzTlTCKSCpRsSSiqr6M4ffqkA66jWFWwtPjtYtYfuZ6KIys489gzWTJ9STBH6zYlWpJwOUAHdx9s\nZqcAzwNHNvQk48eP//x5YWEhhYWF+x2TnQ0jRwa9Wzfe2Oh4RSQOiouLKS4ubtI5VNRUQlFUdDsz\nZgyioOAsAEpL32Do0NlMmHDrfsfefffd/OZPt2NHGwd1a8vaLWs5seuJdOrSibJImXq0UkSmFTU1\nsx7A3939uOjrV4AJ7v6v6OvFwGDgSgB3vyO6/1VgnLu/U8M5693e/fOf8Nvfwttvx+PbiEi8NKat\n05wtSWl33303N99+K1ta7WBr+30sy1nBYe0Oo92ydgzJHaJESxLJoluVvwHDAMzsKKCFu28EXgZG\nmVkLMzsC6A3MburFhw2DSATWrm3qmUQkbEq2JBSjRg2nouJxSkvfoLT0DSoqHmfUqC/d/MV9991H\n0T23UHFcOX5MOZW+E9vYhtK1myk8o5CiG4qUaElCmNnTwH+Ao8xshZn9CHgUONLM5gFPAz8AcPf5\nwHPAfOAVYEw8uuvz8uCCC2Dy5KaeSUTCpmFECc2BJshPnjyZ79zwHXafvBvv4pANbMjFFmVz0K42\nvPV8sRKtFJNpw4iJ0ND27oUX4OGHYerUBAYlIg3SmLZOyZaknEgkwsU/vJhlOcvgBCjrVBbchL8g\ni+yPcph46+3cqFnDKUfJVt0a2t7t2AGHHgrLlsHBBycuLhGpP83ZkrRXVUdri2+h8tBKyveUk7sh\nFysxchZnK9GSZqVNm2Du1pQpYUciIk2h0g+SMiKRCFeNvYpFGxeR0z4Hb+Fk7ciCT6BlSUvuuOkO\nrr322lo/35i6XSKprmqtxB/8IOxIRKSxNIwoKSESiTBm3BjeL3uffQfto5JKWi5sSe6OXAryC7jz\n13cycuTIWj9fvW7Xtm2/45RT+tClS1clXkmSisOIZtYLWOXue6PL7RwH/Nndt4QUT4Pbu61boUcP\nWLQIOndOUGAiUm8aRpS0NWXaFFa2WYn3goruFWTlZ0EL6Ny+M3994q8HTLQAnn32NbKzR1NQcBa5\nuR1YvjyXqVMHMmPGIK6+ehJz5sxJzheRVPMiUGFmvYGHgcMI7iRMG+3bw0UXwZNPhh2JiDSWki1J\nCatXr2b1hg3syXWy93WgcgvkVuRywdALGnzX4Zo1rwE/oFWr0ygoOIvs7NGfDy9Ks1MZXcvwYuA+\nd78JOCTkmBrsiivgT38CDQKIpCclWxKKOXPmcPbZX6dDjwJ6DOjBv995D1q1JGtJHrYsF/u4Nbkb\n89m6vpKiotvr7JmKrdu1a9cyYAOHHqoxF6HMzL4N/BCommaeG2I8jXLGGVBZCTNn1n2siKSehM/Z\nMrPzgHsIErtH3H1Ctfc1Z6uZmTNnDl/72g9YlzsfBkVraJVBx2XDaLf7eHbsWUDl3n2wbwd9+vwP\nABUVjx9w7cSq8z777GusW7ea999fQ9u2P6n3Z6XpUnTOVj/gamCmu/8lWuH9surtUBLjaXR7N3Ei\nfPIJPPponIMSkQZJuTpbZpYFLALOAtYA7wLfcvdPYo5RstXMXHHFDTz61wfgrDI4OgvMYYWT+4/W\nnNQnKJe9bNmvKSi4hZ49LwQOvHZiTXRnYvKlYrIVy8w6AIe5+0chxtDo9m7dOjjmGFixAtq1i3Ng\nIlJvjWnrEl36YRCw2N2XA5jZM8BI4JMDfkrSQmMTmk9XfgK50R6tbKDSYCfk5+UwdGiwpNzRR5/C\nwoWtGx3bwIEDlWAJZlYMXEjQ1r0PrDezf7t72hVr69oVvvpVePZZuPLKsKMRkYZI9JytbsDKmNer\novskzVWVWpgxY1CD7/g7/vijsINzYA+w0uETh9kwqO9XmDDhViZMuJXrrvthnWsnitRDe3ffBnyD\noOTDqcDZIcfUaFdcAY88EnYUItJQKVHUdPz48Z8/LywspLCwMLRYpH5iSy0AlJYG++rTm/T9732f\n5zY+x/ollfhMoMzomtebiRNv+/yYgQMH8uCDY2N6zjTnKtUUFxdTXFwcdhh1yTGzQ4DLgF+GHUxT\nnXceXHUVfPwxDBgQdjQiUl+JTrZWA4fHvO4e3fclscmWZLZdZbuYunUqt3zrFj6b9hkftVhM78OO\n4Sc/+cF+yZSGAlNb9V+MbrvtttoPDs//A14D/u3u75rZkcDikGNqtJwcGD066N2aNCnsaESkvhI9\nQT4bWEgwQX4tMBv4trsviDlGE+TTUPWK7fW5429P+R4mzZxEr4N78c1+38QsZedSSyOk+gT5VBCP\n9u6zz2DwYFi1CvLy4hSYiNRbyt2NCJ+XfriXL0o/3FHtfSVbaaohE+T3lu/l3nfupXu77nx7wLeV\naGWgVEy2zKw7cB9wenTXW8D17r4qpHji0t4NGwZXXw2XXRaHoESkQVIy2aozACVbGW9fxT7un30/\nBa0K+P5x31eilaFSNNmaRrA8T9ViN98Dvuvu54QUT1zau2eegQcfhNSfMieSeZRsZbh0rB1VVlHG\nA+8+QNu8tow+YTRZpkULMlWKJlsfuvsJde1LYjxxae/KyqBXL3jpJTj55DgEJiL1poWoM1hTSi2E\npbyynIfff5iWuS2VaElYNprZ98wsO7p9D9gYdlBNlZsLN9wAd90VdiQiUh8pUfpB6taUUgvJFIlE\nmDJtCpVeye5euznkkEO44sQrlGhJWC4nmLM1CXDgP8DoMAOKlx//GH77W1i2DHr2DDsaETkQ/Q8o\ncROJRLj5zpv5z77/8Nye53jx9Rc5Lf80srOyww5Nmil3X+7uF7p7J3fv7O4XAZeEHVc8tGsXJFwq\nASGS+pRspYlRo4andEX1yZMnc/EPL2b2ztl8lvUZLTu3pFf3Xrz6xqthhyZSXdot1VOb666DJ5+E\nTZvCjkREDkTDiGkizIrqdU3Mnzx5MleOv5JdbXexp+MetqzYQmFeoYYOJVWl1CT+pujWDS68EB56\nCG6t3xrtIhIC3Y0oB1Sf4qXnX3o+H3T8gMrOlWxZvgUq4aC9B3FK61OYeNNE+vfvH07wklSpeDdi\nTcxshbsfXveRCbl23Nu7efNg+HBYulRFTkWSQXcjStzFTswvKDiL7OzRn/dyRSIRJtwzgU+XfcrO\n8p3QDg49/FDyVuRx0NKDlGhJaMxsu5ltq2HbDhxaj88/YmYlZvZRDe/9zMwqzezgmH23mtliM1tg\nZufG+esc0LHHwnHHwVNPJfOqItIQSrakUWInw9tZxq6Nu6h4t4K9G/bScntL7vz1nUq0JDTu3tbd\n29WwtXX3+kyfeAzYb1JktCL9OcDymH19CRa67gucDzxgSa7ce9NNQRmIyspkXlVE6kvJlhxQ9Yn5\nO3bcw+7Ktfx83M/Zc/ge9h66l3a92nFirxPpuKQjJ2w8gT+O/yMjR45s1PXmzJlDUdHtFBXdnvJ1\nxCRzufvbwOYa3poE3FRt30jgGXcvd/dlBAtdD0pshF82bFgwhPjKK8m8qojUlybIywHFTswvLS1h\n4abNrOiczYrPVrB6zWq6dezGoB6D2Lx1M0O+N4SiG4oafa3q88OmT59U5+LWIsliZhcCK919XrWO\nq27AzJjXq6P7khgb/OIXMH48fO1rwWsRSR1KtqROAwcOZODAgUy4ZwIbywrockwXNuVsYtncZZQv\nLGfzvs2URcoYcdOIJl0nXQq3SvNjZi2BXxAMITbJ+PHjP39eWFhIYWFhU08JwCWXwB13wIsvwqWX\nxuWUIgIUFxdT3MSFSJVsSYOt2b6G7S23M7DjQNosb8OQo4Yw4qYRmqMlmawX0BOYG52P1R2YY2aD\nCHqyYu9u7B7dV6PYZCuesrLgf/4Hrr8eLroIctS6i8RF9V+KbrvttgafQz+OUqeqJXhKSkqY/8l8\nNvTcRIceGWcDAAAgAElEQVStB9N6c2vu+s1d9O/f//O5VtD4RbJHjRrO9OmTKC0NXgeFW8fG86uI\nNIRFN9z9Y6Dr52+YLQUGuvtmM3sZeMrM7iYYPuwNzA4hXs49Fw45BJ54Aq64IowIRKQmqrPVzNVV\nsLTqrsPc/rms2LqCjz/5hIPmDSPfD6F93jqeeCLI8OuqxRWveCR1pUudrfows6eBQqAjUAKMc/fH\nYt5fApzs7puir28FrgDKgOvdfWot5014ezdrFnzzm7B4MeTnJ/RSIs1SY9o6JVvNWF0FSyORCD8f\n93OWd1rOoQMP5aOVEfZ93J1Dll5K7+5FlJa+wdChwS/wM2YMiplrFeyfMEElrZuTTEq2EiVZ7d3I\nkXDmmXBjxixMJJI6VNRUGuRABUsnT57MJWMu4f0N77O2ci2zl86m7e72ZJW1CDVmEanbb38LEybA\ntm1hRyIioGRLajB58mR+fNOPWd15NeX9y9m+Zzs5G3LwDfuo/GQZLSoLvrQYdqovki3S3AwYECzh\n87vfhR2JiICGEZu1moYRb7rpQn59/69Z7aspO6qMioMraLe9Hdnzszmp00lcPupy3nvvU+DLc6o0\n10o0jFi3ZLZ3S5fCySfDggXQuXNSLinSLGjOljRYbJJ08sm9efTZR3l/w/uUH1HO9uztZJNN7rZc\nuq3vxosPvKjyDlIrJVt1S3Z7d8MNsGMH/OlPSbukSMZTsiWNVnXX4dKWS1lXsY5te7fR1tpSvqSc\n/HX5/OnOPzV6CR5pHpRs1S3Z7d3WrdCvHzz3HJx+etIuK5LRNEFeGqXqrsOlLZdScEwB5QXltM1v\nS+7SXLpZNyVaImmqfftg3tY110BZWdjRiDRf6tlq5iZPnkzRxCI27dlE2dFlVPSsYEDnAWxZuIUe\nG3pw12131WvoUHO2RD1bdQujvXMPip2edx787GdJvbRIRtIwYjMRr8QmEolwyZhL2HrMVipzKykt\nLSWnPJc2Fe3oy1E89JuH6p1oxauoqaQvJVt1C6u9W7wYhgyBDz6Aww5L+uVFMoqGEZuBqsRmxoxB\nzJgxiKuvnsScOXMada4p06aQ1S+L7MOy2XPoHiw3h8r3W1IxuydbP+3A3r1763WeA9XrEpHw9ekD\nP/1pMGFeRJJPyVaaiXdi07FTR7bt2kb5lkpsd2taVHThhD6P0abNDUqYRDLILbfARx/BK6+EHYlI\n86Nkqxk7fejprNm+hj45fchf3JKc99rQr8sE2rZtWHkHFTUVSX35+fCHPwQ9XLt2hR2NSPOiOVtp\npinzoyKRCFOmTQHgK2d+hX9s+ge9s3qzfu56Vq9ezfRXl9KmzQ0NPm9VXJog37xpzlbdUqG9+973\ngrsU//CHUMMQSVuaIN9MNCaxqaqjlds/lzIvY9GaRRRdWMSPh/24SecVqaJkq26p0N5t2QInnAD3\n3w8jRoQaikhaUrIlNaqqo7W803KOPvVoVuxdQYvNLbgo/yKKbiiq9XNKvqQhlGzVLVXau7ffhm9+\nM7g7sWvXsKMRSS+6G1H2U9WjtaJyBRt9I28ufJOW3pIu2V0O+Ll43vUoIqnljDPgyith9GiorAw7\nGpHMp2Qrw02ZNoXc/rkcNewo9pTvIWtXFhvnbqQsUsaIc2ofQ1A5B5HM9utfB0OK998fdiQimS8n\n7AAkcSKRCMVvF7PMl5EzOId+/fqx9e2t9MjqwcTbJmpRaZFmLDcXnnoKBg+Gr34Vjj027IhEMpeS\nrQwViUS46ldXsaLVGtbkrySr2OiS050eWYfWawmeUaOGM336JEpLg9dBOYexSYhcRJKlVy+46y74\n9rfhnXegdeuwIxLJTBpGzFAPPf4QH5cvpbRjCyq9G2U7WrD5nZb1rgw/cOBAHnxwLEOHzmbo0Nla\nfkckQ/3gB3DyyZq/JZJIuhsxQ31l+Lm8330x3qITFRuOxjcuod2ithx1+E0MHTqbCRNuDTtEyTC6\nG7Fuqdre7dkDw4bB8OEwblzY0Yiktsa0dRpGzEDlleWUH1MGiwyraIuXlcLCClrnHBN2aCKSgvLz\n4aWX4NRToV+/oCyEiMSPkq00VlMdrIrKCv74/h/56plD2P5ma0q3dWTT5m3klLfkoN7HaO6ViNSo\na1f429/g3HOhd2848cSwIxLJHBpGTFM1LdvzwP9ezwd8wN6KvVx98tV89OFHPPvsa5SUrMO9gq5d\nu6k4qSSMhhHrlg7t3QsvwI03wuzZKngqUhNVkG9GiopuZ8aMQRQUnAXAhtJpHPzVhxh+4VcYc8oY\ncrNzG31uVY6XxsikZMvMHgFGACXuflx030Tg68Be4DPgR+6+LfrercDlQDlwvbtPreW8adHe3XYb\n/POf8Prr0KZN2NGIpBZVkG9mdu1axqerJrB41R2sKXiG3baLa065psmJlirHi/AYMLzavqlAf3c/\nAVgM3ApgZv2Ay4C+wPnAA2aW1knnf/839O8PI0fC7t1hRyOS/pRspaFIJMKqDR+zsHQsq9v/hVXH\nPcqGrOe58rjv0SK7RZPOrcrxIuDubwObq+173d2riiPMArpHn18IPOPu5e6+jCARG5SsWBPBDB5+\nGLp0gUsugXpUixGRA1CylWaq1jr8IO8DWg/PpazDIloWbOOUASewcu3KsMMTaS4uB16JPu8GxP7w\nrY7uS2vZ2fDEE8Gdit/5DpSXhx2RSPrS3YgprPrcqby8PH4+7uesqFxB1kFZZB+aTV6XFvTY3J2D\nux4cl2uqcrzIgZnZL4Eyd/9LYz4/fvz4z58XFhZSWFgYn8ASIDcX/vIXuPhi+OEP4c9/DpIwkeak\nuLiY4uLiJp1DE+RTVPW7DXfsuIf2vTezqesmNlduZs/mPZQfUk6rva0oWF9Az3Y9mXjTgdc7rO/E\nd02Ql8bIpAnyAGbWA/h71QT56L7RwJXAMHffG913C+DuPiH6+lVgnLu/U8M507K9270bvvY16NED\n/vhHyNGv6dKM6W7EDFL9bsO5C68k77R/c+KwAUyPTGd3xW46LexE3rY8Lhh8AVf84Io6E63qpSK0\nBI/EUwYmWz0Jkq1jo6/PA34HDHX3jTHH9QOeAk4lGD6cBvSpqWFL5/Zux45g/lZeHjzzDLRqFXZE\nIuFQBfkMtX17hG0755K9tZSS3SV0OrQTFQsrODLvSO56oO5FpeHLE98BSkuDfUq2RPZnZk8DhUBH\nM1sBjAN+AbQApkVvNpzl7mPcfb6ZPQfMB8qAMWmbUR1Amzbw97/DFVfA2WcHzzt2DDsqkfSQsAny\nZjbOzFaZ2Zzodl6irpWJRo0aTkXF46xY8QhzV19BZe/llO3bw3sfv0fBxgJ67+rNXbfVL9ESkYZx\n9++4+6Hunufuh7v7Y+7ex917uPvA6DYm5vjb3b23u/etrcZWJmjRIpg0/5WvwBlnwPLlYUckkh4S\nfTfi3TEN06sJvlZGGThwIDfddCHbKiaQ1XkJR519GIeeeSjd13Wn9fzWdc7Pqq4qeSstfYPS0jei\nE9+rlxESETmwrCyYMAGuuipIuObODTsikdSXsDlbZjYO2OHuv6vjuEzscW+yyZMnUzSxiE05m9jb\nYy/e0jlzwJlk78lmSO4Qim4oavA5NfFdEinT5mwlQqa1d88+Cz/9KdxzD3z3u2FHI5IcKTVBPpps\njQa2Au8BP3P3rTUcl1GNTzxEIhEuGXMJW4/ZSnmrcjZv3Ey7snYcsveQet11KBIGJVt1y8T2bu5c\nuPRSGD4cfve7YAK9SCZL+gR5M5sGdIndBTjwS+AB4P+5u5vZb4C7gStqOk861Z1JhinTppDVLwvr\nZlS2qKTAC6j8oJLDOx3eLBIt9cClh3jUnpH0d/zx8O67MHo0nHkmPP88HHZY2FGJpJaklH6oqV5N\nzHsZ95teU024ZwIvbXqJBeULaNeqHRUrK2j/cXtefPjFZpFoqURFelLPVt0yub1zhzvvhLvvhkcf\nhQsuCDsikcRIqYWozaxrzMtvAB8n6lqZpu+pfdmwYwMDcgbQbnU78j7Mo1+XU/jzn1/O+EWhtTaj\nSHoyg5tvDuZx/eQn8KMfwZYtYUclkhoSeTfiRDP7yMw+BM4EtOZLPSwsXciM7TP4w6g/MPLgkZzd\n5mza7jqWtWtHM2PGIK6+elLGJ1wikr7OPBPmzQuKnh57LLzySt2fEcl0qiCfQj7d9CkPvvcg/3XS\nf3FUx6OA/SvJl5a+wdChs5kw4dYwQ00YDSOmLw0j1q25tXdvvhkUQS0sDCbPHxyfJVxFQpVSw4jS\nMEs3L+XB9x7kihOv+DzRCtOcOXMoKrqdoqLbk9qTNnDgQB58cCxDh85m6NDZSrRE0tiwYUEvV9u2\n0Lcv3H8/lJeHHZVI8qlnKwWs2LqC37/ze0afMJoBnQd86b0wenrUuySNoZ6tujXn9m7ePBg7Ftau\nhUmT4Nxzw45IpHFSqs5WvQNoxo0PwKptq7h31r1897jvckLXE2o8JtmlEJrb0KXEh5KtujX39s4d\nXn4ZfvazoKfrt7+F4/a7R10ktWkh6jSzdvta7p11L6MGjKo10YJgaE29SiKS7sxg5Eg47zz4wx+C\nQqhDhsB//zecUHsTKJL2NGcrJOt3rueeWfdwab9LOfnQk8MO50u0jqKIJFJeHtx4I3z2WbCo9QUX\nBEnY+++HHZlIYmgYMUkikQhTpk0B4PShpzNl0xS+ftTXOf3w00OOrGaq4i4NpWHEujWX9q6hdu+G\nP/4RJk6EXr3g2mvhoosgR2MvkoI0ZytFRSIRbr7zZnL757LP97Fo7SJ+eeEv+dFXfxR2aCJxo2Sr\nbs2hvWuKsjL461/hvvtg2TK45hq48kro1CnsyES+oNIPKWrKtCnk9s/loN4Hsa79Orp06cL6uevD\nDktEJKXk5sJll8FbbwUT6T/7DPr0gW98AyZPhn37wo5QpHGUbCVJmZcxb/08urbpSqds/ZomInIg\nJ54IjzwCy5fD174WrLnYrRv89KcwcyZUVoYdoUj9KdlKgmFfHcbiNYvJ3ZxLizUtKIuUMeKcEWGH\nJSKS8tq3D6rQ/+tfMHs2dOkCP/4xHHYYjBkDr78eDD+KpDLN2UqA2Mnww746jNe2vka7Pe3YFdmF\nmTHinBH0798/5ChF4ktztuqWie1dWBYtCuZ3vfQSfPppUEbi3HPhnHOCHjCRRNEE+RQQOxm+witY\ntHoRV11wFWPPHYuZ/h+SzKVkq26Z1t6litWr4Z//hGnTgp6uQw4Jkq5hw+C006Bjx7AjlEyiZCsF\nTLhnAjPLZtLxqI58vP5jKkoruDT/Us4989wGlVJQ6QVJN0q26pZp7V0qqqiAOXNg6tRg6HHWrGDI\n8Ywzgu3UU6F3b8jSJBppJCVbKWDCPRP4975/s+HgDbTMaUnb9W3psaEHs/61ud5rDWptQklHSrbq\nlmntXTooL4ePPoK33w7ucnz3XdiyBQYOhJNOCrbjjw/uelRdL6kPLdeTAoafNZwn/vgEeRV5dM7u\nTPn8cnZmZ5GdPTpmrUF49tnXak2enn32tQYdLyIiNcvJCRKrgQPhuuuCfaWlQbX6996DZ5+FX/0K\n1qyBo4+GAQPg2GOD50cdFRRZbdEi3O8g6U/JVhyVV5bz1q63GHXuKFosbkGWZTHiphH8+c8vhx3a\nAWnIUuTLzOwRYARQ4u7HRfd1AJ4FegDLgMvcfWv0vVuBy4Fy4Hp3nxpG3FI/BQXBhPrhMauQ7dwJ\n8+fDxx8H24wZwST8FSuge/cg8TriiC9vPXtChw7Bmo8iB6JhxDipqKzgofcfIsuyuHLglWRnZX/+\nXkOHBZM5jKghS4mXTBpGNLMzgB3An2OSrQnARnefaGZFQAd3v8XM+gFPAacA3YHXgT41NWyZ0t41\nJ/v2wdKlsHhx8LhkSfC4dGlQA6ysLEjGDjss2Lp1g65dg0n6VY+dO0Pr1krKMoXmbIWk0iv505w/\nUVZRxlUnX0VO1v4dhg3tPUpWb1NR0e3MmDEoZsjyDYYOnc2ECbcm5HqSuTIp2QIwsx7A32OSrU+A\nM929xMy6AsXufoyZ3QK4u0+IHvdPYLy7v1PDOdO+vZMv274dVq2ClSuDbc0aWLsW1q0LHteuhfXr\nwT1YdqhTpyD56tgRDj74i8eDD4aDDvpia98+eFSSlno0ZysElV7J4x8+zu6y3Yw5ZQw5WTk1JkpV\nW3019HgRSbjO7l4C4O7rzKxzdH83YGbMcauj+6QZaNsW+vYNtgPZuTNIujZsCLZNm4Jt40ZYuDB4\nvmVLsG3d+sXzPXugTRto1y64Vrt2QQLWps0XW+vW0KrVF1vLll/e8vO/eKza8vKCxxYtgue5ubpD\nM5GUbDWBu/N/H/0fW/Zs4dpB15KbnbvfsNz06ZNSelhu1KjhTJ8+idLS4HVFxeOMGjU23KBE0kOj\nuqjGjx//+fPCwkIKCwvjFI6kstatv5jr1RDl5bBjB2zbFvSibdsWJG47dnz5cdeu4Pn69cHrPXtg\n9+5gq3q+Zw/s3Rs8Vm379n2x5eYGyVeLFl88z82tecvJ+eIxdsvO3v+xti0ra//nWVkH3sy+/Fh9\nX/X3q79X0wa1v5edDXl5xRQXFzfp71/DiI3k7vzl47+wettqrjv1OvJy8oD0HJbTBHmJh2YwjLgA\nKIwZRpzu7n1rGEZ8FRinYURJJ+7B/LO9e4PHffu+eNy3L0j6ysqCrep5efmXt7KyoM5ZRcUX+6pe\nV98qK/d/HbtVVAQx1bSv+v7YfbGPscdVva6+VX332racHPj737/8Z6VhxCRxd56f/zwrtq7ghsE3\nfJ5opSsNWYrUyKJblZeB0cAE4IfA5Jj9T5nZJILhw97A7OSFKdJ0Zl/0akn8KdlqIHfnr5/8lcUb\nFzN2yFjyc/K/9L6G5UTSn5k9DRQCHc1sBTAOuAN43swuB5YDlwG4+3wzew6YD5QBY9R9JSKxNIzY\nQH9f+Hc+WPcBPxvyM1q3aF3jMRqWk+Yo04YREyHd2jsR2Z9KPyTYPxf/k3dWv8PPhvyMtnltww5H\nJKUo2apbOrV3IlKzxrR1utGznqZ9No3/rPwPYwePVaIlIiIi9aZkqx6mL51O8bJibhxyI+3z24cd\njoiIiKQRJVt1eGv5W0z9bCo3DrmRDi07hB2OiIiIpBklWwcwc+VM/rH4H4wdMpaOrTqGHY6IiIik\nISVbtXh39bv87ZO/ccPgG+jcunPdHxARERGpgZKtGsxZO4fnIs9x/eDr6dqma9jhiIiISBpTslXN\nRyUf8fS8p7nu1Os4tO2hYYcjIiIiaU7JVozI+gh/nvtnfjropxzW/rCwwxEREZEMoGQr6pPST3js\nw8e45uRr6HlQz7DDERERkQyhCvJRCzYsIDsrm6M6HhV2KCJpSRXk65Yq7Z2INJ6W6xGR0CjZqpva\nO5H0p+V6RERERFKMki0RERGRBFKyJSIiIpJASrZEREREEkjJloiIiEgCKdkSERERSSAlWyIiIiIJ\npGRLREREJIGUbImIiIgkkJItERERkQRqUrJlZpea2cdmVmFmA6u9d6uZLTazBWZ2btPCFBFJDdG2\nLWJmH5nZU2bWwsw6mNlUM1toZq+ZWfuw4xSR1NHUnq15wMXAv2J3mllf4DKgL3A+8ICZpfyaacXF\nxWGH8LlUiSVV4gDFUpNUiaO5MLMewJXAie5+HJADfBu4BXjd3Y8G3gRuDS/KxMmEf2/p/h3SPX7I\njO/QUE1Kttx9obsvBqonUiOBZ9y93N2XAYuBQU25VjKk0j+AVIklVeIAxVKTVImjGdkG7ANam1kO\n0BJYTdDmPRE95gngonDCS6xM+PeW7t8h3eOHzPgODZWoOVvdgJUxr1dH94mIpC133wz8DlhB0K5t\ndffXgS7uXhI9Zh3QObwoRSTV5NR1gJlNA7rE7gIc+KW7/z1RgYmIpBozOxIYC/QAtgLPm9l3CdrE\nWNVfi0gzZu5NbxPMbDrwM3efE319C+DuPiH6+lVgnLu/U8Nn1SiJZAh3T/m5mU1hZpcB57j7ldHX\n3wcGA8OAQncvMbOuwHR371vD59XeiWSAhrZ1dfZsNUDshV8GnjKzSQTDh72B2TV9KNMbZxHJKAuB\nX5tZPrAXOAt4F9gBjAYmAD8EJtf0YbV3Is1Tk5ItM7sIuA8oAKaY2Yfufr67zzez54D5QBkwxuPR\nhSYiEiJ3n2tmfwbeByqAD4CHgbbAc2Z2ObCc4G5sEREgTsOIIiIiIlKzlKkgb2bXRgugzjOzO1Ig\nnp+ZWaWZHRzS9SdG/zw+NLMXzaxdCDGcZ2afmNkiMytK9vVj4uhuZm9GC0nOM7PrwoolGk+Wmc0x\ns5dDjqO9mT0f/XcSMbNTQ4pjvyKfYcSRylLlZ6khzOwRMysxs49i9qVN8dba2o00+w55ZvaOmX0Q\n/R7/E92fNt8B9m8z0zD+ZWY2N/r3MDu6r0HfISWSLTMrBL4OHOvuxwJ3hRxPd+AcguGAsEwF+rv7\nCQR1ypJaJNHMsoD7geFAf+DbZnZMMmOIUQ7c6O79gSHAT0KMBeB6giHysN0LvBKdiH08sCDZAdRS\n5PNbyY4jlaXYz1JDPEYQc6x0Kt5aW7uRNt/B3fcCX3X3E4HjgGFmdjpp9B2iqreZ6RZ/JcENMCe6\ne1XN0AZ9h5RItoBrgDvcvRzA3UtDjmcScFOYAbj76+5eGX05C+ie5BAGAYvdfbm7lwHPEBRuTDp3\nX+fuH0af7yBIKkKp2xZNxC8A/hTG9WPiaAd8xd0fA4gWEN4WQijVi3y2AtaEEEcqS5mfpYZw97eB\nzdV2p03x1lraje6k0XcAcPdd0ad5BP9nbyaNvkMtbWbaxB9l7J8vNeg7pEqydRQw1Mxmmdl0Mzs5\nrEDM7EJgpbvPCyuGGlwO/DPJ16xemHYVKVCY1sx6AicA+5URSZKqRDzsyY5HAKVm9li0e/5hM2uZ\n7CBqKPK5JVrkU76Qkj9LjdQ5HYu3xrQbs0izArTRIbgPgHVAsbvPJ72+Q01tZjrFD0Hs08zsXTP7\ncXRfg75DPEs/HJDVXhz1V9E4Orj7YDM7BXgOODKkWH5BMIQY+16y4/i8YKyZ/RIoc/enExVHujCz\nNsALwPXR31STff2vASXu/mF06DvM2/hzgIHAT9z9PTO7h6Bbe1wyg7D9i3y+YGbf0b/XZiPsXzrq\nVL3dsP1rnaX0d4iOcJwY7c1+Ldr2pMV3qKHNrE1Kxh/jdHdfa2adgKlmtpAG/h0kLdly93Nqe8/M\nrgZeih73bnRiekd335jMWMxsANATmGtmRtDl/L6ZDXL39cmKIyae0QTdr8Pife16WA0cHvO6e3Rf\nKKJDVC8AT7p7jTWMkuB04EIzu4BgTby2ZvZnd/9BCLGsIuiBfS/6+gUgjInXJwP/dvdNAGb2EnAa\noGTrCyn1s9REJWbWJaZ4a9zbxXiqpd1Iq+9Qxd23mdkrBD9z6fIdamoznwTWpUn8ALj72ujjBjP7\nG8HUgAb9HaTKMOLfiCYUZnYUkJuoROtA3P1jd+/q7ke6+xEE/6GdmIhEqy5mdh5B1+uF0UmSyfYu\n0NvMekTvLvsWQbHasDwKzHf3e8MKwN1/4e6Hu/uRBH8eb4aUaBHtvl4Z/XmBoLhmGJP2FwKDzSw/\n+gvKWYQwUT/FpdrPUkMY+xesHh19Xmvx1hRSU7uRNt/BzAqq7nKLThM4h6C2W1p8h1razO8DfycN\n4gcws1bR3lHMrDVwLjCPBv4dJK1nqw6PAY+a2TyCqsyh/AdWAye8oaL7gBYE48QAs9x9TLIu7u4V\nZvZTgrsis4BH3D2U/0Sjd998F5gXnbvgwC/c/dUw4kkh1xGs1JALLAF+lOwADlDkU6JS6WepIczs\naaAQ6GhmKwiGqO8gWA8y5Yu31tZuEFT5T5cCtIcAT0R/kcki6KF7I/p90uU71OQO0if+LsBfo8PP\nOcBT7j7VzN6jAd9BRU1FREREEihVhhFFREREMpKSLREREZEEUrIlIiIikkBKtkREREQSSMmWiIiI\nSAIp2RIRERFJICVbIiKStszsYDP7ILpG6FozWxXzul61JM3sETPrU8cxY8zs2/GJusbzXxxTpFgy\njOpsiYhIRjCz/wZ2uPvdNbxnnsL/4UWXsXkhxOXIJIHUsyUiIpni8xU/zKyXmUXM7P/M7GOgq5k9\nZGazzWyemf0q5ti3zOw4M8s2s81mdruZfWhm/zazgugx/5+ZXRdz/O1m9o6ZLTCzwdH9rczsBTP7\n2MyeN7N3zey4/YI0uzMa24fR85xBsA7u3dEeucPNrLeZvRo9R7GZ9Y5+9kkze8DM3jOzT6JLu2Fm\nA6LfbU70vD0T9qcsDZYqy/WIiIjE29HA99z9AwAzK3L3LWaWDUw3sxfc/ZNqn2kPTHf3W83sd8Dl\nwMSaTu7up5rZ1wmWMjofuBZY6+6XRpOs96t/xsw6A+e7e//o63Yxi0w/7+4vR/e/CVzh7kvN7DTg\nD8Dw6Gm6u/vJ0WHH182sFzAGuNPdn48u4RXWUnNSAyVbIiKSqT6rSrSivhtdyy6HYN3BfkD1ZGuX\nu0+NPn8fOKOWc78Uc0yP6PMzCNb9w90/MrNIDZ/bBFSY2cPAK8CU6gdEF58eDLwYXRcRvjwS9Vz0\nGoui61b2Af4D/Drao/WSu39WS9wSAg0jiohIptpZ9SQ6DHcdUOjuxwOvAfk1fGZfzPMKau+U2FuP\nY/brXXL3cuBk4G/ARcA/avncBncf6O4nRrfjY09T7Vh39/+Lnm8v8Gp0aFJShJItERHJVLHJTjtg\nG7DDzA7hiyG5A32mof4NjAIws2OBvvud3KwN0N7dXwFuBE6IvrU9GiPuvgVYa2YXRT9j1eZ+fTO6\n/yigO7DYzI5w9yXu/nuC3rL95opJeDSMKCIimerzHiB3n2NmC4AFwHLg7ZqOq/a8zvNWcx/wRHRC\n/vzotrXaMe2Bl8wsjyCxGxvd/xfgITO7kaCH6lvAg2Y2HsgF/g/4KHrsajN7D2gNXOnu5Wb2nWhp\nig7DR7AAAAB7SURBVDJgNcE8MkkRKv0gIiISB9GJ9znuvjc6bPka0MfdK+N4jSeJmUgv6UE9WyIi\nIvHRBngjppjqf8Uz0YpSD0kaUs+WiIiISAJpgryIiIhIAinZEhEREUkgJVsiIiIiCaRkS0RERCSB\nlGyJiIiIJJCSLREREZEE+v8BDWQFu2iG7q0AAAAASUVORK5CYII=\n", + "text/plain": [ + "<matplotlib.figure.Figure at 0x7f804039d090>" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "#@test {\"output\": \"ignore\"}\n", + "import tensorflow as tf\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "\n", + "%matplotlib inline\n", + "\n", + "# Set up the data with a noisy linear relationship between X and Y.\n", + "num_examples = 50\n", + "X = np.array([np.linspace(-2, 4, num_examples), np.linspace(-6, 6, num_examples)])\n", + "X += np.random.randn(2, num_examples)\n", + "x, y = X\n", + "x_with_bias = np.array([(1., a) for a in x]).astype(np.float32)\n", + "\n", + "losses = []\n", + "training_steps = 50\n", + "learning_rate = 0.002\n", + "\n", + "with tf.Session() as sess:\n", + " # Set up all the tensors, variables, and operations.\n", + " input = tf.constant(x_with_bias)\n", + " target = tf.constant(np.transpose([y]).astype(np.float32))\n", + " weights = tf.Variable(tf.random_normal([2, 1], 0, 0.1))\n", + " \n", + " tf.initialize_all_variables().run()\n", + " \n", + " yhat = tf.matmul(input, weights)\n", + " yerror = tf.sub(yhat, target)\n", + " loss = tf.reduce_mean(tf.nn.l2_loss(yerror))\n", + " \n", + " update_weights = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)\n", + " \n", + " for _ in range(training_steps):\n", + " # Repeatedly run the operations, updating the TensorFlow variable.\n", + " update_weights.run()\n", + " losses.append(loss.eval())\n", + "\n", + " # Training is done, get the final values for the graphs\n", + " betas = weights.eval()\n", + " yhat = yhat.eval()\n", + "\n", + "# Show the fit and the loss over time.\n", + "fig, (ax1, ax2) = plt.subplots(1, 2)\n", + "plt.subplots_adjust(wspace=.3)\n", + "fig.set_size_inches(10, 4)\n", + "ax1.scatter(x, y, alpha=.7)\n", + "ax1.scatter(x, np.transpose(yhat)[0], c=\"g\", alpha=.6)\n", + "line_x_range = (-4, 6)\n", + "ax1.plot(line_x_range, [betas[0] + a * betas[1] for a in line_x_range], \"g\", alpha=0.6)\n", + "ax2.plot(range(0, training_steps), losses)\n", + "ax2.set_ylabel(\"Loss\")\n", + "ax2.set_xlabel(\"Training steps\")\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "vNtkU8h18rOv" + }, + "source": [ + "In the remainder of this notebook, we'll go through this example in more detail." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "r6rsv-q5gnn-" + }, + "source": [ + "## From the beginning\n", + "\n", + "Let's walk through exactly what this is doing from the beginning. We'll start with what the data looks like, then we'll look at this neural network, what is executed when, what gradient descent is doing, and how it all works together." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "UgtkJKqAjuDj" + }, + "source": [ + "## The data\n", + "\n", + "This is a toy data set here. We have 50 (x,y) data points. At first, the data is perfectly linear." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "cellView": "form", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 398, + "status": "ok", + "timestamp": 1446659128547, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "4896c353dcc58d9f", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "-uoBWol3klhA", + "outputId": "efef4adf-42de-4e6f-e0c3-07ddd3083d85" + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQMAAAEACAYAAAC3RRNlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEGpJREFUeJzt3XuMHeV5x/Hv4xinBgIN2Yq0WCFFJK1EUzC3otAWC4JB\nRHESkUq0qRKIBG2jAE1cZBKoQL0kBGSRtEr/IBdE2iDUEkhMLrJNqS1BBeF+dYA2KbcALYUWIVyu\nT/8447Ase9a7O+/MOTPn+5Esn4Nn552F9Y9553nP80ZmIklLRn0BksaDYSAJMAwkVQwDSYBhIKli\nGEgCCoVBRHw2Iu6NiLsi4lsRsazEeSW1p3YYRMS+wKnAysz8TWApcFLd80pq19IC53gWeBHYLSJe\nBXYFflbgvJJaVPvOIDOfAdYDDwOPAf+TmdfWPa+kdpWYJuwHfBrYF/gVYPeI+IO655XUrhLThEOB\nGzLzaYCIuAp4L3D59IMiwg9BSCOSmbGzY0pUE+4HjoiIX4iIAI4Btg25oNZ+nXfeeY7X0fH6/L2N\nYrz5KvHM4E7gm8CtwJ1AAJfUPa+kdpWYJpCZFwEXlTiXpNHo7QrEVatWOV5Hx+vz9zaK8eYrFjKn\nqDVQRLY1lqTXRATZ0gNEST1gGEgCDANJFcNAEmAYSKoYBpIAw0BSxTCQBBgGkiqGgSTAMJBUMQwk\nAYaBpIphIAkwDCRVDANJgGEgqVJqr8U9I+KfImJbtefib5U4r6T2FGmICnwZ+EFm/l5ELGWwxZqk\nDimxo9IewO9k5qUAmflyZj5b+8qkntm4cSOrV5/I6tUnsnHjxlFfzhvUbogaEQcy2CfhPuBA4Bbg\nzMzcPuM4G6JqYm3cuJEPf/jjbN/+RQCWL1/H1VdfxnHHHdf42G02RF0KHAx8JTMPBp4Hzi5wXqk3\n1q+/pAqCjwODUFi/frz2GirxzOBR4JHMvKV6fyWwbrYDzz///J+/XrVq1dj2j5dK2bhxI+vXX8Kt\nt94JrGllzC1btrBly5YFf12RfRMiYitwamY+EBHnAbtm5roZxzhN0ER5/dTgbuCrwN8A4zlNKFVN\nOAP4VkTsAvwEOKXQeaXOev3UYGCvvf6SQw45kLVr2wmChSi11+KdwGElziV12Y5pAcBTT/33jD99\nD4cc8lM2bfp2+xc2D6XuDKSJN7NisGzZn7Js2Vm8+OLgz5cvX8fatZeN8ArnZhhIhcycFrz4Iqxc\n+VWmpjYAjOXUYDrDQGrQ1NTeYzstmMkwkGra8Zzgqaee7NS0YCa3ZJdqmO05wQEHHMjU1NtYu/a0\nsZgWtF1alCbSbM8JpqY2dGZqMJ39DKRF2PGho8HKwn7wzkBaoNdPDX6VwZq7ga49J5jOMJAWqGsr\nC+fLMJBqG++VhfNlGEjzMH2Z8VFHHcz1169je9Wxo8tTg+ksLUo7MVtjknPOOZ2tW28DGJsS4jCW\nFqVCZj4j2L4dtm7tZvlwLpYWpSH6WD6ci3cG0iz6Wj6ci2EgzaKv5cO5GAZSpcuNSUowDCS635ik\nBMNAovuNSUooFgYRsYTBBiqPZmY7PaGlBnWpMUkJJe8MzmSwq9IeBc8pNaovjUlKKBIGEbECOAH4\na+AzJc4pNW225wQrV15aNSbp/7RgplJ3BhcDZwF7Fjqf1Lg+NSYpoXYYRMT7gScz846IWAUMXQPt\n9moaB6PY8qxNI9teLSI+D/wh8DKwHHgLcFVmfmzGcX5QSSM3LluetWm+H1Qq+qnFiDgKWDtbNcEw\n0DhYvfpENm9ew2srC/+Mvfb6TrWycLw/fbhYfmpRmpf+ryycr6JhkJlbga0lzynVNQmNSUqwuYl6\nreuNSUpwmiAxOY1JSrC5iXpp0hqTlOCdgXpnEhuTlGAYqHcmsTFJCYaBemHSG5OUYBio82xMUoZh\noM6zMUkZhoF6adIak5RgGKizbExSlisQ1UmzPSc44IADq8Yk/V9VuBCuQFSv2ZikPMNAnTF3+VB1\nGQbqBMuHzTMM1AmWD5tnGKizLB+WZRhorFk+bI+lRY0ty4dlWFpU51k+bFft5iYRsSIirouIeyPi\n7og4Y+dfJQ1nY5LRKHFn8DLwmWoTld2BWyNiU2b+uMC5NWFsTDI6tcMgM58AnqhePxcR24B9AMNA\nC2ZjktEp+swgIt4JHATcVPK8mmQ2JmlLsTCopghXAmdm5nOzHeNei5qN+xqUNbK9FgEiYinwPeCH\nmfnlIcdYWtQbuK9B89ouLX4DuG9YEEjDuK/B+ChRWjwS+ChwdETcHhG3RcTx9S9NfWb5cPyUqCbc\nALypwLVoQlg+HE+uQFTrLB+OJ8NArXBfg/FnGKhxNibpBsNAjbMxSTcYBhoJG5OMH8NAjbExSbfY\n3ESNsDHJ+LC5iUbKxiTdU3sFojSdKwu7yzsDFePKwm4zDFSMKwu7zTBQg1xZ2CWGgWqxMUl/WFrU\notmYpBssLapxNibpF0uLWjDLh/3knYEWxPJhfxkGWhDLh/1lGGinbEwyGYqEQdUA9UsMnkF8PTO/\nWOK8Gj0bk0yO2qXFiFgCPAAcA/wMuBk4aeZei5YWu2n16hPZvHkNr00LLqsak+wNWD7sgjZLi4cD\nD2bmQ9XAVwAfxL0We8vGJP1UIgz2AR6Z9v5RBgGhDrMxyeRp9QGiey12w2zPCVauvLRqTGLFYNyN\nbK/FiDgCOD8zj6/enw3kzIeIPjPojtmeExx7rCsLu6rNZwY3A/tHxL7A48BJwO8XOK9aNHf5UJOg\nxPZqr0TEp4BNvFZa3Fb7ytQay4cCP7UoLB/2nZ9aVC2WDyePYTDBLB9qOqcJE8p9DSaH0wTNyX0N\nNJPNTSaMjUk0jHcGE8TGJJqLYTBBbEyiuRgGE83GJHqNYdBz7mug+bK02GPuayCwtCjc10ALY2mx\nhywfajG8M+gZy4daLMOgZywfarEMgx5wXwOVYBh0nI1JVIph0HGzfeBo0JhkA4BTA82bYdBDNibR\nYhgGHWVjEpVWawViRFwIfAB4Afh34JTMfHbIsa5ALMTGJFqI+a5ArBsG7wOuy8xXI+ICBvslfHbI\nsYZBIe5roIWYbxjUWoGYmddm5qvV2xuBFXXOp7m5slBNKvnM4BPAFQXPp2lcWaim7TQMImIzsPf0\nfwQkcE5mXlMdcw7wUmZePte53Gtx8VxZqPka5V6LJwOnAkdn5gtzHOczgxp8TqDFauUjzBFxPHAW\n8LtzBYEWx8YkalPdasKDwDJgx4L4GzPzk0OO9c5gAWxMolJauTPIzHfV+XoNZ2MStc3mJmPG8qFG\nxeXIY8TyoUbJMBgjlg81SobBiNmYROPCMBghG5NonBgGI2RjEo0Tw2DM2JhEo2IYjICNSTSO3F6t\nZTYmUdvcXm1MzfacYGrKlYUaPcOgBXOXD6XxYBg0zPKhusIwaJjlQ3WFYTAClg81jgyDhlg+VNdY\nWmyA5UONE0uLI2T5UF1kc5OCbEyiLvPOoBAbk6jrioRBRKwFLgKmMvPpEufsGhuTqOtqh0FErACO\nBR6qfzl9YmMSdUuJO4OLGeydsKHAuTrFfQ3UJ3U3UVkDPJKZd0fstHLRKzPLh9dfv2NfA1cWqpvq\n7LV4LvA5BlOE6X82VJ/2WnRfA42r1vdajIjfAK4FnmcQAiuAx4DDM/M/Zzm+F4uOdkwNbr31Tp5+\n+s9x70ONu8YXHWXmPcDbpw34U+DgzHxmseccd5YP1Wcl1xkkO5kmdJ3lQ/VZsTDIzP1KnWucuK+B\nJoUrEOdgYxJNEsNgDjYm0SQxDBbIxiTqK8NgFjYm0SSyuckMNiZR39jcZJFsTKJJZXOTio1JNOm8\nM8CVhRIYBoArCyUwDIZwZaEmz8SGgY1JpNebyNLizPLh8uU7GpPcBmAJUb1iaXEONiaR3miiSouW\nD6XhJubOwPKhNLeJCQPLh9Lceh0GNiaR5q+3YWBjEmlhSuyodDrwSeBl4PuZeXbtqyrAxiTSwtTd\nRGUV8AHgPZn5ckRMFbmqhtiYRBqu7p3BnwAXZObLAJn5VP1LqsfGJNLi1FqBGBG3A98Fjge2A2dl\n5i1Djm18BaKNSaQ3KrYCcSfbqy0F3pqZR0TEYcA/AiNrmW5jEmnxdhoGmXnssD+LiD8GrqqOuzki\nXo2It2XmzDoe0K+9FqVx1fpeiwARcRqwT2aeFxHvBjZn5r5Djm19mrB8+TquvtqqgSbbfKcJdcNg\nF+AbwEHAC8DazNw65NhWPrU4faGRzwmklsJgIcbpI8zSJJlvGEzUpxYlDWcYSAIMA0kVw0ASYBhI\nqhgGkgDDQFLFMJAEGAaSKoaBJMAwkFQxDCQBhoGkimEgCTAMJFUMA0mAYSCpYhhIAgwDSZVaYRAR\nh0XEjyLi9ur3Q0tdmKR21b0zuBA4NzNXAucBF9W/pDIW0zfe8cZjvD5/b6MYb77qhsHjwJ7V618E\nHqt5vmL6/h+4z+P1+XsbxXjzVXfj1bOBGyJiPYNt195b/5IkjULdvRZPB07PzO9ExEcYbKgydDs2\nSeOr7o5Kz2bmHtPe/29m7jnkWHdQkUakyC7MO/FgRByVmVsj4hjggToXI2l06obBHwFfiYhlwP8B\np9W/JEmj0Npei5LGW6srECPiLyLizoi4IyKujYgVDY93YURsq8b7dkTssfOvWvRYH4mIeyLilYg4\nuMFxjo+IH0fEAxGxrqlxqrG+HhFPRsRdTY4zbbwVEXFdRNwbEXdHxBkNj/fmiLipWjR3b0R8vsnx\nqjGXRMRtEbGh6bGq8f6j+jt3e0T8aM6DM7O1X8Du016fDnyt4fHeByypXl8AfKHBsX4NeBdwHXBw\nQ2MsAf4N2BfYBbgD+PUGv6ffBg4C7mrp5+PtwEE7flaA+5v8/qpxdq1+fxNwI3Bkw+N9GvgHYENL\n/05/Arx1Pse2emeQmc9Ne7sb8FTD412bma9Wb28EGrsTycz7M/NBBqXXphwOPJiZD2XmS8AVwAeb\nGiwzrweeaer8s4z3RGbeUb1+DtgG7NPwmM9XL9/MIGwb+36rO+ETgK81NcZswzLPGUDrH1SKiL+K\niIeBk4EvtDj0J4AftjheE/YBHpn2/lEa/ssyKhHxTgZ3JTc1PM6SiLgdeALYkpn3NTjcxcBZDNbp\ntCWBzRFxc0ScOteBdasJbzDHIqVzMvOazDwXOLea734JOKXJ8apjzgFeyszLmx5L9UXE7sCVwJkz\n7iaLq+4cV1bPkzbtKJWXHici3g88mZl3RMQqmr2DnO7IzHw8In6JQShsq+743qB4GGTmfFcgXg78\noOnxIuJkBrdmRzc9VgseA94x7f0KxujzICVExFIGQfD3mfndtsbNzGcj4vvAoUDxMACOBNZExAnA\ncuAtEfHNzPxYA2P9XGY+Xv3+XxFxNYOp5qxh0HY1Yf9pbz/E4AFYk+Mdz+C2bE1mvtDkWDOHbui8\nNwP7R8S+1dqOk4Cmn0oH7f1fDAZL2u/LzC83PVBETEXEntXr5QyW0jfyM5mZn8vMd2Tmfgz+u13X\ndBBExK7VXRYRsRuwGrhn2PFtPzO4ICLuquZoq4C1DY/3twyeSm+uyjl/19RAEfGhiHgEOAL4XkQU\nfz6Rma8AnwI2AfcCV2TmttLj7BARlwP/Crw7Ih6OiFpTunmMdyTwUeDoqhR2WxXoTfll4F+qn8cb\nGTzh/+cGx2vb3sD1076/azJz07CDXXQkCbDtmaSKYSAJMAwkVQwDSYBhIKliGEgCDANJFcNAEgD/\nDzcvNav5fpAxAAAAAElFTkSuQmCC\n", + "text/plain": [ + "<matplotlib.figure.Figure at 0x7f8030785b10>" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "#@test {\"output\": \"ignore\"}\n", + "num_examples = 50\n", + "X = np.array([np.linspace(-2, 4, num_examples), np.linspace(-6, 6, num_examples)])\n", + "plt.figure(figsize=(4,4))\n", + "plt.scatter(X[0], X[1])\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "AId3xHBNlcnk" + }, + "source": [ + "Then we perturb it with noise:" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": { + "cellView": "form", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 327, + "status": "ok", + "timestamp": 1446659134929, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "4896c353dcc58d9f", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "fXcGNNtjlX63", + "outputId": "231c945e-e4a4-409e-b75b-8a8fe1fdfc30" + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQkAAAEACAYAAACgZ4OsAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEptJREFUeJzt3X+MpVV9x/H3B5dJx6rYFbNGNqjUX2S1wFbRipWNZHYo\nhtUV02B/iJq4bbBI64QssCZLSmyKdKOGYixRYW3cUH/Rbi3MMgZ3G0ypRUC3gLCt1gKKBrES063L\nj2//uHfWu8OdZ+7Mvec553nu55Xc7Nx7n5nnTLL3O+d8z/eco4jAzGwxR+VugJmVzUHCzCo5SJhZ\nJQcJM6vkIGFmlRwkzKxS8iAh6RJJd0v6tqTPSZpIfU8zG52kQULSi4D3AadExG8Aq4BzU97TzEZr\nVeKf/xhwCPhVSU8BzwR+kPieZjZCSXsSEfFTYAfw38BDwP9ExFdT3tPMRiv1cOME4M+AFwEvBJ4l\n6fdS3tPMRiv1cOM1wNcj4lEASV8G3gDsmr9AkhePmGUUEap6P/Xsxn3A6yX9iiQBZwD3LrwoIhr1\n2L59e/Y2tL3NTWtvU9s8iNQ5iW8BnwW+CXwLEHBNynua2WilHm4QEVcCV6a+j5ml4YrLFdiwYUPu\nJixb09rctPZCM9s8CA06LknWAClyt8FsXEkiMicuzazhHCTMrJKDhJlVcpAws0oOEmZWyUHCzCo5\nSJhZJQcJM6vkIGFmlRwkzKySg4SZVXKQMLNKDhJmVslBwswq1XE4zzGSviDp3u4hPa9LfU8zG506\nehIfB26MiBOBk+izx6VZ2+3Zs4eNG89h48Zz2LNnT+7mLEvSTWckPQe4MyJ+veIabzpjrbZnzx42\nbz6PgwevAGBycis33LCT6enpzC0rY9OZlwCPSLpW0h2SrpE0mfieZkXZseOaboA4D+gEix07mrMf\ndOqNcFcB64H3R8Ttkj4GXAxs773osssuO/z1hg0bWrtXoFlue/fuZe/evcv6ntTDjTXAv0TECd3n\nbwS2RsTZPdd4uGGt1vThRvKNcCXtA94XEfdL2g48MyK29rzvIGGtt2fPnsNDjJmZLUUECCgnSJwE\nfAo4Gvgu8J6I+FnP+w4SVqxSP9yjUkSQWIqDhJUq5TChlODjIGE2hI0bz2FubhOdWQmAnUxN7ebm\nm7801M8tKUcxSJBIfsyfmR3pyClROHiw81qpQxkHCbMevcOA009fz623buXgwc57k5NbmZnZmbF1\neThImHUtHAbceutWtm27gH37dgMwMzOaIcHMzBZuvfW8xgQf5yRs7CyWNEyVg1hOG+rmnIS10jAf\nsKf3Fs7LkjScnp4uNgfxNBGR9dFpgtlgZmdnY3JyTcB1AdfF5OSamJ2dHfj7p6be3v3e6D6ui6mp\nt4/kZzdR9/NX+Rl1T8IaJeXMwPT0NDfcsLOnl1JG6XRuDhI2VpZKGjZqGFATJy6tUUZRiFRK0rAE\nrri0Vqr6kLcxAKT8nQYJEk5cWmu0MfGY+ndigMSlexLWGnXWOdQl9e9UwvZ1ZtZwnt2w1mhaufMg\nSvidPNywxnHist7EZR07Ux0F3A48GBGb+rzvIGEDK2kvhjYoJSdxIXBPDfexMdD07embKGmQkLQW\nOIvOHpdm1kCpexIfBS4CPJ6wkZiZ2cLk5FZgJ7Czm8jbsuT3NfmYvdySzW5Iegvwo4i4S9IGYNFx\njw/nsUGtZBFWKcvDS1DU4TyS/gL4A+AJYBJ4NvDliHjXguucuLSk2lhkNSpZE5cRcWlEHB+d07vO\nBW5ZGCDMrHwuprLWK6EgqclcTGUr1qTCpSa1tU5FFFMtxUGimVzU1A4OEpaMk4HtUErFpZk1mBOX\ntiJOBo4P9yRsxV75ypeyevXlnHLKtc5HtJh7ErZsC5OWBw9uzdwiS8k9CVu23CsxvQ6jXg4S1ijz\nvZi5uU3MzW1i8+bzDgcKB49EltopN/UD75bdODl3pV7smL7UbZqdnY2pqbcfvldb4GP+LIUSj8NL\nefzfuK8idZCwFcl1HN5iU68pcyIpA1ATOEhYo1T1Yly3kYbLsq01Ui3iavM6Fa/dMBuRtq4idZAw\ns0pe4GVmQ0u+pb6kWyTdLWm/pA+kvJ+ZjV7S4YakFwAviM6O2c8Cvgm8NSK+03ONhxtmmWQfbkTE\nwxFxV/frnwP3AselvKdZPy7ZXrnaEpeSXgzsBV7VDRjzr7snYUm1eQpzWIP0JGoppuoONb4IXNgb\nIOb5cB5LadwrJnsVdTjP4RtIq4CvADdFxMf7vO+ehCXl/TgXV0pP4jPAPf0ChFkdvNXecFLPbpwG\n/DOwn86hwQFcGhGzPde4JzHm6qhmbGvF5LBccWnFc1Ixr+xToDbeBpl2zL0V3ii1dZrVS8UtiXHb\nqKXVv+9SW1elfuDt61ppsW3mFsq5Fd4oDfr7loYBtq/zcMOymt9EZmpqN1NTu5f917etXfyiLBVF\nUj9wT6KV6ughlNQLKakty8EAPQkHCRvYcneMTr3DdGld/CbuqD1IkHDi0gaynMTcuNYk5NocOLml\nokjqB+5JNEKqROQwf32b2sUvCe5JWN2Ws5hq2GnDEs//aCMHCRtIivUPo1id2doufkEcJGwgS/3V\nns9DPPLIj5iYuIhDhzqvezFV83nthg1t4bBhYuJPWbfuJI499nmViUuv28jPC7ysFsPs1zCuMyGl\nKGU/CbNFOadQPpdl29BmZrYwObkV2Ans7OYhtuRu1ki47Lue7evOBD5GJyB9OiKuWPC+hxst0MZh\nwzjkTLLnJCQdBdwPnAH8APg34NzwuRvWAOOwN2YJm86cChyIiO9HxOPA9cBbE9/TzEYodeLyOOCB\nnucP0gkcZsXzBrodnt0wW4TLvjtSB4mHgON7nq/tvnYEH87TPm1JZLZtira4w3kkPQO4j07i8ofA\nN4B3RsS9Pdc4cdkyy50VaEtAaaJBEpd1LAU/k06gOABc3Of9Uax4tYIsZzMYL/fOixL2uIyI2Yh4\nRUS8LCL+MvX9rFnq2lLfRVEr58SljVxpswKt3u6+Dkt1NVI/8HCjlQbdcaqO4UZpe2GWBO9MZbkM\nOivgacbyeam4td44rMFYqexrNwbhIGF18DRrfw4SZlaphAVelpin9iw19yQazGNtG5Z7Eg2ykh7B\nKAuR3COxxThIFGC+RzA3t4m5uU1s3nze4Q9qHR/eqvubuZiqAIsV+yxVaDSqQiQXG40vXEzVbEud\ncOVCJKuDg0QBFlvrMEh+YRT7HczMbGHfvnM5dOiTAExMfIeZmeuH+pnWHg4SBajqEdS3UOpo4I+7\nX1+U6B7WRJ4CLVwdlYLjsCu09ecTvFqgbdunWfMkCxKSPgKcDfwC+E/gPRHxWKr72cqVtv+DlSVl\nncTNwLqIOJnO1nWXJLxXI5RasDSfE5ma2s3U1G5XbdoRaslJSHobcE5E/GGf98YiJ+ESaitRSWXZ\n7wVuquleRaprL0ezURsqSEiak/Ttnsf+7r9n91yzDXg8InYN3dqMSh0qmKU2VOIyIqaq3pf0buAs\n4M1V15V+OM8oNlJ1ctBKUNThPJLOBHYAb4qIn1RcV3xOYlR1BN4dyUqTu07iKmACmJMEcFtEnJ/w\nfsVbTs2DA4qVIlmQiIiXpfrZdat7qOBzIqwkLsseUJ1/2V0mbXXJPdxoFZdH27jyzlQjMOrp0ZmZ\nLUxObgV2Aju7w5stQ/9cs5XwcGNIqSopnbi0OvjcjRr0yx+sXn05u3Zd7Q+2Fa+ksuyx8uijz/dm\nstYa7kkMaeFwA+ZzCQ97RsKK555EDeaXWa9efTnwSToBwsMMaw8HiRGYnp5m166rmZz8HvAwnpGw\nNvFwY4Q8I2FN49kNM6vknISZDc1BoiG86Y3l4uFGA3h/TEvFOYmW8KpQS8U5CTMbWvIgIWlG0lOS\nVqe+10o0YazvVaGWU9LhhqS1wKeAVwC/GRGP9rkm23CjSWN912BYCtlzEpK+APw5sJsCg0Tusb4/\n+JZb1p2pJG0CHoiI/d2NcK2H97G0phgqSEiaA9b0vgQE8CHgUmBqwXt95Tp3I+dZGEee6AUHD3Ze\nc5CwlIo5d0PSq4CvAv9LJzisBR4CTo2IHy+4NusUaK4u//r1b+TOO58EXghsoY6l5R7e2EKDDDeI\niOQP4HvAry3yXoyb2dnZmJh4fsB13cexMTHx3JidnU16z8nJNYfvOTm5Jun9rBm6n7/Kz29du2UH\nFcONcbNjxzUcOnQlv0yYwrp11yb9y+7hja1ULUEiIk6o4z5Nduyxz8vdBLO+fO5GBjkSpj6w2FbK\nazcyyZFEdOLSFspeTDWIcQ0STeLg0l4OEja0JpWu2/J5FegimrCoqxRHzop0gsV8r8LGw9glLl0O\nbbY8Y9eTKOkvYxN6NF6mbmPXkyhFU3o084cP/TJxWV4bLa2xS1yWkojLvUzdDDIvFS+V/zKaLc/Y\n9SRK0a9Hs23bBezbdwfgegSrh+skCtdbpHT66ev58Ievyj4MsvHiINEgzlFYDi6mMrOhjV3islRe\npWml8nCjIF5IZXXLnpOQdAFwPvAE8E8RcXGfaxwkzDLJmpOQtAE4G3h1RLwa+KtU9ypVE8quzZaS\nrCch6e+Av4mIW5a4rpU9iVIqO82q5J7deDnwJkm3SfqapNckvFdxSlpIZjaMlIfzrKKzjf7rJb0W\n+DzQd0PcXIfzmI2bYg7nAZB0I3BFROzrPv8P4HUR8ZMF13m4YZZJ1tkNSVuA4yJiu6SXA3MR8aI+\n17UySICnNK18uYPE0cBngJOBXwAz872KBddlDxL+MNu4yl4nMYjcQcLDAhtnDhID8MIqG2e5p0DN\nrAXGfoGXF1aZVRv74QY4cWnjyzkJM6vknISZDc1BwswqOUiYWSUHCTOr5CBRGG9UY6Xx7EZBXCJu\ndfMUaMO4RNzq5ilQMxva2Jdll8Ql4lYiDzcK4xJxq5NzEmZWKfe5G6+V9A1Jd3b/Havdss3aImXi\n8iPAhyLiFGA7cGXCe5lZIimDxA+BY7pfPxd4KOG9zCyRlBvhHg98nc45HALeEBEP9LnOOQmzTAbJ\nSaQ8nOcC4IKI+HtJ76Czc/ZUv5/jw3nM6lHa4TyPRcRzep7/LCKO6XOdexJmmeSuuDwg6fRuQ84A\n7k94LzNLJGXF5R8BV0uaAP4P2JLwXmaWiIupzMZY7uGGmbWAg4SZVXKQMLNKDhJmVslBwswqOUiY\nWSUHCTOr5CBhZpUcJMyskoOEmVVykDCzSg4SZlbJQcLMKjlImFklBwkzqzRUkJD0Dkn/LulJSesX\nvHeJpAOS7pW0cbhmmlkuw/Yk9gObgX29L0o6Efhd4ETgd4BPSKrc2KJJlruRaAma1uamtRea2eZB\nDBUkIuK+iDhAZ5fsXm8Fro+IJyLiv4ADwKnD3KskTfzP0LQ2N6290Mw2DyJVTuI4oPeMjYe6r5lZ\nwyy5EW7F2RrbIuIfUzXMzMowko1wJX0NmImIO7rPLwYiIq7oPp8FtkfEv/b5Xu+Ca5ZR0hO8Fui9\n0W7gc5I+SmeY8VLgG/2+aakGmllew06Bvk3SA8Drga9IugkgIu4BPg/cA9wInO99882aKfu5G2ZW\ntqIqLiXNSHpK0urcbVmKpI90C8XukvQlSc9Z+rvqJ+lMSd+RdL+krbnbsxRJayXdIuluSfslfSB3\nmwYh6ShJd0janbstg5B0jKQvdP8P3y3pdYtdW0yQkLSWzqnj38/dlgHdDKyLiJPp1IFckrk9TyPp\nKOCvgWlgHfBOSa/M26olPQF8MCLWAb8FvL8BbQa4kM7wuik+DtwYEScCJwH3LnZhMUEC+ChwUe5G\nDCoivhoRT3Wf3gaszdmeRZwKHIiI70fE48D1dArdihURD0fEXd2vf07nP2/RNTbdP3BnAZ/K3ZZB\ndHu9vx0R1wJ0ix4fW+z6IoKEpE3AAxGxP3dbVui9wE25G9HHwqK2Byn8A9dL0ouBk4GnTZ0XZv4P\nXFMSfC8BHpF0bXeIdI2kycUuTnmq+BEqirI+BFxKZ6jR+152gxSSSdoGPB4RuzI0sbUkPQv4InBh\nt0dRJElvAX4UEXdJ2kAh/3eXsApYD7w/Im6X9DHgYmD7YhfXIiKm+r0u6VXAi4FvdReBrQW+KenU\niPhxXe3rZ7E2z5P0bjrdzDfX0qDlewg4vuf52u5rRZO0ik6A+NuI+Ifc7VnCacAmSWcBk8CzJX02\nIt6VuV1VHqTTc7+9+/yLwKJJ7eKmQCV9D1gfET/N3ZYqks4EdgBvioif5G5PP5KeAdwHnAH8kE5B\n2zsjYtEkVQkkfRZ4JCI+mLstyyHpdDqVx5tyt2UpkvYB74uI+yVtB54ZEX0DRW09iWUImtFluwqY\nAOa6q+Bvi4jz8zbpSBHxpKQ/oTMTcxTw6QYEiNOA3wf2S7qTzv+HSyNiNm/LWucDdKqijwa+C7xn\nsQuL60mYWVmKmN0ws3I5SJhZJQcJM6vkIGFmlRwkzKySg4SZVXKQMLNKDhJmVun/AUYHTBJb9HcU\nAAAAAElFTkSuQmCC\n", + "text/plain": [ + "<matplotlib.figure.Figure at 0x7f8040158790>" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "#@test {\"output\": \"ignore\"}\n", + "X += np.random.randn(2, num_examples)\n", + "plt.figure(figsize=(4,4))\n", + "plt.scatter(X[0], X[1])\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "3dc1cl5imNLM" + }, + "source": [ + "## What we want to do\n", + "\n", + "What we're trying to do is calculate the green line below:" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": { + "cellView": "form", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 414, + "status": "ok", + "timestamp": 1446659137254, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "4896c353dcc58d9f", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "P0m-3Mf8sQaA", + "outputId": "74e74f19-6ff8-4a8c-81c7-9021a08b78b5", + "scrolled": false + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQMAAAEACAYAAAC3RRNlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHvdJREFUeJzt3Xl4VPX1+PH3CRCNRVCgUpWK4iOlRWWxipVa0BDCIsHt\nZ22lLLbybRECGBYXFBBbQeQBRCoWGkWptrUUxQoJQYQWLVKURREBFUGggKDsS0jm/P6YgULMJDOZ\nO3eZnNfzzNNMcrn3tM2cfO7nc+75iKpijDFpXgdgjPEHSwbGGMCSgTEmwpKBMQawZGCMibBkYIwB\n4kgGIvJHEdkpImtO+d65IrJARNaLSKGI1E1OmMaYZItnZPAckF3me/cDC1X1e8Ai4AGnAjPGuEvi\nKToSkcbA66p6ZeT9x0A7Vd0pIt8BFqtqs+SEaoxJpkTnDM5T1Z0AqroDOC/xkIwxXnB6AtFqm40J\nqJoJ/vudItLwlNuEXdEOFBFLFMZ4RFWlsmPiHRlI5HXCXKB35OtewGuVBBSo18iRIz2PIZXjtZjd\necUqnqXFl4B3gKYiskVE+gBjgSwRWQ9kRt4bYwIo5tsEVf15lB91cCgWY4yHrAKxAu3bt/c6hLgE\nLV6wmP0krjqDhC4kom5dyxjzPyKCJmEC0RiToiwZGGMASwbGmAhLBsYYwJKBMSbCkoExBrBkYIyJ\nsGRgjAEsGRhjIiwZGGMASwbGmAhLBsYYwJKBMSbCkoExBnAoGYjIAyKyVkTWiMifRCTdifMaY9yT\ncDKI7KVwD9BKw/sp1ATuTPS8xhh3JdodGWA/UAx8S0RCwFnAdgfOa4xxUcIjA1X9GpgAbAG2AXtV\ndWGi5zXGuCvhkYGINAEGA42BfcDfROTnqvpS2WNHjRp18uv27dunbC85Y7y0ePFiFi9eHPe/S7gH\noojcAWSp6j2R978A2qhq/zLHWQ9EYzzgZg/E9cC1InKmiAjh/RPWOXBeY4yLnJgzWA28ALwHrCa8\n49IfEj2vMcZd1ird+FZhYSETJoT/ruTl9SU7O9vjiIIp1tsESwbGlwoLC7nlll4cOTIOgIyM4cyZ\nM9MSQhVYMjCB1rHjbRQV5RDezxdgJllZc1mwYLaXYQWSbaJijA8UFhbSseNtdOx4G4WFhV6HUyEn\nKhCNcVxeXl+WLu3FkSPh9xkZw8nLm+ltUHEqe6uzdGkvX9/q2G2C8a2gTyD65VYn1tsEGxkY38rO\nzg5cAggySwbGJEnQbnXsNsGYJPLDrY4tLRpjAFtaNMbEyZKBMQawZGCMibBkYIwPFH1axPYD3nYL\ntGRgjIdUlRdXv8ikdydxtOSop7FYMjCBFKSa/2hCGmLisom8vuF18nPyaXJuE0/jsaVFEzip8Hhz\ncWkxoxaP4stDXzIhewJ1zqiTtGu5urQoInVF5BURWRfZTKWNE+c1pjwTJvwhkgh6AeGkcKKwJwgO\nFh8kd34uJaESpnadmtREEA+nypEnA/NU9f+JSE3CeycYY8r48tCX5Bbk0rJhS4a2HUqa+OdO3YlW\n6XWA61W1N4CqlhDeWMWYpAhazf8Jn+/9nNz5udzc7Gb6tOxDuH+wfzjRKr0F4QaoHwEtgBXAQFU9\nUuY4mzMwjvFDzX881uxcw5AFQxhwzQC6fa+bq9d27dkEEbkKWAb8SFVXiMgkYJ+qjixznI4c+b9v\n2SYqprr45+Z/8uiSR3n0hke57rvXJf16ZTdRGT16tGvJoCHwb1VtEnn/Y2C4qnYrc5yNDEy1M2fd\nHKa9N42J2RP5wbd/4EkMrjU3UdWdIvKFiDRV1Q2EN1H5KNHzGhNkqsr096fzxsY3mN5tOhfVvcjr\nkCrl1GpCLvAnEakFfAb0cei8xgROaaiUsUvHsm73Op7r/hz1Mup5HVJMrOjIGAcdLTnKg28+yLGS\nY4zvOJ6zanm/ym79DIxx2b6j+/jNG7+hdnptJnWa5ItEEA/rgWiMA7Yf2M6A+QNo17gd/a/p76ti\nolhZMjAmQRv2bGBQwSB6tujJnZff6XU4VWbJwJgE/Gfbf3hw0YMMu24YWZdmeR1OQiwZGFNFCz5d\nwPh3xjM2cyxXXXCV1+EkzJKBMVXw0gcvMWvNLH7f5fdcVv8yr8NxRPBmOYyJUTIaoIQ0xKRlk5jz\n8Rzyu+enTCIAqzMwKSoZDVCOlx5n9JLRbD+wnUmdJvmmD0FlrM7A+Fqy25Y53QDlUPEhBhYM5GjJ\nUZ7p+kxgEkE8bM7AuC5oW5XvObyHAfMHcPl5l3P/j+8PZA1BTFTVlVf4UsaoZmXdqvC8gkZez2tW\n1q2OXqOgoEAzMhpGrvO8ZmQ01IKCgrjPs3nvZu32Ujed/t50DYVCjsbolshnr9LPqI0MTErKzs5m\nzpyZpzRAiX/k8eGuD7mv8D76Xd2Pm5vdnIwwfcUmEI3rgtDdeOmWpYxaPIqR7UZyfePrvQ4nIbYL\ns/E1P7ctm7t+Lk8vf5oJHSdwRcMrvA4nYZYMjImTqpK/Mp/X1r/GlM5TaHxOY69DcoQtLRoTh5CG\nGLt0LG9uepP87vm+SgRu7R7lWDIQkTQReV9E5jp1TpM6/Lwd2rGSYwwrGsYX+79gerfpNDirgdch\nnXRifqWoKIeiohxuuaVX8v73i2XJIZYXMBiYBcyN8vNkrJqYAHBqmS8Z9h3dp3e/erc+9OZDWlxS\n7HU43+DEMiwxLi06tb1aI6ALMMOJ85nU4pft0MqOTnYc3MEv5/6SKxpewaM3PEqtGrVcj8lPnKoz\nmAgMBeo6dD5jKhXPikTZ5cx/fngXVw5vwqAbB/HzK37uSrxV4ebuUU5sr9YV2Kmqq0SkPRB11nLU\nqFEnv7ZNVKqPsr/QaWmDadcuL6FzxlvSfNro5Pz3ONbhO3w17yg/H+jfRABVK54qu4lKzGK5l6jo\nBfwO2EK4Rfp/gYPAC+UcV9XbJpMCHnvsMU1Lq69wrUJewvMG8d5Lnzy+SZHyiw7KBY9oWlp938xd\nJBNuzRmo6oOqepGGd1S6E1ikqj0TPa9JLUuWvE8oNAH4N/Ckg/MGhcBtwDR2794Z9ai8vL7I5ffC\nj/JgXgfY/iyhUO9AbeWebFZnYHyrouXIvLy+pKcPAnoAOcCvWbt2Q7nLbqrKhnM2UOe6M2FuTdiz\nHJgJBL+60FGxDB+ceGG3CdVavMuLsRzfqlW7Sm8Vjpce14cXPay9X+2tf/vH33y7xJlM2FOLxk/i\nnQg7fTkSjhwJf+/Uf9OgQf0Kr3n4+GGGFQ2jVlotnun6DGfWPJPac2on9CRjSoslYzjxwkYGJg6x\nTBBWNHrYc3iP9vh7Dx2zZIyWlJZ84/wFBQWalXWrZmXdmvKjA2IcGVgyML4U621FQUGBtmrVTuvV\nu1RbtWqrBQUFumXvFu3+cnd9dsWz5TYk8XNFZDJYMjCBF8tf77If7DMa1dfWE1vr7I9mRz2vG52W\n/CTWZGBzBsa3srOzK72nP21u4bvvcOyGhqQtrc2tg251J8gUYsnApIamr0ObKVD4M869cmWFh7pZ\n4hsk1tzEBFpBQQE5I3/K8Usvgfk9yDj2ZEwt1Pzcaclp1unIpLyQhhj/9njmr5oP82qTXnJGyn+w\nq8KSgUlpxaXFjFg0gv3H9vNkxyepnV7b65B8y9qemZS1/9h++s/rTw2pwVOdn7JE4BBLBiZQdh3a\nxa/m/opmDZrx28zfkl4j3euQUoatJpjA+OzrzxgwfwB3Nr+THlf2QKTSka+JgyUDEwgr/7uS4QuH\nM/jawXS+rLPX4aQku00wvlLeY8uLNi1i2MJhjLlhjCWCJLLVBOMb5W27NnjGL3lf32di9kSaNWiW\ntOumcs1BrKsJTjxz0AhYBKwFPgByoxyXhKprk0pOf2YgpFz9Cz1vwAW6dd/WpF2zOjy0hIvPJpQA\n92m4IWpt4D0RWaCqHztwblMdpZXA9b+Fcz/jBxtbc2GdC5N2qVj6JlQXCScDVd0B7Ih8fVBE1gEX\nApYMTFzy8vryr2U9OfrjWaDCmW9u5P5XXvA6rGrD0dUEEbkYaAm86+R5TfVw9U+upu3jV7Dp/W00\n+aIZQ155Iel/oe2hpf9xbAIxcouwGBijqq+V83N16lom9Wzbv43+8/vTsUlHfv3DX7taQ2ATiJHj\nnPiAikhN4B/AfFWdHOUYHTly5Mn3tomKOWHdl+sYXDiYX7X+Fbf/4Havwwm8spuojB492tVk8AKw\nW1Xvq+AYGxmYb1i2dRkjFo3goesf4oZLbvA6nJTk2oNKItIWuAu4UURWRrZl75ToeU3s/LzdeUXm\nbZzHI289wpMdn7RE4AexrD868cLqDBJWXk/AIK6Th0Ihnblqpnb9U1f99KtPvQ4n5WENUVNLtA99\n0Jp7loZKdfzb4/WOV+7QnQd3eh1OtRBrMrAHlQIiWnFMkBSXFvPIW4/w1ZGvmNFtBmefcXbSrpXq\nKwTJYA8qBVxeXl8yMoYT3jtwZmSdvK/XYX3DgWMH6D+vP6rK012eTloiKCwspHXr9nTpchdFRZdQ\nVJTDLbf0CtRcimdiGT448cJuExJS0dyA33cH2nlwp/70lZ/qE0uf0NJQadKuU/Z/I2ioUOD7W6dk\nI8bbBHtqMUCCOPTd9PUmcgtyuf37t9OzRc+kFhN17HgbRUU5nLiVCo+W5gI5ZGXNZcGC2Um7tp/F\nurRocwYBEsumIn6yesdqhhYNZdC1g+hyWRePotherUuM42FzBqZcidYuLP58MUOKhjC6/WjXEkHZ\n+ZO0tMG0alUjpn0UDDZnYL4p0dqF2R/N1uwXs3XtrrVJjLJ8fp8/8QI2Z2Cqqrx771juuVWVZ997\nloJPCni6y9M0qtMo6bGaytmcgXFVaaiU3/3rd2z8aiP53fOpl1HP65BMnCwZmG+I9xn/I8eP8MCb\nD1AaKmXaTdM4q9ZZLkVqnGS3CaZcsS5j7j26l0EFg2hctzEPt3uYmmn298VvbK9Fk3TbD2yn/7z+\nZF6SSb+r+9mmJj5lcwYmqdbvXs+gwkH0admHO5rf4XU4xgGWDEzclm9bzkOLHuL+tveT2STT63CM\nQ6zoKAUls9lJwScFjFg0gnEdxlkiSDFOtT3rBEwinFz+qKrjyjnG5gxcUN6uRE5V4M1aM4uXP3yZ\npzo9xaX1Lk34fMYdrk0gikgasAHIBLYD/wHu1DKbqFgycEdVC4YqEtIQk5ZNYtnWZUzpPIWGtRs6\nEqtxh5sTiNcAG1V1c+TCfwa6Y5uopITi0mJGLR7Fl4e+ZEbODOqcUcfrkEySODFncCHwxSnvt0a+\nZxwSzxyAk81ODhUfYuD8gZSESpjadaolghRnqwk+V3YOYOnSXhXOAWRnZzNnzsxTCoaqNl+w+/Bu\ncufn0qJhC4a2HUqa2FxzqnMiGWwDLjrlfaPI975h1KhRJ7+2TVRiU5WNQU/te3BiVAGxN0TZvHcz\nA+YP4OZmN9OnZR8rJgqYspuoxCyWRxsregE1gE+AxkA6sAr4fjnHOflUZrWRSPfjqjyKvHrHas16\nIUtf+/g1J8I3PoCbrdKBTsB6YCNwf5Rjkv/fOgUl0lsg3kSy5PMlmjkzU5duXupU+MYHYk0GjswZ\nqGoB8D0nzmVO59QcQGVe/fhVnlnxDJM7Tab5ec0dP7/xP3tQKYWUfdIQqLQASVWZ/v503tj4BlM6\nT+GiuhdVeE5rHxY8sdYZWNuzFBHtdqKiNmAlpSX62JLH9K7Zd+mew3tiPqcJFmx7tdQU7cMd7/zA\nkeNHdHDBYO33j356qPhQuccEbes2U75Yk4HVGQRI2ZqDJUt+QfPmTWnQoCG7d++J+Tz7ju5jcOFg\nGtVpxLgO46hVo1ayQjZBEkvGcOKFjQwSVt5farhW4XlNTz9H09O/XemQfvv+7XrrX27VycsmV7q7\nkd0mpAZsZFBdXAD0orgYWrWaToMGc4HyVx027NnAoIJB9GzRkzsvv7PSM7u1kmH8wVYTAqTsbQIM\nAWYB2VT2dOKK7St44M0HGHbdMLIuzXIpYuMH1gMxRZ1Y6tu9ew9r166muHgSUHHfggWfLmD8O+MZ\nmzmWqy64yu2QjccsGVQDsdQAvPTBS8xaM4vJnSZzWf3Lkn494z+WDKq5kIaY8u4U/rXlX0zpPIXz\nzz4/ofM51UHJEor7rOioGisuKdYRb47QPq/20b1H9jpyTidqDmx1whvEuJpgD6kHXNnGJ4ePH2Zg\nwUAOHz/MM12foe6Zdb0O8aTTH8cOjzJOjBKM92xpMcDKDt3/teIX/OjRy+nQogPD2w6nRloNx64V\n75ZrJoBiGT448cJuExx32tC97mblZy31+z1aaSgUSsr1Et3u3G4TvIEVHVUj530IHfNgxfU0+u62\npHUmOrWDUlX/vRUx+ZetJgRYYWEhOf1/RvF158Pi7mR8me/YHgkmdcS6mpDQBKKIPCEi60RklYjM\nFhFrn+uiYxcf4/J7L+HafeeT9b31lghMQhIaGYhIB2CRqoZEZCzhe5MHohxrIwOHqCr5K/N5bf1r\nTOk8hcbnNPY6JONjrmyioqoLT3m7DLgtkfOZyoU0xBNvP8GanWvI755Pg7MaeB2SSRFOTiDeDfzZ\nwfOZMo6VHGPEohEcOn6I6d2m8630b3kdkkkhlSYDESkCTt1cTwAFHlLV1yPHPAQcV9WXKjqX7ZtQ\ndfuP7WdwwWC+U/s7/C7zd9aQxERV1X0TnNh4tTdwD3Cjqh6r4LjAzxl4VVe/4+AOBswfQNvvtiW3\nTa7tbmTi4sqzCYT3S1gL1I/hWEcLKdzmVcHMxj0btfOszjpr9aykX8ttiRYxmdjgRkNUwpumbAbe\nj7x+X8GxLvzXdtapv6ytWrVzvTnoe9vf0w4vdNCCjan3QbFqRPfEmgwSXU1I7AF5Hytb95+Wlufq\n9Rd+tpBxb4/jtzf+lmsuvMbVa7uhKntImuSycuQoyv6yhkIfkJY2mFAo/PNkPqjzlw//wszVM5na\nZSpN6zdNyjWMKcuSQcyuoEWLH1TYcDRRqsrU/0zlrc/fYkbODC44+4Jyj0uFBiH2FKQPxXIv4cSL\ngM0ZuH1Pe7z0uD6y6BHt/Wpv/frI176JK5lsAtEdxDhnYA8qVcCtv8CHjx9meNFwaqbV5PEOj3Nm\nzTOjHtu69Y9ZubKUcIv0vsCOCrsiR5MKowsTG2t7FhB7Du/RHn/voWOWjNGS0pIKjy0oKNC0tHNP\njgqgoUKetR8zFcL2WvS/LXu3aPeXu+uzK56NqSFJeX0I09Lqx/1Btj0Uq5dYk4GVsnnkoy8/4p7X\n76Fni570vaovIvKNfoaxaNHichviG2fEkjGceGEjg5Pe3vK2Zs7M1CWfLzn5vViG7k4N7+02oXrB\nJhD96R8b/sFT7z7Fkx2f5MqGV578fseOt1FUlMOJuoZTt0s7dbKvXbvWLFnyPpDYxJ9NIFYfrvQz\nMLFTVZ5f9Tx///jvPHvTs1xy7iUx/buylZBLl1Zt85KyEu1naFKPJQMXhDTEk+88ycodK8nPyefb\n3/r2N46JVoRjZbvGLTaBmGTFpcXcv/B+Pv3qU6Z3m15uIoD/dQ7OyppLVtZc62doXGdzBkl04NgB\n8hbkUT+jPqNvGE16jfS4z+HUHoem+rKNVz2269AuBswfQJsL2zDo2kEVNiSpbDLPJvtMIiwZeOiz\nrz8jd34uP23+U3pc2aPCTU3sL79JNksGHlm1YxXDioYx+NrBdL6sc6XHV7SkaIwTXNlE5ZSL5YlI\nSETqOXG+oFq0aRFDi4Yy5oYxMSUCY/wk4WQgIo2ALMLtz1JGvKXBr6x9hfHvjGdK5ym0adQm5uvk\n5fUlI2M4MBOYGVlS7Fv1wI2pqljKFCt6Aa8AVwCbgHoVHOdoiWUyxVOuGwqFdOryqXrLn2/Rrfu2\nVvl69ly/SRbcKEcWkRygvareJyKbgKtU9asox2oi13JTrKXBg+77JSsyVvDJV58wudNkzs0417OY\njYnGsXLkCjZRGQE8SPgW4dSfRRX0TVROm/mvWcyiM27npps68vLdL5NRK8Pr8IwBPNhERUQuBxYC\nhwkngUbANuAaVd1VzvGBGRlEW+6bMOEP4RHDmTnQeSB8dZDMdGXhgjkeR2xMdElfTVDVD1X1O6ra\nRFUvAbYCrcpLBEFTYWnw2V9D97th67WwJIc0q+g2KcKxOgMR+Qz4YSrMGUQz49UZ/N+cewmtuAc+\nupr09EE0b96CBg3qW2Wg8S1X6wwAIiOEchNBKnh367v85cBfeLzrGLIu/C+tWk0HarFyZR+KinK4\n5ZZeMXcnckJVuiIZU6FYlhyceBGgpcWy3tjwhma9kKUr/7vy5Pe87CNonYpMPHBje7VUp6q8uOZF\n/rr2r0y7aRpNzm3idUiAbU1mksOSQRQhDTHx3xNZvn05+d3zOe9b5532c9sRyKQae1CpHMWlxYx8\nayR7juxhQscJnH3G2eUe59Wjxfako4mHPbVYRQeLD5JXmMc5Z57DmBvHVKkhiRusx4GJlSWDKth1\naBe583O56vyryLsur8KGJMYEhXVHjtOmrzeRW5DL7d+/nZ4telbYkMSYVGTJAFizcw1DFgxhYJuB\ndG3a1etwjPFEtR8HL/l8CXkL8hjVfhRdm3a1Yh5TfcVSjODECx8WHc3+aLZmv5ita3etVVX/FvNY\nvwOTCGwX5uhCoZBO+8807f5yd92yd8vJ7/txd2K/JigTHLEmg2o3Z1AaKuXxpY+zYc8G8rvnUy/D\n320brdrQuKVaJYOjJUd5YOEDlIRKmHbTNM6qddZpP7eqQlOdVZs6g71H9zK4cDAX1bmIh9s9TM20\n8vOg34p5rNrQJMqKjk6x/cB2+s/rT+YlmfS7ul/gagj8lqBMsFgyiNiwZwODCgbRu2Vv7mh+h+vX\nN8ZrrjU3EZEBIrJORD4QkbGJns9Jy7ct595595L3ozxLBMZUIqEJRBFpD3QDrlDVEhFp4EhUDij8\npJAJ/57AuA7jaH1+a6/DMcb3El1N+A0wVlVLAFR1d+IhJW7Wmlm8/OHLPNP1GS6td6nX4RgTCIne\nJjQFfiIiy0TkLRH5oRNBVdWJhiRz188lPyffEoExcUh0E5WawLmqeq2IXA38FYjaGyyZm6gUlxYz\nevFodh7ayYycGdQ5o45j5zYmSFzfRAVAROYB41R1SeT9J0AbVd1TzrFJW004VHyIIQuGUDu9No/d\n+Bhn1DwjKdcxJojcWk14FbgxcsGmQK3yEkEy7T68m3tev4eLz7mYcVnjLBEYU0WJTiA+B+SLyAfA\nMaBn4iHFbuv+rfR7ox83N7uZPi37BK6YyBg/CXTR0YFjB1i+bTmZTTIdPa+TrHrQeM0qEH2g7HMF\n6elDad68KQ0aNLTEYFzj+vZqbglSJ6LTHz/uRXHxeFauLPVkOzZjKhOoZHDiL21RUU6AP1AXAOHR\nwonbB2P8IFD9DILW6KNsfwQYAszyMCJjogvUyCBosrOzmTNnJllZc2nV6jnS00uAHcDMSOOUvl6H\naMxJgZpADHqjD1tZMF5I2dUE+0AZE5+UTQbGmPik7NKiMSY5LBkYYwBLBsaYCEsGxhjAkoExJsKS\ngTEGsGRgjIlIKBmIyNUislxEVkb+09OGqMaYqkt0ZPAEMEJVWwEjgfGJh+QfVWkq6aWgxQsWs58k\nmgz+C9SNfH0OsC3B8/lK0P5PD1q8YDH7SaKPMN8PvC0iEwi3UL8u8ZCMMV5IdN+EAcAAVX1VRG4H\n8oGsZARqjEmuRPdN2K+qdU55v09V60Y51p5SMsYjsTyolOhtwkYRaaeqS0QkE9iQSDDGGO8kmgz+\nD5gqIunAUcBa9xgTUK71MzDG+JsnFYgikiciIRGp58X1YyUiT4jIOhFZJSKzRcS3u7mKSCcR+VhE\nNojIcK/jqYiINBKRRSKyVkQ+EJFcr2OKlYikicj7IjLX61hiISJ1ReSVyO/xWhFpE+1Y15OBiDQi\nvOKw2e1rV8ECoLmqtgQ2Ag94HE+5RCQNeBrIBpoDPxORZt5GVaES4D5VbQ78CLjX5/GeaiDwkddB\nxGEyME9Vvw+0ANZFO9CLkcFEYKgH142bqi5U1VDk7TKgkZfxVOAaYKOqblbV48Cfge4exxSVqu5Q\n1VWRrw8S/gW90NuoKhf5Q9YFmOF1LLGIjGSvV9XnAFS1RFX3Rzve1WQgIjnAF6r6gZvXdcjdwHyv\ng4jiQuCLU95vJQAfLgARuRhoCbzrbSQxOfGHLCgTbZcAu0XkucitzR9EJCPawY5volJJkdKDnF6U\n5PlyYwXxPqSqr0eOeQg4rqoveRBiyhKR2sDfgIGREYJviUhXYKeqrhKR9vjgdzcGNYHWwL2qukJE\nJhGuGh4Z7WBHqWq5FYgicjlwMbBawnunNwLeE5FrVHWX03HEKlq8J4hIb8JDwxtdCahqtgEXnfK+\nET5/TkREahJOBC+q6mtexxODtkCOiHQBMoCzReQFVe3pcVwV2Up4JL4i8v5vQNTJZc+WFkVkE9Ba\nVb/2JIAYiEgnYALwE1Xd43U80YhIDWA9kEn44bHlwM9UNepkkddE5AVgt6re53Us8RKRdkCequZ4\nHUtlRGQJcI+qbhCRkcBZqlpuQvByr0XF/0OtKUA6UBQezLBMVft5G9I3qWqpiPQnvPqRBvzR54mg\nLXAX8IGIrCT8u/CgqhZ4G1lKygX+JCK1gM+APtEOtKIjYwxgbc+MMRGWDIwxgCUDY0yEJQNjDGDJ\nwBgTYcnAGANYMjDGRFgyMMYA8P8BB4YEUxpBpuwAAAAASUVORK5CYII=\n", + "text/plain": [ + "<matplotlib.figure.Figure at 0x7f80640172d0>" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "#@test {\"output\": \"ignore\"}\n", + "weights = np.polyfit(X[0], X[1], 1)\n", + "plt.figure(figsize=(4,4))\n", + "plt.scatter(X[0], X[1])\n", + "line_x_range = (-3, 5)\n", + "plt.plot(line_x_range, [weights[1] + a * weights[0] for a in line_x_range], \"g\", alpha=0.8)\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "VYUr2uPA9ah8" + }, + "source": [ + "Remember that our simple network looks like this:" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "cellView": "form", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 170, + "status": "ok", + "timestamp": 1446659140755, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "4896c353dcc58d9f", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "gt8UuSQA9frA", + "outputId": "080025d5-d110-4975-e105-7635afaa3ce9" + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAJYAAABkCAYAAABkW8nwAAAO90lEQVR4Xu2dT5Dc1J3Hv+YQT8VJ\nZUhVdprLWs4FTSrGGv4ql9CuHBCH4GaTFCLZwnIcjOAy8l6Q/1SlU4XHcg6xJgtY2OOik2KxSGoT\nGWrXzYFC2T2MDAtWitRavmQ0e9k2SYGowom4hNRPtqA9TE+rW3/cPfPepcfup6f3fu/Tv9/T+/PV\npo8//vhjsMQsULAFNjGwCrYoKy6xAAOLgVCKBRhYpZiVFcrAYgyUYgEGVilmZYUysBgDpViAgVWK\nWVmhDCzGQCkWGEuwrly5gtf++zW887/vYOn/lnD5T5cT40x9ZQrb/nEbxDtFiHeI2LJlSylGY4X2\nt8BYgUVAvfzqy3i5/TI+vPLhmq37wpYv4AHpATxw3wMMsP4cFJ5jbMAiqA4eOYg/Lv8xMcL26e34\n+vTXk8+vbv1q8n/03TsX38EfLv4h+aRE380dmmNwFY7O2gWOBVgE1Y/2/yjxUls+vwXaY1oS7tZK\n3v94MJ8zceUvV0Dea+H4AoOrQrhGHqxuT0Xjp0P7D2HqH6Yymejyu5dx5PiRZBxGnmt+bj7TdSxT\nfgv0ASuAzglwmyE8pfbZu3VaEDkDdT+AweevzGolvPjvL+LMb84knmr+yHxmqNKyCK7ZQ7OJ5yIo\n+3m6clqx8UrNB1bso2W64FQN9cnijdcdAvNAQWGRPBcLicX3Ua8S84FVcj3PnjuLhRcWkgH63OG5\nXHc7+NTBZEBP47NvffNbucpiF/e3QCaw2g0NfNvES5c+wtQ9u2G0LCj8BLAiFEaeBU0zYJ9fxkfY\njKl7FZgtCzIHIA7QUmXov/g9LmMztt6rwLBMyFROj3TkZ0fgveXh4X96GN//zvf7t2aNHGlI7VlW\n0pYmRC+AKUwAsQu5thOuvIjQEjGBGJ7CQYptdOw6etc6VzXXzcUZwJrGseWt2P28DV2I4OgyDgQK\nFgMTYtQ1xqq10eDuR6j8Fi1NxGTkwpAfRos7h05bQscQIFgibEeHMBHCVhs4EBtY8lQQd6ulvbN7\n8e6f302mC7Z/bXsuo9NkKk1X9PZ+IUyeR0sN4GscYl8DPzOP5VuPYynQwMU+dL4O3wzRbpQQ93O1\nbvQuzgRWS0p/tQA6Nuqcilq7A5u3Px28T7qw7BB1VUHqhEKTB2+pCAIVHZVD3dPgujpE6peOBzes\nQRS5nr/+b//g24nF7JN27qkCGq/J++RknHXm5JlVeiKGr/MQPQMdV0ZkCRBbNUwEMYzQhRyZEHgH\nOv29ynPM6HXtja1Rf7B4AZ7RgZv+SuMAOj+NtrYEX3avfyqMfDi2DdcLEAQBvPOX8MGtR3Ex0MEF\nJiRxP373wWZsvaeBhixDVRrg1/jxlwEWPV3ap+xVrR57Cjgpht2xEDV4mLIFvqkiaoUwwzp4U4Hv\n9/awN7YrR+vuGcAS4ZsdtKV0VNEFVqMLrIkWJGEPPP4hKA0RgiCAc1XsdJQErGQ2Ig7hOQ5sx4Hz\n0u+wvHX2akjtMWCpNhQCiCicq+AcCx1Fh9B2IegcNN6B4Teg1z0EeknzKqPFRe7a9AeLm4ajXvzU\noJEDqUahMESrKxSqbQHbDBGLoXUNlBiuUsNOT8fFQEVsNdHmdOjStTgSGOCnLTQuBDBosLxKqnTw\nntw/glPnoHMS4E6iFVjgbBGcwUGMPAjtawP73GZf/wVkAutYtAvPezYUPoKjipBdGZ5vQOgavGte\nHbfsiXD09TZUIUbg6JD3vITlrU/iYthErPOYaQk44ZhocDF8U0HDqsEOHfQaC7/2X68lyzJVTjd0\nWiJu2XMem++7+tAxSd52+hguTe3GYtjq6V3XPyqDtbA/WLyAtqRg0rHhLceo3avCsk0kjqd7uoEL\n0FJkaC/9Hh/gS9ixS0dTCaDKHVidNhoTNN2gQP/FedAmly/t2IWm2YK2xswqDbj3antzz5oToD/9\n15/i5smbcdo8vfaDQGiC37YfEyeW4KtcMu2g1HbCrp9Dx5Fw3ZCw04ZSb0Jse6CsLH1qgZFfK0zn\nn+hpznzKHGpJRzus4YJ/AX/78G94ofUC7r777pwMxAhdE6pyAK8u78CJJZ+BtcKiIw8Wea0DTx34\nZCH5oHYwM1y0TjhnziXbaWgB+4cP/RCPPfYYtm/fjpMnT+Kmm24aDrDYhdpoQdAbaMtNSB4Da6Uh\nRx4sqnB3SCTPNbtvtu9iMoU/Wg5Kt9p0h8DTp09j3759ePrpp/H4448PB1fylOtC5jTUGVifseFY\ngJXClXou+jcN6Gk2nj7JG1Gi7TG0Hkiz7OlGP/ru6OGjq46rnnjiCSwuLibe66677hocMAZWT5uN\nDVgpXGfbZ5OtybQNZq1EE6G0NXmXtGvNwbrv+4n3uu222wYPjwys9QFW2goKjbQ4Tdth6CAFeSpK\n5J3oQMUwhynS8PjMM89AVdVs3ouBtb7Aytbrw+WiMZfnednCIwOLgTUIZml43LFjB5577rnhnx4H\nuek6yztWY6yqbb+wsJBMTwwUHquu5Ijej4GVoWMoPJ4/fz7xXkM9PWa4x3rLwsDK2KMXLlxIvBeF\nR5qe2LRpU8YrN2Y2BtaA/U7hkaYnnn322exPjwPeYz1kZ2AN2YtpeCTvdeeddw5Zyvq9jIGVo28p\nPJL3ok2NLDxeb0gGVg6w0kvT8HjixIlkHJY1lauaE8GRangwsvD/noKqt+kzsLJSkCEfzdi/8cYb\nifdaKzxWoppDmxJ5FT54NH06YZShAQVmYWAVaEwqKg2PMzMzyfTEyqfHqlRzAoOH6OqwJnXoNQeB\nSWcjq0sMrJJsferUqSQsdofHylRzYg8aLyG0QtiTOvhGhFZglyKD0Mt8DKySwEqLpfD45ptvYn5+\nHr/+z19/sukwj2pOP72vyJXBy4BNME340Pg6AiNAu8IDkQysksGi4t9++2189wffxee++DkIO4Tc\nqjlrSw504Eg81FobYetq+KOwKDgagjVOnRdtBgZW0RZdpbw0BL73/nv4yZM/6bv7tVeVxkk1h4FV\nAVgbUTWHgVUBWGUcvCVV6EP/cuiztQ9NCNsMiIshrPSIeaK3oUNIlXQqaDMDqwIjlyEV0Fv6MoQl\nbENT/FTIhWSXOF2AF5jocei8cCswsAo36WcLLEPchO7yyr+9smrt6TQ3geQmcgcd2CQbIHoIDKGy\nuSwG1joEi06oU+jj3RAWR2HQgFiiTuxqJmRgVQBWGaGQDo78/OjPe9T+qpfSeBeeqIM3JPip4k8F\n7aVbMLAqMHSlg/dr7YkcCZxWg1Jz0G5UL7/EwKoArBuhmoNEbupBvPrRDhxf8qFVLFrCwKoArFQi\n4P3o/VwTpCmgdBi3r2oOIrQbNdwfGljytZ46r2U1n4FVlmW7yn3rrbfwvX/+XrKkMyPM5FLNIS2K\nbCrSNI8loKX48G6AxhIDq2SwaIcDgWWaJn71H78qRDWnlxbF1aaQxJILj6TRjRhm0L4hYrwMrJLA\nos1+BBXtyaLty5SKVs1Zverx1RB4dhIPPe/CVioeXF2rFAOrYLDIOxFQd9xxRwLVytSt90XfFaGa\nU3ATCimOgVWIGa8WkoY9AorA6pUIrqJVcwpsRiFFMbAKMONqYS9LsWWo5mS5bxV5GFg5rExhj8ZP\ndHBitbCXo+ixv5SBNWQXpmGPvNXtt98+ZCnr9zIG1oB9O2zYG/A2Y5+dgZWxC1nYy2goNt2Q3VA0\njqIDESzsZbcZ81hr2CoNe/T56KOPZrcqy8m2zazGAAt7+X8ZzGOtsCELe/mhohLGEqwyVFpY2CsG\nqLSUsQKrDJUWFvaKBWrswCpDpYWFvXKgKiYUxh5U/huwhd8idBqYRARX4bHTldd8Le8gTSpapYWW\nX0is47qnveTdi02I6aFOejlAbSdcOT2fF8NTOEixDTqnV6Uk0CC2GpW8hYTCyFXA72yj8XoAAzoE\n+nsxgNnrZc8DtL7bU9HJlDwqLY9855FkbY8ktS3LWlGLECbPo6UG8DUOsa+Bn5nH8q3HsRRo4GIS\nL6vDN0O0e70SdoB2rfeshYBF71Juyzzu90TcF59FIC8WJvSVvgiT9nnPH5nP/K7CtOPonYWzh2aT\nF2Fu+usmvPjLF3us7cXwdR6iZ6DjyogsAWKrhokghhG6kCMTAu9Ap7+r1l0cQwoLAote4+ugwT+I\nsxO78XrQKkTkqzsEkqeily8Nk0il5cfHfowv3/xlLBxf6Pk2sNhTwEkx7I6FqMHDlC3wTRVRK4QZ\n1sGbCnxfrfxgwjBtvtHXFAZW7OsQZo7hEm7Fkxf8nm+mH6TBlau0RG00OBWcY6Gj6BDaLgSdDn46\nMPwG9Hr15/MGsdco5S0GrDiAIU7D5M/AgIo9gY6Lng4+5wi3jIOea59wieCQzgEnAe4kWoEFzhbB\nGRzEyIPQDmBWpaoxSpQMUZdCwCLh1OlmDWcCBzJsSNzDiIyL8LR8Ur1lHE2nPeZzh+d6mooENW7Z\ncx6b7zuHTlvCJB1Nnz6GS1O7sUhKxDl/LEP00Vhekh8sUjThNUyYAdxr59dCSwSvAWbg5Xq7exkq\nLfRO6TMnz/TurNAEv20/Jk4swaf2xC6U2k7Y9XPoOBIm6crYh6UoaLodABOoSU3YlpLbQ48lQT0q\nnR+sEq1RBlj0dGmfsnPVOtB51IMmfEdGLQ7RkkSYkps8VbJ01QIjDdaNCIVZwOi4DnxOgsRRXIzh\nazwakY3gmphsljLWe56RBqv6wfvg3R0HFqS6CcHxC5kQHrwGo3nFSIN1Q1RaBuinyDchSyYmDRct\nhWPLPF22G2mwuo+k55kgHUylJRtZoa1A0kI0bAdGPRnSszQuYFE90yUdepoznzKHWtLRDmsglZY8\ncHZTE7UVCGqEpmtDScZZLK20wEh7LKpst9YBKQUf1A5mhovWCefMuU9eM9JbWnEQMAIY/DQOXLr+\nmqmHXkfIdj18YpSRByuFa6+2F1f+cgXkuWb3zfZdN6Twt/DCQuKpsgmVDQIXy9vPAmMB1krPRf9e\nryot/TpsXL4fG7BSuNa7Ssu4gNOvnmMFVtqY9azS0q/DxuX7sQRrXIy7kevJwNrIvV9i2xlYJRp3\nIxfNwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3Ixf9d0NIelzdt4X5\nAAAAAElFTkSuQmCC\n", + "text/plain": [ + "<IPython.core.display.Image object>" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from IPython.display import Image\n", + "import base64\n", + "Image(data=base64.decodestring(\"iVBORw0KGgoAAAANSUhEUgAAAJYAAABkCAYAAABkW8nwAAAO90lEQVR4Xu2dT5Dc1J3Hv+YQT8VJZUhVdprLWs4FTSrGGv4ql9CuHBCH4GaTFCLZwnIcjOAy8l6Q/1SlU4XHcg6xJgtY2OOik2KxSGoTGWrXzYFC2T2MDAtWitRavmQ0e9k2SYGowom4hNRPtqA9TE+rW3/cPfPepcfup6f3fu/Tv9/T+/PVpo8//vhjsMQsULAFNjGwCrYoKy6xAAOLgVCKBRhYpZiVFcrAYgyUYgEGVilmZYUysBgDpViAgVWKWVmhDCzGQCkWGEuwrly5gtf++zW887/vYOn/lnD5T5cT40x9ZQrb/nEbxDtFiHeI2LJlSylGY4X2t8BYgUVAvfzqy3i5/TI+vPLhmq37wpYv4AHpATxw3wMMsP4cFJ5jbMAiqA4eOYg/Lv8xMcL26e34+vTXk8+vbv1q8n/03TsX38EfLv4h+aRE380dmmNwFY7O2gWOBVgE1Y/2/yjxUls+vwXaY1oS7tZK3v94MJ8zceUvV0Dea+H4AoOrQrhGHqxuT0Xjp0P7D2HqH6Yymejyu5dx5PiRZBxGnmt+bj7TdSxTfgv0ASuAzglwmyE8pfbZu3VaEDkDdT+AweevzGolvPjvL+LMb84knmr+yHxmqNKyCK7ZQ7OJ5yIo+3m6clqx8UrNB1bso2W64FQN9cnijdcdAvNAQWGRPBcLicX3Ua8S84FVcj3PnjuLhRcWkgH63OG5XHc7+NTBZEBP47NvffNbucpiF/e3QCaw2g0NfNvES5c+wtQ9u2G0LCj8BLAiFEaeBU0zYJ9fxkfYjKl7FZgtCzIHIA7QUmXov/g9LmMztt6rwLBMyFROj3TkZ0fgveXh4X96GN//zvf7t2aNHGlI7VlW0pYmRC+AKUwAsQu5thOuvIjQEjGBGJ7CQYptdOw6etc6VzXXzcUZwJrGseWt2P28DV2I4OgyDgQKFgMTYtQ1xqq10eDuR6j8Fi1NxGTkwpAfRos7h05bQscQIFgibEeHMBHCVhs4EBtY8lQQd6ulvbN78e6f302mC7Z/bXsuo9NkKk1X9PZ+IUyeR0sN4GscYl8DPzOP5VuPYynQwMU+dL4O3wzRbpQQ93O1bvQuzgRWS0p/tQA6Nuqcilq7A5u3Px28T7qw7BB1VUHqhEKTB2+pCAIVHZVD3dPgujpE6peOBzesQRS5nr/+b//g24nF7JN27qkCGq/J++RknHXm5JlVeiKGr/MQPQMdV0ZkCRBbNUwEMYzQhRyZEHgHOv29ynPM6HXtja1Rf7B4AZ7RgZv+SuMAOj+NtrYEX3avfyqMfDi2DdcLEAQBvPOX8MGtR3Ex0MEFJiRxP373wWZsvaeBhixDVRrg1/jxlwEWPV3ap+xVrR57Cjgpht2xEDV4mLIFvqkiaoUwwzp4U4Hv9/awN7YrR+vuGcAS4ZsdtKV0VNEFVqMLrIkWJGEPPP4hKA0RgiCAc1XsdJQErGQ2Ig7hOQ5sx4Hz0u+wvHX2akjtMWCpNhQCiCicq+AcCx1Fh9B2IegcNN6B4Teg1z0EeknzKqPFRe7a9AeLm4ajXvzUoJEDqUahMESrKxSqbQHbDBGLoXUNlBiuUsNOT8fFQEVsNdHmdOjStTgSGOCnLTQuBDBosLxKqnTwntw/glPnoHMS4E6iFVjgbBGcwUGMPAjtawP73GZf/wVkAutYtAvPezYUPoKjipBdGZ5vQOgavGteHbfsiXD09TZUIUbg6JD3vITlrU/iYthErPOYaQk44ZhocDF8U0HDqsEOHfQaC7/2X68lyzJVTjd0WiJu2XMem++7+tAxSd52+hguTe3GYtjq6V3XPyqDtbA/WLyAtqRg0rHhLceo3avCsk0kjqd7uoEL0FJkaC/9Hh/gS9ixS0dTCaDKHVidNhoTNN2gQP/FedAmly/t2IWm2YK2xswqDbj3antzz5oToD/915/i5smbcdo8vfaDQGiC37YfEyeW4KtcMu2g1HbCrp9Dx5Fw3ZCw04ZSb0Jse6CsLH1qgZFfK0znn+hpznzKHGpJRzus4YJ/AX/78G94ofUC7r777pwMxAhdE6pyAK8u78CJJZ+BtcKiIw8Wea0DTx34ZCH5oHYwM1y0TjhnziXbaWgB+4cP/RCPPfYYtm/fjpMnT+Kmm24aDrDYhdpoQdAbaMtNSB4Da6UhRx4sqnB3SCTPNbtvtu9iMoU/Wg5Kt9p0h8DTp09j3759ePrpp/H4448PB1fylOtC5jTUGVifseFYgJXClXou+jcN6Gk2nj7JG1Gi7TG0Hkiz7OlGP/ru6OGjq46rnnjiCSwuLibe66677hocMAZWT5uNDVgpXGfbZ5OtybQNZq1EE6G0NXmXtGvNwbrv+4n3uu222wYPjwys9QFW2goKjbQ4Tdth6CAFeSpK5J3oQMUwhynS8PjMM89AVdVs3ouBtb7Aytbrw+WiMZfnednCIwOLgTUIZml43LFjB5577rnhnx4Huek6yztWY6yqbb+wsJBMTwwUHquu5Ijej4GVoWMoPJ4/fz7xXkM9PWa4x3rLwsDK2KMXLlxIvBeFR5qe2LRpU8YrN2Y2BtaA/U7hkaYnnn322exPjwPeYz1kZ2AN2YtpeCTvdeeddw5Zyvq9jIGVo28pPJL3ok2NLDxeb0gGVg6w0kvT8HjixIlkHJY1lauaE8GRangwsvD/noKqt+kzsLJSkCEfzdi/8cYbifdaKzxWoppDmxJ5FT54NH06YZShAQVmYWAVaEwqKg2PMzMzyfTEyqfHqlRzAoOH6OqwJnXoNQeBSWcjq0sMrJJsferUqSQsdofHylRzYg8aLyG0QtiTOvhGhFZglyKD0Mt8DKySwEqLpfD45ptvYn5+Hr/+z19/sukwj2pOP72vyJXBy4BNME340Pg6AiNAu8IDkQysksGi4t9++2189wffxee++DkIO4TcqjlrSw504Eg81FobYetq+KOwKDgagjVOnRdtBgZW0RZdpbw0BL73/nv4yZM/6bv7tVeVxkk1h4FVAVgbUTWHgVUBWGUcvCVV6EP/cuiztQ9NCNsMiIshrPSIeaK3oUNIlXQqaDMDqwIjlyEV0Fv6MoQlbENT/FTIhWSXOF2AF5jocei8cCswsAo36WcLLEPchO7yyr+9smrt6TQ3geQmcgcd2CQbIHoIDKGyuSwG1joEi06oU+jj3RAWR2HQgFiiTuxqJmRgVQBWGaGQDo78/OjPe9T+qpfSeBeeqIM3JPip4k8F7aVbMLAqMHSlg/dr7YkcCZxWg1Jz0G5UL7/EwKoArBuhmoNEbupBvPrRDhxf8qFVLFrCwKoArFQi4P3o/VwTpCmgdBi3r2oOIrQbNdwfGljytZ46r2U1n4FVlmW7yn3rrbfwvX/+XrKkMyPM5FLNIS2KbCrSNI8loKX48G6AxhIDq2SwaIcDgWWaJn71H78qRDWnlxbF1aaQxJILj6TRjRhm0L4hYrwMrJLAos1+BBXtyaLty5SKVs1Zverx1RB4dhIPPe/CVioeXF2rFAOrYLDIOxFQd9xxRwLVytSt90XfFaGaU3ATCimOgVWIGa8WkoY9AorA6pUIrqJVcwpsRiFFMbAKMONqYS9LsWWo5mS5bxV5GFg5rExhj8ZPdHBitbCXo+ixv5SBNWQXpmGPvNXtt98+ZCnr9zIG1oB9O2zYG/A2Y5+dgZWxC1nYy2goNt2Q3VA0jqIDESzsZbcZ81hr2CoNe/T56KOPZrcqy8m2zazGAAt7+X8ZzGOtsCELe/mhohLGEqwyVFpY2CsGqLSUsQKrDJUWFvaKBWrswCpDpYWFvXKgKiYUxh5U/huwhd8idBqYRARX4bHTldd8Le8gTSpapYWWX0is47qnveTdi02I6aFOejlAbSdcOT2fF8NTOEixDTqnV6Uk0CC2GpW8hYTCyFXA72yj8XoAAzoE+nsxgNnrZc8DtL7bU9HJlDwqLY9855FkbY8ktS3LWlGLECbPo6UG8DUOsa+Bn5nH8q3HsRRo4GISL6vDN0O0e70SdoB2rfeshYBF71Juyzzu90TcF59FIC8WJvSVvgiT9nnPH5nP/K7CtOPonYWzh2aTF2Fu+usmvPjLF3us7cXwdR6iZ6DjyogsAWKrhokghhG6kCMTAu9Ap7+r1l0cQwoLAote4+ugwT+IsxO78XrQKkTkqzsEkqeily8Nk0il5cfHfowv3/xlLBxf6Pk2sNhTwEkx7I6FqMHDlC3wTRVRK4QZ1sGbCnxfrfxgwjBtvtHXFAZW7OsQZo7hEm7Fkxf8nm+mH6TBlau0RG00OBWcY6Gj6BDaLgSdDn46MPwG9Hr15/MGsdco5S0GrDiAIU7D5M/AgIo9gY6Lng4+5wi3jIOea59wieCQzgEnAe4kWoEFzhbBGRzEyIPQDmBWpaoxSpQMUZdCwCLh1OlmDWcCBzJsSNzDiIyL8LR8Ur1lHE2nPeZzh+d6mooENW7Zcx6b7zuHTlvCJB1Nnz6GS1O7sUhKxDl/LEP00Vhekh8sUjThNUyYAdxr59dCSwSvAWbg5Xq7exkqLfRO6TMnz/TurNAEv20/Jk4swaf2xC6U2k7Y9XPoOBIm6crYh6UoaLodABOoSU3YlpLbQ48lQT0qnR+sEq1RBlj0dGmfsnPVOtB51IMmfEdGLQ7RkkSYkps8VbJ01QIjDdaNCIVZwOi4DnxOgsRRXIzhazwakY3gmphsljLWe56RBqv6wfvg3R0HFqS6CcHxC5kQHrwGo3nFSIN1Q1RaBuinyDchSyYmDRcthWPLPF22G2mwuo+k55kgHUylJRtZoa1A0kI0bAdGPRnSszQuYFE90yUdepoznzKHWtLRDmsglZY8cHZTE7UVCGqEpmtDScZZLK20wEh7LKpst9YBKQUf1A5mhovWCefMuU9eM9JbWnEQMAIY/DQOXLr+mqmHXkfIdj18YpSRByuFa6+2F1f+cgXkuWb3zfZdN6Twt/DCQuKpsgmVDQIXy9vPAmMB1krPRf9eryot/TpsXL4fG7BSuNa7Ssu4gNOvnmMFVtqY9azS0q/DxuX7sQRrXIy7kevJwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3Ixf9d0NIelzdt4X5AAAAAElFTkSuQmCC\"), embed=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "Ft95NDUZy4Rr" + }, + "source": [ + "That's equivalent to the function $\\hat{y} = w_2 x + w_1$. What we're trying to do is find the \"best\" weights $w_1$ and $w_2$. That will give us that green regression line above.\n", + "\n", + "What are the best weights? They're the weights that minimize the difference between our estimate $\\hat{y}$ and the actual y. Specifically, we want to minimize the sum of the squared errors, so minimize $\\sum{(\\hat{y} - y)^2}$, which is known as the *L2 loss*. So, the best weights are the weights that minimize the L2 loss." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "RHDGz_14vGNg" + }, + "source": [ + "## Gradient descent\n", + "\n", + "What gradient descent does is start with random weights for $\\hat{y} = w_2 x + w_1$ and gradually moves those weights toward better values.\n", + "\n", + "It does that by following the downward slope of the error curves. Imagine that the possible errors we could get with different weights as a landscape. From whatever weights we have, moving in some directions will increase the error, like going uphill, and some directions will decrease the error, like going downhill. We want to roll downhill, always moving the weights toward lower error.\n", + "\n", + "How does gradient descent know which way is downhill? It follows the partial derivatives of the L2 loss. The partial derivative is like a velocity, saying which way the error will change if we change the weight. We want to move in the direction of lower error. The partial derivative points the way.\n", + "\n", + "So, what gradient descent does is start with random weights and gradually walk those weights toward lower error, using the partial derivatives to know which direction to go." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "W7SgnPAWBX2M" + }, + "source": [ + "## The code again\n", + "\n", + "Let's go back to the code now, walking through it with many more comments in the code this time:" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 718, + "status": "ok", + "timestamp": 1446659172854, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "4896c353dcc58d9f", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "4qtXAPGmBWUW", + "outputId": "0664707f-ea8a-453b-fc3f-48d5ca0f76dc" + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlUAAAEPCAYAAABr+zG+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xl8VPW9//HXJwtLIKwBQVAEEdlETBUXLM2VKlZR22uv\n2NpatNdb6lqtGm17L1j7uwgu1KU+qPe617Vqi3LdEIm4VFHjggNaXNjXsIU1ZPn8/pgJjiEJSWY5\nk5n38/GYR2bOnDnnM0K+fviez/l8zd0RERERkdhkBR2AiIiISDpQUiUiIiISB0qqREREROJASZWI\niIhIHCipEhEREYkDJVUiIiIicRCXpMrMrjezkJl9bGaPmFmbeBxXRCRRzKyvmb0aGbsWmtnlke1d\nzexlM/vMzF4ys85Rn7nezJaY2WIzOyW46EUkFcWcVJlZP+Ai4Ch3HwHkAOfGelwRkQSrAq5y92HA\n8cAlZjYYuA54xd0PB14Frgcws6HAOcAQ4HvA3WZmgUQuIikpHjNV5cAeoIOZ5QB5wOo4HFdEJGHc\nfa27fxh5vh1YDPQFzgIejOz2IPD9yPMzgcfdvcrdlwJLgFFJDVpEUlrMSZW7bwZuBZYDq4At7v5K\nrMcVEUkWMzsEGAm8DRzg7usgnHgBPSO79QFWRH1sVWSbiAgQn8t/A4ArgX7AgUBHM/txrMcVEUkG\nM+sIPAVcEZmxqrt2l9byEpEmyYnDMY4G3nT3TQBm9gxwAvBo9E5mpoFJJE24e1rUEkVKFp4CHnb3\nWZHN68zsAHdfZ2a9gPWR7auAg6I+3jeyre4xNdaJpInmjnXxqKn6DDjOzNpFijbHEq5NqC+4lHhM\nnjw58BhSKQ7FktpxpFosaeY+YJG73x617VlgYuT5z4BZUdvPNbM2ZtYfGAgsqO+gQf8ZpdPft0yM\nPx2+Q2uP371lY13MM1Xu/pGZPQS8D1QDHwD3xHpcEZFEMrPRwHnAQjP7gPBlvt8A04AnzexCYBnh\nO/5w90Vm9iSwCKgELvaWjrwikpbicfkPd78ZuDkexxIRSQZ3fxPIbuDt7zbwmanA1IQFJSKtWkZ2\nVC8qKgo6BCB14gDFUp9UiQNSKxZJf63971trjx9a/3do7fG3lCVr9trMNFMukgbMDE+TQvVE0Fgn\nkh5aMtZl5EyViIiISLwpqRIRERGJAyVVIimuoqqixbf3iohI8iipEklhVTVV3LXgLhasqrcdkoiI\npBAlVSIpyt154MMH6NCmA8f0OSbocEREZD+UVImkqGcWP8OmXZv4+VE/J8v0qyoikuo0UoukoHlf\nzeOjdR9xyTGXkJudG3Q4IiLSBEqqRFLMB2s+4MXPX+TyYy+nQ5sOQYcjIiJNpKRKJIV8sekL/vLx\nX7j4mIspyCsIOhwREWkGJVUiKWLd9nXMfG8mFxx1Af269As6HBERaSYlVSIpoLyinDveuYPvD/4+\nw3sODzocERFpASVVIgGrqKrgrgV3cVzf4xh98OigwxERkRZSUiUSoBqv4Z7376Fvp76MHzQ+6HBE\nRCQGSqpEAuLuPPLxIzjOeUech1mzFkMXEZEUo6RKJCDPL3me5VuX84tv/YLsrOygwxERkRgpqRIJ\nwFsr3uLNFW9y2bGX0TanbdDhiIhIHCipEkmy0PoQzyx+hsuPvZxObTsFHY6IiMRJXJIqM+tsZn81\ns8VmFjKzY+NxXJF0s3zrcu774D4mHT2JXh17BR1ORjOze81snZl9HLVthJm9ZWYfmdksM+sY9d71\nZrYkMs6d0tix3RMZuYikKvM4/Pab2QPAa+5+v5nlAHnuXl5nH4/HuURKS0t54omXAJgwYRyFhYUB\nR9Q0G3duZPqb05kwfAKFvVtHzPUxM9y91VfVm9mJwHbgIXcfEdm2ALjK3d8ws4nAAHf/LzMbCjwC\nHAP0BV4BDqtvUDMz377d6aAVhkRatZaMdTHPVJlZJ+Db7n4/gLtX1U2oROKltLSUSZNmMH/+KObP\nH8WkSTMoLS0NOqz92rFnB3e8cwfjBo5r1QlVOnH3N4DNdTYfFtkO4cTp7MjzM4HHI+PbUmAJMKqh\nY2/bFudgRaRViMflv/5AmZndb2alZnaPmbWPw3FF9vHEEy+RnT2RgoKxFBSMJTt74t5Zq1RVWV3J\n3e/ezREHHMFJ/U8KOhxpXMjMzow8P4fwrBRAH2BF1H6rItvqpaRKJDPlxOkYhcAl7v6emf0RuA6Y\nHIdji7Rq7s59H9xHl3ZdOHvI2fv/gATtQuBOM/tP4FlgT0sOcsstU+jdO/y8qKiIoqKieMUnIglS\nUlJCSUlJTMeIR1K1Eljh7u9FXj8FFNe345QpU/Y+10AjLTFhwjjmzZtBWVn4dXX1A0yYcGWwQdUR\nCoWYPWc2ALlDcqnJr+GKY69otc094zHQtBbu/k9gHICZHQacHnlrFXBQ1K59I9vq9eMfT+E730lU\nlCKSCHXzkhtuuKHZx4hXofprwEXu/k8zm0y4UL24zj4qVJe4eOSRR7jjjicAuPzyCZx33nlxPX4s\nhfChUIhrb76W3GG5rK9ez4a1G3jsgsc4+sij4xpjkNKlUB3AzA4BnnP3IyKve7j7BjPLAu4H5rn7\nA1GF6scSvuw3h0YK1Z97zhmvVYdEWrWWjHXxmKkCuBx4xMxygS+BC+J0XJFvKC0t5fbbXyQ7+woA\nbr/9AYYMGbI38Yn1zsDaQvjs7IkAzJs3g5kzr2zycWbPmU3usFyyDspi55adHJZ1GHPnzU2rpCpd\nmNmjQBHQ3cyWEy5ZyDezSwAHnnH3BwDcfZGZPQksAiqBixv7V6JqqkQyU1ySKnf/iPCtxiIJFV2o\nDlBWFt5WWFgYc0K0v+M31faa7WzYvIHhPYezY9uOJn9Oksvdf9zAW3c0sP9UYGpTjq2kSiQzqaO6\npI1UuDPw2BOP5ct1X1JQXsCOL3dQGapk/Mm6DpRplFSJZKZ4Xf4TSYpEF6rHcvwtu7fw8taX+cP3\n/8CK0vDd9+OvGc+wYcPiFp+0DkqqRDJTXArVm3QiFapLnDRUN1X38l919QPNvvzX2PEbs7tqN7e8\ndQuFvQs57bDTmnW+1iadCtUTwcz8qqucW28NOhIRiUVLxjolVZJWgljCprqmmrsW3EX3vO6cd8R5\nrbZ1QlMpqWqcmflFFzn33BN0JCISiyDv/hNJCYWFhQlLpOpL2Nydhz9+mOysbH58xI8xs1a7NqHE\njy7/iWQmJVUiTdDQnYUrO6xkzbY1XHX8VWRZVlzuQJTWr1yrn4pkJCVVIk1QX6uFW/46kwEn96R4\ndDFtc9o2uF9zWzJI66eZKpHMpJYKIi2ws+OXfJ79KZcfezn5bfODDkdSjJIqkcykmSqRJpgwYRzP\nPz+ZVRsfp6rDNvYMfZf7xs2gZ4ee++yX6msTSuIpqRLJTEqqROoRXWx+9NED+cd7/2Bb3kJqRi5k\nV8E2Dll7MId2O3SfzxUWFjJz5pVRheqqp8pESqpEMpNaKojUEV1svnPnUpbt/C+6DWzLjn472N1t\nN0f1O4ou5V04Pvd4in9VvN/jpRu1VGicmXn79s7OnUFHIiKxaMlYp5oqkTqii833ZJWRNfgQdloF\ne7ruoW37tuzatCvoECXFVVRAVVXQUYhIsimpEtkPx/HDHVtlZK3MYtvybVrTTxrVsSNs3x50FCKS\nbEqqROqYMGEcZWU38nrpKFasepid2z8mv2MeI3qMoMsnXTi1y6lMv2a61vSTBuXnq65KJBOpUF2a\nJRO6hb/55pus3PMO1X2q4VDDKmDUulEcOfRIxl+oBZJl/5RUiWQmJVXSZJnQLTwUCvH7//k9NSfW\nkNM9h6rtVXSo6cjmTZszsihdWkZJlUhmUlIlTZYJ3cJnz5mNHW5YX6Mmv4bcmlwqP6+Envv/rEgt\nJVUimUk1VSJ19OnXh6rKKtgGVZurqFpWRfuaAoqLp1JaWhp0eNIKKKkSyUxKqqTJJkwYR3X1A5SV\nzaWsbG6kW/i4oMOKq29/59uUV5dzePbhdPqyE3kf5NHDh7NmzUTmzx/FpEkzlFjJfimpEslMcbv8\nZ2ZZwHvASnc/M17HldSRrt3CQ6EQs+fMptIrWdt3LcXji9m8cDP0gX92WsOiRWek9SXPTGVm9wLj\ngXXuPiKy7RjgT0AuUAlc7O7vRd67HrgQqAKucPeXGzq2kiqRzBTPmqorgEVApzgeU1JMYWFhWiUU\noVCIa2++luyh2XxV9RU5JTlM+uUkho8dDkBx8dSAI5QEuh+4E3goatt04Hfu/rKZfQ+4GfgXMxsK\nnAMMAfoCr5jZYQ0tE5GfD+XliQ1eRFJPXJIqM+sLnAb8P+CqeBxTJFGi20LsqllDztActvTYQmfv\nTPec7vzfK//H8OHhpEoLJKcvd3/DzPrV2bwG6Bx53gVYFXl+JvC4u1cBS81sCTAKeKe+Y2umSiQz\nxWumagZwDV8PRiIpqW5biBUbrqbLuZV06NqBI3oeQdnGsm/sn66XPKVB1wFvmtmtgAEnRLb3Af4R\ntd+qyLZ65efDunUJi1FEUlTMSZWZnU64JuFDMysiPBDVa8qUKXufFxUVUVRUFOvpRZrliSdeoqKi\niD1Z7wFgvY9i1bJnGd3pOMq2lIWXn7nmm8vPpNslz+YqKSmhpKQk6DCS5V7gMnf/u5n9ELgPOLm5\nB3n11SksWwZTpmisE2kt4jHWWQMlAU0/gNl/Az8hXLzZHsgHnnH38+vs11D5gUjS/Pznv+KZ+W/T\n9sheVLbbyo49nzHWRzNm9NEAjD9ZHdP3pyUrt6eqyOW/56IK1cvdvVPU+1vcvYuZXQe4u0+LbH8R\nmOzu+1z+MzN/8knn8cfh6aeT9EVEJO5aMtbFPFPl7r8BfhMJ4DvAr+smVCK1YlnmJh5L5HToVgOH\nr6Sqex678pfR5ssaDj20t7qlZy7jm7PrS8zsO+7+mpmNBZZEtj8LPGJmMwhf9hsILGjooJ06qaZK\nJBOpo7okTSzL3MRjiZxQKMSSr5bQqWc22woWcpAdQO8hB8Oer+/yS9f1DGVfZvYoUAR0N7PlwGTg\nP4C7zawNsDvyGndfZGZPEr7DubbVQoNT7ypUF8lMcU2q3P014LV4HlPSRyzL3MS6RE5t64Qd/XZQ\ntrGMNmva0LfvAVQsrGDe51/RseNZQHquZyj1c/cfN/DWsQ3sPxVoUo8NJVUimUkzVZIRZs+ZTdbQ\nLHZ33c3IbSPZ8sYW2pS34ZBuR7Coo5p7SnwpqRLJTEqqJGla2vOptLSUtWtXsXTpf7N9+w46duzQ\n7H5RNV7DsqpldMvtxqCBg1hfvZ7jc49n05qaln4dkQYpqRLJTEqqJGla0vMpupaqR4/hbNhwE9/6\n1jFcfnnTL9G5O1mDs6h6oYpOOZ1Yv2H93tYJFRUVau4pcVebVLmDpcV9kiLSFDG3VGjyidRSQVqg\nuHgq8+ePiro8N5cxYxYwbdr1TT7GC0te4P0173Nal9OY8+oc4JutE+JxV2EmSaeWColQO9a1bQtb\nt0K7dkFHJCItEUhLBZFU9vbKt3l9+etcO/paurTrQuGIfROmTG/uKYlRO1ulpEokc2QFHYBIYyZM\nGEd19QOUlc2lrGxu5PLcuCZ9dvGGxTy16CkuG3UZXdp1SXCkIt+kuiqRzKOZKklpza3DCoVCzJ4z\nm601W1ndazW/Hfdbeuf3Tla4InspqRLJPEqqJOU19fJcbS8qH+osqVxCz5Ke7DlyD3RPQpAideTn\nQ3l50FGISDLp8p+kjdlzZmNDjXWd1zHokEH0OKwHs+fMDjosyVCaqRLJPEqqJG1UezVLq5bStX1X\n+uT3CTocyXBKqkQyj5IqSQvuTuXASmyN0WFtB9Z/FulFdfL4oEOTDKWkSiTzqKZK0sIzi58hr1se\nD//Hw7w0N1zUPv6ar3tRiSRbp05KqkQyjZKqNJVJDS1f/epVPl73MdeOvpYObTow8oiRQYckopkq\nkQykpCoNRS/tAjBv3gxmzmz6si6pbtasWcx8eCYAp5x9CmsL1u5NqERSRX4+bNgQdBQikkxKqtLQ\nE0+8RHb2xKilXcLbmpJUpfoM16xZs7hoykVkj8qmKruK155/jTvOvIPueeqbIKklPx++/DLoKEQk\nmVSoLnvVznDNnz+K+fNHMWnSDEpLS4MOa69QKMQ1N17DrvxdtOnRhppDasjrlMfTTzwddGgi+9Dl\nP5HMo5mqNDRhwjjmzZtBWVn4dXhplyv3+7lYZrgSrbax55b+W6hoV8GaVWsosAK8Sot0S2pSUiWS\neZRUpaHmLu2Saupegmzbti1XT76aZT2Wcejhh/Le2vdgF2x7bRvtt7Vn0pRJAUcssi8lVSKZR0lV\nmmrq0i7RWjrDFU91i+yff34ynQduZrNtZlPNJtaUr+GQnoew5cMtdN3ZlZun3MxZZ52V1BhFmkJJ\nlUjmibmmysz6mtmrZhYys4Vmdnk8ApPkq53hGjNmAWPGLAjkjsHoS5AFBWPZWtGLdZ03MeSkIVTu\nrKS6opqclTmM6jCKvz34NyVU0mJmdq+ZrTOzj6O2PW5mpZHHV2ZWGvXe9Wa2xMwWm9kp+zu+kiqR\nzBOPmaoq4Cp3/9DMOgLvm9nL7v5pHI4tSdaSGa5k2N5lO71H9Kb67WoOsUOYfsN0NfaUWN0P3Ak8\nVLvB3c+tfW5mtwBbIs+HAOcAQ4C+wCtmdpi7N1jUp6RKJPPEnFS5+1pgbeT5djNbDPQBlFRJs9Ve\ngvznP99k3fbZVFV+RcGmDny57EsOyz0M8mH6NUqoJHbu/oaZ9Wtkl3OAosjzs4DH3b0KWGpmS4BR\nwDsNfVhJlUjmiWtNlZkdAoykkYFGWqdk9a8qLCykqKg3M578A1ZotC1ow9pd2zhjxRkcPfhoLT0j\nSWFm3wbWunttp6k+wD+idlkV2dagjh1h506oqYEsNa8RyQhxS6oil/6eAq5w9+317TNlypS9z4uK\niigqKorX6SWBktmhPRQKcf9z95M12sg5JIfdlbvJ35jP9rLtFP+qOO7nk/0rKSmhpKQk6DCS7UfA\nYy39cO1Yl50NL7xQxOmnF8UnKhFJmHiMddZISUDTD2KWA8wGXnD32xvYp7HyA0lhxcVTmT9/VFT/\nqrmMGbOAadOub9HxGpv1mvbHadz6t1vZNngbfpCTtTuL7M+yOdFP5IWnXoj9y0jMzAx3t6DjiIfI\n5b/n3H1E1LZswjNRhe6+OrLtOsDdfVrk9YvAZHffZ1Y+eqzr3Rveew/6NDqnJSKpqCVjXbwmpe8D\nFjWUUInUakrX9oOHHkzljkp8rVP9VTX+kTPpp+pFJQlhkUe0k4HFtQlVxLPAuWbWxsz6AwOBBfs7\neKdOqqsSySTxaKkwGjgPOMnMPojcinxq7KFJqpgwYRzV1Q9QVjaXsrK5kf5V41p0rLotE7KzJ+6d\ntQIYN3Yc29tsp39Bfzov7Eyn0k5MvWyqWidI3JnZo8BbwCAzW25mF0TemkCdS3/uvgh4ElgEPA9c\n3JSpdxWri2SWeNz99yaQHYdYJEUlq0O7u/NOxTv86Ls/ou3nbbFDjPEnB1OYnuoLS0vs3P3HDWy/\noIHtU4GpzTmHkiqRzKKO6tIk8epf1VDXdnfnr4v+yvY92/nt6b8lJyu4v5rJLMyX9KakSiSzKKmS\npGpo1uuVL19h8YbFXDP6mkATKkjthaWldVFSJZJZlFRJ0tWd9Xpv9Xu88uUrXDv6WvJy8wKMTCS+\nlFSJZBYlVZJ0oVCI2XNmA3DEcUfwavmr/Oq4X9GtfbeAIwtLhYWlJT0oqRLJLEqqJKlmzZpF8W3F\nZA3NoluPbvz5yT/zx3/7I3079Q06tL2SVZgv6U9JlUhmUVIlSRMKhSieXszW4VvJ7ZPLyp0rGdRx\nEIvfWcyZx58ZdHjfkKoLS0vrkp8Py5YFHYWIJItWpJKkmT1nNtkHZNOmYxt2td1F+/bt2blhZ9Bh\niSSMZqpEMouSKkmqHv17UL6pHC9zbKVRs6iG8SePDzoskYRQUiWSWZRUSdKc/t3T2VSxiR4H9KDn\nkp50+aQL066aFkhzT5FkUFIlkllUUyVx1Vgn8i9yvuCkfzmJHst7kNMrJ7Bu6SLJoqRKJLMoqZK4\naawT+evLXufd1e9y4/gbyW+bH2icIsmSnw/l5UFHISLJoqRK4qahTuQ5fXJ49rNnuWb0NUqoJKN0\n6qSZKpFMoqRK4m7bthDrts5mx47P+WpzNQ9++CCXjrqUnh16Bh2aSFLp8p9IZlFSJd/QWE3U/kyY\nMI5nnrmKlTu/hJ5G9sE7eSunLbd3vJ3+XfsnKmSRlKWkSiSzKKlKA7EkQnWP01BNVFO0bduWym5L\nyTp6M1l52ThV9OjYl8/f/RxGtygkkVatbVuoqYE9e6BNm6CjEZFEU0uFVq42EZo/fxTz549i0qQZ\nlJaWtuhY0TVRBQVjyc6euDdZa4rZc2aTV5hHwbButB/chryCPPZ8vqdFsYg0lZkdamZtI8+LzOxy\nM+sSdFwAZpqtEskkSqpauVgToXjr1aMX5dvLqdldQ9aWLKrXVau5pyTa00C1mQ0E7gEOAh4NNqSv\nKakSyRxKqgQIz3itW7eWpUv/k2XLZlJWNpfq6geYMGFck49x+ndPZ/PGzRS0LaDH6h50+bQL065V\nc09JuBp3rwJ+ANzp7tcAvQOOaS8lVSKZQzVVrdyECeOYN28GZWXh1+FE6Momfba2Fmvt2lW8//5q\n8vMvoaBgLBs23M5RRw3hiiuaXk8FsKrdKkYXjabXyl7kds1l/KVq7ilJUWlmPwJ+BpwR2ZYbYDzf\noKRKJHPEJakys1OBPxKe+brX3afF47iyf4WFhcyceWVUoXrTEqHoovRly56kvPwUCguP5pBDOtOx\nYwd69VrQrIRqwaoFzFs6jxvH30jX9l1b+nVEWuICYBLw/9z9KzPrDzy8vw+Z2b3AeGCdu4+I2n4Z\ncDFQBfyfu18X2X49cGFk+xXu/nJTglNSJZI5Yk6qzCwLuAsYC6wG3jWzWe7+aazHlqYpLCxs9h1/\n0bVYGzYsoLy8B6tXr6dz585N+nwoFGLabdN4+6O36di3IwefcjB3//RuJVSSdO6+CLgcwMy6AvlN\n/Ifd/cCdwEO1G8ysiPBs1xHuXmVmBZHtQ4BzgCFAX+AVMzvM3X1/J1FSJZI54lFTNQpY4u7L3L0S\neBw4Kw7HlSQ58MBxwEPs3PlWk2qpQqEQ5196Pk+88QRLhy3lo4M/Yt5f5vHuq+8mL2iRCDMrMbNO\nZtYNKAX+x8xu29/n3P0NYHOdzb8EborUaOHukQvrnAU87u5V7r4UWEJ47NsvJVUimSMeSVUfYEXU\n65WRbZLCJkwYR3X1A5SVzaWycjP9+lVyyimljBmzoNHeVKFQiKsnX82nyz6FYZB1WBZtu7elZkAN\nMx+eCYQvLRYXT6W4eGqL2zuINENndy8H/hV4yN2PBb7bwmMNAsaY2dtmNs/MvhXZXnecW0UTxzkl\nVSKZI6mF6lOmTNn7vKioiKKiomSeXqLsW4v1h/1eQgyFQlx787Us67GMyqGVVLapJHtDDlkdHas2\nIPYGopJ6SkpKKCkpCTqMxuSYWW/Cl+d+G+uxgK7ufpyZHQP8FRjQ3INEj3WbNhWxbVtRjGGJSKLF\nY6yzJpQENH4As+OAKe5+auT1dYDXrWkws6aUH0iKqp2hWtZjGX2G9uGNL95g9/rd4YsgfXPILjWm\nX3ET69ZVMH/+qKhFlecyZswCpk27PtgvIHFjZri7BR1HLTP7N+A/gTfd/ZdmNgC42d3PbsJn+wHP\n1Raqm9nzwDR3fy3yeglwHHARgLvfFNn+IjDZ3d+p55jfGOumToWtW+Gmm2L8oiKSVC0Z6+IxU/Uu\nMDAyOK0BzgV+FIfjSoqonaFaXrOcTTWbWLtqLXm5+VSt64AthY5rB9Mj7xTWrasIOlTJQO7+V8Iz\nSrWvvwT2m1BFWORR6+/AScBrZjYIaOPuG83sWeCRSK1WH2AgsKApJ+jUCVas2P9+ItL6xVxT5e7V\nwKXAy0CIcDHn4liPK6lj9pzZ5A7LZejYoVRtq6KyshL/3Om4fCAnDHuN0SPfoFu38OJ+0bVaLWkg\nKtJcZtbXzP5mZusjj6fNrG8TPvco8BYwyMyWm9kFwH3AADNbSLgr+/mw9w7DJ4FFwPPAxU2deldN\nlUjmiEtNlbu/CBwej2NJ6trddTc9R/bEFzjddnSjvF03KirWUlGxdm/T0Zb2zRKJwf2EE6B/i7z+\nSWTbyY19yN1/3MBbP21g/6nA1OYG16lT+PKfiKS/mGuqmnwi1VS1WqFQiEm3T2Jtv7UMzB1I1qIs\npl8znYqKiqjkaVxck6fabu+JOLbEJgVrqj5095H725bEeL4x1n30EZx3HnzySRDRiEhLtWSsU1Il\n9QqFQsyeMxuAI48/kse/eJxea3vRNasr409O7PIzde8grK5+QHcQppAUTKrmEp6Zeiyy6UfABe4+\nNqB4vjHW7dwJ3buHLwHmaGEwkVYjqEJ1STO1hem5w3LZ7bv58+N/5rZ/u43v//j7STl/dLd3gLKy\n8DYlVdKACwl3Rp8BOOE6qYlBBhQtLw969YKlS2HgwKCjEZFEUlIl31DbOmF5zXIOa3cY63LX0aem\nD58t+AxOCDq62OiSYnpy92XAmdHbzOxXhNcjTQmDB8OnnyqpEkl38eioLmkiFApx8eSLed/eZ1nO\nMua+O5e8nXl0z+6e1DgScQdh7SXF+fNHMX/+KCZNmqFu7+ntqqADiFabVIlIetNMVZqJZTbm3r/c\ny6e5n5IzKIeK3RX4emfNC2vo0qcL468Zn6iQ95GIOwh1STHjpEzNF4STqne1NKZI2lNSlUZiWSKm\ntjB9R/8d5LXNo3P7zuxetZuczTlM/+P0hBam16ewsFAJj8Qipe6KGTwYHn446ChEJNGUVKWRls7G\n1BamVx9aTUV5BRXLKyjIKiB7eTbjxzZ8p19rqlGaMGEc8+bNoKws/Lq2r5a0Xma2jfqTJwPaJzmc\nRunyn0hmUFKV4aLX9Cs4soB1y9bBR1BdXs3gvoP5+fk/r/dzrW3hZDUlTT/unh90DE3VsydUV4f/\noVNQEHTQ56HpAAAgAElEQVQ0IpIoSqrSSHNnY2bNmkXx9GI27d5EZZdKVm5YydEDjmZ9xXr6bejH\nLTfc0uAsVWusUdIlRQmK2dezVSeeGHQ0IpIoSqrSSHNmY2bNmsW//+e/s/vw3eS2zWXbjm10WteJ\n9ZvX039Xf6bfkPw6KpF0pqRKJP0pqUpBsdQqNWU2JhQKUTy9mN0jdlN1YBUVVJC/Pp+cD3Po178f\n028IL0FTXDy1wRhUoyTSPKqrEkl/SqpSTDJqlWbPmU32Adl07NaRje02YnuMqh1VHNDuAG654RYq\nKir2G4NqlESaZ/BgeP31oKMQkURSUpViklWr1KN/D1ZtXkXHNh2pXFtJu8/aMe3GaQwbNozi4qlN\nikE1SiJNp5kqkfSnpCqD1PaiWrtmLavLV9OrZy+y/plFzboapt04jbPOOivoEEXS1oABsHIl7N4N\n7doFHY2IJIKSqhSTqFql6EWSV3dZTdWmKs6wM/BDnR1ds3jrrUUcdNBBFBYWql5KJAFyc6F/f/j8\ncxg+POhoRCQRzD05jYfNzJN1rtauvkL1WBttTvvjNP5R+Q+qDqxizfY19N7amwEbBvD2a5v31k5V\nVz+wt3aqNTX2lOQyM9w9pZaBSSWNjXU/+AGcdx788IdJDkpEmq0lY52SqhTRWBJTt3g9OvnZn9pL\nfiVvlFDWv4zKQys58oAj2fr5VjbP2c2ebddE1U7NZcyYBUybdn3cv5+kDyVVjWtsrLv+eujQAX73\nuyQHJSLN1pKxTpf/UsD+7vhrafH6rFmzKL6tmKyhWeQeksuStUsY6SPZum0rlaFKBh40mEWLEvjF\nROQbBg+GOXOCjkJEEiUrlg+b2XQzW2xmH5rZ02bWKV6BZZLopKmgYCzZ2RP3zlq1VG0vqq2Dt7Kt\nzza+avMVA3sNpPPSzhyfezzTr5nOJZecT3X1A5SVzaWsbG6kdmpcXL6TSKozs3vNbJ2ZfRy1bbKZ\nrTSz0sjj1Kj3rjezJZEx75SWnFN3AIqkt1hnql4GrnP3GjO7Cbg+8pA4aknh+L0P3cum3ZvYs2wP\nHAB5eXlUraqi6MQiin9VvHc/9ZqSDHY/cCfwUJ3tt7n7bdEbzGwIcA4wBOgLvGJmhzW3puHww+Gz\nz8A9vHSNiKSXmJIqd38l6uXbwNmxhZOZ9pc0NbfRZigU4vm3n6fmiBp2tNuBlRo57XOoWV/D+EvH\nf2Nf9ZqSTOXub5hZv3reqi/dOQt43N2rgKVmtgQYBbzTnHN26QIdO8KqVdC3b/NjFpHUFs+aqguB\nx+N4vIzRlKSpqclPKBTi6slXs2fAHvYU7KFju45UbqzEPjKm3TxN6/mJ7N+lZvZT4D3g1+6+FegD\n/CNqn1WRbc1WewlQSZVI+tlvUmVmc4ADojcBDvzW3Z+L7PNboNLdH23sWFOmTNn7vKioiKKiouZH\nnKbiMWNUW5i+kY3s7r0bM6Ov96W6oJpTzz61VTX3VEuH1FFSUkJJSUnQYSTL3cDv3d3N7A/ArcC/\nN/cgjY11tUnVd78bc6wiEkfxGOtibqlgZhOBi4CT3L2ikf3UUiGBQqEQZ//H2WwdvpWKjhVsW72N\nzu0707NtT/rv6s/0a6a3mlmqWFpISOKlU0uFyOW/59x9RGPvmdl1gLv7tMh7LwKT3X2fy3/7G+tu\nvx2WLIG77orb1xCRBGjJWBfr3X+nAtcAZzaWUEli1V7y27R7E5VeSdYBWXTv2Z2sL7Lot6Ffq0qo\nIDF3Q4o0wIiqoTKzXlHv/SvwSeT5s8C5ZtbGzPoDA4EFLTmh7gAUSV+x1lTdCbQB5lj4Vpa33f3i\nmKOSJqu95Lep5yZ2j9jNzm076f5pdyzX6EY3brnhlnoTKl1ek0xnZo8CRUB3M1sOTAb+xcxGAjXA\nUuAXAO6+yMyeBBYBlcDFLZ16V1Ilkr7UUb0Vi77kV1NQw8ZdG8nblEe7he3o1q4b066tf5HkVL+8\nlurxZbp0uvyXCPsb62pqID8f1q4N/xSR1JT0y38SnFmzZvGDn/2AFWtWsKd6D1Wdq+iW1412O9rx\nrf7f4ul7nm6wMD3VL6/V3g05ZswCxoxZoIRK0kpWFgwaFO5XJSLpRcvUtEKzZs3ioikXsWvYLiq2\nVrBz4046fdWJNjVt6Ly+M7fcXf8lv9ZE/bMkndVeAjz66KAjEZF40kxVKzTz4Zlkj8qmx1E9yBqZ\nRVaHLKpeqqLzJ52ZdtX+e1FNmDBOy9OIBGjIEFi4MOgoRCTeVFPVCn3vh9/jg+4fwGFQXVXNrk92\nceCnB/K3B//W5BkqFapLS6mmqnFNGeveew/OPTfcWkHL1YikppaMdUqqWqG///3vnP+/58MgyNuR\nR82CGv5nyv+0quae0nopqWpcU8Y69/AlwIcegmOPTVJgItIsSqrSWCgUYvac2QB0G9GNV5e8ysZX\nNpLt2Uz66SQlVJI0Sqoa19Sx7sYbYf16uPPOJAQlIs2mpCpN1faiyhqaRYceHdi0bRMPn/8wJxx1\nQtChSQZSUtW4po51X34Jxx0XXlw5NzcJgYlIs6ilQhoKhUIUTy9m6+CtbO2zlc+qPqNXfi9ef+31\noEMTkRgMGAADB8KcOUFHIiLxoqQqxc2eM5vsA7LJzs9md5vd5Ofls3nD5qDDEpE4+MlP4C9/CToK\nEYkXJVWtQLf+3SgvL6dNWRtqVtRQs6iG8SePDzosEYnROefA88/Dtm1BRyIi8aCkKsWdfNLJrKte\nR/+u/en2ebcm96ISkdRXUADf/jb8/e9BRyIi8aBC9RRUe6dftVdT3r+cbm27UfNpDWbG+JPHM2zY\nMPWZksCoUL1xzR3rnngC7rsPXkqdlaJEBN39lxZCoRDX3nwtOUNzWF69nMrVlTz2i8c4YvgRe/cp\nLS3l/PP/wKZN4TYK3brN4qGHfqfESpJCSVXjmjvW7dwJffrA4sXQq1cCAxORZtHdf61cKBTi6slX\n81X7ryjLL6NNQRsGHDSA5195/hv73XHHgyxdegq7dp3Grl2nsXTpKdxxx4NJjbW0tJTi4qkUF0+l\ntLQ0qecWSSd5eXDWWfD440FHIiKxUlKVImbNmsXZ/3E273/1PqtrVvPJik84MPdAsmzfP6LFi78C\netCmTfgBPSLbkqO0tJRJk2Ywf/4o5s8fxaRJM5RYicRAdwGKpAclVSkgFApRfFsxW4dvxQud7eXb\nyVmTw2fvfEZlqHKfO/0GDz4YeIg9e+ayZ89c4KHItuR44omXyM6eSEHBWAoKxpKdPXFvfZeINN+/\n/AusXg2ffhp0JCISCyVVKWD2nNlkDc0i+6BsqgdW0y2vG9kfZ9NvQz+mXzN9nzv9rrjiQvr1q6R9\n+ydp3/5J+vWr5IorLgwoehGJVXY2XHghTJsWdCQiEoucoAOQsO49urNy50o65nXEc5zO7Tpzyw23\n1Ns6obCwkIcf/kPU3X+/SGqR+oQJ45g3bwZlZeHX1dUPMGHClUk7v0g6uvZaGDIE3n47vHyNiLQ+\nuvsvBSz4cAHnPXAe+Z3y2b1hNzWLaph21bSUXiRZLR0yl+7+a1wsY91f/gJ//CO880549kpEghNY\nSwUz+zVwM1Dg7psa2EdJVT32VO/htn/cRt7OPHYt2gWwtxeVSCpKl6TKzO4FxgPr3H1Enff2GdPM\n7HrgQqAKuMLdX27guC0e69zDzUB/9jO46KIWHUJE4iSQpMrM+gL/CxwOfEtJVdPVeA0z35tJ+5z2\nTBw5EbNW//8pyQBplFSdCGwHHopOquob08xsCPAocAzQF3gFOKy+QS3Wse6DD+B73wv3reratcWH\nEZEYBdWnagZwTRyOk1Hcncc/eZyKqgp+euRPlVCJJJm7vwHUtzp5fWPaWcDj7l7l7kuBJcCoRMR1\n1FHwr/8K//VfiTi6iCRSTIXqZnYmsMLdFyopaJ6XvniJLzZ9wdUnXE1OVviPQXVKIsFqZEzrA/wj\n6vWqyLaEuPFGGDoU/v3f4cgjE3UWEYm3/SZVZjYHOCB6E+DA74DfACfXea9BU6ZM2fu8qKiIoqKi\npkeaRt5Z+Q6vLX2N4hOLaZ/bHvi6oWZ29kQA5s2bwcyZV8acWClRk1iVlJRQUlISdBgJZ2bt2XdM\na5FYx7ru3eGGG+Cyy+C110D/ZhVJvHiMdS2uqTKz4YTrCnYSTqb6Ev7X2yh3X1/P/hlbU1W7QDLA\n0GOH8tq217jq+Ks4MP/AvfsUF09l/vxRFBSMBaCsbC5jxixg2rTrW3zeuoladfUDcUnUJLOlS00V\ngJn1A55z9xGNjWmEC9Rx95sin3sRmOzu79RzzLiMddXVcOyxcMEFcMklMR9ORJqpJWNdiy//ufsn\nwN7lP83sK6DQ3eurUchYtQsk5w7LZVfNLv781z9z5zl3fiOhSpTozucAZWXhbUqqRPayyKPRMc3M\nngUeMbPbCF/2GwgsSGRg2dnh9QBPOCFcZ3XCCYk8m4jEQzw7qjv7ufyXie596F6Wli9lxRcrWGbL\nOKjnQXzy9if77Ddhwjiqqx+grGwuZWVzIw01xwFavFgkEczsUeAtYJCZLTezC+rssndMc/dFwJPA\nIuB54OJkTL0PHAj33w/nnANr1iT6bCISKzX/TKBQKMTZF5/NliFb2N11N22+asPwPsMZd9A4in9V\nvM/+9dU/xXIJT5f/JBHS6fJfIiRirPv97+Hll+HVV6FNm7geWkQaEFjzzyadKAOTqml/nMaLm17k\ng8oPyM3LJXtlNl0+6cLT9zzd5OaesdZaqVBd4k1JVeMSMdbV1MD3vw8HHwx33RXXQ4tIA5JaUyX7\n5+5sab+FQ3seSnZ5NtuztnPqmFOT2i29sLBQiZRIK5eVBQ8/DMccAw8+GO64LiKpR0lVArUb1o5d\n/7eLQdmDyMrNonJXJT+/9OfNOoYWLxYRgM6d4e9/h6Ii6N8fxowJOiIRqUuX/xJk3lfzmLd0Hmd0\nO4NX570KtHxNP13Ck1Siy3+NS/RY9+qrcO65MGsWHH98wk4jkvFUU5UiPljzAY998hjXjr6WgryC\noMMRiSslVY1Lxlj34otw/vnwf/8XviQoIvGnpCoFfLn5S/604E9cfuzl9OvSD0jeTJNmtCQZlFQ1\nLllj3XPPhZexeeklGDky4acTyThKqgK2bvs6bnnrFn428mcM7zkcSF5bA7VPkGRRUtW4ZI51Tz8N\nl14Kc+bA8OFJOaVIxmjJWBfP5p8ZrbyinDveuYOzBp+1N6ECuP32+1i5Mo8NGxaQm9uV7OyJe2eT\n4im6e3pBwdiEnUdEUsfZZ8Ntt8Epp8C77wYdjYjo7r84qKiq4K4Fd3Fc3+M48eAT924vLS3l5ZcX\ns23bxeza1YWyshkcfPDoACMVkXTzox9BXh6cdhr87//CWWcFHZFI5lJSFaMar+Ge9++hT34fxg8a\n/433nnjiJQoKrmDXroOAvlRVfZ8NG25lwoT4d+9T6wWRzHXWWdCnT/jn0qVwxRVBRySSmZRUxcDd\neXThozjOT0b8BLN9L7127NiBESMGsHr1enbu3M4ppxyTkDqnwsJCZs68MqpQXfVUIpnk6KPhrbfg\n9NPhiy9gxozwoswikjwqVI/B80uep3RNKVefcDXtctrt837QxeN17wYEdHegxEyF6o0LeqzbuhV+\n+MPwGoEPPQTduwcWikirprv/kuitFW8x+5+zKR5dTOd2nRvcL6g2B3UTuvLyWzFrR37+JYDuDpSW\nU1LVuFQY6yor4Te/gSeegL/8Rd3XRVpCa/8lyaINi3hm8TP8+vhfN5pQQXBr70XfDQiwbNmTwAn0\n71+7MHN4HyVVIuknNxduvhlOOgkmTIBJk+B3v9PlQJFEU0uFZlqxdQX3lt7LpKMn0Tu/d9DhiIg0\n6Hvfg9JSmD8fxo6FlSuDjkgkvSmpaoaNOzdy14K7OG/EeQzsNjDocBo1YcI4qqsfoKxsLmVlc+na\ndQXdus3a+zp8d+C4oMMUkQTr3Rtefjncy+qoo+BPf4Lq6qCjEklPqqlqoh17dnDzWzfz7YO/zdgB\nY4MOp0lUqC6JoJqqxqXyWLdoEfziF7BnD/z5z1reRqQxKlRPkMrqSm5/53b6de7Hvw37t6DDEQmU\nkqrGpfpYV1MD998fLmT/yU/ghhugY8egoxJJPYEsU2Nml5nZYjNbaGY3xXq8VOPu3P/h/XRu25kf\nDv1h0OGISJyY2b1mts7MPo7a9nsz+8jMPjSzV8ysb9R715vZksh4d0owUccuKwt+/nP45JPwDSuH\nHRa+JLhnT9CRibR+MSVVZlYEnAEc4e5HALfEI6hU8tSipyivKGfiyIn1NvcUkVbrfqBuYeF0dz/S\n3UcCs4DJAGY2FDgHGAJ8D7jbWvmA0KMHPPggPP88zJ4NgweH2y+o3kqk5WKdqfolcJO7VwG4e1ns\nIaWOuV/OJbQhxC+P/iW52blBhyMiceTubwCb62zbHvWyA7Ax8vxM4HF3r3L3pcASYFQy4ky0o46C\nF14IXxK8++5wndVf/wpVVUFHJtL6xJpUDQLGmNnbZjbPzI6OR1Cp4P3V7/PyFy9z2ajL6NCmQ9Dh\niEiSmNkfzGw5MBGYGtncB1gRtduqyLa08Z3vwJtvwn//N9x+e/iy4B13wPbt+/+siITtN6kyszlm\n9nHUY2Hk55mEm4d2dffjgGuBJxMdcDIs2biExz55jEtHXUr3PK3xIJJJ3P137n4w4cuDfww6nmQy\ngzPOgDfegMceg9dfh0MOgeuuCy/ULCKN229HdXc/uaH3zGwS8Exkv3fNrMbMurv7xvr2nzJlyt7n\nRUVFFBUVNTfehFuzbQ1/fv/PXHjUhRzU+aCgwxEJXElJCSUlJUGHEYRHgecjz1cB0QNC38i2erWG\nsW5/jjsufBnwyy/hzjvhmGPgyCPhwgvhBz+A9u2DjlAkvuIx1sXUUsHM/gPo4+6TzWwQMMfd+zWw\nb0rfZgywdfdWpr05jTMGncHxBx0fdDgiKSmdWiqY2SHAc5EbbTCzge7+eeT5ZcAod/9ppFD9EeBY\nwpf95gCH1TeotYaxriV274Znn4X77oN334VzzoFzz4UTT9TyN5Kekt6nysxygfuAkUAF8Gt3f62B\nfVN6oNldtZtb3rqFwt6FnHbYaUGHI5Ky0iWpMrNHgSKgO7CO8J1+pwOHA1XAl8Av3X19ZP/rgZ8D\nlcAV7v5yA8dN6bEuHpYvh4cfhqeegtWrwzNXP/xhuC4rV/f0SJpQ888Wqq6p5k/v/olu7btx3hHn\nqXWCSCPSJalKlFQe6xLhiy/g6afDj88/h+9+F8aNCz/6pFUpv2QaJVUt4O489NFDbNuzjYuPuZgs\n03KIIo1RUtW4VB3rkmHVqvA6gy++CK+8El538JRT4NvfhtGjoWfPoCMUaTolVS3w3GfPsXD9Qn59\n/K9pm9M26HBEUp6Sqsal6liXbNXV8N57MHdu+G7Ct96CXr3CNVgnnABHHw1Dh0LOfm+XEgmGkqpm\nemP5G7yw5AWKTyymU9tOQYfTInUXTdYiyZJoSqoal4pjXSqorg4vjfP66/D22/D+++HarBEj4Fvf\nCjcdHT4chg2D/PygoxVRUtUsn6z/hAc/fJCrT7iaAzoeEHQ4LVJaWsqkSTPIzp4IQHX1A8yceaUS\nK0koJVWNS7WxLpWVl8MHH4QTrI8+glAIFi8OL6EzfDgcfjgMGhRuRDpoEBx4YHjtQpFkaMlYl5ET\nr8u2LOP+D+7nklGXtNqECuCJJ14iO3siBQVjgfDiqE888ZKSKhFpFTp1Ct8x+J3vfL2tuhq++io8\nq/XZZ+H2DY8+Cv/8ZzgJ69cv3JA0+mffvuGi+AMPhLaq4pAAZVxSVbazjD+9+yd+euRPGdB1QNDh\niIhIlOxsGDgw/Khr+/ZwZ/fax7Jl4bqtlSvDRfJr10LnzuEE64AD9n107w4FBV//7NQp3EVeJF4y\nKqmq8Rr+tOBPnHbYaYzsNTLocGI2YcI45s2bQVlkGevq6geYMOHKYIMSEUmQjh3DlwWHD6///Zoa\n2LAhnGCtW/f1Y/Vq+PBD2LgxPKNf+3PXrnAS1qULdO0a/tmlSzjZ6tQp/F6nTuEar44dv3506BB+\n5OV9/bN9ezVBlQysqdqwYwM9OvQIOoy4UaG6JJtqqhqXKmOd7F9lJWzZAps3f/1z69bwZcby8q+f\nb9/+zce2bbBz59ePHTvCCVpuLrRrF06w2rcPP699tG379c82bb7+WfvIzQ0/op/n5obvjox+XvvI\nzm74Z+0jK2vf59E/G3qY7fvc7JvP63uvsUdrpEJ1EUk4JVWN01iXmdyhoiK8nM+uXeHH7t1fb4v+\nuWdP+FH7vKIinODVPvbsCf+sqvrm9urq8LboR3X119ujf9bUfP1e7evabbU/3b/eXneb+9evo7fV\n/mxoW91HXfUlW409r7ut7vaGttX3Xt04on/+7Gdw661191FSJSIJpqSqcRrrRPbVUMLV2PO62+pu\nb2hbfe/VjaXue23b7tvKQ3f/iYiISMppzZcBm0MdP0RERETiQEmViIiISBwoqRIRERGJAyVVIiIi\nInGgpEpEREQkDpRUiYiIiMSBkioRERGROFBSJSIiIhIHMSVVZnaMmS0wsw8iP4+OV2AiIolkZvea\n2Toz+zhq23QzW2xmH5rZ02bWKeq9681sSeT9U4KJWkRSWawzVdOB37n7UcBk4ObYQ0q8kpKSoEMA\nUicOUCz1SZU4ILViSSP3A+PqbHsZGObuI4ElwPUAZjYUOAcYAnwPuNssfftDt/a/b609fmj936G1\nx99SsSZVa4DOkeddgFUxHi8pUuUPO1XiAMVSn1SJA1IrlnTh7m8Am+tse8XdayIv3wb6Rp6fCTzu\n7lXuvpRwwjUqWbEmW2v/+9ba44fW/x1ae/wtFevaf9cBb5rZrYABJ8QekohISrgQeCzyvA/wj6j3\nVkW2iYjstd+kyszmAAdEbwIc+B1wGXCZu//dzH4I3AecnIhARUSSxcx+C1S6+2P73VlEJMLcveUf\nNit39+hCzq3u3rmBfVt+IhFJKe6eFvVEZtYPeM7dR0RtmwhcBJzk7hWRbdcB7u7TIq9fBCa7+zv1\nHFNjnUiaaO5YF+vlvyVm9h13f83MxgL/jFdgIiJJYJFH+IXZqcA1wJjahCriWeARM5tB+LLfQGBB\nfQfUWCeSuWJNqn4B/MnM2gC7gf+IPSQRkcQzs0eBIqC7mS0nfAfzb4A2wJzIzX1vu/vF7r7IzJ4E\nFgGVwMUeyzS/iKSlmC7/iYiIiEhY0juqm9llkeZ5C83spmSfv04svzazGjPrFmAMDTYbTNL5TzWz\nT83sn2ZWnMxz14mjr5m9amahyN+Ny4OKJRJPlpmVmtmzAcfR2cz+Gvk7EjKzYwOM5fpIDB+b2SOR\nGWqJkiq/T83RQBPUrmb2spl9ZmYvmVm9tbKpoKGxo7V8BzNra2bvRJpoh8zsvyPbW0X8teqOma0w\n/qVm9lFtM/PItmZ/h6QmVWZWBJwBHOHuRwC3JPP8dWLpS/hOxWVBxRBRb7PBZDCzLOAuwg0QhwE/\nMrPByTp/HVXAVe4+DDgeuCTAWACuIHypJ2i3A8+7+xDgSGBxEEFECrovAo6KFHXnAOcGEUuqSrHf\np+aorwnqdcAr7n448CpJHJdaoKGxo1V8h0jt3r9EmmiPAE4ys9G0kvij1B0zW1v8NUCRux/l7rU9\n6Jr9HZI9U/VL4CZ3rwJw97Iknz/aDMIFqYFqpNlgMowClrj7MnevBB4Hzkri+fdy97Xu/mHk+XbC\nyUMgfYAiCfdpwP8Gcf6oODoB33b3+wEijSfLAwqnHNgDdDCzHCAPWB1QLKkqZX6fmqO+JqiE434w\n8vxB4PtJDaoZGhg7+tK6vsPOyNO2hP+/vJlWFH8DY2ariT/C2DcnavZ3SHZSNQgYY2Zvm9k8C2it\nQDM7E1jh7guDOH8jLgReSOL5+gArol6vJAUaGprZIcBIYJ/b1ZOkNuEOuuCwP1BmZvdHptXvMbP2\nQQTi7puBW4HlhBtfbnH3V4KIJYWl5O9TC/V093UQTlqAngHH0yRRY8fbwAGt5TtELp19AKwFStx9\nEa0ofuofM1tT/BCOfY6ZvWtm/x7Z1uzvEOvdf/uwxpuF5gBd3f04MzsGeBIYEO8YmhDHb/hmk9KE\n3gLdSCy/dffnIvvUNht8NJGxpDoz6wg8BVwR+Vdnss9/OrDO3T+MXK4O8vb4HKAQuMTd3zOzPxKe\njp6c7EDMbABwJdAP2Ao8ZWY/zvS/rxkk6H9g7FfdscP27ReWst8hcrXiqMjs9EuRsadVxF/PmNmQ\nlIw/ymh3X2NmPYCXzewzWvBnEPekyt0b7KhuZpOAZyL7vRspEu/u7huTFYeZDQcOAT4yMyM8Tfy+\nmY1y9/XxjqOxWKJimkh46vSkRJy/EauAg6Ne9yXA9Rsjl5WeAh5291kBhTEaONPMTgPaA/lm9pC7\nnx9ALCsJz6i+F3n9FBBU8fPRwJvuvgnAzJ4hvCyVkqqvpdTvU4zWmdkB7r7OzHoBCRkb46WBsaNV\nfQcAdy83s+cJ/761lvjrGzMfBta2kvgBcPc1kZ8bzOzvhC/nN/vPINmX//5OJHEws0FAbiISqsa4\n+yfu3svdB7h7f8L/4zoqUQnV/tjXzQbPrNNsMBneBQaaWb/InVznEm5yGJT7gEXufntQAbj7b9z9\nYHcfQPi/x6sBJVREpp1XRH5XAMYSXPH8Z8BxZtYu8o+RsQRUNJ/CUu33qTm+0QSVcNwTI89/BgT1\nj5ymqm/saBXfwcwKau8qi1zePxn4gFYSfwNj5k+B52gF8QOYWV5kphMz6wCcAiykBX8GcZ+p2o/7\ngfvMbCFQAQTyP6s6nGAv8dxJPc0Gk3Fid682s0sJ34GYBdzr7kHdXTYaOA9YGKktcOA37v5iEPGk\nkMsJd/LOBb4ELggiCHf/yMweAt4HqgkP+vcEEUuqSqXfp+aw+pug3gT81cwuJHyH9DnBRdi4hsYO\nYMjnsCMAAAPrSURBVBrwZCv4Dr2BByP/WMkiPNs2N/JdWkP8DbmJ1hP/AcDfIpeMc4BH3P1lM3uP\nZn4HNf8UERERiYOkN/8UERERSUdKqkRERETiQEmViIiISBwoqRIRERGJAyVVIiIiInGgpEpEREQk\nDpRUiYhIyjOzbmb2QWQdzDVmtjLqdZN6LprZvWZ22H72udjMfhSfqOs9/g+iGvpKmlGfKhERaVXM\n7L+A7e5+Wz3vmafw/9giS7g8FeBSXJJAmqkSEZHWZu8qGGZ2qJmFzOwvZvYJ0MvM/mxmC8xsoZn9\nLmrf181shJllm9lmM5tqZh+a2ZtmVhDZ50Yzuzxq/6lm9o6ZLTaz4yLb88zsKTP7xMz+ambvmtmI\nfYI0uzkS24eR45xIeJ3X2yIzbAeb2UAzezFyjBIzGxj57MNmdreZvWdmn0aWNMPMhke+W2nkuIck\n7L+yNFuyl6kRERGJt8OBn7j7BwBmVuzuW8wsG5hnZk+5+6d1PtP5/7d3PyE2hWEcx79PjSjTzE6p\nKUmjkAZNkSzsNAs1CyJ2xEJRZmehrMmGFHYypUwmicnInw1WxmKKKRqymGah5H8NM34W9xmuM/di\n6tTMrd+nTr333Pd9z1k+Pc/beYAHko5FxGlgH3Cy1uaSNkbEdiotfLqAw8C4pB0ZTA0V10TEEqBL\n0pr83VLVMLlP0o28fx/YL+l1RGwGzgHbcps2SZ1ZLrwbESuAQ8ApSX3Zvmou26xZgYMqMzNrdKPT\nAVXam/3amqj01lsNFIOqr5Lu5HgI2FJn7/6qOctyvIVKbzskDUfEsxrr3gFTEXERGABuFidkI+VN\nwLXs/Qd/VpCu5jNeZF/GduAxcDwzVP2SRuu8t80Bl//MzKzRfZkeZPnsCLBVUgcwCCyqseZb1XiK\n+kmGif+YMyNbJGkS6ASuA93ArTrr3kraIGl9Xh3V2xTmSlJv7jcB3M6Sos0TDqrMzKzRVQc1LcBH\n4HNELOV3Ke1va2brEbALICLWAqtmbB7RDLRKGgB6gHX516d8RyS9B8YjojvXROFs1s68vxJoA15G\nxHJJrySdoZL9mnGWy+aOy39mZtbofmV0JD2NiBFgBHgDPKw1rzD+574FZ4FLeTD+eV4fCnNagf6I\nWEglgDua968AFyKih0rGaTdwPiJOAAuAXmA4545FxBNgMXBA0mRE7MlPPnwHxqic87J5wp9UMDMz\nm4U8AN8kaSLLjYNAu6QfJT7jMlUH2q0xOFNlZmY2O83AvaqPjh4sM6BKzng0IGeqzMzMzErgg+pm\nZmZmJXBQZWZmZlYCB1VmZmZmJXBQZWZmZlYCB1VmZmZmJXBQZWZmZlaCn3k/n05X32zbAAAAAElF\nTkSuQmCC\n", + "text/plain": [ + "<matplotlib.figure.Figure at 0x7f8047207c10>" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "#@test {\"output\": \"ignore\"}\n", + "import tensorflow as tf\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "\n", + "# Set up the data with a noisy linear relationship between X and Y.\n", + "num_examples = 50\n", + "X = np.array([np.linspace(-2, 4, num_examples), np.linspace(-6, 6, num_examples)])\n", + "# Add random noise (gaussian, mean 0, stdev 1)\n", + "X += np.random.randn(2, num_examples)\n", + "# Split into x and y\n", + "x, y = X\n", + "# Add the bias node which always has a value of 1\n", + "x_with_bias = np.array([(1., a) for a in x]).astype(np.float32)\n", + "\n", + "# Keep track of the loss at each iteration so we can chart it later\n", + "losses = []\n", + "# How many iterations to run our training\n", + "training_steps = 50\n", + "# The learning rate. Also known has the step size. This changes how far\n", + "# we move down the gradient toward lower error at each step. Too large\n", + "# jumps risk inaccuracy, too small slow the learning.\n", + "mu = 0.002\n", + "\n", + "# In TensorFlow, we need to run everything in the context of a session.\n", + "with tf.Session() as sess:\n", + " # Set up all the tensors.\n", + " # Our input layer is the x value and the bias node.\n", + " input = tf.constant(x_with_bias)\n", + " # Our target is the y values. They need to be massaged to the right shape.\n", + " target = tf.constant(np.transpose([y]).astype(np.float32))\n", + " # Weights are a variable. They change every time through the loop.\n", + " # Weights are initialized to random values (gaussian, mean 0, stdev 1)\n", + " weights = tf.Variable(tf.random_normal([2, 1], 0, 0.1))\n", + "\n", + " # Initialize all the variables defined above.\n", + " tf.initialize_all_variables().run()\n", + " \n", + " # Set up all operations that will run in the loop.\n", + " # For all x values, generate our estimate on all y given our current\n", + " # weights. So, this is computing y = w2 * x + w1 * bias\n", + " yhat = tf.matmul(input, weights)\n", + " # Compute the error, which is just the difference between our \n", + " # estimate of y and what y actually is.\n", + " yerror = tf.sub(yhat, target)\n", + " # We are going to minimize the L2 loss. The L2 loss is the sum of the\n", + " # squared error for all our estimates of y. This penalizes large errors\n", + " # a lot, but small errors only a little.\n", + " loss = tf.reduce_mean(tf.nn.l2_loss(yerror))\n", + "\n", + " # Perform gradient descent. \n", + " # This essentially just updates weights, like weights += grads * mu\n", + " # using the partial derivative of the loss with respect to the\n", + " # weights. It's the direction we want to go to move toward lower error.\n", + " update_weights = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)\n", + " \n", + " # At this point, we've defined all our tensors and run our initialization\n", + " # operations. We've also set up the operations that will repeatedly be run\n", + " # inside the training loop. All the training loop is going to do is \n", + " # repeatedly call run, inducing the gradient descent operation, which has the effect of\n", + " # repeatedly changing weights by a small amount in the direction (the\n", + " # partial derivative or gradient) that will reduce the error (the L2 loss).\n", + " for _ in range(training_steps):\n", + " # Repeatedly run the operations, updating the TensorFlow variable.\n", + " sess.run(update_weights)\n", + " \n", + " # Here, we're keeping a history of the losses to plot later\n", + " # so we can see the change in loss as training progresses.\n", + " losses.append(loss.eval())\n", + "\n", + " # Training is done, get the final values for the charts\n", + " betas = weights.eval()\n", + " yhat = yhat.eval()\n", + "\n", + "# Show the results.\n", + "fig, (ax1, ax2) = plt.subplots(1, 2)\n", + "plt.subplots_adjust(wspace=.3)\n", + "fig.set_size_inches(10, 4)\n", + "ax1.scatter(x, y, alpha=.7)\n", + "ax1.scatter(x, np.transpose(yhat)[0], c=\"g\", alpha=.6)\n", + "line_x_range = (-4, 6)\n", + "ax1.plot(line_x_range, [betas[0] + a * betas[1] for a in line_x_range], \"g\", alpha=0.6)\n", + "ax2.plot(range(0, training_steps), losses)\n", + "ax2.set_ylabel(\"Loss\")\n", + "ax2.set_xlabel(\"Training steps\")\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "lSWT9YsLP1de" + }, + "source": [ + "This version of the code has a lot more comments at each step. Read through the code and the comments.\n", + "\n", + "The core piece is the loop, which contains a single `run` call. `run` executes the operations necessary for the `GradientDescentOptimizer` operation. That includes several other operations, all of which are also executed each time through the loop. The `GradientDescentOptimizer` execution has a side effect of assigning to weights, so the variable weights changes each time in the loop.\n", + "\n", + "The result is that, in each iteration of the loop, the code processes the entire input data set, generates all the estimates $\\hat{y}$ for each $x$ given the current weights $w_i$, finds all the errors and L2 losses $(\\hat{y} - y)^2$, and then changes the weights $w_i$ by a small amount in the direction of that will reduce the L2 loss.\n", + "\n", + "After many iterations of the loop, the amount we are changing the weights gets smaller and smaller, and the loss gets smaller and smaller, as we narrow in on near optimal values for the weights. By the end of the loop, we should be near the lowest possible values for the L2 loss, and near the best possible weights we could have." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "dFOk7ERATLk2" + }, + "source": [ + "## The details\n", + "\n", + "This code works, but there are still a few black boxes that are worth diving into here. `l2_loss`? `GradientDescentOptimizer`? What exactly are those doing?\n", + "\n", + "One way to understand exactly what those are doing is to do the same thing without using those functions. Here is equivalent code that calculates the gradients (derivatives), L2 loss (sum squared error), and `GradientDescentOptimizer` from scratch without using those functions." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 657, + "status": "ok", + "timestamp": 1446499870301, + "user": { + "color": "", + "displayName": "", + "isAnonymous": false, + "isMe": true, + "permissionId": "", + "photoUrl": "", + "sessionId": "0", + "userId": "" + }, + "user_tz": 480 + }, + "id": "_geHN4sPTeRk", + "outputId": "85c49bf6-8d07-401a-ae08-79c6933adff5" + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlUAAAEPCAYAAABr+zG+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xl8VPXZ///XlYUlLEEIIIsCCqKAqCmgFkvTUkUrLnft\nXay2va12obZK9RZTWr8/oO19K2pLXWqxLWqxWnGrWEpVVCK3K2pcBxSs7MgSdgiELNfvj5ngGJKQ\nZCZzZnk/H4/zyJkzZ865RuDjlc/5fK6PuTsiIiIiEpusoAMQERERSQdKqkRERETiQEmViIiISBwo\nqRIRERGJAyVVIiIiInGgpEpEREQkDuKSVJnZFDMLmdm7ZvaAmbWJx3VFRFqLmc02s01m9m7UseFm\n9rKZvWNm88ysY9R7U8xshZktM7OzgolaRJJZzEmVmfUDvg+c4u7DgRzg4livKyLSyu4FxtU59mfg\nenc/Cfg7cD2AmQ0BvgGcAJwD3GVmlsBYRSQFxKOnahdwAOhgZjlAHrAhDtcVEWk17v4isL3O4UGR\n4wDPAhdF9s8HHnL3KndfBawARiUkUBFJGTEnVe6+HfgNsAZYD+xw92djva6ISABCZnZ+ZP8bQN/I\nfh9gbdR56yPHREQOisfjv2OAa4B+QG+go5ldEut1RUQCcDnwYzN7HehAuBdeRKRJcuJwjRHAS+6+\nDcDMHgc+DzwYfZKZaZFBkTTh7mk5nsjdlxMZZ2Vmg4BzI2+tB46KOrVv5Ngh1NaJpI/mtnXxGFP1\nIXCambWLDNwcCyxrILik2KZOnRp4DMkUh2JJ7jiSLZY0Y5Et/MKse+RnFnADMCvy1pPAxWbWxswG\nAAOBJQ1dNOg/o3T6+5aJ8afDd0j1+N1b1tbF3FPl7u+Y2RzgTaAaeAv4Y6zXFRFpTWb2IFAEdDOz\nNcBUoJOZ/Rhw4HF3vw/A3Zea2cPAUqASuNJb2uqKSNqKx+M/3P0W4JZ4XEtEJBHcvaGxn7c3cP6N\nwI2tF5GIpLqMrKheVFQUdAhA8sQBiqU+yRIHJFcskv5S/e9bqscPqf8dUj3+lrJE9WCbmXrLRdKA\nmeFpOlA9HtTWiaSHlrR1GdlTJSIiIhJvSqpERERE4kBJlUiSq6yubPH0XhERSRwlVSJJrKqmijuX\n3MmS9Q2WRBIRkSShpEokSbk7c96ZQ7ucdozsMzLocERE5DCUVIkkqSc+eIIte7fwvcLvkWX6pyoi\nkuzUUoskoZJVJZR+UsqPR/2Y3OzcoMMREZEmUFIlkmTe2fgOC1YsYNJpk+jYpmPQ4YiISBMpqRJJ\nIh9v/5j7372fH4/8MQV5BUGHIyIizaCkSiRJbN67mT+8/gcuO/ky+nXpF3Q4IiLSTEqqRJLA7ord\n3PbqbZw/+HyG9RgWdDgiItICSqpEAlZRVcGdS+7k1L6n8oV+Xwg6HBERaSElVSIBqvEa/lz6Z3p3\n6s15x50XdDgSJzU1QUcgIkFQUiUSEHfnwfcepKqmim8N/xZmzVoMXZLY7t1BRyAiQVBSJRKQf330\nL1btWMUPR/yQ7KzsoMORONqxI+gIRCQISqpEAvDK2ld4cc2LXDXqKtrltAs6HIkzJVUimUlJlUiC\nLduyjMeWPcZVo64iv11+0OFIK1BSJZKZ4pJUmVm+mT1iZsvMLGRmp8bjuiLpZu3Otcx+azYTR0yk\nV6deQYcjrURJlUhmyonTdW4DFrj7f5pZDpAXp+uKHKK0tJS5c58GYMKEcRQWFgYcUdNsLd/KnUvu\n5JITL2Fg14FBhyOtaOfOoCMQkSDEnFSZWWfgC+5+GYC7VwG7Yr2uSH1KS0uZOHEm2dmXAbBo0Uxm\nzbom6ROr8spy7lhyB2cdexaFvZI7VomdeqpEMlM8Hv8NAMrM7F4zKzWzP5pZ+zhcV+QQc+c+TXb2\nZRQUjKWgYCzZ2Zcd7LVKVpXVldz1+l0M7T6UsceMDTocSQAlVSKZKR5JVQ5QCPze3QuBcuBncbiu\nSMpzd+57+z7y2+bz9SFfDzociWJms81sk5m9G3VspJktMbO3Ij9HRL03xcxWRMaOntXYtZVUiWSm\neIypWgesdfc3Iq8fBYrrO3HatGkH94uKiigqKorD7SWTTJgwjkWLZlJWFn5dXX0fEyZcE2xQdYRC\nIeYvnA9A7gm51HSqYdKpk1K2uGdJSQklJSVBh9Ea7gXuAOZEHbsZuMHdnzGzc4BbgC+Z2RDgG8AJ\nQF/gWTMb5O5e34WVVIlkppiTKnffZGZrzew4d18OjAWW1ndudFIl0hKFhYVMmnQ2t99+GwCTJk2I\n+3iqWAbCh0Ihrr/lenKH5rKlegub/7WZB7/7ILnZuXGNMZHq/gI0ffr04IKJI3d/0cz61Tn8CVBb\n56ILsD6yfz7wUGTM6CozWwGMAl6r79pKqkQyU7xm/10NPGBmucDHwHfjdF2RzygtLeW2254iO3sS\nALfddh8nnHDCwcQn1pmBsQ6En79wPrlDc8k+Opu92/cyKGsQzy96npEnjWxWHBKYnwEvmdlvAAM+\nHzneB3gl6rz1kWP10uw/kcwUl6TK3d8B9H8NaXXRA9UBysrCxwoLC+MyM7Cx6zfVnpo9bNm2hWE9\nhrF3994mf06SwmzgKnd/wsy+DtwDnNnciyxdOo3ajnkNdRBJDfEY6hCvniqRwMUjIYrVqWecyt0P\n3k1/+rN3z14qQ5WMnzw+YfeXmJ3q7mcCuPujZvbnyPH1wFFR5/Xl00eDh8jL+zSpEpHUEI+hDkqq\nJKW09kD1WK6/c/9Ontn5DL+68FesK10HwPjJ4xk6dGjc4pO4s8hWa4WZfdHdXzCzscCKyPEnCQ9x\nmEn4sd9AYElDF9WYKpHMZA1MXon/jcwamigj0iwNjZuq+/ivuvq+FhUGbcm4rP1V+7n15Vsp7FXI\nVwd9tVn3SzVmhrun5lTGKGb2IFAEdAM2AVOBd4G7gDbAfuBKd38rcv4U4AqgEpjk7s80cF3PznYq\nKyFFJ3yKCC1r65RUSVoJYgmb6ppqfv/67+naviuXnnhpypZOaKp0Sapai5l5hw7Oxo3QsWPQ0YhI\nSympEmlF9SVs7s6cd+aw+8Burhx5JVmWlbJrEzaVkqrGmZn36eO8+ir07Rt0NCLSUi1p6zSmSqQJ\nGppZuL7Dejbs3sC1p197MKFKxbUJJb66dAmPq1JSJZJZlFSJNEF9Mwt/88jdDDizO8Wji2mb07bB\n8xI9A1GCl5+vweoimSgea/+JZJzyjitZkb2Mq0+9mk5tOwUdjiSZ2p4qEcks6qkSaYIJE8axYMFU\n1m99iKoOuzkw5HXuGTeTHh16HHJesq9NKK1PSZVIZlJSJVKP6MHmI0YM5JU3XmF33nvUnPwe+wp2\n03/j0Rzb9dhDPldYWMisWddEDVTXeKpM1KWLlqoRyURKqkTqiB5sXl6+irsf/yldB7Zl3yn72N91\nP4X9Csnvnc/8hfPrLexZWFioRCrDqadKJDNpTJVIHdGDzQ9klZF1fH/KrYIDRxygbfu2lG8rDzpE\nSXJKqkQyk5IqkcNwHB/s2Hoja10Wu9fsDq/pd6bW9JP6afafSGbS4z+ROiZMGMfjj1/LsjVTqKos\np7LLKrp1PJK+Wcex8dWNnD3mbK74yRVa008apJ4qkcykpEqaJd2rhQO89NJLrDvwGtV9quFYwypg\n1KZRnDTkJMZfrgWS5fA0UF0kMympkibLhGrhoVCIX/7pl9ScUUNOtxyq9lTRoaYj27dtp/inxUGH\nJylCPVUimUlJlTRZJlQLn79wPjbYsL5GTacacmtyqfyoEnoc/rMitZRUiWQmDVQXqaNPvz5UVVbB\nbqjaXkXV6ira1xRQXHwjpaWlQYcnKUAD1UUyk5IqabIJE8ZRXX0fZWXPUVb2XKRa+Ligw4qrL3zx\nC+yq3sXg7MF0/rgzeW/l0d2H8cknl7F48SgmTpypxEoOqzapcg86EhFJJPM4/as3syzgDWCdu59f\nz/ser3tJcNJxoHooFGL+wvlUeiUb+25keNfhbH9vOwDL3/uEpUvPi3rk+RxjxixhxowpQYYcKDPD\n3S3oOJJVbVvXvj1s3Qp5eUFHJCIt0ZK2Lp5jqiYBS4HOcbymJJl0qxYeCoW4/pbryR6SzcdVH5Nb\nksvEH01k2NhhABQX3xhwhJKqamcAKqkSyRxxSarMrC/wVeB/gGvjcU2R1hLd27av5hNyhuSwo/sO\njvAj6JrTlX8++0+GDQsnVVogWVqqdrB6r15BRyIiiRKvnqqZwGQgP07XE2kVdctCrN1yHV0urqTD\nER0Y3nM4W7Zu+cz5WiBZWkozAEUyT8xJlZmdC2xy97fNrAho8PnjtGnTDu4XFRVRVFQU6+1FmmXu\n3KepqCjiQNYbAFivk1m/+h+M7nwaW3ZsCS8/M/mzy8+k2yPP5iopKaGkpCToMFKOZgCKZJ6YB6qb\n2f8C3wKqgPZAJ+Bxd/9OnfM0UF0Cd8UVP+Xxxa/S9qQjqWy3k70HPmSsj2bM6BEAjD9TFdMPRwPV\nG1fb1l18MVxwAXzzm0FHJCItEchAdXf/OfDzSABfBP67bkIlUiuW2YPxmHnYoWsNDF5HVbc89nVa\nTduPazj22F6qli5xp6VqRDKPKqpLwsSyzE08lsgJhUKsWLmCzj2y2V3wHkdZT3qdcDQc+HSWX7qU\niZDgaUyVSOaJa/FPd3+hvhpVIvDZZW4KCsaSnX3ZwZ6n1vwsfFo6YW+/vZTVlOGfVNE3qydVS6tY\n9NRKFi8epeKeGcbMZpvZJjN7N+rYQ2ZWGtlWmllp1HtTzGyFmS0zs7MOd30lVSKZRz1VkhHmL5xP\n1pAs9h+xn5N3n8yOF3fQZlcb+nc9kaUdz0vr9QylQfcCdwBzag+4+8W1+2Z2K7Ajsn8C8A3gBKAv\n8KyZDWpsoGh+Pqxa1TqBi0hyUlIlCdPSmk+lpaVs3LieVav+lz179tKxY4dm14uq8RpWV62mW243\nBg0cxObqzZyeezrbPqlp6deRFOfuL5pZv0ZO+QZQFNm/AHjI3auAVWa2AhgFvNbQh9VTJZJ5lFRJ\nwrSk5lP0WKru3YexZctNfO5zI7n66qaPp3J3bLBR/VQ1nXM6s3nL5oOlEyoqKlTcUw5hZl8ANrr7\nx5FDfYBXok5ZHznWICVVIplHSZUkVHNrPtUdS9WhwyB69lzSrGv866N/UdWxivu/dz8Ln18IwPjJ\nn5ZOUHFPqcc3gb+19MPTpk1j7Vp4/30oKVFNPpFUEI+afEqqJK29vPZlXlzzIsWji8lvl0/h8EMT\npkwv7imfZWbZwNeA6L8U64Gjol73jRyr17Rp01i2DF5+GZRPiaSGukXJp0+f3uxrxHX2n0i8TZgw\njurq+ygre46ysucij+fGNemzS7cs5fFlj3PVqKvIb6cVlKRexqGrQJwJLHP3DVHHngQuNrM2ZjYA\nGAgsaezCevwnknlirqje5Buporq0UHOKfoZCIeYvnM+Omh18cuQn3HD2DQzsOjBRoWaEdKmobmYP\nEh6I3g3YBEx193vN7F7gFXf/Y53zpwBXAJXAJHd/poHrurtTXg7dusG+fa36NUSklbSkrVNSJWmj\nthaVD3FWVK6g5+qe3D3pbi07E2fpklS1ltq2zh3atoXdu8M/RSS1tKSt0+M/SRvzF87Hhhib8jcx\nuP9gCgYVMH/h/KDDkgxlpqVqRDKNkipJG9VezaqqVXRt35U+nRud7S6SEBpXJZJZlFRJWnB3Dgw8\ngH1i5G3MY9MHm8K1qM4cH3RoksGUVIlkFpVUkLTw6NJH6ditI/f/4H6efi48qD26FpVIEPLzlVSJ\nZBIlVWmqOTPmUt2zHz9LaEuI60dfT15uHiefeHLQIYkA6qkSyTRKqtJQ9NIuAIsWzWTWrPSpFD5v\n3jxm3T8LgK9c9BU2F2ym+Ixi8nLzAo5M5LOUVIlkFiVVaSh6aReAsrLwsaYkVcnewzVv3jy+P+37\nZI/Kpiq7ihf++QK3X3A7Xdt3DTo0kUNo9p9IZtFAdTmotodr8eJRLF48iokTZ1JaWhp0WAeFQiEm\n/2oy+zrtI7d7LjX9a+iQ34HH5j4WdGgi9VJPlUhmUU9VGpowYRyLFs2krCz8Ory0yzWH/VwsPVyt\nrbaw544BO6hoV8HGDRspsAK8SgVlJXl16QIbNhz+PBFJD0qq0lBhYSGzZl0T9RgvtcZT1X0E2bZt\nW66beh2ru6/m2MHH8samN6Acdr+wm/a72zNx2sSAIxapn2b/iWQWJVVpqrCwsNmJVEt7uOKp7iD7\nBQumkj9wO9ttO9tqtvHJrk8Y0H0A29/ezhHlR3DLtFu44IILEhqjSFPp8Z9IZol5TJWZ9TWz580s\nZGbvmdnV8QhMEq+2h2vMmCWMGbMkkBmD0Y8gCwrGsrPiSDblb+OEL59A5b5KqvdXk70um1EdRvH3\nv/xdCZUkNQ1UF8ks8eipqgKudfe3zawj8KaZPePuH8Th2pJgLenhSoTd+bvpfWJvql6tor/15+bp\nN6uwpyQ99VSJZJaYkyp33whsjOzvMbNlQB9ASZU0W+0jyOXLX2LTnvlUVa6kYFsHVq1ZxcDcgdAJ\nbp6shEpSg5IqkcwS1zFVZtYfOBl4LZ7XleAlqn5VYWEhRUW9mPnwr7FCo21BGzbu2815a89jxPEj\ntPSMpBQNVBfJLHFLqiKP/h4FJrn7nvrOmTZt2sH9oqIiioqK4nV7aUWJrNAeCoW49x/3kjXayOmf\nw/7K/XTa2ok9ZXso/mlx3O8nh1dSUkJJSUnQYaSkjh1h/36orITc3KCjEZHWFpekysxyCCdU97v7\nvIbOi06qJHXEu35VY71e8xfOx3oYWR2zqG5fTa7lUrmrMubvIC1X9xeg6dOnBxdMijEL91bt3AkF\nBUFHIyKtLV4V1e8Blrr7bXG6nqSpplRtP3rI0VTurcQ/capXVuPvOBO/rVpUkpo0A1Akc8SjpMJo\n4FLgy2b2lpmVmtnZsYcmyWLChHFUV99HWdlzlJU9F6lfNa5F16pbMiE7+7KDvVYA48aOY0+bPRzT\n7Rjy38+nc2lnbrzqRpVOkJSlweoimSMes/9eArLjEIskqURVaK/xGl7e/zKXnnkpuStysQHG+DOD\nGZie7AtLS+pQUiWSOVRRXZokXvWrGqra7u7MfX8uFVUVTPnqFHKygvurmciB+ZL+NANQJHMoqZKE\naqjX6+mPnuajbR9x3eevCzShguReWFpSj3qqRDKHkipJuLq9XkvWL6FkVQnFZxTTPrd9gJGJxJ8G\nqotkDiVVknChUIj5C+cDMOTUIbyw+wWuPf1aurTrEnBkYcmwsLS0PjObDYwHNrn78KjjVwFXEl6C\n65/u/rPI8SnA5ZHjk9z9mabcRz1VIplDSZUk1Lx58yj+bTFZQ7Lo2r0rdz9yN3d84w56d+oddGgH\nJWpgvgTuXuAOYE7tATMrAs4DTnT3KjMriBw/AfgGcALQF3jWzAa5ux/uJl26wEcftUL0IpJ0lFRJ\nwoRCIYpvLmbnsJ3k9MlhXfk6BncczPuvvs+5p50bdHifkawLS0v8uPuLZtavzuEfATe5e1XknEh/\nJRcAD0WOrzKzFcAomrAklwaqi2SOeBX/FDms+Qvnk90zm9yOuexvu5/27duzd8veoMMSiXYcMMbM\nXjWzRWb2ucjxPsDaqPPWR44dVrdusGVLnKMUkaSknipJqO4DurN++3pyc3KxHUbNBzWM/8n4oMMS\nqZUDHOHup5nZSOAR4JjmXiR6Sa5jjy3iww+L4hWfiLSSeKxzak0YEhAXZtaU4QeSxt5//30umXUJ\nFR0ryF6eTc2mGmZcP0PV0lOMmeHuFnQc8RB5/PeP2oHqZrYAmOHuL0RerwBOA74P4O43RY4/BUx1\n90Me/9Vt66qqoFOncGmODh1a+xuJSLy0pK1TT5XEVWOVyJdnL+fML59Jt9XdyD4yO7Bq6SJRLLLV\negL4MvCCmR0HtHH3rWb2JPCAmf2W8GO/gcCSptwgJwcGDYIPPwQN0xNJb0qqJG4aq0ResqqEtze+\nzS/H/5IObfTrugTPzB4EioBuZrYGmEp4cfh7zew9oAL4DoC7LzWzh4GlQCVwZXO63ocMgaVLlVSJ\npDslVRI3DVUit17GghULuH709UqoJGm4+yUNvPXtBs6/EbixJfeqTapEJL0pqZK42707xKad89m7\n9yM+3l7FX98t4+pTr6YgryDo0EQCMWQI/PWvQUchIq1NSZV8RmNjog5nwoRxPP74tawr/xh6GNlH\nl/Nybhsu7fR7+nWpWw5IJHOop0okMyipSgOxJEJ1r9PQmKimaNu2LZVdV5E1YjtZedk4VfTscBQf\nLvkQPt+ikETSwsCBsGYN7N8P7doFHY2ItBYV/0xxtYnQ4sWjWLx4FBMnzqS0tLRF14oeE1VQMJbs\n7MsOJmtNMX/hfPIK8ygY2pV2x+eSV5BHxUcVLYpFpKnM7FgzaxvZLzKzq80sORaSjGjTBo45BpYv\nDzoSEWlNSqpSXKyJULwd2f1Idu3Zhe93snZkUb2pmvFnqrintKrHgGozGwj8ETgKeDDYkA41dCiE\nQkFHISKtSUmVAOEer02bNrJq1f9j9epZlJU9R3X1fUyYMK7J1zj3K+eybes2urftTvcN3enyQRdm\nXD9DtaiktdVE1uT7D+AOd58M9Ao4pkNoXJVI+tOYqhQ3YcI4Fi2aSVlk2ddwInRNkz5bOxZr48b1\nvPnmBjp1+jEFBWPZsuU2TjnlBCZNavp4KoDVbVYz5ktj6Lm2JzlH5DD+JyruKQlRaWbfBP4LOC9y\nLDfAeOo1ZAg8/HDQUYhIa4pLUmVmZwO/I9zzNdvdZ8TjunJ4hYWFzJp1TdRA9aYlQtGD0levfphd\nu86isHAE/fvn07FjB448cslhrzNv3jxm3T8LgKKvFbGz505+fd6v6dy2c8zfS6QZvgtMBP7H3Vea\n2QDg/oBjOoR6qkTSX8xJlZllAXcCY4ENwOtmNs/dP4j12tI0hYWFzZ7xFz0Wa8uWJeza1Z0NGzaT\nn5/fpM/fcccdTLlzCjbSyOmQwwsLXuCuC+9SQiUJ5+5LgasBzOwIoFMy/mJ33HGwciUcOBAeuC4i\n6SceY6pGASvcfbW7VwIPAVohN4X07j0OmEN5+ctNGksVCoX45R2/pHJEJTWDatjTcw/t8tox96G5\niQtaJMLMSsyss5l1BUqBP0XW6EsqbdtCv36wYkXQkYhIa4nH478+wNqo1+sIJ1qSxOqOxerXr5KR\nI0vp2XNDo48QQ6EQ1029jvKqctgH1dnV5JJLZXnlwXPiVTdLpIny3X2XmX0PmOPuU83s3aCDqk/t\nI0ANNRRJTwkdqD5t2rSD+0VFRRQVFSXy9hLl0LFYvz5s8hMKhbj+lutZ3X01uaflUr6vHHs/iyw3\ncktzmXjrxJgLiEryKSkpoaSkJOgwGpNjZr2AbwC/CDqYxmhclUh6i0dStR44Oup138ixQ0QnVRK8\n5ozFqu2hWt19NX2G9GHt2rVkfZKNP5uLVbWle/tBHHXUUQ0uqqykKnXV/QVo+vTpwQVTv18CTwMv\nufvrZnYMkJQP2YYMgSefDDoKEWkt8UiqXgcGmlk/4BPgYuCbcbiuJInaHqo1NWvYVrONjes30tbb\nU7m5K53bDWFIv1upqNgYaNFRyVzu/gjwSNTrj4GLgouoYUOGwE03BR2FiLSWmAequ3s18BPgGSAE\nPOTuy2K9riSP+Qvnkzs0lyFjh1C1p4qqA1XUfOS0/7gXQ/rdSqdOnw4QmTBhHNXV91FW9lyLCoiK\nNJeZ9TWzv5vZ5sj2mJn1DTqu+gweDB99BFVVQUciIq0hLmOq3P0pYHA8riXJq7xLOT1O6oG/5nTd\n25Vd7bpSUbGRioqNB4uOtrRulkgM7iW8LM1/Rl5/K3LszMAiakBeHvTuHU6sjj8+6GhEJN7M3RNz\nIzNP1L0kvkKhED+47Qds7reZQbmDsKXGzZNvpqKiotVm+WkGYfIyM9zdgo6jlpm97e4nH+5YAuNp\ntK077zz47nfha19LYFAi0mwtaeu0TI3UKxQKMX/hfABOPO1EBhUNYszGMXTJ6sL4yZ8uP9MayY5m\nEEozbTWzbwF/i7z+JrA1wHgaVTsDUEmVSPpRUiWHqB2Ynjs0l/2+n7sfvpvf/efvOP+S8xNyf80g\nlGa6HLgDmAk48DJwWZABNWbIEHhaczpE0lI8KqpLGqktnbBq1yq8nbOp8yb69ujLstdSf+5BaWkp\nxcU3Ulx8I6WlpUGHI3ESWc3hfHfv7u493P1CknT2H4QLf6pWlUh6UlIlB4VCIa6ceiVv2puszlnN\n8288T4d9Heia3TWhcbTGDMLaR4qLF49i8eJRTJw4U4lVers26AAacvzxsHw5VFcHHYmIxJse/6WZ\nWAZ4z/7rbD7I/YCc43Ko2F+Bb3I2LNhAfp98xk8e31ohH6I1ZhDqkWLGSZqB9HV17Ag9eoQXVx44\nMOhoRCSelFSlkVgGeNcOTN87YC/t27Qnv30++9fvJ2d7Djf/7uaDA9MTpTnV3kXqkdRTjWsHqyup\nEkkvevyXRqJ7YwoKxpKdfVmTqpzXDkyvPraaivIKdqzdQe6aXDqs6cD4seMbTKhSaYySipKmHzPb\nbWa76tl2A72b8PnZZrYpevFlM5tqZuvMrDSynR313hQzW2Fmy8zsrFhiHzIEQqFYriAiyUhJVYar\nHZi+sv1Kup3WjbaD2tL2w7ZUv1bN4CMGc8V3rqj3c6k2Rqn2keKYMUsYM2aJSjSkAXfv5O6d69k6\nuXtTeuHvBerLrH/r7oWR7SkAMzuB8ILNJwDnAHeZWYsfMZ5+Oixa1NJPi0iy0uO/NDJhwjgWLZpJ\nWVn4dW2V84bMmzeP4puL2bZ/G5VHVFK9pZoRx4xgc8Vm+m3px63Tb22wlyoVxyjpkaJEc/cXI2uW\n1lVfsnQB4SW4qoBVZrYCGAW81pJ7jxsXLgC6Ywd06dKSK4hIMlJPVRppTm/MvHnz+N7/+x7re6+n\nakgVu/fuJmdTDpuXbWbAvgGNJlQiae4nZva2mf3ZzPIjx/oAa6POWR851iIdO0JRESxYEEOUIpJ0\n1FOVhGLoBo+PAAAgAElEQVSZwdeU3phQKETxzcXsH76fqt5VVFBBp02dyHk7h34D+nHz9PASNMXF\nNzYYQ3N7xURSxF3AL93dzezXwG+A7zX3ItOmTTu4X1RURFFR0SHnXHghPPEEXHJJi2MVkTgqKSmh\npKQkpmto7b8kU3cGX3X1fXEf/zPjdzOYs3gO2/puY2uXrdgBo82/29BnQx8e++NjVFRUNCkGrc+X\nmZJt7b9YRB7//cPdhzf2npn9DHB3nxF57ylgqrsf8vivqW3dli0waBBs3Ajt2sX8VUQkzrT2XxpI\n1Fil7gO6s37HejrmdqRyUyXtPmzHjF/NYOjQoRQX39ikGDRGSdKAETWGysyOdPeNkZdfA96P7D8J\nPGBmMwk/9hsILInlxt27w/Dh8NxzcO65sVxJRJKFkqoMUluLauMnG1m/ez29uvfClhs1m2qY8asZ\nXHDBBUGHKJIwZvYgUAR0M7M1wFTgS2Z2MlADrAJ+CODuS83sYWApUAlcGY+u99pHgEqqRNKDHv8l\nmdZ6/FdbiypnSA7rq9ez/ePtnFdwHjVVNezdlkVBQc+Dj/AS8QhSUlc6Pf5rDc1p6z7+OFxeYcMG\nyM5u5cBEpFla0tYpqUpC9Y1VinX80ozfzeCVylc40PsAW/Zu4cgdRzJgywBefWF7vcmTxktJQ5RU\nNa65bd1JJ8Fdd8Ho0a0YlIg0m5KqFNZYEhNLz1HtI7+SF0vYfMxmao6p4aSeJ7Hjox1sX7ifA7sn\nR42deo4xY5YwY8aUuH8/SR9KqhrX3LZu6lQoL4dbbmnFoESk2VrS1qlOVRI4XHXyli4/M2/ePC66\n8iL+8uFfWD9gPcs2LCPvozx2fLSDylAlA486vvW+lIg0yYUXwt//DvqdUyT1xTRQ3cxuBs4DKoB/\nA991913xCCyTtMaMv9paVDuH7SS7Tza7yncxsNdAOq/szOl9T2f85PEHSyeo1pRIcE4+GaqqwmsB\nDhsWdDQiEotYZ/89A/zM3WvM7CZgSmSTOGpJoc3Zc2azbf82KlZXYD2NDnkdqFpfRdEZRRT/tPjg\nebNmXRP12FGD0UUSzezTWYBKqkRSW9zGVJnZhcBF7v7tBt7XmKoGNGXMVHMGjodCIS668iLKBpSx\ns/1ObLXRuUNnCjYX8Nhdj2n5GYmJxlQ1riVtXUkJXHcdvPFG68QkIs0X6EB1M3uS8IKjDzbwvpKq\nRsRjtl0oFGL2nNnMf24+5YPL2dV7FzntcjjwwQHav9OeP9/yZ9WikpgpqWpcS9q6qio48kh46y04\n6qhWCkxEmqVVKqqb2UKgZ/QhwIFfuPs/Iuf8AqhsKKGq1ZT1sDJVrNXJQ6EQP/r5j/hw+4fszdvL\n/o77aVfTjj7eh+qCas6+6OyUSqhU0iF5xGM9LGlcTg6cfz787W9w/fVBRyMiLRVzT5WZXQZ8H/iy\nu1c0cp56qlpJKBTiuqnX8ebKN8kZlkP10dVs+3gbba0tR3c6mgH7BnDz5JtT5rGfio8mN/VUNa6l\nbd2bb4bHVn38MeTmtkJgItIsCS+pYGZnA5OB8xtLqKT11FZKX919NfuO2UfZ5jIO2AG6detG3to8\n+m3pl1IJFbS8hIRIKvvc58ILLM+dG3QkItJSsc7+uwNoAyw0M4BX3f3KmKOSJpv919msbL+SnG45\n5OTkUF5VTuVLlbTp0YYTup/ArdNvrTeh0uM1keRz3XXw85/DpZeGZwWKSGqJqafK3Qe5ez93L4xs\nSqgSKBQKsWDxArbXbGdru60cyDtAj/Ie9N7Zm0v7X8pd0+9qMKFqrNho0CZMGEd19X2UlT1HWdlz\nkRIS44IOS6TVnX02VFbCc88FHYmItESsPVUSkHnz5jH5V5MpKy/DPjCqulXRfk978lfl89hfGi+b\n0BrFRuOpsLBQ9bMkI2VlwX//N9x6K3zlK0FHIyLNpaQqBc2bN4/vT/s++4buo8IqqC6vpteKXuRX\n5zPutHEpNX6qIbHOhhRJVZdeCjfcAO++C8OHBx2NiDSH1v5LQbPun0X2qGy6n9Id+kNWtywOfHyA\n/p37c8V3rjjs5/V4TSR5tW0LV10Fv/1t0JGISHPFrfjnYW+kkgpxc87Xz+Gtbm/hgxyvdsrfK6f3\nB735+1/+3uReKg1Ul5ZSSYXGxaOt27YNBg6E996DPn3iFJiINEugFdUPeyMlVXHzxBNP8J17vgMD\nIW9vHjVLavjTtD+lVHFPSV1KqhoXr7Zu0iRo1w5mzIhDUCLSbEqq0lgoFGL+wvkAdBrWiZf//TJb\nnt1Clmcx8dsTlVBJwiipaly82rqVK2HEiPDPzp3jEJiINIuSqjQ1b948in9bTNaQLPK657Fj9w4e\n+K8HOPXkU4MOTTKQkqrGxbOt+/a3YcAA+OUv43I5EWmGhFdUl9YXCoUovrmYncfvZEefHSyvWk6v\nTr20FptIBvjf/4Xf/x5Wrw46EhFpCiVVSW7+wvlk98wmu1M2+3P30ymvE1u3bA06LBFJgKOOCs8E\nLC4OOhIRaQolVSmg6zFd2bV7F23L2lKztoaapTWMP3N80GGJSAJcfz28/DK8+GLQkYjI4SipSnJf\n/tKX2Vi1kWPzj6Xrv7uS/34+M66dkRYFPkXk8PLy4Kab4Kc/hZqaoKMRkcZooHoSqp3pV+3VbO23\nlaM6HEXFsgoAxp85nqFDh6rOlAQmXQaqm9lsYDywyd2H13nvv4FbgAJ33xY5NgW4HKgCJrn7Mw1c\nN+5tnTuMHg3f/z5897txvbSINECz/9JAKBTi+luuJ2dIDqurVlP9STV/++HfGDZs2MFzSktL+c53\nfs22beEyCl27zmPOnBuUWElCpFFSdQawB5gTnVSZWV/gz8Bg4HPuvs3MTgAeBEYCfYFngUH1NWqt\n1da9/jpccAF8+CF06hT3y4tIHZr9l+JCoRDXTb2Ole1XsrnjZtr3aM+Aowbwz2f/+Znzbr/9L6xa\ndRb79n2Vffu+yqpVZ3H77X9JaKylpaUUF99IcfGNlJaWJvTeIvHg7i8C2+t5ayYwuc6xC4CH3L3K\n3VcBK4BRrRvhZ40cCWeeGZ4RKCLJSUlVkpg3bx4X/eAi3lz5Jut9PUvXLqVXTi+y7NA/omXLVgLd\nadMmvEH3yLHEKC0tZeLEmSxePIrFi0cxceJMJVaSFszsfGCtu79X560+wNqo1+sjxxLqxhvhT3+C\nDz5I9J1FpClygg5AIrWoflvMzmE7qampYe+OveRX5PPBjg8YsG8A4yd/dqbf8ccfTSg0hwMHukSO\nzOH4449OWLxz5z5NdvZlFBSMBaCsLHxMjx8llZlZe+DnwJmxXmvatGkH94uKiigqKor1kgD07g2/\n+lW4KOjLL0NublwuKyJASUlJzDUglVQlgfkL55M1JIusPll4G6fbB93wd5x+A/px8/SbD5npN2nS\n5bzxxg1s3/4wAL17VzJp0uVBhC6STo4F+gPvmJkRHjtVamajCPdMRf/m0jdyrF7RSVW8TZwI8+aF\nHwNOndpqtxHJOHV/AZo+fXqzr6GkKkl07d6VdeXr6EQnanJryG+Xz63Tb623dEJhYSH33//rqNl/\nP0xoL9GECeNYtGgmZWXh19XV9zFhwjUJu79IHFlkw93fB448+IbZSqDQ3beb2ZPAA2b2W8KP/QYC\nSwKIFzO45x445RT46lfDY61EJDlo9l8SeOWtV/j2nG/TpVMXyreUU7O0hhnXzkjqRZJV0iFzpdHs\nvweBIqAbsAmY6u73Rr3/MTCiTkmFK4BKElxSoT5z54Z7qkpLw7WsRCS+AiupUF9Nl3rOUVJVj/1V\n+7nlpVvoXtmdne/vBD6tRSWSjNIlqWotiWzrLrkECgrg9tsTcjuRjBJIUlVfTZcGzlNSVUdVTRV3\nLrmT7nndueTESwgP4xBJbkqqGpfItm7bNjjpJLj3XvjKVxJyS5GMEVSdqvpqushhuDv3v3M/uVm5\nfPPEbyqhEpFm69oVZs8OV1nftCnoaEQkpoHq0TVdlBQ0z5MfPsnGPRu59vRrD9ai0jglEWmus86C\nyy6Diy6C55+HNm2Cjkgkcx02qTKzhUDP6EOAAzdwaE2XRjOr1qrdkmoWr17MGxve4PrR19M2py3w\naUHN7OzLAFi0aCazZl0Tc2KlRE1iFY/aLdK6pk+H996Dn/wE7r47PENQRBKvxWOqzGwY4fWvygkn\nU7V1W0a5++Z6zs/YMVW1CyQDDBo5iFf2vsLk0ZPp0aHHwXOKi29k8eJRUQU1n2PMmCXMmDGlxfet\nm6hVV98Xl0RNMpvGVDUuqLZu9244/XS48srwJiKxaUlb1+LHf43VdGnpNdNR7QLJuUNzKa8p5+7H\n7uYPF//hMwlVa1Hlc5HM0alTuCjo6NEwZAhk6IMAkUDFc+0/5zCP/zLR7DmzWbVrFWs+XsNqW02/\nnv14++W3DzlvwoRxVFffR1nZc5SVPRcpqDkO0OLFItI0xx4LDzwAF18MKxO3HKiIRMStorq7HxOv\na6WLUCjEglcXsP2E7VQcUUGbN9rQ56g+cNSh5xYWFjJr1jVR45/Cj+liGWulyucimWfsWPjFL2D8\neFi8GLp1CzoikcyhZWpa0fyF8+n1xV5srNxIm/ZtyD4imw2vbmD85ePrPb+wsPCQZCmWR3gNJWoi\nkt5+8hPYsCE8M/C556BLl8N/RkRip6SqFdV4DdvabWNQz0HYTmNP1h7OHnN2Qqul15eoiUh6Mwsv\nuFxeDuecA888Ex5zJSKtK55jqiSKu5N9fDaVGyvptbcXvXN7M2DfAK741hXNuk5jY61ERBpiBr/7\nHZx4Ipx/fjjBEpHWpQWVW8lTHz3F6+tf59wjzmXh8wuBlq/pp1pTkkxUUqFxydbWVVeHi4Nu2RKe\nHdi2bdARiaSGwBZUbtKNkqyhaU2vrXuNJz54guIziunSToMZJL0oqWpcMrZ1VVXwzW/C3r3wyCPQ\noUPQEYkkPyVVSeCDsg/4c+mfufb0a+ndqTeQuJ4m9WhJIiipalyytnWVlfDDH4Yrr//zn9Cj9Uvl\niaQ0JVUBW7drHb979Xf84HM/4LhuxwGJq2qu6umSKEqqGpfMbZ17eEmbv/4V/vUvGDQo6IhEkldL\n2joNVI+T7fu2c+eSO7l42MUHEyqA2267h3Xr8tiyZQm5uUeQnX3Zwd6keIouvVBQMLbV7iMiqcsM\npk2DKVNgzBh49dWgIxJJL0qq4qC8spzbX7udsQPGMqL3iIPHS0tLeeaZZezadRZbt47i3Xdnsnfv\nigAjFRGBK66Ae+4Jzwp85JGgoxFJH0qqYlRVU8UfXv8Dxxccz1eO+cpn3ps792kKCiaRk3MMMJyq\nqgvZsmVOq5REUOkFEWmOc86Bp56C4mKYNAkqKoKOSCT1KamKgbtz39v30aFNB/5z6H9iduij144d\nOzB8+DF067aDzp33MG7cyFYZ51RbPX3MmCWMGbNE46lE5LAKC+HNN2HNGjjjDK0XKBIrDVSPwWNL\nH+Pf2//NNaddQ2527iHvBz14vO5sQECzAyVmGqjeuFRs69zhttvCVdjvvhv+4z+CjkgkeJr9l0CL\nVi5i0apFFI8upkObhou+BFXmoG5Ct2vXbzBrR6dOPwY0O1BaTklV41K5rXvtNbj4YjjzTLj5Zq0Z\nKJmtJW2d1v5rgbc+eYunPnqKyaMnN5pQQXBr79VdiHn16oeBzzNgQPMXZhaRzHDqqfD22/Czn8HQ\noXDHHfC1rwUdlUjq0JiqZvr3tn/z13f/ypUjr6QgryDocERE4io/H/7wB3joIfj5z8NJ1YYNQUcl\nkhqUVDXDpj2bmPXGLC4/5XL6dekXdDiNqjsb8Igj1tK16zzNDhSJMLPZZrbJzN6NOvZLM3vHzN42\ns2fNrG/Ue1PMbIWZLTOzs4KJOnG+8IVwr9WwYXDSSfCb38D+/UFHJZLcNKaqiXZV7GLGizP46qCv\nMvro0UGH0yQaqC6tIV3GVJnZGcAeYI67D48c6+jueyL7VwHD3f37ZjYEeAAYCfQFngUG1deopXpb\nV59ly8KPBN95B379a7jkEsjSr+SS5jRQvZVUVFXwm1d+w/Cewxl/3PigwxEJVLokVQBm1g/4R21S\nVee9nwFd3P1nkX139xmR9/4FTHP31+r5XMq2dYfzf/8HkyeHa1rNmBEe0F5PJRmRtBDIMjVmdlWk\nO/w9M7sp1uslm+qaau5+8276du7LuYPODTocEWllZvZrM1sDXAbcGDncB1gbddr6yLGM8oUvwCuv\nwA03wFVXwWmnweOPQ3V10JGJJIeYZv+ZWRFwHnCiu1eZWVqN3HZ3HnjvAQzj0hMvrbe4p4ikF3e/\nAbjBzIqB3wHfbe41pk2bdnC/qKiIoqKieIUXODO46CK48EKYNy/cYzVlClx3HXz729CuXdARirRM\nSUkJJSUlMV0jpsd/ZjYXuNvdn2/CuSnXJT5/+Xze2fgO133+OtrmtA06HJGkkEGP/44CFrj7ifU8\n/nsKmJppj//q4w4vvBCua1VaCt/9Lnzve3DssUFHJhKbIB7/HQeMMbNXzWyRmY047CdSxEtrXuKV\nta9w1alXKaESSV8W2cIvzAZGvXch8HZk/0ngYjNrY2YDgIHAkoRFmcTMoKgIFiyARYvgwAE4/XQY\nOzZclkFrCkomOWxPlZktBHpGHwIcuAH4H+B5d59kZiOBue5+TAPXSZnf3kKbQ9z39n1c9/nr6Nmx\n5+E/IJJB0qWnysweBIqAbsAmYCpwLjAYqAI+Bn7k7psj508BrgAqgUnu/kwD102Ztq61VFSEHw3+\n6U/h3qsLLoBvfCOcaOUeuqKXSFJK+Ow/M1sAzHD3FyKvPwJOdfet9ZzrU6dOPfg6WccZrNm5htte\nvY0rR17JsV3Vfy1Sd5zB9OnT0yKpai1Kqj5r3Tp49FF4+GFYvjw8FutrX4MvfQnatw86OpGGBZFU\n/QDo4+5Tzew4YKG711sVMxUamq3lW7n5pZu5eNjFnNLrlKDDEUlK6dJT1VpSoa0Lypo14QRr3jx4\n6y0YPRrOOSe8DRoUdHQinxVEUpUL3AOcDFQA/13ba1XPuUnd0Ow9sJcZL83gS/2/xJcGfCnocESS\nlpKqxiV7W5csduyAZ5+Ff/0rvLVpA2PGwBe/GP45cKBqYEmwVPyzhSqrK5n56kyOPeJYLhpyUdDh\niCQ1JVWNS+a2Llm5w4cfwuLF4ZmEL7wQrn11+ukwcmR4GzECunQJOlLJJEqqWqDGa/jjm38kJyuH\nK065QrWoRA5DSVXjkrWtSyXusGoVvPYavP56eHvrLejdO7wO4YknfroNGKAlc6R1KKlqJnfn4dDD\nrNu1jkmnTSInK6ZaqCIZQUlV45KxrUsH1dXhNQjffRfee+/TbevW8HiswYPhuOPC26BB4WSre3c9\nQpSWU1LVTAv/vZCX177M5NGTycvNCzqcFqm7aLIWSZbWpqSqccnY1qWzXbvCswqXLw8/QqzdX7ky\nXDOrf/9wgtW/P/TtC336hH/27Rvu+cpLzaZfEkBJVTO8vv51Hlv2GMWjizmi/RFBh9MipaWlTJw4\nk+zsywCorr6PWbOuUWIlrUpJVeOSra3LZDt3hh8jrlwJq1fD+vXhEg+124YN4QHyPXvCkUeGtx49\noKDgs1vXrnDEEeEtP1+PGzNFS9q6jHzetXzrcuaG5vLT036asgkVwNy5T5OdfRkFBWMBKCsLH1NS\nJSISToBOOim81cc93NO1ceOn25Yt4bb0ww/hpZfCr7dtg+3bw9uePdC5c/jatVvt644doVOnT392\n6PDplpf36c/27Q/dcnP1qDIdZFxStWH3Bv745h/5XuH36Nu5b9DhiIhIQMw+TYwGD27aZ6qqwj1g\nO3eGE7Lo/T17YPfu8M8NG8L75eWwd+9nf+7bd+hWXQ1t24a3du0+3W/T5tOfbdqEk6/an/VtOTmf\n/ozesrPDW/R+3S0rq+GfdffNDt2veyz6eFO32vNr/3yit9pj0e/Vtx/9s6nHOnSIz+zSjEqqamf6\nfX3I1zm+4Pigw4nZhAnjWLRoJmVl4dfV1fcxYcI1wQYlIpLGcnKgW7fwFk81NeHlffbvD28VFeEx\nYQcOhPcrKqCyMrzVHq99XbtVVX36M3qr/Ux1dXirqvp0v3arqWn4Z91j7uH92p8N7bsfun+4raYm\n/N+j7vHaY9Hv1bcf/bM5xy69NLwoeKwybkzVropddG7bOegw4kYD1SXRNKaqccnS1olIbDRQXURa\nnZKqxqmtE0kPLWnrNIdBREREJA6UVImIiIjEgZIqERERkThQUiUiIiISB0qqREREROJASZWIiIhI\nHCipEhEREYkDJVUiIiIicaCkSkRERCQOlFSJiIiIxEFMSZWZjTSzJWb2VuTniHgFJiLSmsxstplt\nMrN3o47dbGbLzOxtM3vMzDpHvTfFzFZE3j8rmKhFJJnF2lN1M3CDu58CTAVuiT2k1ldSUhJ0CEDy\nxAGKpT7JEgckVyxp5F5gXJ1jzwBD3f1kYAUwBcDMhgDfAE4AzgHuMrO0Xf8w1f++pXr8kPrfIdXj\nb6lYk6pPgPzIfhdgfYzXS4hk+cNOljhAsdQnWeKA5IolXbj7i8D2OseedfeayMtXgb6R/fOBh9y9\nyt1XEU64RiUq1kRL9b9vqR4/pP53SPX4Wyonxs//DHjJzH4DGPD52EMSEUkKlwN/i+z3AV6Jem99\n5JiIyEGHTarMbCHQM/oQ4MANwFXAVe7+hJl9HbgHOLM1AhURSRQz+wVQ6e5/O+zJIiIR5u4t/7DZ\nLnePHsi5093zGzi35TcSkaTi7mkxnsjM+gH/cPfhUccuA74PfNndKyLHfga4u8+IvH4KmOrur9Vz\nTbV1ImmiuW1drI//VpjZF939BTMbCyyPV2AiIglgkS38wuxsYDIwpjahingSeMDMZhJ+7DcQWFLf\nBdXWiWSuWJOqHwK/N7M2wH7gB7GHJCLS+szsQaAI6GZmawjPYP450AZYGJnc96q7X+nuS83sYWAp\nUAlc6bF084tIWorp8Z+IiIiIhCW8orqZXRUpnveemd2U6PvXieW/zazGzLoGGEODxQYTdP+zzewD\nM1tuZsWJvHedOPqa2fNmFor83bg6qFgi8WSZWamZPRlwHPlm9kjk70jIzE4NMJYpkRjeNbMHIj3U\nEiVZ/j01RwNFUI8ws2fM7EMze9rM6h0rmwwaajtS5TuYWVszey1SRDtkZv8bOZ4S8deq22amYPyr\nzOyd2mLmkWPN/g4JTarMrAg4DzjR3U8Ebk3k/evE0pfwTMXVQcUQUW+xwUQwsyzgTsIFEIcC3zSz\n4xN1/zqqgGvdfShwOvDjAGMBmET4UU/QbgMWuPsJwEnAsiCCiAzo/j5wSmRQdw5wcRCxJKsk+/fU\nHPUVQf0Z8Ky7DwaeJ4HtUgs01HakxHeIjN37UqSI9nDgy2Y2mhSJP0rdNjPV4q8Bitz9FHevrUHX\n7O+Q6J6qHwE3uXsVgLuXJfj+0WYSHpAaqEaKDSbCKGCFu69290rgIeCCBN7/IHff6O5vR/b3EE4e\nAqkDFEm4vwr8OYj7R8XRGfiCu98LECk8uSugcHYBB4AOZpYD5AEbAoolWSXNv6fmqK8IKuG4/xLZ\n/wtwYUKDaoYG2o6+pNZ3KI/stiX8/+XtpFD8DbSZKRN/hHFoTtTs75DopOo4YIyZvWpmiyygtQLN\n7Hxgrbu/F8T9G3E58K8E3q8PsDbq9TqSoKChmfUHTgYOma6eILUJd9ADDgcAZWZ2b6Rb/Y9m1j6I\nQNx9O/AbYA3hwpc73P3ZIGJJYkn576mFerj7JggnLUCPgONpkqi241WgZ6p8h8ijs7eAjUCJuy8l\nheKn/jYzleKHcOwLzex1M/te5Fizv0Oss/8OYY0XC80BjnD308xsJPAwcEy8Y2hCHD/ns0VKW3UK\ndCOx/MLd/xE5p7bY4IOtGUuyM7OOwKPApMhvnYm+/7nAJnd/O/K4Osjp8TlAIfBjd3/DzH5HuDt6\naqIDMbNjgGuAfsBO4FEzuyTT/75mkKB/wTisum2HHVovLGm/Q+RpxSmR3umnI21PSsRfT5vZkKSM\nP8pod//EzLoDz5jZh7TgzyDuSZW7N1hR3cwmAo9Hzns9Mki8m7tvTVQcZjYM6A+8Y2ZGuJv4TTMb\n5e6b4x1HY7FExXQZ4a7TL7fG/RuxHjg66nVfAly/MfJY6VHgfnefF1AYo4HzzeyrQHugk5nNcffv\nBBDLOsI9qm9EXj8KBDX4eQTwkrtvAzCzxwkvS6Wk6lNJ9e8pRpvMrKe7bzKzI4FWaRvjpYG2I6W+\nA4C77zKzBYT/vaVK/PW1mfcDG1MkfgDc/ZPIzy1m9gThx/nN/jNI9OO/J4gkDmZ2HJDbGglVY9z9\nfXc/0t2PcfcBhP/HdUprJVSHY58WGzy/TrHBRHgdGGhm/SIzuS4mXOQwKPcAS939tqACcPefu/vR\n7n4M4f8ezweUUBHpdl4b+bcCMJbgBs9/CJxmZu0iv4yMJaBB80ks2f49NcdniqASjvuyyP5/AUH9\nktNU9bUdKfEdzKygdlZZ5PH+mcBbpEj8DbSZ3wb+QQrED2BmeZGeTsysA3AW8B4t+DOIe0/VYdwL\n3GNm7wEVQCD/s6rDCfYRzx3UU2wwETd292oz+wnhGYhZwGx3D2p22WjgUuC9yNgCB37u7k8FEU8S\nuZpwJe9c4GPgu0EE4e7vmNkc4E2gmnCj/8cgYklWyfTvqTms/iKoNwGPmNnlhGdIfyO4CBvXUNsB\nzAAeToHv0Av4S+SXlSzCvW3PRb5LKsTfkJtInfh7An+PPDLOAR5w92fM7A2a+R1U/FNEREQkDhJe\n/FNEREQkHSmpEhEREYkDJVUiIiIicaCkSkRERCQOlFSJiIiIxIGSKhEREZE4UFIlIiJJz8y6mtlb\nkXUwPzGzdVGvm1Rz0cxmm9mgw5xzpZl9Mz5R13v9/4gq6CtpRnWqREQkpZjZ/wfscfff1vOeeRL/\nj/Hvu0AAAAM/SURBVC2yhMujAS7FJa1IPVUiIpJqDq6CYWbHmlnIzP5qZu8DR5rZ3Wa2xMzeM7Mb\nos79PzMbbmbZZrbdzG40s7fN7CUzK4ic8yszuzrq/BvN7DUzW2Zmp0WO55nZo2b2vpk9Ymavm9nw\nQ4I0uyUS29uR65xBeJ3X30Z62I42s4Fm9lTkGiVmNjDy2fvN7C4ze8PMPogsaYaZDYt8t9LIdfu3\n2n9labZEL1MjIiISb4OBb7n7WwBmVuzuO8wsG1hkZo+6+wd1PpMPLHL3KWb2G+By4Ob6Lu7up5rZ\neYSX8DkHuAr4xN2/Hkmm3qz7GTPrAZzj7kMjrztHLZj8iLs/GTn+PHCFu680s88DvwfGRS7T191H\nRB4XPmtmxwJXAre4+yOR5auCXGZN6lBSJSIiqe7ftQlVxKWR9dpyCK+tNwSom1SVu/szkf03gTMa\nuPbjUef0i+yfQXhtO9z9XTML1fO5bUC1mf0RWADMr3tCZCHl04DHImv/wWefID0cucfyyLqMg4CX\ngf8X6aF63N3/3UDcEgA9/hMRkVS3t3Yn8vjsaqDI3U8Cngba1fOZA1H71TTcyVDRhHMO6S1y9yr+\n/3bun7WKIArD+HNAsUgwtWCTIgEL8Q9+kDSSYJ/0ySfIZ0gaLcWAndgoWthpFywsTCEIFiFFGjU2\nQfG1uINeNtFwYcBceH6wsOzOzG75cuYwcAd4CiwBz/4y7zDJ7SS32nVjfJnB2CTZaesdAy/alqLO\nCUOVJGnajYeay8BX4FtVXeHPVtq/5kzqDbAMUFXXgWsnFq+aBeaSPAc2gJvt1VH7R5J8Bg6qaqnN\nqUFv1t32fBG4CnyoqvkkH5NsMap+nejl0v/j9p8kadr9rugkeVtVe8Ae8Al4fdq4wf2Z6w5sAw9b\nY/z7dn0ZjJkDnlTVJUYBbr09fww8qKoNRhWnFeB+VW0CF4Ed4F0bu19Vu8AMsJrkR1Xda0c+fAf2\nGfV56ZzwSAVJkibQGuAvJDlu240vgYUkPzt+4xFjDe2aDlaqJEmazCzwauzQ0bWegaqx4jGFrFRJ\nkiR1YKO6JElSB4YqSZKkDgxVkiRJHRiqJEmSOjBUSZIkdWCokiRJ6uAX5JpS92QHQFAAAAAASUVO\nRK5CYII=\n", + "text/plain": [ + "<matplotlib.figure.Figure at 0x7f8040158550>" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "#@test {\"output\": \"ignore\"}\n", + "\n", + "# Use the same input data and parameters as the examples above.\n", + "# We're going to build up a list of the errors over time as we train to display later.\n", + "losses = []\n", + "\n", + "with tf.Session() as sess:\n", + " # Set up all the tensors.\n", + " # The input is the x values with the bias appended on to each x.\n", + " input = tf.constant(x_with_bias)\n", + " # We're trying to find the best fit for the target y values.\n", + " target = tf.constant(np.transpose([y]).astype(np.float32))\n", + " # Let's set up the weights randomly\n", + " weights = tf.Variable(tf.random_normal([2, 1], 0, 0.1))\n", + "\n", + " tf.initialize_all_variables().run()\n", + " \n", + " # mu is the learning rate (step size), so how much we jump from the current spot\n", + " mu = 0.002\n", + " \n", + " # The operations in the operation graph.\n", + " # Compute the predicted y values given our current weights\n", + " yhat = tf.matmul(input, weights)\n", + " # How much does this differ from the actual y?\n", + " yerror = tf.sub(yhat, target)\n", + " # Change the weights by subtracting derivative with respect to that weight\n", + " loss = 0.5 * tf.reduce_sum(tf.mul(yerror, yerror))\n", + " gradient = tf.reduce_sum(tf.transpose(tf.mul(input, yerror)), 1, keep_dims=True)\n", + " update_weights = tf.assign_sub(weights, mu * gradient)\n", + " \n", + " # Repeatedly run the operation graph over the training data and weights.\n", + " for _ in range(training_steps):\n", + " sess.run(update_weights)\n", + "\n", + " # Here, we're keeping a history of the losses to plot later\n", + " # so we can see the change in loss as training progresses.\n", + " losses.append(loss.eval())\n", + "\n", + " # Training is done, compute final values for the graph.\n", + " betas = weights.eval()\n", + " yhat = yhat.eval()\n", + "\n", + "# Show the results.\n", + "fig, (ax1, ax2) = plt.subplots(1, 2)\n", + "plt.subplots_adjust(wspace=.3)\n", + "fig.set_size_inches(10, 4)\n", + "ax1.scatter(x, y, alpha=.7)\n", + "ax1.scatter(x, np.transpose(yhat)[0], c=\"g\", alpha=.6)\n", + "line_x_range = (-4, 6)\n", + "ax1.plot(line_x_range, [betas[0] + a * betas[1] for a in line_x_range], \"g\", alpha=0.6)\n", + "ax2.plot(range(0, training_steps), losses)\n", + "ax2.set_ylabel(\"Loss\")\n", + "ax2.set_xlabel(\"Training steps\")\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "TzIETgHwTexL" + }, + "source": [ + "This code looks very similar to the code above, but without using `l2_loss` or `GradientDescentOptimizer`. Let's look at exactly what it is doing instead.\n", + "\n", + "This code is the key difference:\n", + "\n", + ">`loss = 0.5 * tf.reduce_sum(tf.mul(yerror, yerror))`\n", + "\n", + ">`gradient = tf.reduce_sum(tf.transpose(tf.mul(input, yerror)), 1, keep_dims=True)`\n", + "\n", + ">`update_weights = tf.assign_sub(weights, mu * gradient)`\n", + "\n", + "The first line calculates the L2 loss manually. It's the same as `l2_loss(yerror)`, which is half of the sum of the squared error, so $\\frac{1}{2} \\sum (\\hat{y} - y)^2$. With this code, you can see exactly what the `l2_loss` operation does. It's the total of all the squared differences between the target and our estimates. And minimizing the L2 loss will minimize how much our estimates of $y$ differ from the true values of $y$.\n", + "\n", + "The second line calculates $\\sum{x_i (\\hat{y} - y)}$. What is that? It's the partial derivative of the L2 loss, the same thing as what `gradients(loss, weights)` does in the earlier code. Not sure about that? Let's look at it in more detail. The gradient calculation is going to get the partial derivatives of loss with respect to each of the weights so we can change those weights in the direction that will reduce the loss. L2 loss is $\\frac{1}{2} \\sum (\\hat{y} - y)^2$, where $\\hat{y} = w_2 x + w_1$. So, using the chain rule and substituting in for $\\hat{y}$ in the derivative, $\\frac{\\partial}{\\partial w_i} = \\sum{(\\hat{y} - y)\\, x_i}$. `GradientDescentOptimizer` does these calculations automatically for you based on the graph structure.\n", + "\n", + "The third line is equivalent to `weights -= mu * gradient`, so it subtracts a constant the gradient after scaling by the learning rate (to avoid jumping too far each time, which risks moving in the wrong direction). It's also the same thing that `GradientDescentOptimizer(learning_rate).minimize(loss)` does in the earlier code. Gradient descient updates its first parameter based on the values in the second after scaling by the third, so it's equivalent to the `assign_sub(weights, mu * gradient)`.\n", + "\n", + "Hopefully, this other code gives you a better understanding of what the operations we used previously are actually doing. In practice, you'll want to use those high level operators most of the time rather than calculating things yourself. For this toy example and simple network, it's not too bad to compute and apply the gradients yourself from scratch, but things get more complicated with larger networks." + ] + } + ], + "metadata": { + "colabVersion": "0.3.2", + "colab_default_view": {}, + "colab_views": {}, + "kernelspec": { + "display_name": "Python 2", + "language": "python", + "name": "python2" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 2 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython2", + "version": "2.7.6" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/tensorflow/tools/docker/notebooks/3_mnist_from_scratch.ipynb b/tensorflow/tools/docker/notebooks/3_mnist_from_scratch.ipynb new file mode 100644 index 0000000000..b51983b71c --- /dev/null +++ b/tensorflow/tools/docker/notebooks/3_mnist_from_scratch.ipynb @@ -0,0 +1,1985 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "9yupXUk1DKOe" + }, + "source": [ + "# MNIST from scratch\n", + "\n", + "This notebook walks through an example of training a TensorFlow model to do digit classification using the [MNIST data set](http://yann.lecun.com/exdb/mnist/). MNIST is a labeled set of images of handwritten digits.\n", + "\n", + "An example follows." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "collapsed": false + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAMYAAABFCAYAAAARv5krAAAYl0lEQVR4Ae3dV4wc1bYG4D3YYJuc\nc8455yCSSIYrBAi4EjriAZHECyAk3rAID1gCIXGRgIvASIQr8UTmgDA5imByPpicTcYGY+yrbx+t\nOUWpu2e6u7qnZ7qXVFPVVbv2Xutfce+q7hlasmTJktSAXrnn8vR/3/xXmnnadg1aTfxL3/7rwfSP\nmT+kf/7vf098YRtK+FnaZaf/SS++OjNNathufF9caiT2v/xxqbTGki/SXyM1nODXv/r8+7Tb+r+l\nnxZNcEFHEG/e3LnpoINXSh/PWzxCy/F9eWjOnDlLrr/++jR16tQakgylqdOWTZOGFqX5C/5IjXNL\njdt7/NTvv/+eTjnllLT//vunr776Kl100UVpueWWq8n10lOmpSmTU5o/f0Fa3DDH1ry9p0/++eef\naZ999slYYPS0005LK664Yk2eJ02ekqZNnZx+XzA/LfprYgGxePHitOqqq6YZM2akyfPmzUvXXXdd\nHceoic2EOckxDj300CzPggUL0g033NC3OKy00krDer3pppv6FgcBIjvGUkv9u5paZZVVhoHpl4Mv\nv/wyhfxDQ0NZ7H7EQbacPHny39Tejzj88ccfacqUKRmHEecYf0Nr8GGAQJ8gMHCMPlH0QMzmEBg4\nRnN4DVr3CQIDx+gTRQ/EbA6BgWM0h9egdZ8g8PeliD4RutfF/Ouvfz9OtZy8aNGiNH/+/GGWl112\n2XzseYuVNKtqsaI23Ghw0DYCA8doG8JqO+AUG2+8cVq4cGHaY4890vLLL5/WXXfdfI6jvPDCC3lJ\n8amnnkoezP3000/pl19+GThHtWpIPekYomTxFS7HnkqKjMsss0yGgFE4r62tSBFVJ02aNPyconi9\nV4/JwzHwT9ZNNtkkeZ6w5ZZbph133DH99ttv6ccff8zXX3nllcRRnHNfv2cNGMQWGRaOrWbUrjsG\nBRLAA6U4Lhoqw9h2223ztRBq6aWXzsbgvueffz4Lu9NOO2UnYTgrr7xy7tO9nOH111/Pbb744ov0\nww8/jAvngAdFMvQDDjggG/0GG2yQX1GZNm1aziCCwzrrrJPl3muvvXKwePnll9M333wzHDCKWPbL\nMbuAkfISjnvvvXcW/emnn85lqCBqa4a65hiYR/Gk2RNGRlwm3n7ggQfmdrKD9sqJtdZaKxvCnDlz\n8n3Tp09PXmPYeuutc0SVNQjvnmuvvTa3efzxx9N33303PGZ5rF75DBvvqq233nrp22+/TWeddVby\nikpgxCE4vQDhlQUBRfDw2esbs2fPTquvvnqviNN1PuIdJ4GErVx44YUZowsuuCB9+umn6eeff84B\nspmsWqljhPFDxjGGYx/lDkN33udajCoVlAjRzl4U8LjefRwnPjsXG8OJqKBd8NB1LTU5IHyCd7LJ\nGOYXNoGjFqaGIKtrERDIDKtukfGMH/zRZa1A101+YBF44KfMYzO8VOYYjDWiukiGqc022yyXOUqd\nzTffPJ/z1ialeqNVxA9gi0wzlOJ5juJlR8JeddVV+ZrIKTq4ZvJp/8EHH+SU+txzz+W2SqmxVFZR\nplrH5DTRXmGFFdKuu+6azjjjjOzosl5g6D54CQCI4mGjhNQO5occckh2LvLTA6fqJOEnyhU6kNlk\nZmUuvrtNcFx77bUzhsZWXgoSsm6t4Dsa/tp2DErCmA04HAI4FLjaaqtlBhmnSKiNY4rDtHZFB6jF\nMMH0RVDH+nCPYxtDCFJnKkniRbDitWjTK3sykQUuMLPn3DZGX8SFnCG/fVyz5zCCBtIHTLshdzif\n8fERn8cKXxjCNOwCTu3Qf6yqhV4AQokiP489//zzM0DxnQYKwqAtIkko1kQzFFxvaNcJ6u3Pe+65\nJ/cRRvDee+9lA2BInIyRff/997nNO++8k7t0vl2A6vHWynmyiPJ43WKLLbIijz/++LTddtvlTCdz\nwIWSg9yjxBJ0GN/DDz+c7zv77LOzbEceeWSekwVGgsOsWbNyNo0+qt7DfPvtt8/dmtvIGnPnzk3P\nPPPMsJ6rHrNef/BBeJA90RprrJEDcNhctMkXR/mnbccwuCjNGTbaaKMc8TBZprITxOdgOvbuKxqG\nz6LSJ598kseJ9Gi1CYmSv/76a3YyJZWMZJ6Ceskp8EMusihFEAyUmVaa8G2rxTNHIrd733///eH7\nYeaLNe5xrEzlWNF/HqQDf0Tm+GIbvYdD43MsKAIo/JDgE0G5aFfN8NaWYxiUshikqGYTTUSt0TCk\njXsYNqJQQso+rgGa0vX58ccf56hQTtk+48F92rmvlnE1A0on2uKP0Yrw+Nxzzz0zn+ZhjKwRXq6v\nueaa2TmUiRQfS7SyNeMks9IV9vrvJOl/q622yo4Mfw5Pvm6TMclLdit6shh+YAMnq1E29tEsteUY\nBgMSgxa5MOAzJZcVXQs4bUR8XxhCHIwzMALCBuCcx5q0tF3u133l8XrRMchFiRYNyMxBKM/5IjZl\nWVzjULKwACISytIWFsi56aab5mvOKyEikmdAO/iHY+BDCRUZuoPD1e1akECyLseA7d13352DhdKa\nk8Cmlt3U7TSl9p58FwejYK8ncAwKpDTnGDcARbWiAUjHiNEHsITSPlagpEZChcfrZzwSOfBOiQwX\nLuR3PjAhtwAD08iAMCO/a+5xPTIm3ALjwERf0V+c69QeT7ZujVdLDhgKBrANXAMreMESRkU7rdVP\nrXNtZ4xIpSLH1VdfnR3j4IMPzkbw2Wefpa+//jovo5188slZsZjArAcvFP3YY4+lSy+9NEdTdTTy\n0I5xHHfccfm1CH2LtuORKEqmkwVlVU+sBY+IdJRmE0zeeOONnEXuu+++7AhnnnlmWn/99XMJ5brt\nzTffzHMJx/o555xzkgdb0U8rRtAKrnTYqtG1Ml6teyxInHDCCdlGYByBmG2Z97ChVvFo2zEwbHCR\nTbqP7EDxPjN2pUBEe86AXAcsg+f10TYMSTvnRM1ulQe1wG/nHEXZZEJZUIYQ5cgWMsEgMgqclFdk\ndh+MbFFyuddnWMLNfTYkcuuXHlBkpFYNI3dS+mMMfCHHsZWadfUjmQVn8iLywscG21apMscQwR55\n5JEM3KuvvpoZ5LHOmzgjAvBwzFt2/Oijj3Lm4Ayin/MU/eGHH+b2N998c/5MGSaZ44nw7OEd5Rx7\n7LE5+1EehYXxkpes5li2K6+8Mhv8Lrvsko381ltvzcEBfvHQKh5auk9GPvHEE3NJAx+/eKL/HXbY\nIQcbK3nwN067xAk4s5VHdbvsx0nxrYQeKxJMZAfBA7GlRx99NC9EtCN7JY4RoPBeAHIAyrB3jpHY\nwqu1d02d7HpZcfqINo5dL7eJMXtxTzk2sgWFM/gcsnCakI2cFOk+523O+Qw7WaeYHYpYRp9xn4Bk\nbPdWSfgJXYYM+ne+2xRj2sdx8EDu8rm4Ntp9pY4RSmb0CIPOAVNGoLA47yU4S2xen37ppZdy9CkL\nE/3lm8bJHzJbbiavt2Q9p7AkK7oyXAZOLk7gs9c4PJC0AOE8DDyrgJkaWgYQkSPYuAdpWySfteU8\nHhqKouYq+io6ZfGeZo7xpbT1+jt+jGULfprpq922ePHMBibwjWVq523KVrzBsIzTaMeu1DFi0HI0\nYyyYtAekY5MltbRyihFJiROBKIYTwMCTWJNubwdQFCXFapK9z96mtbjgs3thFKWnUgjBzNZIya5F\nOyUcPG36q4LwRgZ6Ix8HtBk3tirGGU0feAkslHfk5PzBh2cXSkvtWqWOOEaRGcoSHdXDMoYn1tK8\nyaON0ahbCWgFS/vxSnjn5F4ItLeiFAGAzCKc7MDA1OlIjc4pLFKE7FEyxb5ZPNTbtuiv2fvrtddf\nOFsYXcwj8d8qv/XGq3femLvvvnvOvrIYPPEjG+PDseDbDnXcMXiyiGiyyACOPvrovN95552zV3/+\n+ef5zVveznlEo6CICvG5l/d4JSvHP+qoo7JjKDs4PkVSGPm9HSz9W5rlPEoCQYHjVFXyRGnBOcKA\n28VOP/qTBWX6YnS2IKB8qYL/enyGHPbKziOOOCLj6sGeslGW8L6Y4ANr2MY99fpsdL7jjmFwkSTS\nr6gDVCk+tmDQedcJ5LgdwaLPbu7xjJRRNlErSsiQhVHJlOEQoh182o1wRTnharwYs3itnWP9Rd/R\nD5mLW5yveh/YRhYMjItyBh/wjPat8tEVx6B00RKo5513XpIl7rzzzuwEourMmTOz95uIcyBfTSXY\niy++mCOrSFS1klsFrNZ9eGPoJtmeyRx00EE5cpGbIi21XnbZZbkMee2117KMHIKMIVcotVb/vXoO\nz6I0+URoMlVFcBFE7L1+IjNYIo6v/fo+D3tC+FCR+FHuwNUCgfOtUlccI5hnJMoIBhN1sBICqMoN\nNaLP3pkiFGciIIBC4HaEbRWk0dyHb3Mp/EY0I6+NsytvyKxsKhpQr8ozGpm1IZ8IbV+PyllGuyh1\nYBXXOQEcy6R8M5eAHzuxxX3GRvbaCKJ4aRfXrjkG5jEbk00Prxi8SZTJKmc5/PDDc5v99tsvC+hB\njWtqStmD0F4Ma1foMvDtfqZMUc3/lYjMSFFW3NS7JtyyoKzSiTocHoFJHMc+MlK7Mta7n9NbATJe\nrbEYvQWIWCVitIyaXrV3nsG7H2Y2GVcbxyj6NX+waKEPmOvbfShwtjhQDDz5Ygt/uuoY+OPtnICD\nEMBTWsAQUu0NBBsDEgFEWOADAiDaVRERWsCq5i34IRN+TbTJgn8KwzOFuR4KDUXW7Kyik53Ep8w/\n+RkxWeO5S1EM5wVABguXMGp69dk1x87D0ObdL32GHI5tsDQGHtwbm/Hw4TpnKvNY5Ge0x113DEwT\n3tIsIdSnDIfxcxJAevCHfE9cXcmotHXfAw88kIFUdgFjLMn4HuZRuh9FExmjRCCnZxRqcPxz8ioU\nVk9eRhJkPAYHV8ZVFRkjjFSfAtw222yTy2OZ0iv15fHcQ4dKaMcwsBdEEL26RzaIh5+yK7LSBGPn\no8yOZX+vzRhfXzZ8cRrtyzzkzpr803XHwB8wTJYIRol+VY8zqMMBbP0f+cExE1qTdbU7x3jwwQdz\nVBYdesExKNiEWx2MfwoOAyCbJ9uRHZvUTcPmsENhGNE4HBKOHKNqZzQu3KNfX9H1nRABQZlbNkpt\n4SNo4DWIIesDj9qYnwki2giWqol3330348kZLPm7xvi1Pffcc7MzhA3gy/0oeIuxWtmPiWNgNCIF\nYwcCAa2FA1ikJZz1aeUVsBmge9TyoqGoIqKUFdEKCFXcU0/pHJizVMUnXBiBh6IicdTTzsEOnuZk\nDE/2rcJI4KMf/TF+0TucwDhkZ+DGL4/nGkPGV/AIC+2RvfP6ZPTI4gu5XNM/Um7RPzuIFyn1zW7w\npQ9UHj+fbOHPmDlGCOGBGIeQQfwuq0jnISBQfOHft7JEHN94Q5xF6XLFFVfkyKIEGyuiGAo3r6BI\nx0imcM6k+6GHHspOEQbcDq+UTl4BwRu7PstUiPEJFsa9/PLL83nXg6d2xnUvoxS5L7744uGyh/wy\nRpRF9YwSHsHjE088kWWADQeRFThZkTgBstensZG5h4m56oEdcAp9CwTOVUlj6hgECcGBpA6XDaze\niLKhVABQAhKB3cNxbEAL4KoEppm+gjf3OMafDf+UW7zeTL/ltqIiAxBMOIIxnLOHgbFsMGQ4InhE\n0nJfrXw2hnIRD3SFBKmYWDfqE49woFvOzZno3NxM0HDciMjBDsjEBgLTsJHYN+qjmWtj7hjBLKFF\nQgL7qRz14jHHHJPBcC2M3wRPVDT5ohzZRv0Z16O/sdozAKmdopUH5kftTrzJpl+lk29CcgpLw3Bg\npMbwwqF/S80pGJ6xO0WM+8Ybbxw2TuOEoTYakwyovB/JKdzDMVQOHvCRzXju890fL11aGhcMqqIx\ndwwCRkYQDZAaE7lWBhyosQEmQM439MgffDHm0Si8EcuBC0ezcQSZVKYktzFEW+3sfQ4natRvu9eM\nTS9F7IvHo+m/2fb6LNuCc0WsW+mzHq9j6hgE9YCHp5tkez2EAVjlMOmyUlU2Lis8ygVR0rykyolt\nPZCaOY9fr32Qp50X6xi7pWCGbsHBvwLgGIcddljGxvcsjOU1GseyiKjJQWydpiqNsBlei85BfhNx\neJunVCl31x0jBOMAjJ9jRC3OEERDS7QMI0qQohIYgLSq7FJuMZbi9WZA7kRbvFAWx5Dyy449mjED\nG/dyDPW4VSiy2iNvBcCSUdxyyy35OYHrqJUx843j8I/qQpA074BVVdR1x+AIHCIiIGewsqIuds41\ntSSlOxeOFHuOQ/E+2zPEuFYVKM32U3RMvGy44YbZMTg2B2+GOIXXJcjpR9lkUy/QyZ7GUU8zAD9R\nCiuR0oQYVv1IMAk7qFL+rjkGg7GZQPLufffdN69QKJtkCAKKjNGu1p7gMgWDYEDRpkpAmu0rnMLe\nhie/RavcI49Sr1ZW0w6V91ac/IsxmdHPB0U5pQ+4+TExDudNUhPufnaKIn7N6m2k9h11jKLRqP+U\nQJb2eHh4uYjK0LW1D0MpCq0NR4g24RTR/0hCdvM6/m14FtljeTL4D/liedFeO7LYcyh7eMGDY8X1\n6IM8Vp9kWjj2GwWG5IZb2FKVOHTMMTCvDKBgD2Z22223bNynnnpqVrZXBFxjQDZUFJiwIqKHN8qH\nO+64IxvN/fffn9vG/VWC0UpfeC5uZMEbg/ctM/8SzYOxZ599Nhs4ebSx0ECpcDFvMCdRggkesoQ+\nzaHU0N4EgAEnue2227JTON+LgaEVDFu5h+w2Wdl33GFkEUIQqYIqdYwwbJGO8q2xOydqUiTFWpJV\nPzsuUwhlzzFETxlGdFSCqaMB4XwvUzgKWU3AyW4uwFns4QMbilUyxbq8p/4cw3UEB8FDGQUDx/ac\nqB8zRS2dw5qthe3VatPKucocg6JiYu3lP2nfawvekKVITzgJQLH24QTBtPZeE2D89957b27jwZ1I\nwIm8R2OMWHmJ+3pxTzaK8l+HyMrgTzrppMxqOIEsGoZvz0nsyWiliRMUl2G9aOk6POyLZVUvYtBp\nniL4wA1m9lVSW46BOQqKpTLK9FnUsxftvW4swssa4dkhCGFCMNfcp08lhM9KKc4h0obgsa8ShHb6\nCv5DJnu8IwHB9TB852DkOlzIRV6kXbSVMfQj48BWdhE0TLr1Fe3zQR/+gRMK5yjuq4KjZccQ2SlY\njexHmCnSkiLjtsesmlnpQ5naFo1A5GMAHoJxBI709ttv54ygntZWmWEcQMS9VQleRT9kNmfAG0P3\nHRPGbHnVudg4gEyJOAYiE0wikHAAcxHyxndO4KI/WHEK/Qzo7wjAXfaFNdurikaNtIERRTqmYIYd\nE2tGEs8hfJ8iFB/3xV67MCjG8NZbb6Unn3wyC+XfDxfnDxFp496qhK6qn5CDA5twK/fIRH5Gb0MM\nOhxCFgkKjOBoHqKEkmWvueaanG04iTHcP3CKQO0/e3ZhgceP2smqcKyKRuUYlEKhPDL+d5z1c4qV\nFTDnmBIZMwZ9DiKAzTmvCetPNFR7W7fXXt/KLddqTcyjr17bRybkEF5XiQhPHnMuDlF07MCB3I49\nl4EDxTrnfsFBJBxQbQSKeGoROqjdurWzIzoGJqRxS2KUf/rpp2flcRDRjRKVCdpFhCwz7rOVKE5z\n++235/7uuuuuXDq5P5yKEY0np8B3TKb9K1/vLTF0/7MiJtyRPYrq4fx+7R2e7vFDDzDyfx1goPwc\nUGMEYG/rFI3oGAYW0UUyimQIcRwGzbgpVsZAUTYE065xCtc5GUeSHTyg4kzKs/FKoSBljyhvTz6y\n2gseZAwlwgI+cNBGtpV9ZRj4BobjFY9O8g0bQcXWaRpxBE5hHuFnJ0XB6dOn56ge2QGDlK2dFSSG\n4b8kxVzEdSWGVxgYQLzrxJkIGgbTaUE73b9MZ/KNfIMOJpdcckndYZWmFAwv+wgydW/o8wsCK3xn\nz56dFzx8oxPGtk7QiI5h0FBaeGzRKYIpjDN2ig6lB9OiprmI60qNieIMIXvsQy7yotjH9eI+2hbP\nDY4bI8D+2JdnWTYY+iwDs78qaUTHEM0sI1pClAVMnqX9ImGQszB6DHoNOLzZNZlGRlEq9JNB9JOs\nRXvoxDGnsDTudwFUHTNmzMjDqEaU9xYvGgWiZnka0TEo16CeNyCM1SLtwmt5cNEoCOUa5xjQAIFW\nEGBP5rbKdTRr1qwcfGUMthXVTCt917pnRMdwE6ZiQm0JckADBMYCgWLwtXjTSeq/d5Y7ieag7wmD\nwMAxJowqB4JUicDAMapEc9DXhEFgcjxcM7vvR4on7bHS1q84WNkpUr/iEL+aOLRw4cIlQCmuIhUB\nmsjHlpQ9c7EmzjEsN1vd6DeCg8UVT+qRd7b6EQey8wMT+6El8RSu36xhIO8AgQYI9F94bADG4NIA\ngUDg/wHX+3lgThDIegAAAABJRU5ErkJggg==\n", + "text/plain": [ + "<IPython.core.display.Image object>" + ] + }, + "execution_count": 1, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from IPython.display import Image\n", + "import base64\n", + "Image(data=base64.decodestring(\"iVBORw0KGgoAAAANSUhEUgAAAMYAAABFCAYAAAARv5krAAAYl0lEQVR4Ae3dV4wc1bYG4D3YYJucc8455yCSSIYrBAi4EjriAZHECyAk3rAID1gCIXGRgIvASIQr8UTmgDA5imByPpicTcYGY+yrbx+tOUWpu2e6u7qnZ7qXVFPVVbv2Xutfce+q7hlasmTJktSAXrnn8vR/3/xXmnnadg1aTfxL3/7rwfSPmT+kf/7vf098YRtK+FnaZaf/SS++OjNNathufF9caiT2v/xxqbTGki/SXyM1nODXv/r8+7Tb+r+lnxZNcEFHEG/e3LnpoINXSh/PWzxCy/F9eWjOnDlLrr/++jR16tQakgylqdOWTZOGFqX5C/5IjXNLjdt7/NTvv/+eTjnllLT//vunr776Kl100UVpueWWq8n10lOmpSmTU5o/f0Fa3DDH1ry9p0/++eefaZ999slYYPS0005LK664Yk2eJ02ekqZNnZx+XzA/LfprYgGxePHitOqqq6YZM2akyfPmzUvXXXddHceoic2EOckxDj300CzPggUL0g033NC3OKy00krDer3pppv6FgcBIjvGUkv9u5paZZVVhoHpl4Mvv/wyhfxDQ0NZ7H7EQbacPHny39Tejzj88ccfacqUKRmHEecYf0Nr8GGAQJ8gMHCMPlH0QMzmEBg4RnN4DVr3CQIDx+gTRQ/EbA6BgWM0h9egdZ8g8PeliD4RutfF/Ouvfz9OtZy8aNGiNH/+/GGWl1122XzseYuVNKtqsaI23Ghw0DYCA8doG8JqO+AUG2+8cVq4cGHaY4890vLLL5/WXXfdfI6jvPDCC3lJ8amnnkoezP3000/pl19+GThHtWpIPekYomTxFS7HnkqKjMsss0yGgFE4r62tSBFVJ02aNPyconi9V4/JwzHwT9ZNNtkkeZ6w5ZZbph133DH99ttv6ccff8zXX3nllcRRnHNfv2cNGMQWGRaOrWbUrjsGBRLAA6U4Lhoqw9h2223ztRBq6aWXzsbgvueffz4Lu9NOO2UnYTgrr7xy7tO9nOH111/Pbb744ov0ww8/jAvngAdFMvQDDjggG/0GG2yQX1GZNm1aziCCwzrrrJPl3muvvXKwePnll9M333wzHDCKWPbLMbuAkfISjnvvvXcW/emnn85lqCBqa4a65hiYR/Gk2RNGRlwm3n7ggQfmdrKD9sqJtdZaKxvCnDlz8n3Tp09PXmPYeuutc0SVNQjvnmuvvTa3efzxx9N33303PGZ5rF75DBvvqq233nrp22+/TWeddVbyikpgxCE4vQDhlQUBRfDw2esbs2fPTquvvnqviNN1PuIdJ4GErVx44YUZowsuuCB9+umn6eeff84BspmsWqljhPFDxjGGYx/lDkN33udajCoVlAjRzl4U8LjefRwnPjsXG8OJqKBd8NB1LTU5IHyCd7LJGOYXNoGjFqaGIKtrERDIDKtukfGMH/zRZa1A101+YBF44KfMYzO8VOYYjDWiukiGqc022yyXOUqdzTffPJ/z1ialeqNVxA9gi0wzlOJ5juJlR8JeddVV+ZrIKTq4ZvJp/8EHH+SU+txzz+W2SqmxVFZRplrH5DTRXmGFFdKuu+6azjjjjOzosl5g6D54CQCI4mGjhNQO5occckh2LvLTA6fqJOEnyhU6kNlkZmUuvrtNcFx77bUzhsZWXgoSsm6t4Dsa/tp2DErCmA04HAI4FLjaaqtlBhmnSKiNY4rDtHZFB6jFMMH0RVDH+nCPYxtDCFJnKkniRbDitWjTK3sykQUuMLPn3DZGX8SFnCG/fVyz5zCCBtIHTLshdzif8fERn8cKXxjCNOwCTu3Qf6yqhV4AQokiP489//zzM0DxnQYKwqAtIkko1kQzFFxvaNcJ6u3Pe+65J/cRRvDee+9lA2BInIyRff/997nNO++8k7t0vl2A6vHWynmyiPJ43WKLLbIijz/++LTddtvlTCdzwIWSg9yjxBJ0GN/DDz+c7zv77LOzbEceeWSekwVGgsOsWbNyNo0+qt7DfPvtt8/dmtvIGnPnzk3PPPPMsJ6rHrNef/BBeJA90RprrJEDcNhctMkXR/mnbccwuCjNGTbaaKMc8TBZprITxOdgOvbuKxqGz6LSJ598kseJ9Gi1CYmSv/76a3YyJZWMZJ6Ceskp8EMusihFEAyUmVaa8G2rxTNHIrd733///eH7YeaLNe5xrEzlWNF/HqQDf0Tm+GIbvYdD43MsKAIo/JDgE0G5aFfN8NaWYxiUshikqGYTTUSt0TCkjXsYNqJQQso+rgGa0vX58ccf56hQTtk+48F92rmvlnE1A0on2uKP0Yrw+Nxzzz0zn+ZhjKwRXq6vueaa2TmUiRQfS7SyNeMks9IV9vrvJOl/q622yo4Mfw5Pvm6TMclLdit6shh+YAMnq1E29tEsteUYBgMSgxa5MOAzJZcVXQs4bUR8XxhCHIwzMALCBuCcx5q0tF3u133l8XrRMchFiRYNyMxBKM/5IjZlWVzjULKwACISytIWFsi56aab5mvOKyEikmdAO/iHY+BDCRUZuoPD1e1akECyLseA7d13352DhdKak8Cmlt3U7TSl9p58FwejYK8ncAwKpDTnGDcARbWiAUjHiNEHsITSPlagpEZChcfrZzwSOfBOiQwXLuR3PjAhtwAD08iAMCO/a+5xPTIm3ALjwERf0V+c69QeT7ZujVdLDhgKBrANXAMreMESRkU7rdVPrXNtZ4xIpSLH1VdfnR3j4IMPzkbw2Wefpa+//jovo5188slZsZjArAcvFP3YY4+lSy+9NEdTdTTy0I5xHHfccfm1CH2LtuORKEqmkwVlVU+sBY+IdJRmE0zeeOONnEXuu+++7AhnnnlmWn/99XMJ5brtzTffzHMJx/o555xzkgdb0U8rRtAKrnTYqtG1Ml6teyxInHDCCdlGYByBmG2Z97ChVvFo2zEwbHCRTbqP7EDxPjN2pUBEe86AXAcsg+f10TYMSTvnRM1ulQe1wG/nHEXZZEJZUIYQ5cgWMsEgMgqclFdkdh+MbFFyuddnWMLNfTYkcuuXHlBkpFYNI3dS+mMMfCHHsZWadfUjmQVn8iLywscG21apMscQwR555JEM3KuvvpoZ5LHOmzgjAvBwzFt2/Oijj3Lm4Ayin/MU/eGHH+b2N998c/5MGSaZ44nw7OEd5Rx77LE5+1EehYXxkpes5li2K6+8Mhv8Lrvsko381ltvzcEBfvHQKh5auk9GPvHEE3NJAx+/eKL/HXbYIQcbK3nwN067xAk4s5VHdbvsx0nxrYQeKxJMZAfBA7GlRx99NC9EtCN7JY4RoPBeAHIAyrB3jpHYwqu1d02d7HpZcfqINo5dL7eJMXtxTzk2sgWFM/gcsnCakI2cFOk+523O+Qw7WaeYHYpYRp9xn4BkbPdWSfgJXYYM+ne+2xRj2sdx8EDu8rm4Ntp9pY4RSmb0CIPOAVNGoLA47yU4S2xen37ppZdy9CkLE/3lm8bJHzJbbiavt2Q9p7AkK7oyXAZOLk7gs9c4PJC0AOE8DDyrgJkaWgYQkSPYuAdpWySfteU8HhqKouYq+io6ZfGeZo7xpbT1+jt+jGULfprpq922ePHMBibwjWVq523KVrzBsIzTaMeu1DFi0HI0YyyYtAekY5MltbRyihFJiROBKIYTwMCTWJNubwdQFCXFapK9z96mtbjgs3thFKWnUgjBzNZIya5FOyUcPG36q4LwRgZ6Ix8HtBk3tirGGU0feAkslHfk5PzBh2cXSkvtWqWOOEaRGcoSHdXDMoYn1tK8yaON0ahbCWgFS/vxSnjn5F4ItLeiFAGAzCKc7MDA1OlIjc4pLFKE7FEyxb5ZPNTbtuiv2fvrtddfOFsYXcwj8d8qv/XGq3femLvvvnvOvrIYPPEjG+PDseDbDnXcMXiyiGiyyACOPvrovN95552zV3/++ef5zVveznlEo6CICvG5l/d4JSvHP+qoo7JjKDs4PkVSGPm9HSz9W5rlPEoCQYHjVFXyRGnBOcKA28VOP/qTBWX6YnS2IKB8qYL/enyGHPbKziOOOCLj6sGeslGW8L6Y4ANr2MY99fpsdL7jjmFwkSTSr6gDVCk+tmDQedcJ5LgdwaLPbu7xjJRRNlErSsiQhVHJlOEQoh182o1wRTnharwYs3itnWP9Rd/RD5mLW5yveh/YRhYMjItyBh/wjPat8tEVx6B00RKo5513XpIl7rzzzuwEourMmTOz95uIcyBfTSXYiy++mCOrSFS1klsFrNZ9eGPoJtmeyRx00EE5cpGbIi21XnbZZbkMee2117KMHIKMIVcotVb/vXoOz6I0+URoMlVFcBFE7L1+IjNYIo6v/fo+D3tC+FCR+FHuwNUCgfOtUlccI5hnJMoIBhN1sBICqMoNNaLP3pkiFGciIIBC4HaEbRWk0dyHb3Mp/EY0I6+NsytvyKxsKhpQr8ozGpm1IZ8IbV+PyllGuyh1YBXXOQEcy6R8M5eAHzuxxX3GRvbaCKJ4aRfXrjkG5jEbk00Prxi8SZTJKmc5/PDDc5v99tsvC+hBjWtqStmD0F4Ma1foMvDtfqZMUc3/lYjMSFFW3NS7JtyyoKzSiTocHoFJHMc+MlK7Mta7n9NbATJerbEYvQWIWCVitIyaXrV3nsG7H2Y2GVcbxyj6NX+waKEPmOvbfShwtjhQDDz5Ygt/uuoY+OPtnICDEMBTWsAQUu0NBBsDEgFEWOADAiDaVRERWsCq5i34IRN+TbTJgn8KwzOFuR4KDUXW7Kyik53Ep8w/+RkxWeO5S1EM5wVABguXMGp69dk1x87D0ObdL32GHI5tsDQGHtwbm/Hw4TpnKvNY5Ge0x113DEwT3tIsIdSnDIfxcxJAevCHfE9cXcmotHXfAw88kIFUdgFjLMn4HuZRuh9FExmjRCCnZxRqcPxz8ioUVk9eRhJkPAYHV8ZVFRkjjFSfAtw222yTy2OZ0iv15fHcQ4dKaMcwsBdEEL26RzaIh5+yK7LSBGPno8yOZX+vzRhfXzZ8cRrtyzzkzpr803XHwB8wTJYIRol+VY8zqMMBbP0f+cExE1qTdbU7x3jwwQdzVBYdesExKNiEWx2MfwoOAyCbJ9uRHZvUTcPmsENhGNE4HBKOHKNqZzQu3KNfX9H1nRABQZlbNkpt4SNo4DWIIesDj9qYnwki2giWqol3330348kZLPm7xvi1Pffcc7MzhA3gy/0oeIuxWtmPiWNgNCIFYwcCAa2FA1ikJZz1aeUVsBmge9TyoqGoIqKUFdEKCFXcU0/pHJizVMUnXBiBh6IicdTTzsEOnuZkDE/2rcJI4KMf/TF+0TucwDhkZ+DGL4/nGkPGV/AIC+2RvfP6ZPTI4gu5XNM/Um7RPzuIFyn1zW7wpQ9UHj+fbOHPmDlGCOGBGIeQQfwuq0jnISBQfOHft7JEHN94Q5xF6XLFFVfkyKIEGyuiGAo3r6BIx0imcM6k+6GHHspOEQbcDq+UTl4BwRu7PstUiPEJFsa9/PLL83nXg6d2xnUvoxS5L7744uGyh/wyRpRF9YwSHsHjE088kWWADQeRFThZkTgBstensZG5h4m56oEdcAp9CwTOVUlj6hgECcGBpA6XDazeiLKhVABQAhKB3cNxbEAL4KoEppm+gjf3OMafDf+UW7zeTL/ltqIiAxBMOIIxnLOHgbFsMGQ4InhE0nJfrXw2hnIRD3SFBKmYWDfqE49woFvOzZno3NxM0HDciMjBDsjEBgLTsJHYN+qjmWtj7hjBLKFFQgL7qRz14jHHHJPBcC2M3wRPVDT5ohzZRv0Z16O/sdozAKmdopUH5kftTrzJpl+lk29CcgpLw3BgpMbwwqF/S80pGJ6xO0WM+8Ybbxw2TuOEoTYakwyovB/JKdzDMVQOHvCRzXju890fL11aGhcMqqIxdwwCRkYQDZAaE7lWBhyosQEmQM439MgffDHm0Si8EcuBC0ezcQSZVKYktzFEW+3sfQ4natRvu9eMTS9F7IvHo+m/2fb6LNuCc0WsW+mzHq9j6hgE9YCHp5tkez2EAVjlMOmyUlU2Lis8ygVR0rykyoltPZCaOY9fr32Qp50X6xi7pWCGbsHBvwLgGIcddljGxvcsjOU1GseyiKjJQWydpiqNsBlei85BfhNxeJunVCl31x0jBOMAjJ9jRC3OEERDS7QMI0qQohIYgLSq7FJuMZbi9WZA7kRbvFAWx5Dyy449mjEDG/dyDPW4VSiy2iNvBcCSUdxyyy35OYHrqJUx843j8I/qQpA074BVVdR1x+AIHCIiIGewsqIuds41tSSlOxeOFHuOQ/E+2zPEuFYVKM32U3RMvGy44YbZMTg2B2+GOIXXJcjpR9lkUy/QyZ7GUU8zAD9RCiuR0oQYVv1IMAk7qFL+rjkGg7GZQPLufffdN69QKJtkCAKKjNGu1p7gMgWDYEDRpkpAmu0rnMLehie/RavcI49Sr1ZW0w6V91ac/IsxmdHPB0U5pQ+4+TExDudNUhPufnaKIn7N6m2k9h11jKLRqP+UQJb2eHh4uYjK0LW1D0MpCq0NR4g24RTR/0hCdvM6/m14FtljeTL4D/liedFeO7LYcyh7eMGDY8X16IM8Vp9kWjj2GwWG5IZb2FKVOHTMMTCvDKBgD2Z22223bNynnnpqVrZXBFxjQDZUFJiwIqKHN8qHO+64IxvN/fffn9vG/VWC0UpfeC5uZMEbg/ctM/8SzYOxZ599Nhs4ebSx0ECpcDFvMCdRggkesoQ+zaHU0N4EgAEnue2227JTON+LgaEVDFu5h+w2Wdl33GFkEUIQqYIqdYwwbJGO8q2xOydqUiTFWpJVPzsuUwhlzzFETxlGdFSCqaMB4XwvUzgKWU3AyW4uwFns4QMbilUyxbq8p/4cw3UEB8FDGQUDx/acqB8zRS2dw5qthe3VatPKucocg6JiYu3lP2nfawvekKVITzgJQLH24QTBtPZeE2D89957b27jwZ1IwIm8R2OMWHmJ+3pxTzaK8l+HyMrgTzrppMxqOIEsGoZvz0nsyWiliRMUl2G9aOk6POyLZVUvYtBpniL4wA1m9lVSW46BOQqKpTLK9FnUsxftvW4swssa4dkhCGFCMNfcp08lhM9KKc4h0obgsa8ShHb6Cv5DJnu8IwHB9TB852DkOlzIRV6kXbSVMfQj48BWdhE0TLr1Fe3zQR/+gRMK5yjuq4KjZccQ2SlYjexHmCnSkiLjtsesmlnpQ5naFo1A5GMAHoJxBI709ttv54ygntZWmWEcQMS9VQleRT9kNmfAG0P3HRPGbHnVudg4gEyJOAYiE0wikHAAcxHyxndO4KI/WHEK/Qzo7wjAXfaFNdurikaNtIERRTqmYIYdE2tGEs8hfJ8iFB/3xV67MCjG8NZbb6Unn3wyC+XfDxfnDxFp496qhK6qn5CDA5twK/fIRH5Gb0MMOhxCFgkKjOBoHqKEkmWvueaanG04iTHcP3CKQO0/e3ZhgceP2smqcKyKRuUYlEKhPDL+d5z1c4qVFTDnmBIZMwZ9DiKAzTmvCetPNFR7W7fXXt/KLddqTcyjr17bRybkEF5XiQhPHnMuDlF07MCB3I49l4EDxTrnfsFBJBxQbQSKeGoROqjdurWzIzoGJqRxS2KUf/rpp2flcRDRjRKVCdpFhCwz7rOVKE5z++235/7uuuuuXDq5P5yKEY0np8B3TKb9K1/vLTF0/7MiJtyRPYrq4fx+7R2e7vFDDzDyfx1goPwcUGMEYG/rFI3oGAYW0UUyimQIcRwGzbgpVsZAUTYE065xCtc5GUeSHTyg4kzKs/FKoSBljyhvTz6y2gseZAwlwgI+cNBGtpV9ZRj4BobjFY9O8g0bQcXWaRpxBE5hHuFnJ0XB6dOn56ge2QGDlK2dFSSG4b8kxVzEdSWGVxgYQLzrxJkIGgbTaUE73b9MZ/KNfIMOJpdcckndYZWmFAwv+wgydW/o8wsCK3xnz56dFzx8oxPGtk7QiI5h0FBaeGzRKYIpjDN2ig6lB9OiprmI60qNieIMIXvsQy7yotjH9eI+2hbPDY4bI8D+2JdnWTYY+iwDs78qaUTHEM0sI1pClAVMnqX9ImGQszB6DHoNOLzZNZlGRlEq9JNB9JOsRXvoxDGnsDTudwFUHTNmzMjDqEaU9xYvGgWiZnka0TEo16CeNyCM1SLtwmt5cNEoCOUa5xjQAIFWEGBP5rbKdTRr1qwcfGUMthXVTCt917pnRMdwE6ZiQm0JckADBMYCgWLwtXjTSeq/d5Y7ieag7wmDwMAxJowqB4JUicDAMapEc9DXhEFgcjxcM7vvR4on7bHS1q84WNkpUr/iEL+aOLRw4cIlQCmuIhUBmsjHlpQ9c7EmzjEsN1vd6DeCg8UVT+qRd7b6EQey8wMT+6El8RSu36xhIO8AgQYI9F94bADG4NIAgUDg/wHX+3lgThDIegAAAABJRU5ErkJggg==\"), embed=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We're going to be building a model that recognizes these digits as 5, 0, and 4.\n", + "\n", + "# Imports and input data\n", + "\n", + "We'll proceed in steps, beginning with importing and inspecting the MNIST data. This doesn't have anything to do with TensorFlow in particular -- we're just downloading the data archive." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 110, + "status": "ok", + "timestamp": 1446749124399, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "716a6ad5e180d821", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "w5vKZqr6CDz9", + "outputId": "794eac6d-a918-4888-e8cf-a8628474d7f1" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Already downloaded train-images-idx3-ubyte.gz\n", + "Already downloaded train-labels-idx1-ubyte.gz\n", + "Already downloaded t10k-images-idx3-ubyte.gz\n", + "Already downloaded t10k-labels-idx1-ubyte.gz\n" + ] + } + ], + "source": [ + "import os\n", + "import urllib\n", + "\n", + "SOURCE_URL = 'http://yann.lecun.com/exdb/mnist/'\n", + "WORK_DIRECTORY = \"/tmp/mnist-data\"\n", + "\n", + "def maybe_download(filename):\n", + " \"\"\"A helper to download the data files if not present.\"\"\"\n", + " if not os.path.exists(WORK_DIRECTORY):\n", + " os.mkdir(WORK_DIRECTORY)\n", + " filepath = os.path.join(WORK_DIRECTORY, filename)\n", + " if not os.path.exists(filepath):\n", + " filepath, _ = urllib.urlretrieve(SOURCE_URL + filename, filepath)\n", + " statinfo = os.stat(filepath)\n", + " print 'Succesfully downloaded', filename, statinfo.st_size, 'bytes.'\n", + " else:\n", + " print 'Already downloaded', filename\n", + " return filepath\n", + "\n", + "train_data_filename = maybe_download('train-images-idx3-ubyte.gz')\n", + "train_labels_filename = maybe_download('train-labels-idx1-ubyte.gz')\n", + "test_data_filename = maybe_download('t10k-images-idx3-ubyte.gz')\n", + "test_labels_filename = maybe_download('t10k-labels-idx1-ubyte.gz')" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "gCtMhpIoC84F" + }, + "source": [ + "## Working with the images\n", + "\n", + "Now we have the files, but the format requires a bit of pre-processing before we can work with it. The data is gzipped, requiring us to decompress it. And, each of the images are grayscale-encoded with values from [0, 255]; we'll normalize these to [-0.5, 0.5].\n", + "\n", + "Let's try to unpack the data using the documented format:\n", + "\n", + " [offset] [type] [value] [description] \n", + " 0000 32 bit integer 0x00000803(2051) magic number \n", + " 0004 32 bit integer 60000 number of images \n", + " 0008 32 bit integer 28 number of rows \n", + " 0012 32 bit integer 28 number of columns \n", + " 0016 unsigned byte ?? pixel \n", + " 0017 unsigned byte ?? pixel \n", + " ........ \n", + " xxxx unsigned byte ?? pixel\n", + " \n", + "Pixels are organized row-wise. Pixel values are 0 to 255. 0 means background (white), 255 means foreground (black).\n", + "\n", + "We'll start by reading the first image from the test data as a sanity check." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 57, + "status": "ok", + "timestamp": 1446749125010, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "716a6ad5e180d821", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "P_3Fm5BpFMDF", + "outputId": "c8e777e0-d891-4eb1-a178-9809f293cc28" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "magic number 2051\n", + "image count 10000\n", + "rows 28\n", + "columns 28\n", + "First 10 pixels: [0 0 0 0 0 0 0 0 0 0]\n" + ] + } + ], + "source": [ + "import gzip, binascii, struct, numpy\n", + "import matplotlib.pyplot as plt\n", + "\n", + "with gzip.open(test_data_filename) as f:\n", + " # Print the header fields.\n", + " for field in ['magic number', 'image count', 'rows', 'columns']:\n", + " # struct.unpack reads the binary data provided by f.read.\n", + " # The format string '>i' decodes a big-endian integer, which\n", + " # is the encoding of the data.\n", + " print field, struct.unpack('>i', f.read(4))[0]\n", + " \n", + " # Read the first 28x28 set of pixel values. \n", + " # Each pixel is one byte, [0, 255], a uint8.\n", + " buf = f.read(28 * 28)\n", + " image = numpy.frombuffer(buf, dtype=numpy.uint8)\n", + " \n", + " # Print the first few values of image.\n", + " print 'First 10 pixels:', image[:10]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "7NXKCQENNRQT" + }, + "source": [ + "The first 10 pixels are all 0 values. Not very interesting, but also unsurprising. We'd expect most of the pixel values to be the background color, 0.\n", + "\n", + "We could print all 28 * 28 values, but what we really need to do to make sure we're reading our data properly is look at an image." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 887, + "status": "ok", + "timestamp": 1446749126640, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "716a6ad5e180d821", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "F_5w-cOoNLaG", + "outputId": "77dabc81-e3ee-4fcf-ac72-88038494fb6c" + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEACAYAAABI5zaHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnWuQbFd1339rHt3T3dM974eu5iKJCINExRGULYeSXRli\nLAtCJIoPisDlAuQkrgIZqpQHuny5oziJEAk42C7ZQWBKUBBZOIUlV1FGKNQ4RQISGInXFfIF50r3\nzp3Hnff0PLqnu1c+dO+j3We6586jex5n1q9qV59z5pyz9+mZ+ffqtddeS1QVwzAMI1q0HPYADMMw\njMZj4m4YhhFBTNwNwzAiiIm7YRhGBDFxNwzDiCAm7oZhGBHExN2INCLyCyLyvIh8v/K6JCIfFpEe\nEXlaRF4Ska+LSJd3zRkROS8iL4rI7Yc5fsPYK2Jx7sZJQURagEvArwD3AXOq+gkR+SjQo6oPiMjN\nwJeAXwZGgGeA16n9oxjHDLPcjZPE24Cfq+pF4C7gscrxx4B3VbbvBB5X1YKqXgDOA7ce9EANY7+Y\nuBsniX8BfLmyPaSq0wCqOgUMVo5fC1z0rpmoHDOMY4WJu3EiEJF2ylb5VyqHwm4Wc7sYkaLtsAdg\nGAfE24G/VdXZyv60iAyp6rSIDAMzleMTwGnvupHKsS2IiH0gGE1FVWWv15rlbpwU3gP8D2//KeD9\nle33AU96x+8RkZiI3ADcCDxX76aq2tR29uzZSPQRpWc5qPdrv5jlbkQeEUlSnkz9197hh4EnRORe\n4GXgbgBVPSciTwDngE3gg9qI/zTDOGBM3I3Io6prwEDo2Dxlwa91/kPAQwcwNMNoGuaWMYwjzOjo\naCT6OKh+otJHI7BFTIaxR0TEPDZG0xAR1CZUDcMwDB8Td8MwjAhi4m4Y+6CtLb6l/cmffOawh2UY\nFi1jGPuhWFwOHfmv/OxnPz+UsRiGj4m7YeyLeGjf/qWMo4G5ZQzDMCKIibthGEYEMXE3DMOIICbu\nhmEYEcTE3TAMI4KYuBuGYUQQE3fDMIwIYuJuGIYRQUzcDcMwIoiJu2EYRgQxcTcMw4ggJu6GYRgR\nxMTdMAwjgpi4G4ZhRBATd8MwjAhi4m4YhhFBTNyNyCMiXSLyFRF5UUR+IiK/IiI9IvK0iLwkIl8X\nkS7v/DMicr5y/u2HOXbD2Csm7sZJ4NPA11T1JuAfAT8FHgCeUdXXA98EzgCIyM3A3cBNwNuBR0RE\nDmXUhrEPTNyNSCMiGeDXVPXzAKpaUNUl4C7gscppjwHvqmzfCTxeOe8CcB649WBHbRj7x8TdiDo3\nALMi8nkR+b6IfEZEksCQqk4DqOoUMFg5/1rgonf9ROWYYRwrrJqvEXXagDcDH1LV74nIH1B2yWjo\nvPD+Dhnztkf3dgvDAMbHxxkfH2/Y/UzcjahzCbioqt+r7P9PyuI+LSJDqjotIsPATOXnE8Bp7/qR\nyrE6jIX2n23AkI2TyOjoKKOjo8H+gw8+uK/77cstIyJ3iMhPReTvROSj+xqJYTSBiuvlooj8QuXQ\nrwM/AZ4C3l859j7gycr2U8A9IhITkRuAG4HnDm7EhtEY9my5i0gL8MeU/1kuA98VkSdV9aeNGpxh\nNIgPA18SkXbg74EPAK3AEyJyL/Ay5QgZVPWciDwBnAM2gQ+q6h5dNoZxeOzHLXMrcF5VXwYQkccp\nRyBUibuI2D+G0VRUddtQRVX9AfDLNX70tjrnPwQ81IChGcahsR+3TDiq4BJ1ogpUFVXl7NmzwXaz\nm/V1vPraa3+GYdTGQiENwzAiyH7cMhPAa7z9ulEFY2NjwKuhPv6MsGHshkaHixlGVNmPuH8XuFFE\nrgMmgXuA99Q60Rf3gxL2g/wAsb4Orr9Gh4sZRlSR/fgtReQOynk7WoDPqerHa5yj5hs1moWIoFeZ\nUG1i37p17dPD3H//PJ/85MOHMSQjQuz3b3tfi5hU9a+B1+/nHoZhGEbjsQlVwzCMCGLibhiGEUFM\n3A3DMCKIibthGEYEMXE3DMOIICbuhmEYEcTE3TAMI4KYuBuGYUQQE3fDMIwIYuJuGIYRQUzcDcMw\nIoiJu2EYRgQxcTcMw4ggJu6GYRgRxMTdMAwjgpi4G4ZhRBATd8MwjAhi4m5EHhG5ICI/EJHnReS5\nyrEeEXlaRF4Ska+LSJd3/hkROS8iL4rI7Yc3csPYOybuxkmgBIyq6ptU9dbKsQeAZ1T19cA3gTMA\nInIzcDdwE/B24BEROZQarYaxH/Yl7rUsIsM4gghb/9bvAh6rbD8GvKuyfSfwuKoWVPUCcB64FcM4\nZuyrQDavWkQLjRiMYTQJBb4hIkXgv6vqZ4EhVZ0GUNUpERmsnHst8G3v2onKMcM4VuxX3GtZRIZx\n1LhNVSdFZAB4WkReoiz4PuH9HTLmbY/u7RaGAYyPjzM+Pt6w++1X3H2L6DOq+mgDxmQYDUVVJyuv\nV0TkLym7WaZFZEhVp0VkGJipnD4BnPYuH6kcq8NYaP/ZBo3aOGmMjo4yOjoa7D/44IP7ut9+xd23\niL4hIi+q6rfCJ42NjQXb4QcwjN2wW+tGRJJAi6pmRSQF3A48CDwFvB94GHgf8GTlkqeAL4nIH1B2\nx9wI2HyScewQ1T1+Gw3fSOQssKKqnwod10b1YRhhRARVrRvNIiI3AF+l/C2zDfiSqn5cRHqBJyhb\n6S8Dd6vqYuWaM8DvAJvAR1T16Tr31q3enIe5//55PvnJh/f7aMYJ52p/21djz5b7NhaRYRwZVPX/\nAbfUOD4PvK3ONQ8BDzV5aIbRVPbjlhkCvlq2XgKLqKaFYxiGYRwsexb3ehaRYRiGcfjsd0L1yKKq\nlEqlqqaqFAoFisUihUIh2HZzAiJS1Rz+dktLC62trUFra2ujpaUlOMe/ttarLXY0DOMgiLS4OxH3\nxTyXy5HL5djY2Ai2i8UiLS0tVS0s0m67ra2NWCxGPB4nFosFLfzB4N+j1geGYRhGMzkR4p7P59nc\n3CSfz7O6urqlFQqFwBL3LXPYas3HYjESiQTJZDJ4VdUtHwzhDws3JhN4wzAOgkiLe6lUCkTdWerL\ny8ssLS2xtLQUbG9ubtLW1ha4Wdx2LUu8o6ODdDpNPp+nWCxuEXL/A0JVA5ePE3jDMIyDINLi7ix3\n54ZZX19neXmZhYUF5ubmmJ+fZ35+nnw+T1tbG+3t7VWvvqi711QqVSXs7gPBWfulUikQ9rD1b/H+\nhmEcFJEVd2e1r6+vk81mWVlZYWVlhbm5uS0tl8ttEXZ/otR/TSaTVf5696HhC7wv+O5e/jeCo0q9\nCeCwi8l9C6k3cWwYxuETaXHf2NhgeXk5sNDn5+dZWFgI2uLiIouLi+Tz+S2C7NwrYZ97IpEgm82y\ntLTE3Nwc6XSaVCq1RdxbW1tpb2/f0traju5bHp4zEJFgAtm1eDxOe3t7zQ8+wzCODkdXafZJsVhk\nY2ODlZUV5ufnmZqaYmpqiuXl5cDXvrKyEvjcwxOq9aJdak2odnR0bBF29y3Aj6xxwnhUCYd5trS0\nEI/HSSaTQSsWiySTyap5BfdqlrthHB0iK+6+5T43N8fU1BQXL14km80GbXV1lWw2S6FQqBvhEnY5\nOMEOW7JhcW9tbSUWi9HR0UFHRwfxeJyOjg5isdihvSdXw3cdue1EIkEmkyGTyQTvk/uZqlbNTRiG\ncXSIrLj7lvvc3ByTk5NcvHiRtbU11tbWWF9fD5oTre1i0t1+2Lqt11paWujo6CCRSJBIJOjo6CCZ\nTBKPxw/j7dgR7tuG31KpFBsbG8EEcnt7Ox0dHcE1/gehYRhHh8iKu7Pcs9ksCwsLzMzMMDExUTUZ\nms/ng0VMe8W5I3zRd9tO0P3mC+NRIxaLbflmkslkAmF3lnw+nw+uceJukUCGcbSIrLi3trYGLoX+\n/n5OnTq1ZWVqeIVqeGWpi5UPv4a3S6USQCBwvtC5qJ2NjQ1Ulc3NzZr+fHeui433Y+R3Q61vHLVW\n2ob7UdWqSV+3XSgUqvbdB4D7NuI+3I7yXIJhnEQiK+4tLS0kEgm6uroYGBggl8uhqoG4+68uNj3c\nisXillYrN029bREJxN3F3efz+ZqLntzPw/lwdkM9YQ9HwQBb+imVSjUXcrkFXr64x2IxCoUCpVIp\nsOjNcjeMo0Vkxd25RTKZDLlcjlKpRHt7O+vr62xsbGwRdz++3W1vbm5SKBSCV7edz+erUhpsbm5u\n+TaQz+cD0dzc3KwS9vDCp7a2Nkql0pYPj/2Iu9sORwC5OPtaH1y1zt3c3KwS9ng8TjwerxL2jo6O\nXY/VMIzmEmlxd5a7i+pIpVJV4u62VbXKz+xcD07EfTEPu3bcvdxErRNPJ+phNw5Q84OkVCpVfZC4\nD4TdUGtRUa1FVUDwYeW3WrHrvrg7YY/H44Gwx+Px4JuJYRhHh8iLu/Mlp1Ipenp6ggiZsLi7cEU/\nZLGekK+vr1dF3KytrbGyshKEBZZKJfL5fGCF+5Z/sVismrgsFou0t7cH5/nfCAqFwq6euZZ/PbxC\n1o1xc3NzS3PX+a/5fH6LsMfj8UDYU6kUhULBxN0wjhiRFXc/bM/fdgLtu2aALeLlxD3c/BBKv6VS\nqarm0hSE3TiFQqEqRt5tF4vFwJ3j2l7F3RfocM6c9vZ2SqVSMG5gS157eHVSOBaLVY3HuYvCefIN\nwzhaRFrcnd/Yj+Twk3054YLqMEAngv5K03g8Tj6fD8IbfcHf2NhgdXWVtbW1qlTCvlg7kXeWe7g5\nn3zYDbSb53WvfquVEK1QKLC0tMTi4iJLS0uBJe9H68CrAu/PE/gfSu5+FuduGEePq4q7iHwOeCcw\nraq/WDnWA/w5cB1wgXLl+KUmjnPXOHF3PmZf7J1I+e4I3w/u+8OdsPsTqr77xG3X8uXncrkqy73Z\n4l5L4P3ncNu5XI6ZmRk6OjoCv/rq6mpNi9x/H/1JVXc/P1WDYRhHh51Y7p8H/gj4gnfsAeAZVf2E\niHwUOFM5dqRwYl5L2P0oEaBmbhkXweKam/QMR7XUEuZazXfLhCs51XLL7EXca1nuYYFfX18P3FWb\nm5tks1na2tqqJnCdsIe/AYUtd/deGYZxtLiquKvqt0TkutDhu4B/Utl+DBjniIm7H9PtJ7kKL0qq\nVT/VWaLh8+otaKoVxuiHTfrW+9Usd1/g9+Nz385yX1lZCYR9dXWV+fn5LYuQ/NDG8Aejm3D2xd0s\nd8M4WuzV5z6oqtMAqjolIoMNHFND8IW62YTF3227En9hn3t4iX8j3TLbWe5ue2lpKShcMjs7SyKR\nCAqNuPG7by61rHYXWXScfO4i0gJ8D7ikqndu51oUkTPAvUAB+IiqPn04ozaMvdOoCdVtwyXGxsaC\n7dHRUUZHRxvU7dEivIio1oSuWwka9lm71Ln+as/dFvaoJ/BOqN03i3B+HecucguT3Jja29uDnDip\nVIrOzs6guTw5sVgsCK88CMbHxxkfH9/LpR8BzgGZyn5N16KI3AzcDdwEjADPiMjr1EKCjGPGXsV9\nWkSGVHVaRIaBme1O9sU9yvgC5wTbL87hRLZWrdZak7+7WcRUK3+Mcy255uLRnbj7wu5cQLXKCrrW\n2dlJOp0mnU4H4u5cMwcl7mHj4MEHH7zqNSIyArwD+E/A/ZXD9VyLdwKPq2oBuCAi54FbgWcb8gCG\ncUDsVNyl0hxPAe8HHgbeBzzZ2GEdP5yQhgXe33ax5OEMkmG3hrOe97KkP7wIyU9l4EQ8bLW71bB+\n9SXXfKvdt96dayYWix2ouO+RPwD+HdDlHRuq41q8Fvi2d95E5ZhhHCt2Egr5ZWAU6BORV4CzwMeB\nr4jIvcDLlL/Gnnh8ga/lFnEul/ASfyfm7nzf/73b/sP7LseNs9D9VbfhxUnt7e2BuDv/uhN3X9jT\n6fSW0oFHVdxF5J9RDuN9QURGtzl1j26XMW97u9sbxvbsw+VYk51Ey7y3zo/e1rBRRIiw393f98U6\nbGG7bRfV0yj8rJRugjfslnHC7z5g/CpS9dwy4QRjR1XcgduAO0XkHUACSIvIF4GpOq7FCeC0d/1I\n5VgdxkL75r0x9sZeXI7bcfTDHI4B9SYynWXup/cNu2NqnbuTSk+1Kj/5zd3TTaC6oiVXrlxhZmaG\n+fl5lpeXWVtbC8S/tbU1yBfT1dVFX18ffX19dHd3k8lkSKVSVWGQ/pzBURV3Vf2Yqr5GVV8L3AN8\nU1V/G/gryq5FqHYtPgXcIyIxEbkBuBF47oCHbRj7JrLpB04q4fQBuVyObDbL/Pw8c3NzzM3NMT09\nzfT0NPPz82Sz2SDXvV8ztbe3l97eXoaGhujv76erq4tUKkUsFqv5IXYM+TjwRNi1qKrnROQJypE1\nm8AHLVLGOI6YuEeIWtWVcrkcKysrzM/PMz09zeTkJNPT01y5ciUQ93w+v0Xc+/r6GBoaYmhoiL6+\nPrq6ukgmk8RisapvBscJVf0b4G8q2/PUcS2q6kPAQwc4NMNoOCbuEcMX9lKpVGW5T01NcfHiRaam\nplhaWmJpaSkQdyCIa3fiPjw8zNDQEL29vVWWu4vuqTVvYBjG0cDEPUKEhT0s7tPT01y6dInJycmq\nnPS13DK+uKfTaTKZTGC5m6gbxtHHxD1C+KLuEp1tbGywsrISTKZOTk4yOTm5pXwgEJTMS6fT9PT0\n0N/fz8DAAIlEgmQySSKRoL29/VikGzCMk46Je0QI57JxbXFxkZWVFdbW1tjY2KiqEOXna6+V6jgc\nFWMJwgzj+GDiHhFUlc3NzSCfvGtLS0usrKywurpaJe7OfeMnV6tVtckvXGLibhjHBxP3iOAWKeVy\nOVZXV8lms2Sz2UDcfcvdz1njx9ib5W4Y0cHEPSL4lvva2hrLy8tBKT1nua+vrweWe60UCGFx99ML\nuNS+Ju6GcTywmbGI4Cx3V891eXmZhYWFLZa7n97Xd8v4mSr9knpmuRvG8cQs92NIeKESENRxzWaz\nLC4uMjc3x8zMDLOzsywuLgYrUZ3V7nLH+G1wcDBYsJRKpYKCHOF0CYZhHH1M3I8h4RJ/LuTR+did\nsF++fJnZ2VkWFhbIZrNsbGxQLBZpbW0lkUjQ1dUVtEwmw8jICNdccw19fX2k02ni8XhgtZvlbhjH\nCxP3Y4gTd79uq5tIXVpaYn5+PhD3xcVFlpeXWVlZIZfLBcVCkskk3d3dDA4OMjAwwMDAQFW6AV/c\nzWo3jOOHifsxxK/R6hYjOV+7E3eXRyabzQYrUZ3lHovFSCaT9PT0MDQ0xMjICCMjI/T09ATNF3c/\nosYE3jCOBybuxxBVpVgsUiwWg3zs9dwy/iSqs/KdW6a7u5uhoSFe85rX8NrXvjaotuRyt3d0dFSV\n/vNfDcM42pi4H0N8y91fuLS6usrKygpLS0ssLCwwOzsbpBZwtLS0EIvFqnK2Dw8PMzIyUlU6z59M\nNQzj+GHifgxx1ZWcqK+trQXhji6WvV691tbWVrq6ukin06RSqaDQtRPz41CAwzCMq3PVOHcR+ZyI\nTIvID71jZ0Xkkoh8v9LuaO4wDR/njsnlcqytrZHNZuumGGhpaaG9vT2osJTJZAJx7+zsDMS9Vky7\nYRjHl538B38e+M0axz+lqm+utL9u8LiMbXAuGSfuKysrLC8v1xR3t/LU1UP1xT2VSpFIJAJxj8Vi\ntmDJMCLCTgpkf0tErqvxI/vPPyScW6aW5V7LLdPe3h6Ie2dnJ5lMZovlHo/Ht9RiNXE3jOPLfr57\n3yciL4jIZ0Wkq2EjMq6K73N3lvvV3DL1LPd6bhkTdsM43ux1QvUR4D+oqorIfwQ+BfxOvZPHxsaC\n7dHRUUZHR/fY7cmiXl1m53P3xd13y2xublIqlQACYe/s7KS7u5u+vj4GBgaCWHa/AIc/iXpUxX18\nfJzx8fHDHoZhHHn2JO6qesXbfRT4q+3O98Xd2DtO7H1xd+GP4eRgpVIJEamqi9rb28vg4GCwCtWV\nzmtvbz8Wwg5bjYMHH3zw8AZjGEeYnbplBM/HLiLD3s/eDfy4kYMyqvEThLkFTPl8/qqWuxN3VxfV\nF/fe3l4ymUxguTuOsrAbhrFzrmq5i8iXgVGgT0ReAc4CbxWRW4AScAH43SaO0YC64u4s9+3E3bfc\nXf6YWkWvTdgNIzrsJFrmvTUOf74JYzFq4PvdnQVfyy1zNcs9nU5XWe6JRCJozi0TRUQkDvxvIFZp\nT6rqx0SkB/hz4DrKBsrdqrpUueYMcC9QAD6iqk8fxtgNYz/YSpUjjhNzl2rAWexuZaoTd5fS1wm7\nHyWTTCZJp9N0dXUFicEymQypVIqOjo4gxYCz3v123FHVHPBWVX0T8IvAPxWR24AHgGdU9fXAN4Ez\nACJyM3A3cBPwduARicIbYZw4LP3AEcZVV8rn80GCsHw+z8LCAouLi0GVJZf50dVGdZZ4LBYLYtqT\nyWTVgqVwEY4oo6prlc04ZYNmAbgL+CeV448B45QF/07gcVUtABdE5DxwK/DsQY7ZMPaLifsRxtVF\nzeVyrK+vB2l7nbgvLy+zvLxMNptlfX09SMvrSuS1tLQEi5Wcle4Sg7myeidhsZKItAB/C/wD4E9V\n9ZyIDKnqNICqTonIYOX0a4Fve5dPVI4ZxrHCxP0IE66Lurq6SjabDWqj+pb7+vp6kELAibuz3N1i\npUQisaV03klYsKSqJeBNIpIBvi4io0B4EUHtRQVXZczbHt3bLQyDxq/hMHE/wjhx96ssuXS+foUl\n55ZxrhgXIeNWpPp5ZJzl7hffiLq4O1R1WUS+BvwSMO2s90po70zltAngtHfZSOVYHcZC++a9MfZG\no9dwRNvZeszxLfe1tTWWl5eZn5+v6XNfX18Pcrc7cQ9nf/TF3bfeoyzwItLv0mOISAL4DeB54Cng\n/ZXT3gc8Wdl+CrhHRGIicgNwI/DcgQ7aMBqAWe5HhFKpFIQ6uu1cLrelutLMzAxXrlypKnrtomNc\nEQ4X097f309fXx9dXV2kUqkgOVhUhbwO1wCPVSJeWoAvqur/EpHngSdE5F7gZcoRMlT88U8A54BN\n4INaLw+EYRxhTNyPCH7pPNfW19dZWVlhcXGR2dnZoC7q3NwcCwsLrK6uks/nUdUgra/LIdPf38/Q\n0BD9/f1bxP0koao/At5c4/g88LY61zwEPNTkoRlGUzFxPyKE49kLhUKQWmBhYYG5uTmmp6eZmJgI\nImSy2Wwg7rUShA0PD9Pb20t3dzednZ3E4/HIhz0ahlHGxP2I4NwxLpY9n8/XtNwvX74cWOyuAVWW\ne1dXV2C5++l9T6LlbhgnFRP3I4JvuYeTgi0uLgaW++XLl8nlcoF/3rWwW6avr4+hoSE6OzuDNAMu\n9t0wjOhj4n7AbJej3VnrLqZ9YWGhZisWi0FRjba2tiB/jF9pyaUacKLuImRM3A3jZGDifoj4mR7z\n+Tyrq6ssLi6yuLgY+NmnpqaYn59nZWUlsNj9otcutLG7u5tMJlMV9lir6PUJi5QxjBOLifsh4btU\nADY3NwNxd+GOMzMzTE9PV4l7qVQKrHWXFCyZTG4R93CqASfuhmGcDEzcD4Gwv9y33BcWFpiZmeHy\n5ctMTU0xNzfH/Pw82Ww2sNxbW1uromPS6XRNcfddMVb02jBOFibuh0gtcV9cXOTKlStMTExw+fLl\nIMWAb7m7BUvOz57JZKrE3U810NbWFqxANXE3jJODifsh4Qu7C4EMi/ulS5fY2Nggl8sFr77PvaOj\ng1QqRVdXV13L3UIfDeNkYuLeRGpFxqhqINR+m5iYYHp6mtnZWRYXF6uKbxSLRUSEtrY2VJVEIhHE\ns/f19TEwMMDAwMCWuqhRzhljGMb27KSG6gjwBWCIcs3UR1X1D7crU2bUp1QqkcvlgtJ4zuUyNTXF\n5OQks7OzLC0tsba2FrhhVDUQ95aWlqCyUnd3d1XpvJ6eHtLpdORL5xmGcXV2Ej5RAO5X1TcCbwE+\nJCJvoE6ZMmN7SqUSGxsbLC8vMzc3x+XLl3n55Zd55ZVXAnFfXFwMVqEWi8Ut4Y++uDvLfXBwkN7e\n3ipxNwzj5LKTAtlTwFRlOysiL1LOcV2vTJlRB+djd+I+OzvL5OQkk5OTXLlyhbm5Oebm5qosdxej\n7rdkMlm1EnVwcJDBwcEgLDKZTNLWZh43wzjJ7EoBROR64BbgO0C9MmVGDZz/3bllnLg7y31+fp6l\npaWgdJ6z3IEgyqWtrY1YLLbFcnfi7ldhMreMYZxsdizuItIJ/AXwkYoF36AyZdHGX4Xq8sfkcrmg\n+IbL0+5S+K6trbG2tha4ZEQkKL7hWk9PDz09PXR3d9PV1RUkBwtb+IZhnFx2JO4i0kZZ2L+oqq5i\nTb0yZVsYGxsLtsOlpKJOeLFSsVgMil5vbGwEuWTW1tbY2Nggn89TKBQCP3s8HiedTgfhjl1dXZw6\ndYqBgYEgT3t4FWqUo2QaXWfSMKLKTi33PwPOqeqnvWOuTNnDVJcp24Iv7ieJcCx7qVQK8rW7JGFr\na2uBuOdyucBiL5VKiAjxeJzOzk56e3uDkMfh4WEGBgbo6emhs7MzEHd/JWpUaXSdScOIKjsJhbwN\n+C3gR5XSZAp8jLKobylTZlTjC7ursORb7mtra0GBa1ekw1nura2tgeXe29vL8PAwp06dCsrndXd3\nV+Vp9632qFruhmHsjJ1Ey/wfoJ4Dt2aZMqOMb7U7YXeWuy/uznJ3FrtrYct9aGiI06dP09PTQyaT\nIZPJVLllfFE3cTeMk43FyzUAt8io3opU33J3xThyuVzglnGWe5iWlhY6OjpIp9NB2bzTp08Hq1AT\niQTJZDLIIWMYhuEwRWgQfvpe9+qiYlxbX19ncXGRS5cuceXKFZaWltjY2AgqKflhjLFYjHQ6XeVf\nT6fTNTM+mpVuGEYYE/cGUSuNr0sz4ApwuHJ5Fy9eZGZmJhB3l6M9mUySSqWC1t3dHUTGOFeMy/YY\ni8WCBU4m7oZhhDFxbyC+j90XdxfL7opvTE9PMzMzw/LyciDu8XicRCIRhDz29PQEk6i1LPe2tjYT\nd8Mw6hLdmLkDJhzy6BYrraysBCtRL1y4wM9//vMqt8z6+nrglkkmk3R1dTEwMMCpU6c4ffo011xz\nTVXGR+cXRHVQAAAPJ0lEQVRjN7fMzhCRERH5poj8RER+JCIfrhzvEZGnReQlEfm6iHR515wRkfMi\n8qKI3H54ozeMvWOWewMJC7xLMzA3N8fk5CQXLlzg4sWLgf99fX19i1vGF/fh4WH6+vro7e2tyvjo\nF+CwsMer4hLfvVBZZf23IvI08AHKie8+ISIfpZz47gERuZlyWO9NlHMoPSMir9N6lc0N44hi4t4g\n/DBH95rNZgOf+/z8PHNzc8zOzpLP54OFTJubm0FMu583pqenh/7+/iC1gHPHtLe3W2qBXbCHxHd3\nAo+ragG4ICLngVuBZw946IaxL0zcG4CfM2ZjYyMoxrG4uMjS0lIQ6ujSC7gPAOebh3JcuhN4v/C1\nK3RtLpj9s8PEd9cC3/Yum6gcM4xjhYl7gygUCkHo4+rqalDsenl5eYu4Oyvf5Wp3eWRcOKTL2e5q\nofqRMcbeaF7iuzFve3RvtzAMGp83ycS9Aahqlbi7KkuLi4uBuDv/ej6fr5p0LZVKwFbL3S1Qspj2\n/bPLxHcTwGnv8pHKsTqMhfbNe2PsjUbnTbJomQbhi/vy8jILCwtV4r66urqtW6aW5Z5MJoO4drPc\n98V2ie+gOvHdU8A9IhITkRuAG4HnDmqghtEozHLfJfVSDDhxX11dDcTdd8usr6+Ty+XY3NyseV9n\nube3t1dZ7s5q9y13s953zm4T36nqORF5AjgHbAIftEgZ4zhi4r4HnMXtwh4LhULgjnGrUKenp5md\nnWVhYYFsNsvGxgaFQuGwh37i2EviO1V9CHioaYMyjAPAxH2XhBcquXh23x0zOzvLzMwMV65cYXFx\nMRD3YrF42MM3DOOEYOK+S/wUvi73unPHhC33+fl5VlZWTNwNwzhwTNx3ie+K2dzcDHKzh90yri6q\nK6WXy+XMLWMYxoFh4r4HfMt9O3FfWlqqWrVqlrthGAeFifsu8S33fD4fVFNyzS1gctWV/MpKLqYd\n2JIbxs/yGI6KsegYwzB2y1Xj3Gtk1fu9yvGzInJJRL5faXc0f7iHTy1hX11drVqk5Mex+5E1jpaW\nliDsMRaLEY/HgxQDLneMX+zaEoQZhrFbdmK518qq943Kzz6lqp9q3vCOHi6PjHPHuFJ5fnoB54Jx\nwu4vVoJXxd231v0CHH6edhN1wzD2wk4KZNfKqucSKZ041fHFvZbl7hYq+eLuNxGpEndnrcdisWDB\nkm+5m2vGMIy9sKv0A15WPZdA4z4ReUFEPusXO4gybjLVL3Jdyy1TLBa3CLzD+dhruWXqWe5mwRuG\nsRt2LO7hrHrAI8BrVfUWypb9iXDPOMs97HevlzfGt9id1e6E3aX27ezsDDJAOpEP+90NwzB2w46i\nZWpl1VPVK94pjwJ/Ve/6sbGxYDuc+SzKODF3bhgn7Ol0mnQ6TSaTIZ1O09nZyTXXXMOpU6fo7+8n\nk8kEdVJruWhOMo1Oi2oYUWWnoZBbsuqJyHDFHw/wbuDH9S72xf0k4ScDc+6WeDweFL/228DAAIOD\ngwwMDATi3traWjM08iTT6LSohhFVriru22TVe6+I3AKUgAvA7zZxnMcS3wXjJkyTySQ9PT0MDg4y\nODjI0NAQQ0ND9Pb20t3dTVdXV5Xl7oTdLHfDMHbDTqJl6mXV++vGDyda+JZ7PB6no6ODVCpFd3c3\nAwMDXHvttYyMjHDttdeSyWSCHO4uj7tfCNswDGM32ArVJhIW90QiQWdnZ2C5nzp1iuuuu47rr7+e\nVCoV+NidC8esdcMw9oqJ+y5xk6N+Iet0Ok13dzfr6+tBGb2WlhZUlUQiETR37vDwMAMDA/T399PT\n00MmkyGRSFS5X9y2YRjGXjBx3yUtLS2BsHd2dgZhjqVSiba2NhKJBJlMhr6+PlQ1cMe4WPZUKsXw\n8DCDg4N0dXWRSCSqYtoBc8UYhrFvTNx3SUtLC+3t7SQSCVQ12G9vbyeZTNLV1UV/fz9LS0uoalWZ\nPPeh0NvbS09PD93d3YG4OyvdRN0wjEZg4r5LnJg7YY/FYoHLJZPJsL6+HuSbAQIfugtrjMVipFKp\noCWTyWDiFMxqNwyjMRyouI+Pjx/YAqZm9eXE3Ql1qVRifHyct7zlLVW5211hDt+P7hYjOUvezwK5\nU1GPwnt4VPozjChj4r5LaqUD+N73vsc73/nOhvdViyi8h0elP8OIMhaOYRiGEUFM3A3DMCKI+Klo\nm9KBSHM7ME48qrrtZIWIfA54JzCtqr9YOdYD/DlwHeX0GXer6lLlZ2eAeykXqvmIqj5d575azsbh\n8zD33z/PJz/58D6eyDDK83VX+9vejqb73PczOMNoEJ8H/gj4gnfsAeAZVf2EiHwUOAM8ICI3A3cD\nNwEjwDMi8jptthVkGA3G3DJG5FHVbwELocN3AY9Vth8D3lXZvhN4XFULqnoBOA/cehDjNIxGYuJu\nnFQGVXUaglKSg5Xj1wIXvfMmeLWspGEcG2wRk2GU2aPbZczbHm3AMIyTSqML0RyIuIvIHcB/o/xN\n4XOq2rTZJhG5ACxRzjO/qaoN/Uq928m5JvR1FvhXwEzltI+p6r7TL4vICGWf9BDl9+5RVf3DZjxb\njb4+o6p/1Kxnq8O0iAyp6rSIDHt9TgCnvfNGKsfqMBbaf7bWSYZxVRpdiKbpbhkRaQH+GPhN4I3A\ne0TkDU3ssgSMquqbGi3sFT5P+Vl83OTc64FvUp6ca1ZfAJ9S1TdXWqPErwDcr6pvBN4CfKjye2rG\ns4X7us/7m2jGswFIpTmeAt5f2X4f8KR3/B4RiYnIDcCNwHMNHIdhHAgH4XO/FTivqi+r6ibwOOXJ\nrGYhNPG5djk514y+oFqkGoKqTqnqC5XtLPAiZau14c9Wpy/n1274s4nIl4H/C/yCiLwiIh8APg78\nhoi8BPx6ZR9VPQc8AZwDvgZ80CJljOPIQbhlwhNUl2hu9IEC3xCRIuWv+482sS9H1eSciAxe7YJ9\ncp+I/DbwPeDfNMIF5CMi1wO3AN8Bhpr5bF5fzwK/ShOeTVXfW+dHb6tz/kPAQ/vt1zAOkyhGy9ym\nqm8G3kHZtfCrhzCGZlp6jwCvVdVbgCngU428uYh0An9BefFOlq3P0rBnq9FXU5/NME4SByHuE8Br\nvP2rTFDtD1WdrLxeAb7KwcQoT4vIEEBocq7hqOoVz03wKPDLjbq3iLRRFtsvqqrzQTfl2Wr11cxn\nM4yTxkGI+3eBG0XkOhGJAfdQnrRqOCKSrFiDiEgKuB34cTO6YmeTcw3vqyKwjnfT2Of7M+Ccqn7a\nO9asZ9vSV5OfzTBOFAeRfqAoIvcBT/NqKOSLTepuCPhqJZ9NG/ClenlB9kplcm4U6BORV4CzlCfj\nviIi9wIvU16+3qy+3ioit1COCroA/G6D+roN+C3gRyLyPGX3y8eAh4EnGvls2/T13mY8m2GcRJqe\nOMwwooolDjOayX4Th0VxQtUwDOPEY+JuGIYRQUzcDcMwIoiJu2EYRgQxcTcMw4ggJu6GYRgRxMTd\nMAwjgpi4G4ZhRBATd8MwjAhi4m4YhhFBTNwNwzAiiIm7YRjGNgwPX4+I1GzDw9cf9vDqciAFsg3D\nMI4r09MvU69GzfR0w6tCNgyz3A3DMCKIibthGEYEMXE3DMOIICbuhmEYEcTE3TBqICJ3iMhPReTv\nROSjhz0ew9gtJu6GEUJEWoA/Bn4TeCPwHhF5w2GMZXx8PBJ9HFQ/B/MsB9HH/jFxN4yt3AqcV9WX\nVXUTeBy46zAG8s53vqvpMdbHWdzDMehvfetbr/re1Itb3/n7Od6AkTcfE3fD2Mq1wEVv/1Ll2I74\n0z/9bMMEeXV1iXKM9dY2PT117BbWNJpXY9BdO0v5vXl5F9ds/34eV2wRk2Hsg0zmn1ft53I/Y21t\nnvqLXjpqCkZLS5JSaW2Xvedq9lOvD4ChoeuYmrqw5fjw8PV1BbHe2LYbc72f/f7v/5ddX7O39ya+\nB2Gu/X7C8RR4Ua39R2gYJxUR+cfAmKreUdl/AFBVfTh0nv3zGE1FVff8yWLibhghRKQVeAn4dWAS\neA54j6q+eKgDM4xdYG4ZwwihqkURuQ94mvK81OdM2I3jhlnuhmEYEcSiZQxjlzRrgZOIXBCRH4jI\n8yLyXOVYj4g8LSIvicjXRaRrD/f9nIhMi8gPvWN17ysiZ0TkvIi8KCK376OPsyJySUS+X2l37LOP\nERH5poj8RER+JCIfbtKzhPv5vUY/j4jEReTZyu/6JyLynxv+LKpqzZq1HTbKBtHPgOuAduAF4A0N\nuvffAz2hYw8D/76y/VHg43u4768CtwA/vNp9gZuB5ym7bK+vPKvssY+zwP01zr1pj30MA7dUtjsp\nz4u8oQnPUq+fRj9PsvLaCnwHuK2Rz2KWu2HsjmYucBK2fpu+C3issv0Y8K7d3lRVvwUs7PC+dwKP\nq2pBVS8A5yk/8176gNpxhHftsY8pVX2hsp0FXgRGmvAstfpx6xwa+TwuvjNO+fe+0MhnMXE3jN2x\nrwVOV0GBb4jId0XkX1aODanqNJRFBxhsUF+Dde4bfr4J9vd894nICyLyWc/FsO8+ROR6yt8UvkP9\n96iR/TxbOdSw5xGRFhF5HpgCxlX1XCOfxcTdMI4Ot6nqm4F3AB8SkV9j66qaZkVANOO+jwCvVdVb\nKAvYJxtxUxHpBP4C+EjFsm7Ke1Sjn4Y+j6qWVPVNlL99/JqIjNLAZzFxN4zdMQG8xtsfqRzbN6o6\nWXm9Avwl5a/d0yIyBCAiw8BMI/ra5r4TwGnvvD0/n6pe0YrDGHiUV90Ie+5DRNooC+4XVfXJyuGG\nP0utfprxPJX7LgNfA36pkc9i4m4Yu+O7wI0icp2IxIB7gKf2e1MRSVYsRUQkBdwO/Khy7/dXTnsf\n8GTNG+ygC6r9xfXu+xRwj4jEROQG4EbKi7h23UdFnBzvBn7cgD7+DDinqp9u8rNs6aeRzyMi/c6t\nIyIJ4DcoT5g27ll2O/NuzdpJb8AdlCMozgMPNOieN1COvHmesqg/UDneCzxT6e9poHsP9/4ycJly\n8pRXgA8APfXuC5yhHI3xInD7Pvr4AvDDynP9JWV/8n76uA0oeu/T9yu/i7rvUYP7adjzAP+wct/n\ngR8A//Zqv+/d9mGLmAzDMCKIuWUMwzAiiIm7YRhGBDFxNwzDiCAm7oZhGBHExN0wDCOCmLgbhmFE\nEBN3wzCMCGLibhiGEUH+P+TZ+wgUbGx9AAAAAElFTkSuQmCC\n", + "text/plain": [ + "<matplotlib.figure.Figure at 0x7f22841b4d50>" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "%matplotlib inline\n", + "\n", + "# We'll show the image and its pixel value histogram side-by-side.\n", + "_, (ax1, ax2) = plt.subplots(1, 2)\n", + "\n", + "# To interpret the values as a 28x28 image, we need to reshape\n", + "# the numpy array, which is one dimensional.\n", + "ax1.imshow(image.reshape(28, 28), cmap=plt.cm.Greys);\n", + "\n", + "ax2.hist(image, bins=20, range=[0,255]);" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "weVoVR-nN0cN" + }, + "source": [ + "The large number of 0 values correspond to the background of the image, another large mass of value 255 is black, and a mix of grayscale transition values in between.\n", + "\n", + "Both the image and histogram look sensible. But, it's good practice when training image models to normalize values to be centered around 0.\n", + "\n", + "We'll do that next. The normalization code is fairly short, and it may be tempting to assume we haven't made mistakes, but we'll double-check by looking at the rendered input and histogram again. Malformed inputs are a surprisingly common source of errors when developing new models." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 531, + "status": "ok", + "timestamp": 1446749126656, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "716a6ad5e180d821", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "jc1xCZXHNKVp", + "outputId": "bd45b3dd-438b-41db-ea8f-d202d4a09e63" + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAEACAYAAABMEua6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXmQbHd13z9nlu7p7unu2Wf0NE9bxF4GQRlhClwMBmOB\niaTiD0XgwoCcMlWsFScOEpUq6aWSElAu8EKRhE0RBCILHCLZcYyQxdhFAggZie0J+ZnwxFtmebNP\nz9I9M33yR9/f5dfbm56Z7lnunE/Vr/r27Xvv73f7zfve0+d3fueIqmIYhmFEg7aDHoBhGIbRPEzU\nDcMwIoSJumEYRoQwUTcMw4gQJuqGYRgRwkTdMAwjQpioG5FGRJ4rIk+KyPeD10UR+YCI9IrIIyLy\njIh8XUSy3jl3icgZEXlaRN5wkOM3jJ0iFqduHBdEpA04D7wCeB8wq6ofE5EPAb2qeqeIvBD4EvBy\nYBR4FHiO2n8U44hglrpxnHg98DNVPQfcAtwf7L8fuDXYvhl4QFU3VfUscAa4cb8Hahi7xUTdOE78\nC+DLwfawqk4BqOokMBTsvxI4551zIdhnGEcCE3XjWCAinZSs8K8EuyrdKeZeMSJBx0EPwDD2iTcC\n/6CqM8H7KREZVtUpERkBpoP9F4CT3nmjwb4qRMQeBEZLUVXZ6TlmqRvHhbcC/917/zDwzmD7HcBD\n3v7bRSQmItcC1wOP17uoqra03X333ZHoI0r3sl/f124xS92IPCKSpDRJ+vve7o8CD4rIHcCzwG0A\nqnpaRB4ETgMbwHt0L//DDGOfMVE3Io+qrgKDFfvmKAl9rePvBe7dh6EZRtMx94thHGLGxsYi0cd+\n9ROVPvaCLT4yjF0iIuaZMVqGiKA2UWoYhnG8MVE3DMOIECbqhrEPPPPMMySTWTo64lVtcHCUXC53\n0EM0IoKJumHsAxMTE8RiL2Fra6mq5XIFVlZWDnqIRkSwkEbD2DfagHjVXpEdz4UZRl3MUjcMw4gQ\nJuqGYRgRwkTdMAwjQpioG4ZhRAgTdcMwjAhhom4YhhEhTNQNwzAihIm6YRhGhDBRNwzDiBAm6oZh\nGBHCRN0wDCNCmKgbhmFECBN1wzCMCGGibhiGESFM1A3DMCKEibphGEaEMFE3Io+IZEXkKyLytIj8\nREReISK9IvKIiDwjIl8Xkax3/F0iciY4/g0HOXbD2Ckm6sZx4E+Av1bVFwAvAX4K3Ak8qqrPAx4D\n7gIQkRcCtwEvAN4IfEqsNJFxhDBRNyKNiGSAX1fV+wBUdVNVF4FbgPuDw+4Hbg22bwYeCI47C5wB\nbtzfURvG7jFRN6LOtcCMiNwnIt8XkU+LSBIYVtUpAFWdBIaC468EznnnXwj2GcaRwApPG1GnA3gZ\n8F5VfUJEPkHJ9aIVx1W+b4h77rkn3B4bG2NsbGx3ozSOPePj44yPj+/5OibqRtQ5D5xT1SeC939B\nSdSnRGRYVadEZASYDj6/AJz0zh8N9tXEF3XD2AuVRsGpU6d2dZ09uV9E5CYR+amI/KOIfGgv1zKM\nVhC4WM6JyHODXa8DfgI8DLwz2PcO4KFg+2HgdhGJici1wPXA4/s3YsPYG7u21EWkDfgkpf8kF4Hv\nichDqvrTZg3OMJrEB4AviUgn8P+AdwHtwIMicgfwLKWIF1T1tIg8CJwGNoD3qOquXDOGcRDsxf1y\nI3BGVZ8FEJEHKEUUlIm6iNh/CKOlqOplQw5V9QfAy2t89Po6x98L3NuEoRnGvrMX90tllMB56kQJ\nqCqqyt133x1ut7pZX0err932ZxhGORbSaBiGESH24n65AFzlva8bJeAiBFzIjoV9GbulWWFfhhFV\n9iLq3wOuF5GrgQngduCttQ70RX2/BH0/HxzW1/7116ywL8OIKrIXv6SI3EQpr0Yb8DlV/UiNY9R8\nn0arEBF0m4nSFvbd8N/2+Pg4t956D4uL41WfJRLD/PznP2R4eLjJIzSOMrv9297T4iNV/RvgeXu5\nhmEYhtE8bKLUMAwjQpioG4ZhRAgTdcMwjAhhom4YhhEhTNQNwzAihIm6YRhGhDBRNwzDiBAm6oZh\nGBHCRN0wDCNCmKgbhmFECBN1wzCMCGGibhiGESFM1A3DMCKEibphGEaEMFE3DMOIECbqhmEYEcJE\n3TAMI0KYqBuRR0TOisgPRORJEXk82NcrIo+IyDMi8nURyXrH3yUiZ0TkaRF5w8GN3DB2jom6cRwo\nAmOq+lJVvTHYdyfwqKo+D3gMuAtARF4I3Aa8AHgj8CkROZAaqIaxG/Yk6rUsIMM4hAjVf+u3APcH\n2/cDtwbbNwMPqOqmqp4FzgA3YhhHhD0VnuaXFtB8MwZjGC1CgW+IyBbwX1T1s8Cwqk4BqOqkiAwF\nx14JfNs790KwzzCOBHsV9VoWkGEcNl6lqhMiMgg8IiLPUBJ6n8r3DXHPPfeE22NjY4yNje12jMYx\nZ3x8nPHx8T1fZ6+i7ltAn1bVz+x5RIbRZFR1Ini9JCL/k5I7ZUpEhlV1SkRGgOng8AvASe/00WBf\nTXxRN4y9UGkUnDp1alfX2auo+xbQN0TkaVX9VuVBZs0YzWKn1oyIJIE2Vc2JSAp4A3AKeBh4J/BR\n4B3AQ8EpDwNfEpFPUHK7XA/YfJFxZBDVXf3qrL6QyN3Asqp+vGK/NqsPw6hERFDVutEpInIt8DVK\nvyo7gC+p6kdEpA94kJJV/ixwm6ouBOfcBfwesAF8UFUfqXPthv+2x8fHufXWe1hcHK/6LJEY5uc/\n/yHDw8MNXcs4Hmz3t12PXVvql7GADOPQoKo/B26osX8OeH2dc+4F7m3x0AyjJezF/TIMfE1EfAuo\npkVjGIZh7A+7FvV6FpBhGIZxcOx1ovTQoqoUi8Wypqpsbm6ytbXF5uZmuO38oiJS1hz+dltbG+3t\n7WHr6Oigra0tPMY/t9arLU40DKOVRFrUnXj7Ip7P58nn86yvr4fbW1tbtLW1lbVKcXbbHR0dxGIx\n4vE4sVgsbJUPBP8atR4UhmEYreBYiHqhUGBjY4NCocDKykpV29zcDC1v3xKHaus9FouRSCRIJpPh\nq6pWPRAqHxJuTCbshmG0kkiLerFYDMXcWeZLS0ssLi6yuLgYbm9sbNDR0RG6U9x2Lcu7q6uLdDpN\noVBga2urSsD9B4Oqhq4dJ+yGYRitJNKi7ix1525ZW1tjaWmJ+fl5ZmdnmZubY25ujkKhQEdHB52d\nnWWvvpi711QqVSbo7kHgrPtisRgKeqW1b/H6hmG0msiKurPS19bWyOVyLC8vs7y8zOzsbFXL5/NV\ngu5PgPqvyWSyzB/vHha+sPtC767l/wI4rNSb2K10JblfHfUmhA3DODgiLerr6+ssLS2FFvnc3Bzz\n8/NhW1hYYGFhgUKhUCXEzo1S6VNPJBLkcjkWFxeZnZ0lnU6TSqWqRL29vZ3Ozs6q1tFxeL/yyjkB\nEQknhl2Lx+N0dnbWfOAZhnHwHF6F2SNbW1usr6+zvLzM3Nwck5OTTE5OsrS0FPrSl5eXQ5965URp\nveiVWhOlXV1dVYLurH4/UsYJ4mGlMlyzra2NeDxOMpkM29bWFslksmzewL2apW4YB09kRd231Gdn\nZ5mcnOTcuXPkcrmwrayskMvl2NzcrBuxUulacEJdablWinp7ezuxWIyuri66urqIx+N0dXURi8UO\n7DvZDt9F5LYTiQSZTIZMJhN+T+4zVS2bezAM4+CJrKj7lvrs7CwTExOcO3eO1dVVVldXWVtbC5sT\nq8vFlLv3ldZsvdbW1kZXVxeJRIJEIkFXVxfJZJJ4PH4QX0dDuF8XfkulUqyvr4cTw52dnXR1dYXn\n+A9AwzAOnsiKurPUc7kc8/PzTE9Pc+HChbJJzkKhEC4+2i3O7eCLvdt2Qu43XxAPG7FYrOqXSCaT\nCQXdWe6FQiE8x4m6RfYYxuEgsqLe3t4eug4GBgY4ceJE1UrSyhWllStBXax75WvldrFYBAiFzRc4\nF4Wzvr6OqrKxsVHTX++OdbHtfoz7Tqj1C6PWytjKflS1bDLXbW9ubpa9d8Lvfn24h9phniswjONE\nZEW9ra2NRCJBNptlcHCQfD6Pqoai7r+62PLKtrW1VdVq5Y6pty0ioai7uPlCoVBzsZL7vDJfzU6o\nJ+iVUS1AVT/FYrHmAiy3MMsX9VgsxubmJsViMbTgzVI3jMNBZEXduT8ymQz5fJ5isUhnZydra2us\nr69Xibofn+62NzY22NzcDF/ddqFQKEs9sLGxUWX9FwqFUCw3NjbKBL1ywVJHRwfFYrHqobEXUXfb\nlRE9Lk6+1gOr1rEbGxtlgh6Px4nH42WC3tXVteOxGobRGiIt6s5Sd1EaqVSqTNTdtqqW+ZGdi8GJ\nty/ilS4cdy03AetE04l5pbsGqPkAKRaLZQ8Q9yDYCbUWA9VaDAWEDym/1Yo990XdCXo8Hg8FPR6P\nh79EDMM4eCIv6s5XnEql6O3tDSNeKkXdhR36oYf1BHxtba0sgmZ1dZXl5eUwvK9YLFIoFEKr27f0\nt7a2yiYkt7a26OzsDI/zfwFsbm7u6J5r+c8rV7S6MW5sbFQ1d57/WigUqgQ9Ho+Hgp5Kpdjc3DRR\nN4xDQmRF3Q+/87edMPsuGKBKtJyoVzY/FNJvqVSqrLl0ApXums3NzbIYd7e9tbUVum1c262o+8Jc\nmdOms7OTYrEYjhuoyisPv5zsjcViZeNxbqHKPPWGYRwOIi3qzi/sR2b4SbicYEF5OJ8TP39laDwe\np1AohGGKvtCvr6+zsrLC6upqWUpfX6SduDtLvbI5n3ulu2cn9+te/VYrUdnm5iaLi4ssLCywuLgY\nWu5+9A38Utj9eQD/YeSuZ3HqhnF42FbUReRzwJuBKVV9cbCvF/hz4GrgLKVK7IstHOeOcaLufMi+\nyDtx8t0Ovp/b93c7QfcnSn03iduu5avP5/NllnqrRb2WsPv34bbz+TzT09N0dXWFfvOVlZWaFrj/\nPfqTpe56fkoFwzAOnkYs9fuAPwO+4O27E3hUVT8mIh8C7gr2HSqciNcSdD/qA6iZ+8VFpLjmJjMr\no1RqCXKt5rtfKisn1XK/7EbUa1nqlcK+trYWuqU2NjbI5XJ0dHSUTcw6Qa/8xVNpqbvvyjCMw8G2\noq6q3xKRqyt23wK8Jti+HxjnkIm6H5PtJ5+qXExUqz6pszwrj6u3EKlWOKIf/uhb69tZ6r6w78Wn\nfjlLfXl5ORT0lZUV5ubmqhYP+SGKlQ9EN5Hsi7pZ6oZxONitT31IVacAVHVSRIaaOKam4At0q6kU\nfbftSulV+tQrl+I30/1yOUvdbS8uLoYFQ2ZmZkgkEmGBDzd+90ullpXuIoWOkk9dRNqAJ4Dzqnrz\n5VyIInIXcAewCXxQVR85mFEbxs5p1kTpZcMf7rnnnnB7bGyMsbGxJnV7uKhc/FNrotat3Kz0SbsU\ntv7qzJ0W1Kgn7E6g3S+Jyvw3zi3kFhS5MXV2doY5a1KpFN3d3WFzeWxisVgYJrkfjI+PMz4+vptT\nPwicBjLB+5ouRBF5IXAb8AJgFHhURJ6jFuJjHBF2K+pTIjKsqlMiMgJMX+5gX9SjjC9sTqj9ohhO\nXGvVQq01qbuTxUe18rs4F5JrLp7cibov6M7VU6t8n2vd3d2k02nS6XQo6s4Fs1+iXmkUnDp1attz\nRGQUeBPwH4E/CHbXcyHeDDygqpvAWRE5A9wIfLcpN2AYLaZRUZegOR4G3gl8FHgH8FBzh3X0cAJa\nKez+tosFr8zoWOm+cNbybpbeVy4e8lMOOPGutNLd6lW/2pFrvpXuW+vOBROLxfZV1HfJJ4A/BLLe\nvuE6LsQrgW97x10I9hnGkaCRkMYvA2NAv4j8Argb+AjwFRG5A3iW0s/VY48v7LXcH861UrkU34m4\nO973b++0/8r3LgeNs8j9VbKVi4o6OztDUXf+cyfqvqCn0+mqEn2HVdRF5LcpheM+JSJjlzl0V+6V\n4+JaNFrPHlyLZTQS/fK2Oh+9fs+9R5BKv7r/3hfpSovabbsonWbhZ4l0E7eV7hcn+O7B4ldtqud+\nqUz8dVhFHXgVcLOIvAlIAGkR+SIwWceFeAE46Z0/GuyryXFxLRqtZzeuxVoc/rCFI0C9CUpniftp\ndivdLrWObaSyUq1KS35z13QTo65YyKVLl5ienmZubo6lpSVWV1dD0W9vbw/zuWSzWfr7++nv76en\np4dMJkMqlSoLZ/TnBA6rqKvqh1X1KlW9DrgdeExV3w78JSUXIpS7EB8GbheRmIhcC1wPPL7PwzaM\nXRPZNAHHlcpl/vl8nlwux9zcHLOzs8zOzjI1NcXU1BRzc3Pkcrkw17xfk7Svr4++vj6Gh4cZGBgg\nm82SSqWIxWI1H15HkI8AD1a6EFX1tIg8SClSZgN4j0W+GEcJE/UIUauaUT6fZ3l5mbm5OaamppiY\nmGBqaopLly6Fol4oFKpEvb+/n+HhYYaHh+nv7yebzZJMJonFYmW/BI4Sqvp3wN8F23PUcSGq6r3A\nvfs4NMNoGibqEcMX9GKxWGapT05Ocu7cOSYnJ1lcXGRxcTEUdSCMS3eiPjIywvDwMH19fWWWuovW\nqTUvYBjGwWKiHiEqBb1S1Kempjh//jwTExNlOeFruV98UU+n02QymdBSNzE3jMOLiXqE8MXcJSBb\nX19neXk5nCSdmJhgYmKiqkwfEJamS6fT9Pb2MjAwwODgIIlEgmQySSKRoLOz80ikBTCM44qJekSo\nzDXj2sLCAsvLy6yurrK+vl5WkcnPl14r5XBllIsl7jKMw4+JekRQVTY2NsJ87q4tLi6yvLzMyspK\nmag7N42f9KxWlSS/YIiJumEcfkzUI4JbXJTP51lZWSGXy5HL5UJR9y11P6eMHyNvlrphHH1M1COC\nb6mvrq6ytLQUlqxzlvra2lpoqddKVVAp6n4aAJdi10TdMA43NuMVEZyl7uqlLi0tMT8/X2Wp+2l2\nffeLnznSL11nlrphHC3MUj+CVC4wAsI6qblcjoWFBWZnZ5menmZmZoaFhYVw5aiz0l1uF78NDQ2F\nC41SqVRYCKMyrYFhGIcXE/UjSGUpPRe66HzoTtAvXrzIzMwM8/Pz5HI51tfX2draor29nUQiQTab\nDVsmk2F0dJQrrriC/v5+0uk08Xg8tNLNUjeMo4GJ+hHEibpfF9VNkC4uLjI3NxeK+sLCAktLSywv\nL5PP58MiHclkkp6eHoaGhhgcHGRwcLAsLYAv6malG8bRwUT9COLXQHWLiJwv3Ym6y/OSy+XClaPO\nUo/FYiSTSXp7exkeHmZ0dJTR0VF6e3vD5ou6HyFjwm4YhxsT9SOIqrK1tcXW1laYD72e+8WfHHVW\nvXO/9PT0MDw8zFVXXcV1110XVjdyudO7urrKSuz5r4ZhHE5M1I8gvqXuLzhaWVlheXmZxcVF5ufn\nmZmZCVMAONra2ojFYmU500dGRhgdHS0rUedPkhqGcXQwUT+CuGpGTsxXV1fDsEUXi16vHmp7ezvZ\nbJZ0Ok0qlQoLSDsRPwqFLwzDqM+2ceoi8jkRmRKRH3r77haR8yLy/aDd1NphGj7O7ZLP51ldXSWX\ny9VNBdDW1kZnZ2dY0SiTyYSi3t3dHYp6rZh0wzCOHo38z70P+K0a+z+uqi8L2t80eVzGZXCuFyfq\ny8vLLC0t1RR1t1LU1Rv1RT2VSpFIJEJRj8VittDIMI44jRSe/paIXF3jI/sff0A490stS72W+6Wz\nszMU9e7ubjKZTJWlHo/Hq2qdmqgbxtFjL7+x3yciT4nIZ0Uk27QRGdvi+9Sdpb6d+6WepV7P/WKC\nbhhHk91OlH4K+PeqqiLyH4CPA79X7+B77rkn3B4bG2NsbGyX3R4v6tU7dj51X9R998vGxgbFYhEg\nFPTu7m56enro7+9ncHAwjEX3C1/4k6OHVdTHx8cZHx8/6GEYxqFlV6Kuqpe8t58B/vJyx/uibuwe\nJ/K+qLswxsqkXcViEREpqzva19fH0NBQuGrUlajr7Ow8EoIO1UbBqVOnDm4whnEIadT9Ing+dBEZ\n8T57C/DjZg7KKMdP3OUWHhUKhW0tdSfqru6oL+p9fX1kMpnQUnccZkE3DGN7trXUReTLwBjQLyK/\nAO4GXisiNwBF4Czw7haO0YC6ou4s9cuJum+pu/wutYpJm6AbxtGnkeiXt9XYfV8LxmLUwPerO4u9\nlvtlO0s9nU6XWeqJRCJszv0SRUQkDvw9EAvaQ6r6YRHpBf4cuJqSYXKbqi4G59wF3AFsAh9U1UcO\nYuyGsRtshckhx4m4SwngLHS3ktSJukut6wTdj3pJJpOk02my2WyYsCuTyZBKpejq6gpTAThr3W9H\nHVXNA69V1ZcCLwZ+Q0ReBdwJPKqqzwMeA+4CEJEXArcBLwDeCHxKovBFGMcGSxNwiHHVjAqFQpi4\nq1AoMD8/z8LCQljVyGVidLVHneUdi8XCmPRkMlm20Kiy+EWUUdXVYDNOyZCZB24BXhPsvx8YpyT0\nNwMPqOomcFZEzgA3At/dzzEbxm4xUT/EuLqj+XyetbW1MH2uE/WlpSWWlpbI5XKsra2F6XFdKbq2\ntrZwkZGzyl3CLle+7jgsMhKRNuAfgH8G/GdVPS0iw6o6BaCqkyIyFBx+JfBt7/QLwT7DOBKYqB9i\nKuuOrqyskMvlwtqjvqW+trYWLvV3ou4sdbfIKJFIVJWoOw4LjVS1CLxURDLA10VkDKhcBFB7UcA2\n2BoMo1k0aw2Gifohxom6X9XIpdX1Kxo594tzubiIF7eC1M/z4ix1v+hF1EXdoapLIvLXwK8CU85a\nD0J0p4PDLgAnvdNGg301sTUYRrNo1hqMaDtTjzi+pb66usrS0hJzc3M1fepra2th7nQn6pXZGH1R\n9631KAu7iAy4NBYikgB+E3gSeBh4Z3DYO4CHgu2HgdtFJCYi1wLXA4/v66ANYw+YpX5IKBaLYcii\n287n81XVjKanp7l06VJZMWkX7eKKX7iY9IGBAfr7+8lms6RSqTBpV1QFvA5XAPcHESxtwBdV9W9F\n5EngQRG5A3iWUsQLgb/9QeA0sAG8R+vlazCMQ4iJ+iHBL1Hn2traGsvLyywsLDAzMxPWHZ2dnWV+\nfp6VlRUKhQKqGqbXdTleBgYGGB4eZmBgoErUjxOq+iPgZTX2zwGvr3POvcC9LR6aYbQEE/VDQmU8\n+ubmZpgCYH5+ntnZWaamprhw4UIY8ZLL5UJRr5W4a2RkhL6+Pnp6euju7iYej0c+fNEwjjsm6ocE\n53ZxseiFQqGmpX7x4sXQQncNKLPUs9lsaKn7aXaPo6VuGMcNE/VDgm+pVybrWlhYCC31ixcvks/n\nQ/+7a5Xul/7+foaHh+nu7g7TAbjYdcMwoouJ+j5zuRzpzjp3Menz8/M129bWVljMoqOjI8zv4lc2\ncikBnJi7iBcTdcOINibqB4ifebFQKLCyssLCwgILCwuhH31ycpK5uTmWl5dDC90vJu1CFHt6eshk\nMmXhi7WKSR+zyBfDOHaYqB8QvusEYGNjIxR1F7Y4PT3N1NRUmagXi8XQOnfJupLJZJWoV6YEcKJu\nGEa0MVE/ACr94b6lPj8/z/T0NBcvXmRycpLZ2Vnm5ubI5XKhpd7e3l4W7ZJOp2uKuu9ysWLShnE8\nMFE/QGqJ+sLCApcuXeLChQtcvHgxTAXgW+puoZHzo2cymTJR91MCdHR0hCtGTdQNI/qYqB8QvqC7\nUMZKUT9//jzr6+vk8/nw1fepd3V1kUqlyGazdS11C2E0jOOFiXoLqRXpoqqhQPvtwoULTE1NMTMz\nw8LCQlnRi62tLUSEjo4OVJVEIhHGo/f39zM4OMjg4GBV3dEo53QxDKM2jdQoHQW+AAxTqkn6GVX9\n08uVAzPqUywWyefzYQk651qZnJxkYmKCmZkZFhcXWV1dDd0tqhqKeltbW1jJqKenp6xEXW9vL+l0\nOvIl6gzDqE8j4RCbwB+o6ouAVwLvFZHnU6ccmHF5isUi6+vrLC0tMTs7y8WLF3n22Wf5xS9+EYr6\nwsJCuGp0a2urKozRF3VnqQ8NDdHX11cm6oZhHD8aKTw9CUwG2zkReZpSjul65cCMOjgfuhP1mZkZ\nJiYmmJiY4NKlS8zOzjI7O1tmqbsYc78lk8mylaNDQ0MMDQ2F4Y3JZJKODvOsGcZxZEf/80XkGuAG\n4DtAvXJgRg2cf925X5yoO0t9bm6OxcXFsESds9SBMGqlo6ODWCxWZak7UferHpn7xTCOJw2Luoh0\nA18FPhhY7E0pBxZ1/FWjLr9LPp8Pi164POkule7q6iqrq6uh60VEwqIXrvX29tLb20tPTw/ZbDZM\n2lVp0RuGcfxoSNRFpIOSoH9RVV2FmHrlwKo4znUcKxcZbW1thcWk19fXw1wvq6urrK+vUygU2Nzc\nDP3o8XicdDodhi1ms1lOnDjB4OBgmCe9ctVolKNemlXH0TCiSqOW+ueB06r6J94+Vw7so5SXA6vi\nuNZxrIxFLxaLYb50l7xrdXU1FPV8Ph9a6MViEREhHo/T3d1NX19fGLo4MjLC4OAgvb29dHd3h6Lu\nrxyNKs2q42gYUaWRkMZXAb8D/CgoAabAhymJeVU5MKMcX9BdRSPfUl9dXQ0LR7viGM5Sb29vDy31\nvr4+RkZGOHHiRFimrqenpyxPum+lR9VSNwzj8jQS/fJ/gHoO2prlwIwSvpXuBN1Z6r6oO0vdWeiu\nVVrqw8PDnDx5kt7eXjKZDJlMpsz94ou5ibphHE8s7q0JuMVB9VaQ+pa6K4KRz+dD94uz1Ctpa2uj\nq6uLdDodlqc7efJkuGo0kUiQTCbDHC+GYRimBE3CT6PrXl2Ui2tra2ssLCxw/vx5Ll26xOLiIuvr\n62HlIj8cMRaLkU6ny/zn6XS6ZgZGs8oNw3CYqDeJWul0XToAV/jClaU7d+4c09PToai7HOnJZJJU\nKhW2np6eMNLFuVxc9sVYLBYuTDJRNwzDYaLeRHwfui/qLhbdFb2YmppienqapaWlUNTj8TiJRCIM\nXezt7Q0nR2tZ6h0dHSbqhmFUEd3Yt32mMnTRLTJaXl4OV46ePXuWn/3sZ2Xul7W1tdD9kkwmyWaz\nDA4OcuKtFEXOAAAQFklEQVTECU6ePMkVV1xRloHR+dDN/dIYIjIqIo+JyE9E5Eci8oFgf6+IPCIi\nz4jI10Uk651zl4icEZGnReQNBzd6w9g5Zqk3kUphd+kAZmdnmZiY4OzZs5w7dy70r6+trVW5X3xR\nHxkZob+/n76+vrIMjH7hCwtf3BaXkO6pYFX0P4jII8C7KCWk+5iIfIhSQro7ReSFlMJzX0Apx9Gj\nIvIcrVcx3DAOGSbqTcIPV3SvuVwu9KnPzc0xOzvLzMwMhUIhXIC0sbERxqT7eV16e3sZGBgIUwA4\nt0tnZ6elANgBu0hIdzPwgKpuAmdF5AxwI/DdfR66YewKE/Um4Od0WV9fD4tgLCwssLi4GIYsujQA\nTvid7x1KceVO2P2C0q6AtLla9k6DCemuBL7tnXYh2GcYRwIT9SaxubkZhjCurKyERaSXlpaqRN1Z\n9S5Xusvz4sIaXc50V2vUj3QxdkerEtId57xGRnNpVl4jE/UmoKplou6qGi0sLISi7vznhUKhbDK1\nWCwC1Za6W1hkMel7Z4cJ6S4AJ73TR4N9NTmueY2M5tOsvEYW/dIkfFFfWlpifn6+TNRXVlYu636p\nZaknk8kwLt0s9T1xuYR0UJ6Q7mHgdhGJici1wPXA4/s1UMPYK2ap75B6qQCcqK+srISi7rtf1tbW\nyOfzbGxs1Lyus9Q7OzvLLHVnpfuWulnrjbPThHSqelpEHgROAxvAeyzyxThKmKjvAmdhu/DFzc3N\n0O3iVo1OTU0xMzPD/Pw8uVyO9fV1Njc3D3rox47dJKRT1XuBe1s2KMNoISbqO6RygZGLR/fdLjMz\nM0xPT3Pp0iUWFhZCUd/a2jro4RuGEXFM1HeIn0rX5T53bpdKS31ubo7l5WUTdcMw9g0T9R3iu1w2\nNjbC3OiV7hdXd9SVrMvn8+Z+MQyj5Zio7wLfUr+cqC8uLpatMjVL3TCMVmOivkN8S71QKITVi1xz\nC49cNSO/kpGLSQeqcrf4WRcro1ws2sUwjEbZNk69Rpa79wf77xaR8yLy/aDd1PrhHjy1BH1lZaVs\ncZEfh+5Hyjja2trC8MVYLEY8Hg9TAbjcLn4RaUvcZRhGozRiqdfKcveN4LOPq+rHWze8w4fL8+Lc\nLq4knZ8GwLlanKD7i4zgl6LuW+d+4Qs/T7qJuWEYO6GRwtO1sty5BEfHTm18Ua9lqbsFRr6o+01E\nykTdWeexWCxcaORb6uaCMQxjJ+woTYCX5c6lIX2fiDwlIp/1iwxEGTdJ6hePruV+2draqhJ2h/Oh\n13K/1LPUzWI3DKMRGhb1yix3wKeA61T1BkqW/LFwwzhLvdKvXi+vi2+hOyvdCbpLsdvd3R1mZHTi\nXulXNwzDaISGol9qZblT1UveIZ8B/rLe+cc1PakTceducYKeTqdJp9NkMhnS6TTd3d1cccUVnDhx\ngoGBATKZTFiHtJYr5jjTrPSkhhFVGg1prMpyJyIjgb8d4C3Aj+udfFzTk/pJupxbJR6Ph0Wl/TY4\nOMjQ0BCDg4OhqLe3t9cMcTzONCs9qWFElW1F/TJZ7t4mIjcAReAs8O4WjvNI4rta3ERoMpmkt7eX\noaEhhoaGGB4eZnh4mL6+Pnp6eshms2WWuhN0s9QNw2iERqJf6mW5+5vmDyda+JZ6PB6nq6uLVCpF\nT08Pg4ODXHnllYyOjnLllVeSyWTCHOouj7pfYNowDKMRbEVpC6kU9UQiQXd3d2ipnzhxgquvvppr\nrrmGVCoV+tCdq8asc8MwdoqJ+g5xk55+geh0Ok1PTw9ra2thubq2tjZUlUQiETZ37MjICIODgwwM\nDNDb20smkyGRSJS5Wdy2YRjGTjBR3yFtbW2hoHd3d4fhisVikY6ODhKJBJlMhv7+flQ1dLu4WPRU\nKsXIyAhDQ0Nks1kSiURZTDpgLhfDMHaNifoOaWtro7Ozk0QigaqG7zs7O0kmk2SzWQYGBlhcXERV\ny8rRuYdBX18fvb299PT0hKLurHITc8Mw9oKJ+g5xIu4EPRaLha6VTCbD2tpamA8GCH3kLjwxFouR\nSqXClkwmwwlRMCvdMIy9sa+iPj4+vm8Lj1rVlxN1J9DFYpHx8XFe+cpXluVOdwUxfD+5W0TkLHc/\nK2OjYh6F7/Cw9GcYUcREfYfUWrb/xBNP8OY3v7npfdUiCt/hYenPMKKIhVcYhmFECBN1wzCMCCF+\nStiWdCDS2g6MY4+qXnYyQkQ+B7wZmFLVFwf7eoE/B66mlObiNlVdDD67C7iDUoGYD6rqI3Wuq43+\n/xkfH+fWW+9hcXG86rNEYpif//yHDA8PN3Qt43ggItv+bdei5T713QzKMJrMfcCfAV/w9t0JPKqq\nHxORDwF3AXeKyAuB24AXAKPAoyLynIbV2zAOGHO/GJFHVb8FzFfsvgW4P9i+H7g12L4ZeEBVN1X1\nLHAGuHE/xmkYzcBE3TiuDKnqFIQlG4eC/VcC57zjLvDL8o2GceixxUeGUWJX7pXjWgDGaD5NKwBT\nWRi5FQ24Cfgp8I/Ah1rc11ngB8CTwOMtuP7ngCngh96+XuAR4Bng60C2hX3dDZwHvh+0m5rU1yjw\nGPAT4EfAB1p1bzX6en8r7y249tUV3+PTwHCwPQI8HWzf6f+NUkox/Yo619RG+eY3v6nZ7GsUtKol\nEkM6OTnZ8LWM40Hw97Xjv/WWu19EpA34JPBbwIuAt4rI81vYZREYU9WXqmorfKH3UboXHzfp9jxK\nYnVXC/sC+Liqvixozcprvwn8gaq+CHgl8N7g36kV91bZ1/u8v4lW3BuABM3xMPDOYPsdwEPe/ttF\nJCYi1wLXA483cRyG0VL2w6d+I3BGVZ9V1Q3gAUqTVK1CaOF96c4m3VrRF5SLU1NQ1UlVfSrYzlGy\nZEdpwb3V6cv5rZt+byLyZeD/As8VkV+IyLuAjwC/KSLPAK8L3qOqp4EHgdPAXwPvCawmwzgS7IdP\nvXLi6TytjSZQ4BsisgV8WlU/08K+HGWTbiIytN0Je+R9IvJ24AngX2sQX90sROQa4AbgO5RcFC27\nN6+v7wKvpgX3pqpvq/PR6+scfy9w7177NYyDIIrRL69S1ZcBb6LkQnj1AYyhlZbdp4DrVPUGYBL4\neDMvLiLdwFcpLbrJUX0vTbu3Gn219N4M4ziwH6J+AbjKez8a7GsJqjoRvF4Cvsb+xBhPicgwgIiM\nANOt6khVL3nugM8AL2/WtUWkg5LIflFVnY+5JfdWq69W3pthHBf2Q9S/B1wvIleLSAy4ndJkVNMR\nkWRg/SEiKeANwI9b0RWNTbo1va9AWB1vobn393ngtKr+ibevVfdW1VeL780wjgX7kSZgS0TeRyks\nrg34nKo+3aLuhoGvBflmOoAvaZ28HbslmHQbA/pF5BeUwvA+AnxFRO4AnqW0zLxVfb1WRG6gFOVz\nFnh3k/p6FfA7wI9E5ElKbpYPAx8FHmzmvV2mr7e14t4M4zjR8oRehhFVLKGX0Up2m9ArihOlhmEY\nxxYTdcMwjAhhom4YhhEhTNQNwzAihIm6YRhGhDBRNwzDiBAm6oZhGBHCRN0wDCNCmKgbhmFECBN1\nwzCMCGGibhiGESFM1A3DMJrEyMg1iEhVGxm5Zt/GsB+VjwzDMI4FU1PPUquOzNRU06s01sUsdcMw\njAhhom4YhhEhTNQNwzAihIm6YRhGhDBRN4waiMhNIvJTEflHEfnQQY/HMBrFRN0wKhCRNuCTwG8B\nLwLeKiLPP4ixjI+PR6KP/eonKn3sBRN1w6jmRuCMqj6rqhvAA8AtBzGQ3QjITmOlj4Oo1/tOLve9\n1DvnzW++tXU30ARM1A2jmiuBc97788G+lvErv/LymgLyR3/0xzu+1i9jpcvb1NRk0/o4atT7Tkrf\ny7M7OmdlJVf3AXEYsMVHhrEPdHZ2sr7+IzKZf1712erqIpcu5am1aGVlpaOmWLS1JSkWV3c4ip31\nATA8fDWTk2er9o+MXFNTDC8/rg5OnTrV8DmXu1b9z2r3cXniOxTkLWp9jyUOXthN1A2jmgvAVd77\n0WBfFTu1zvL5v7rMp41fa3tBr3etnY13aurZHd3j5ce1uaNzLnet+p/V7qPEbgR3N99j7c/2y5IX\n1XpPHMM4nohIO/AM8DpgAngceKuqPn2gAzOMBjBL3TAqUNUtEXkf8AileafPmaAbRwWz1A3DMCKE\nRb8YRgOISK+IPCIiz4jI10UkW+e4rIh8RUSeFpGfiMgrWtFPcGybiHxfRB5udh8iMioijwX38CMR\n+UCD19520ZaI/KmInBGRp0Tkhp2MvdF+RORtIvKDoH1LRH6l2X14x71cRDZE5C2t6ENExkTkSRH5\nsYh8c9uLqqo1a9a2acBHgX8bbH8I+Eid4/4r8K5guwPItKKf4PN/Bfw34OFm9wGMADcE292U5hie\nv81124B/Aq4GOoGnKs8B3gj8r2D7FcB3dvFv0Ug/vwZkg+2bdtpPI314x/0t8FfAW1pwH1ngJ8CV\nwfuB7a5rlrphNMYtwP3B9v1A1QoUEckAv66q9wGo6qaqLjW7n6CvUeBNwGd3eP2G+lDVSVV9KtjO\nAU+zfax+I4u2bgG+EFz3u0BWRIZ3OP5t+1HV76jqYvD2Ow2Mfcd9BLwf+CowvcPrN9rH24C/UNUL\nAKo6s91FTdQNozGGVHUKSoIHDNU45lpgRkTuC9winxaRRAv6AfgE8IfUD5huRh8AiMg1wA3Ad7e5\nbiOLtiqPuVDjmO3Y6eKwfwn872b3ISIngFtV9T+xu3jJRu7juUCfiHxTRL4nIm/f7qIW/WIYASLy\nDcC3GoWSaP67GofXEtMO4GXAe1X1CRH5Y+BO4O5m9iMivw1MqepTIjJGDUFpwr2463RTskQ/GFjs\nRwoReS3wLuDVLbj8H1NyX4XdtaAP9zf1G0AK+LaIfFtV/+lyJxiGAajqb9b7TESmRGRYVadEZITa\nP7fPA+dU9Yng/Vcp/0/frH5eBdwsIm8CEkBaRL6gqr/bxD4QkY7gHr6oqg/Vu55HI4u2LgAntzmm\nGf0gIi8GPg3cpKrzLejjV4EHpLSqaAB4o4hsqGqjE9eN9HEemFHVdWBdRP4eeAklX3xNzP1iGI3x\nMPDOYPsdQJXIBS6NcyLy3GDX64DTLejnw6p6lapeB9wOPOYLejP6CPg8cFpV/6TB634PuF5ErhaR\nWDC2SoF7GPhdABH5NWDBuYJ2wLb9iMhVwF8Ab1fVn+3w+g31oarXBe1aSg+/9+xA0Bvqg9K/zatF\npF1EkpQmly+/ZmKnM8/WrB3HBvQBj1KKAnkE6An2XwH8lXfcS4L/rE8B/4MgAqPZ/XjHv4adR79s\n2welXwNbwX08CXyfksW73bVvCq57Brgz2Pdu4Pe9Yz5JydL8AfCyXf57XLYf4DPAbDDuJ4HHm91H\nxbGfZ4fRLzv4vv4NpQiYHwLv3+6atvjIMAwjQpj7xTAMI0KYqBuGYUQIE3XDMIwIYaJuGIYRIUzU\nDcMwIoSJumEYRoQwUTcMw4gQJuqGYRgR4v8DDPSR5usfYD4AAAAASUVORK5CYII=\n", + "text/plain": [ + "<matplotlib.figure.Figure at 0x7f22841b4550>" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# Let's convert the uint8 image to 32 bit floats and rescale \n", + "# the values to be centered around 0, between [-0.5, 0.5]. \n", + "# \n", + "# We again plot the image and histogram to check that we \n", + "# haven't mangled the data.\n", + "scaled = image.astype(numpy.float32)\n", + "scaled = (scaled - (255 / 2.0)) / 255\n", + "_, (ax1, ax2) = plt.subplots(1, 2)\n", + "ax1.imshow(scaled.reshape(28, 28), cmap=plt.cm.Greys);\n", + "ax2.hist(scaled, bins=20, range=[-0.5, 0.5]);" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "PlqlwkX-O0Hd" + }, + "source": [ + "Great -- we've retained the correct image data while properly rescaling to the range [-0.5, 0.5].\n", + "\n", + "## Reading the labels\n", + "\n", + "Let's next unpack the test label data. The format here is similar: a magic number followed by a count followed by the labels as `uint8` values. In more detail:\n", + "\n", + " [offset] [type] [value] [description] \n", + " 0000 32 bit integer 0x00000801(2049) magic number (MSB first) \n", + " 0004 32 bit integer 10000 number of items \n", + " 0008 unsigned byte ?? label \n", + " 0009 unsigned byte ?? label \n", + " ........ \n", + " xxxx unsigned byte ?? label\n", + "\n", + "As with the image data, let's read the first test set value to sanity check our input path. We'll expect a 7." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 90, + "status": "ok", + "timestamp": 1446749126903, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "716a6ad5e180d821", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "d8zv9yZzQOnV", + "outputId": "ad203b2c-f095-4035-e0cd-7869c078da3d" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "magic number 2049\n", + "label count 10000\n", + "First label: 7\n" + ] + } + ], + "source": [ + "with gzip.open(test_labels_filename) as f:\n", + " # Print the header fields.\n", + " for field in ['magic number', 'label count']:\n", + " print field, struct.unpack('>i', f.read(4))[0]\n", + "\n", + " print 'First label:', struct.unpack('B', f.read(1))[0]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "zAGrQSXCQtIm" + }, + "source": [ + "Indeed, the first label of the test set is 7.\n", + "\n", + "## Forming the training, testing, and validation data sets\n", + "\n", + "Now that we understand how to read a single element, we can read a much larger set that we'll use for training, testing, and validation.\n", + "\n", + "### Image data\n", + "\n", + "The code below is a generalization of our prototyping above that reads the entire test and training data set." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 2 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 734, + "status": "ok", + "timestamp": 1446749128718, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "716a6ad5e180d821", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "ofFZ5oJeRMDA", + "outputId": "ff2de90b-aed9-4ce5-db8c-9123496186b1" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Extracting /tmp/mnist-data/train-images-idx3-ubyte.gz\n", + "Extracting /tmp/mnist-data/t10k-images-idx3-ubyte.gz\n" + ] + } + ], + "source": [ + "IMAGE_SIZE = 28\n", + "PIXEL_DEPTH = 255\n", + "\n", + "def extract_data(filename, num_images):\n", + " \"\"\"Extract the images into a 4D tensor [image index, y, x, channels].\n", + " \n", + " For MNIST data, the number of channels is always 1.\n", + "\n", + " Values are rescaled from [0, 255] down to [-0.5, 0.5].\n", + " \"\"\"\n", + " print 'Extracting', filename\n", + " with gzip.open(filename) as bytestream:\n", + " # Skip the magic number and dimensions; we know these values.\n", + " bytestream.read(16)\n", + " \n", + " buf = bytestream.read(IMAGE_SIZE * IMAGE_SIZE * num_images)\n", + " data = numpy.frombuffer(buf, dtype=numpy.uint8).astype(numpy.float32)\n", + " data = (data - (PIXEL_DEPTH / 2.0)) / PIXEL_DEPTH\n", + " data = data.reshape(num_images, IMAGE_SIZE, IMAGE_SIZE, 1)\n", + " return data\n", + "\n", + "train_data = extract_data(train_data_filename, 60000)\n", + "test_data = extract_data(test_data_filename, 10000)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "0x4rwXxUR96O" + }, + "source": [ + "A crucial difference here is how we `reshape` the array of pixel values. Instead of one image that's 28x28, we now have a set of 60,000 images, each one being 28x28. We also include a number of channels, which for grayscale images as we have here is 1.\n", + "\n", + "Let's make sure we've got the reshaping parameters right by inspecting the dimensions and the first two images. (Again, mangled input is a very common source of errors.)" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + }, + { + "item_id": 2 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 400, + "status": "ok", + "timestamp": 1446749129657, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "716a6ad5e180d821", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "0AwSo8mlSja_", + "outputId": "11490c39-7c67-4fe5-982c-ca8278294d96" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Training data shape (60000, 28, 28, 1)\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW0AAAC2CAYAAAASj9x6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztfVtsa9t13VikSPEhviVKR/f63rjIh31RBEaB+scBoqJB\nYRQBXOTDCFIUdhIE+WjaAAlQJ/655wb5SPJhwA2Qj7hOYBcNkjZAaidAUidohcIF0joPN07jPK7t\n+zpHD0p8k+JTqx9HY525Fxd1KImkuKk1gIW9RUncpDT34FzzMabSWsPDw8PDIxyI3PcL8PDw8PCY\nHZ60PTw8PEIET9oeHh4eIYInbQ8PD48QwZO2h4eHR4jgSdvDw8MjRLgTaSulPqqU+hul1N8ppT41\nrxfl4XHf8LbtsapQt63TVkpFAPwdgH8K4CmArwH4Ia3138zv5Xl4LB/etj1WGRt3+N0PA/h7rfXb\nAKCU+i0AHwMQMGyllO/e8VgotNZqzk/pbdtjJeCy7buER14C8K74+r2rx1wXhtYar7/+ujlf9PLX\nCte1bnu9BeFGtu3/J/5ai7jWNNzF054Zjx8/BgAcHh7i8PAQBwcHy7isxxqCNrQqePz4MQ4PD/H4\n8WMcHBx42/a4NWa17buQ9hMAr4ivX756bAIkbRq2h8dtYRPjG2+8sYjL3Mi2uTw87oJZbfsu4ZGv\nAfhupdSrSqk4gB8C8OUXvahlwV8rXNe6j+tdgxvZtv+f+Gst81q3rh4BnpVFAfgsnpH/57XWv+j4\nGX2Xa3h4XAelFPT8E5Hetj3uHdNs+06kPeOFvWF7LAyLIu0Zr+1t22NhmGbbviPSw8PDI0TwpO3h\n4eERInjS9vDw8AgRPGl7eHh4hAietD08PDxCBE/aHh4eHiGCJ20PDw+PEMGTtoeHh0eI4Enbw8PD\nI0TwpO3h4eERInjS9vDw8AgRlqKn7eHh4SEhNVvk+eXlpRkC8KJzpRSUUohEIhPn9mOEUsp5HiZ4\n0vbw8LgXuAh5OBzOvDY2NhCLxWZaLmIPKzxpe3h43Au01hiPx7i8vMTl5SXG4zH6/T4uLi5wcXGB\nXq9nzl2PbW5uIplMIplMIpFImHP7MQCIRCKIRqPG6w4zcXvS9vDwuBdIsh6PxxiNRri4uEC73Ua7\n3Uar1TJH13kqlcLW1hYymQwymYw5l0cAiEaj2NjYgNYa0Wg0tGRNeNL28PBYOmRoZDweYzgcYjQa\nodfrod1uo16vo16vo1arBY7yPJPJIJ/Po1AoIJ/PB85HoxEAYGNjA5ubmyZurpTC5eVlIM4dNnjS\n9vDwuBeQtEejEUajEYbDofG06/U6zs7OAqtSqQS+zufz2N7eNmtnZwf9fn+CsFOpFACYeHbYB1d4\n0l4gZjGO2xiQ3FJyXV5eOp/TlXmPRCLY2NgwiRyeS4+HS25d5fUkuN3kNlSuWCyGaDQ6scLs6Xjc\nDLQ9LgAYjUbodDrodrvodDpm0ZOuVquBY61WQ7PZRKfTQa/Xw3A4RK/XQ7fbRavVwsbGhvGih8Mh\n+v0+ut0u2u02Go0Gtra2kE6nA8doNHrPf5nb4U6krZR6C0ADwCWAodb6w/N4UesOGq59nBXD4RCD\nwSCwRqNR4Pl4Lkmd59FoNJCwSaVSiEQiJgnEZE+v10Ov10O/3w+swWAAACYbz/NYLDaRDEomk4jH\n49jc3DQrDITtbXt+kA4D12AwQLvdNoQsibnZbKLRaJhzLhI2bX08HqPX66HT6RgPmiGWTqeDZrOJ\nWq2GbDaLUqmEYrGIUqkEpRQSiQTi8fh9/2luhbt62pcADrTWtXm8mIcAF7HelLRHo1Egy35xcYF+\nvz/h0dCI7bWxsYFsNotsNovxeIxIJIJ4PI5+v49Op2OSPa1Wy3hA9Ii63S663a4hbEncyWQSmUwG\n2Ww2cEylUkin09BaIxKJIBaLzfePuhh4254j6DTwOBgM0Gq1cH5+jtPTU5ycnODk5CRgc3LR7mT4\ng8/T6XQChE3vu16vI51OI51Oo91uYzAYIBKJIJFIIJfL3fNf5Pa4K2kr+K7KmWETtr1lnBUk7U6n\nYzLqvV4vEArhkV45a1sHgwFisZjxWJRS2NzcDNwAjUbDbEvr9fqEx9NsNicaGJRSSKfTxqPh4rXZ\nDBGLxcISU/S2PSdIL5thtn6/j3a7jWq1iqOjI7z77rt49913jTctF3d3clcJwDwP7Zzx8GazaUr+\nuAaDgfGwc7ncRIgvTLgraWsAf6SUGgP4Na315+bwmtYS1xH2TYlbehTNZhP1eh3dbjew/eSNQoOX\n4Y14PI7xeGwIe2try/ws44pnZ2c4PT3F2dmZiS1y1Wq1ic6zSCSCbDaL3d1d7O7uotvtGrKWHnYy\nmQzE31cY3rbnBNoAw3N0HuhpHx0d4e2338a3vvUtdLtdk5TkkiE+LgAmTDIcDhGJREwttqu5BoAh\n7HK5bIg/jLgraX9Ea32klNrBMwP/ptb6q/N4YcuEizCvI1E7tHEdCbvOZXyZ5zchbW795Op0OhOE\nzdihjEX3+31sbm4iFoshHo8jkUgglUohlUqh0WigVqvh/PwclUoFJycnqFQqAcKWpM2EIs+HwyHi\n8TiSySS2trZMwojJTb62kGAtbHuZcLWmy7AF7bDX66HRaASqQhgi6fV6EzkYl80wJGcnxpVSxi5l\n0rtYLGJ3dxftdtvsMum42M/pwirVdt+JtLXWR1fHilLqdwF8GMCEYT9+/NicHxwc4ODg4C6XXQhm\njTXLT3s7TmdXW8iqDUnYDBnILd9NyEzGnRkeubi4cOozyHIqng8GA9TrdUPeTOKQqE9PT015Va1W\nM89Pz9lVJbKxsTHR6MBMfSqVQiKRMJUkd7kBDg8PcXh4eOvfnxXrZNvLhCtExxizXLVaDe+99x5O\nTk7MTpH3j3Q+JOwciotw7Z/h8/CelGFCxrjtHeN9YVbbVrf1fJRSKQARrXVbKZUG8BUAb2itv2L9\nnF5178omuuuIW5KzTO7RCKRB0EuQRjwejydac0m4s4KhkYuLC5OgIfG73oftgScSiUB9KxdDIiTt\n09NTc0PJa3W7XcTj8YlVKBSwv7+P/f19PHr0CPv7+9jb2zNJT7nmlYxUSkFrPVc3aJ1se5kgOdrO\nTLPZRLVaxfn5uTmen5/j+PjYrJOTExwfH2M4HE7sFoHnZGznUfg9eXThAx/4AF577TV88IMfxGuv\nvYbXXnsN5XI5sGOUpaj2c92Hpz3Ntu/iae8C+F2llL56nv9kG3WYYBMejcXe8smYnDzayRNuwWzP\nfDQaBVp0uW6SGGEdKsMdMtkn3wuTf7aBD4dD42EDMO+JYRHZxNBqtQIfSPS0ZXglkUiY2LhrpVIp\n49WHpDZ2rWx7mbDDf6PRCN1u1+RJSNKnp6cm3NZoNAKetivPY3vDUjtkmnft2nHS26ZNs72ddd68\nJ3jvrCJuTdpa6+8A+NAcX8u9wQ5hyGYVm7TpVcvEHj1fuyyOSRTpeQyHQ9TrdTQaDTQaDXN+k8SI\nbHKRW0pXTFE2tGxsbJjYM0MirBrp9XrGCzo7OzNHO8F5eXlpnmtzc9OI8lAHwrWYyZ9HeGQZWCfb\nXjbshCP1RBqNBiqVCp48eYJ3330Xx8fHAadFkrZ8LkJ62nLZpaf8HVcYUyY3ZViSBM17gva5qsTt\nOyKvYIcSJAnKT33+w+lNM7zhErRhiESuwWAwsU2sVqsYDocLeV/0iHmMx+PmWjRkNiPIKhG+rl6v\nF0g4ygy9TGSm0+lALFsuNtXE4/FQkLbH7SDvIRk27Ha7hrSfPn2Kt956C0+ePJnI68gGMQnpUUsb\ntElbtqjLowxlymvyPuDvche4qmRNPDjSdlV5sMbTloCc9rN2hyBJT3ranU7HxOektz0cDtFoNALJ\nvZtWj9wEGxsbSCQSpsmAixUj8pxkzJ/P5XImWSMz8dFoFKlUynjYPC8UCiiXyyiVSsjlckin04as\n6WWHoRvS4zmm2aXr3hiPxxOt6d1uF0dHRzg6OsLZ2ZkJhZCk7ZCICyRVO/FNu5KOCROfMgfDvoZm\ns4lKpYKtrS3EYjFTzy1tmPcBr8HzVSLxB0na9taJ2zeGKthQ4sqE03uw49qy7XuWmLbUUFgkYrEY\n0um0UUHL5/PIZDImFi11h3mkelq73cZoNHJqh8jGBa5sNotCoYBisRggbeqb0MtepRvAY3ZIUrXt\nmqG/Vqtl7iHeTycnJxOkzdCJK7Rng2V8cofH0lKbcC8vL831I5GICc+QtE9PT7GxsYHRaIRarWY6\nd7kymUxAcmFzc3Pl8jAPlrRlnTTrRlkxweoJV92zq3JEVo/IJX9fXpdhFbbkLsrLZgdiKpVCLpcz\nSmj5fN7ogchjKpVCJpMJaI8whm3HxeXv8TyVSgWMnwlI+Xve0w4nXGEHu9SVeiLVatUktCuVSiBP\n0mw2AzvMaeV9Nuhp09YSiUSAcHmutcbp6SkikQjG4zG63S4AoN/vo9FoIBqNmh1BtVo1TkaxWDSO\nFj8ItH7eFLZKxP3gSZufxPV6Haenp3jvvffw7rvv4unTpxNTNeTRfswuBbQ9CEncMiGyKNKmN0vS\nzufz2NnZwf7+PorFYkDhj0e7E43VInKbyKP8XZ7LRh16PhSIspNHHuHDNMLm6vV6aLVaqFarODk5\nwZMnT3B0dBQQgpKk7Srtc0GGR2KxWIC0SbqFQgGlUsmoWLK0tl6vQyllPG2qC56fn6NQKGBnZ8c0\n3DAJynuShL1qZZ0PlrTpNTMRx3jXe++9h29/+9t46623JqRPJRG7yuum1XrLawPBreV1sbzbQpKi\n7Wnv7++jVCo5u8ZciVil1ER8j+f2kltXrlgsNlGa5RFe2LtG+UHvIu23337bELVM3LtKVK+DTdrs\nuqWmdrlcRrlcNiFMlhlSya/f7xsPu1arIRqNIpfLod1um+/R1nm9VZVdeHCkTdgJFMalKcB+fn7u\n9J4X+anrKmECJgv7p31IuJoNZCIyl8uhWCxie3vbObl62mtykbbtPXO5vHKPcMOlqe6SR2i32ya8\nyBBjpVJBp9MJiJYxNHJT2CTP8B/tO5vNAoCpYuKOj4JoLASQFS78AJCjy2S4b5Hhy9viQd5RssSH\n239buJ8dezYhLgouwpOJO7lccquyc4xHkjZL85LJpBGAdz2v6z3KJJBcrmYHVymWR7ihtTakLJPs\nrM6g58xW9ePjYxwdHZn4db/fv1HScRpIuP1+39RjZzIZkxviB4EsSaU3nsvlAuFILkne3CnI55P3\n1SrhwZG2XaRvk5Ekb+nNLou0ZU01uxZtgmSik56O/GBxkbbcUrLkbxbC5vPY3rTrw8RF4J601wPD\n4dCQst3NS/1r9ifIGv9Wq+Uk7ZuCu+HhcGgIm0nGXq9nvHiGOKgPT3vP5XIBwSoAZtfMHQTvJ4Zx\nWJK4iiJnD5K0Gb9lRlgStSRuxu0kKS3qHyhJm15xPB6f8GA5YYaDCBjD4+u0l6z0kNtA/i1m0W2Y\nJqozzVv3hL0+YG8CG2Q4YYZlsezsZZLRlmdgWaurae0mr8EObfA1kYxJ2ryHGTbZ2tpCNps1IT3g\n+fAE6WlL0mbIZxkh0dvgwZE2MDngc5pqHUdzLYOE6BVLjziRSEzMVYxGo7i4uAh4HP1+3xkPl562\nDI/Q05bXftFrm5ZMdMXRZ/kg8AgHGB6hdjsnzZyfn0/McGw0GhNlr7J5bFqCfpbXID1jyjBIT1sS\nrPS0GR6xCVsm3mWXsw+PrCCkN8ivXfHsWCxmPG2buKeR0V0+kekdMAHCRIqd1OPPATAGJ6fQuEh7\nmqft4WHDVfEkPW2q85G4pU5NvV53Pue0XMlN4CoLlKEMmdzk7pKkzZAIZZF7vV7A6WFMm3F7etp3\nicEvEg+OtAEjeRio/0wmk8jlciiVSkbZzm5rv7i4MMkOO3braq6Z9s92VV2w5lSuTCbjTALKMWOt\nVsuUVNl14+PxGPl83og2SWU/Dw8b02QbKIjGYbkMjbTbbVxcXBiCmwZXMtsljRCNRp0dx9NCf7K0\nlN2LrCSRr4nXIGlfXFxMvQ9WjaBdeJCkDQQ/6Una2WwW29vbJrnnUu67vLyciIFHIhHTms5Ynq1Y\nJsEaUPk82WwWxWIROzs7ZuVyuQmtX4ZHXINPXWVYbFuXMXIPDxdcEg/0TLvdrpm/yPZ0xqzZ/TsN\ntHdbvMx1pI4Pl51Xkou/Jwk7mUwGiJ47U+5KLy4uzIxTSdAusl5VAn9wpG3HY7XWgQYUFtpvbGwE\nEipcWuuJBpJIJGJieyzJ63a7TtKmITFswUXSLpfLePToEfb29lAsFieSkExEyl0AS6/syemdTsd7\n2h43gt0xzLCBHJpL0qbdvajumrtZW6tGDt/lebvdRq1WM4UAdEBkmS7XNE9betjM55CwW62WuWft\nGmy7VX9V8eBIG5iMp0lPm+GPZDJpsuEyDAHAGAdXNBo17dok7OvIkUYsn8cm7fe9733Y2dlxltHZ\n8TdWkzCLL49SIMp72h7XwdWefp2nTRuc1dNmvsZWnOTa2toy3YrUBGq1WgCCvRUy/yQJm3kbWTXF\ncAkJu1qtIh6PBzhg1UnaxoMkbWCya5AKYVSwy2QygXImzjkEMCFLSulG1o5SYWzadV161DI8sr+/\nj1deeQW7u7uBEjrZXGMLune73YmRThsbG97T9pgZrhb1aaTdbDZNhcWsnjZJm3NEXWPokskktNbo\n9/totVqmLNcmbZdkAj126WGnUikz+b1WqyGVSoXeeXlwpO3KWst/Mr+Ox+MTxsHvk7Cl/i5j3vTI\nKVRjT88AYAyYrbeZTMaI3hQKBeTzeeRyOWSzWWcCxhbq4YeF3TC0sbExIZO6SmplHvcDl2IfgEBX\nIFe73UalUkG1WkW9Xker1TKldlKXh88hd4Q8Z4MLpYELhQJyuVxgEDQX9UGYI+p0OlBKTShKxuNx\n7OzsGPvmsA0SMsOP/GCh47K5uWmcLFk9IidQseuTiVCZn5pW+rpMPDjSdsFOWLCyRAomsYQIQCA0\nQiLM5/OBqTXdbhebm5uBahImOBlDp3dNtTF6xYlEwhiWq2tReh0EpVXZgMBa1Ww2i3w+b0g7zB6G\nx/xga9dIiWLZNMOp6cfHx6jVama4h0s8TZbPytJZat6USiWzcrkcksnkhJ47c0tMrI/HYxPes2Pg\ne3t7eOmll7C9vW28dO4mpeMyHo9NeFDq5rCxhmTNey6bzZrXwOoY1obb5cL3gReStlLq8wB+AMCJ\n1vp7rh4rAPhtAK8CeAvAx7XWjQW+zoVCJgdJ1GzVtms+AUxsy5RSZmgAE4G9Xg+xWMx44AxrjMdj\nQ7Ayjr29vW3K/Fykbb9eaTzAs5uQimR8zfYYMO9pB/EQbHsaZBhEDgOh2qVrVatVtNvtQIejLA/k\n/WLnfDjRSFZGFQqFgMQvz7n7ZJw8Eong4uJiYspSOp1GqVTC9vY2tre3kcvlDDHzPZGwLy8vzffk\nnFKWNPb7fXQ6HQDParnpgHFQCUlb6sHfp/Mzi6f9GwB+BcAXxWM/C+CPtda/rJT6FICfu3oslCAB\nyk9pGhMJm94ygAljAxAgbHZV8XkY0uj3+7i8vAx42qVSCbu7u8aQpactDcMmblcrud0Nxikc9FA8\naU9g7W17GmyJYupPywG81MOm191oNAKetqvDUcaumWAslUrY2dnB3t4ednd3sbe3h0KhYH5H2jZr\nrEnY8Xgcg8HAhFI4d9QVF6enzUSm3EnQ/uUEJUpA9Ho9AM9FqYrFotkty45LisvdZ2gEmIG0tdZf\nVUq9aj38MQDfd3X+BQCHCLFh2541PW7GsaUsJYCJxgCtNTqdjmnCkWVKkrB5DSqQSdLmlpE11dfN\npePj8lOfKx6PByQ07RZ9Hx55jodg29Ngl/aRvCRpv/XWW3jnnXcCzWVSS8SOiUspBiYcaeM7OzvY\n3d3FSy+9hP39fRQKBadaJT1j6XyMx2OT4+GiZ00ypncvhd742kja0zxt4Hl7O7X16WnLtnYpynaf\nzs9tY9plrfUJAGitj5VS5Tm+pqVDdmwRdmeYNFIZniApZ7NZ42Xz01lOO2+32wHSppgN4330spnd\nto1CNgPc9yf9mmOtbHtaOZs9xMDWFzk+PsY777yD73znO85hIK7n5W5V6lvn83lTFUUvm9OTmPCT\n5avyeWRIkklMmdC0+xfskKEEk5QuT5uNN0xgsttTkjY7nHmN+9QkmVci8tpCx8ePH5vzg4MDHBwc\nzOmyy4PdlOOKM9vVGy5tadkaLAcusNKEBmuXUIWtlnRRODw8xOHh4TIvGXrbtr1OltTJSpFer4ez\nszPUajVTIcJGM6nVPk3hkXZLVb1cLhcYBWbPDJXJf1ta2C6JBZ4NNqCIGsOY9r11U0eG15R/G3aB\nssSxXq/j7OwMAEw8nffpvDGrbd+WtE+UUrta6xOl1B6A0+t+WBp2mOAiZnnu+trVuSWNS8bRaBTp\ndNr8zKqOOFoV2MT4xhtvzPsSa2fb9o6RnYZSdqHdbhvRp1ar5dQUkfYtF208Ho8b0mZpX6lUQqFQ\nQDabNbtIhulcyX+bsJlHImFK0nZVVt32bwPA6OZTn4S13WdnZ6apTYZu5o1ZbXtW0lZXi/gygE8C\n+CUAnwDwpVu8xtDAZRB2veY0g7a9AQABT7vRaJhYHCs+6OF4LAVrbds2WTO8wS5aWd53fn4eqMWe\nJtZEu7bVJzc3N02SUHra7Mp1edo2YXPAdCKRMHFuACZuLYeDSOXNu/6NCKlR0mq1jKfN3BUJO5VK\n3fm6t8UsJX+/CeAAQEkp9Q6A1wH8IoD/opT6UQBvA/j4Il/kfeI6o7A97WnhEWlcLk9bVqlsbW29\nUDXNYz54KLbt6nSkp01vkvKqDI8wlstktgxdkLBZQcUj7dflaUvSZpJdkjerquwyRN4HvAY9cdkp\nKY+3+dvY5/a8WEpVSMLmh8l9YJbqkR+e8q3vn/NrWWnMYhQu4ra3cYxpU+CJJYH0UvL5vPe0l4SH\nYtuuShHpaTMEYHvadB5k0l0Stq3QJytG6GmzKopJdnra0nOd1qFpN+7Y99I8/z6E7WlzF0zHin0P\n93l/+o7IKbiJUdBbkAI1rCah7jUbA3q9nonjDQYDU9SfTqedBf321tRXjXhMgytZTRKScr39fh/n\n5+fGu+aqVqtoNpuGsBnjZUWIXHZLOXeKbBRjezk9bLsjcdVsWf7tbHVDqmaywuW+J9p40p4TZBkf\nP4Wl4A4z9aPRyJT0jcdj9Ho9XF5eBsqMZOmgLRK/SobusdqQoTipUU09kUqlgtPTU1QqlYkkJOuS\nKaZGD5pH1kfLzuBkMmmqRYrFoml42dzcNOGNecWhFwm5M5HDGO46UX5e8KQ9B7CpgKQNIEDKLKvq\ndDomG876bhL61taW8bRlfajsuryuDtXDg7C9Rpk/4ZIeNhcJ2yZtOdVpe3sbpVLJhDpsPWtbACrM\npC3laamo6Ul7jUDSlufUc+AWq91um+4qe8lJ1tLTjsfjgaJ+D49ZQRU7ypwyds34tb1YMcIFPHM+\nSNo7Ozt49OgRHj16FFDVk0c7jMISPbuxZZVhe9r0tr2nvUZg3A+AKX1Kp9NGMpVbUsYLOdWGXlCn\n08HW1lagC0t2iMmSKNkQ4OFhw07iSU+7VquhUqng+PgY1Wp1YlGFUv6+PYpvf38fr776KrLZ7ARh\nM+xnL3s036rbr5209eGRNYUsCaLBj0Yjk4hkA8PFxQUAmAnXsvvKtWwjkeVXhKvsadZSRY/1g2ym\nsctLSdq1Wg31et0c6/U6RqPRxGQYDgShfPDe3h5efvll5HK5ifCI7G6Uy27CWXX7kx94DJF40l5D\nuAxRxgMZJwRgdK25de10OgGVtdPTUySTSYxGo8CwBTl0wZ4daXs2tpaKx/rDbqKR08dZk02NbOZP\nWA0hk46sdGJ52/ve9z7s7+9je3sb+Xw+0Jlox6pZ1npduZ7H3eBJewGgYUoJVpnYkYRNISkpQn96\neopoNIperxe4gXjO+KDd4OAaOOxvkocDux6bNkaRf0naDNPJEjaG9ThBqVgsolgsGknVnZ0d5HK5\nQJOMHfpg+E6GWTxpzxeetOcI2yClpw3ADDeVhC0nT9PTZuVJu902+sFbW1tm+Kmsk5WZe8pYAs+V\n0jweDuyuRzbR0NOmbEK9Xg8MhZYORTqdNkRNsi6VSmbyDEk7kUg4u39dHYYAFtYY8xDhSXvOoJfB\n5CQ1CnieTCYDhE3tbEnanOher9cnSqhYRiVHL3FMEwmbrb4eDws2abPqwRUesXWsZXikUChgb28P\nr776Kl5++eWJGu1UKhUQbZpGyDI8Io8ed4Mn7TlAbgf5NfDc0+bNkM1mTZVItVoNjBZjEw4rThqN\nBjY3N01LsBR/l2OXuKiFIIcU+0qThwUXaTM8QtJm2zp/Xi6O+yJpv/LKK3j/+98fmM/Io1265+1s\nefCkPSe4jJY6DSzZY/s6s/GlUgnlchnNZhOdTicwbIETQoDnnW28Ce3kJIV6KKdJz0kmK13iVf6m\nWy/IIbXsDTg/Pw8kHmlXdkVHJBJBKpUyKn10EHK53ESuJKwTkOyKj+ucmlXWr/ekvWBIOUsAxpvJ\n5/Mol8umtb3ZbAba3Xu9nolJ2pOj7RFLFJuS+gicLkIxH7nk5A6Z8ffEHW6wjFROVD8+Psb5+Tka\njUaggknOQaVdcCeYTqcDc0Vtuwk7XHH3697XqhG4J+0FQxI2291TqZTpMBuNRohEIqhWq0bXGEBA\n3IeeNssDXQprtqcNPPP02UYswyWu+OM63IwPHZK0KQZ1fHyMs7MzNBoN0/EIBMNodAAYt2b4TQ6D\nlhUi64CbeN2rBk/aCwSNgCEJxg1TqRTy+bwhbJZabW5uAgD6/T6azaYJc3BogqsWm0dOsaaHzdAM\nJTFtsXn5QeKxHpCkfXZ2hqOjI5ycnODs7Mx048rcB1UpmReZ5mm7hnqEFbaXbb+XadUvqwRP2guG\njB/LDH0+n0ckEjEdZ9QtoU5EJBIxcWxCbuVsjWO2zdMQqf/LDi4S9ubmppGVlHW1HuHHNNKu1WoT\nnrY9hFeGjIl6AAAfkElEQVRWiNDTTiaTiMfjaycNPI24w3IveNJeIKaVOjFUEY1GA8qAzPI3m03T\nRCNrbmV5luw401ojlUoF4pPc8srBrLxRZWKUXZPUNZn2HjxWB/awAJ7LwQbVatXIrlJKgXmSafrv\n0tOWvQDriBd53PJDyh4beN/3hCftJUNO/4jH44ZQs9ksSqWSKfuLRCLIZrMmKSkXyZuETmKml16t\nVo0OBDvf+DyDwQCZTMYp9iMN8r4N0+N62GO5WOfP/zeFyrrdrqkYkeL9JO1kMmlIW048p/jTOuJF\n+RxW1fDesJOy9x3bn2VG5OcB/ACAE63191w99jqAH8fzSdWf1lr/4cJe5ZpBkja/zmQy2N7exuXl\npfHA8/m8EY6i6FSr1QpMIWHYhaTdbrdNYw3FqHjj8uaV7fGpVMqEU2xPYt2JO6y2LcWM5E5MkjZ1\n2fm/Z4LaRdqc65jNZg1ps3lm3WGTt3SquGNlfbok7fu8N2bxtH8DwK8A+KL1+Ge01p+Z/0tab3Br\nyu0ZDYQ3FOPc9LypxEY1to2NDSPfSsIeDocBT5ujzKjNzTJA3riDwSAwPHhjYyOQcJKvdc0RStuW\nIv1S/9ombJK2/H/bOQ+SNhu4JGmvo6c9zcOW5/S0Sdr0tFmpdd916rMM9v2qUupVx7fW/o5eFOym\nG6nFLQm72WyiUqkYvRE250idBzZK0IOWsyfj8biZgCNF3LmA4MAG+foeAGGH1ralp80dF9vV5UxD\nLumRXxcekaRNwbF1xYvCI/xQI2nbnvZKk/Y1+Eml1L8C8KcAfkZr3ZjTa1p7yG40AKYUkIRNz6jT\n6RitEY4cY/afA1t7vZ5pvuHNK7d5Mp4ptYDpbTERJadLS3nNh0DeDqy0bdueNlvVp8W0+T9n7BsI\nlvzR05Yx7XUMj9hhv2m2LcMjsnU/TOERF34VwM9rrbVS6hcAfAbAj0374cePH5vzg4MDHBwc3PKy\n4ceLGlrk1iwSiRj5TIZAAJh2Y7k6nc7ECDN6X+1220hp0kO3dZfH47GpOJHJyVW7cQ8PD3F4eLjI\nS4TCtvmhKv9/dpURP6j589zZKaXM/1i2rsvJ6WEIj7iqnbgLsRelIqSqIR0b2cq/sbGBTCaDXC6H\nfD5v5CY4qJiytIv428xq27ciba11RXz5OQC/d93PS8P2mA5pRAAQj8dN0wNLtdhu3Gw2zVxJHim/\nKbfFspOS3ro9SoliVVICljXlqwabGN944425Pn+YbNsmbtfUGEKGvOzGGtZpb21tmTDAfcdtbwMS\ntgwZ8chJ86xVZ++C1KLnkXriHGS8s7ODUqmEfD4fCFXOG7Pa9qxXVhBxPqXUntb6+OrLHwTwV7d6\nlR4GJGzZQHN5eWlCF/SOUqkUWq1WoJpESm5Wq1VEo1Ej78oZgUx+0gOnNy5F8ovFIvr9Pi4vL41C\n4QNAKG3bRdDTlmzCojfJWO000g5TeER+ONnTehgqqlaraDabRuqBSX/+LShxnEwmzQAIEna5XEax\nWDR/o0WR9qyYpeTvNwEcACgppd4B8DqAf6KU+hCASwBvAfiJBb7GBwO5feUNw5K8eDyOdDqNXC5n\nvGk5S7JeryOZTJpOym63C6WUEZkiYTNZRS9ExkJpzCRsmZxcR6yDbdvetr1klRIlDOhRkqQYHpET\n1sNE2gT/Dix15S6UcrT0tGnn3LkyTMQhI/l83gx9IGnn83lD7IsKj8yKWapHftjx8G8s4LU8aMjk\niNQs4bBgamZPK+2qVqsBwq7X61BKGU+bhB2JRNButwOEzQ8ASdi5XC6QnFxHhNm2Z/WygWDYza4/\nJlmxuUYO9g0DacvuUBkeYWcoy2VJ2tLTlsn4VCplWvnpaTM8Ui6Xkc1mA+PVVtrT9lgOXO2xMikp\nPSpbM7nb7SKRSBjdiXQ6bRp3GL+WGiYMgfBxxrQpXMVBxExkzZp191g+bkPc0tuWJW0cbiDFoVYB\n04SbXDsM5m9arRbq9TrOzs5wdnZmwiNSNMtVq854Nlc+nzex7FVpZfekHQJMK/yX5Xv9fn8iHimN\ny9ZakPMDudVjnJzNO71ez4RcbA0Gj/DBJTTmUvALy/9XjlRjjqbdbqNSqaBSqeD09NTor5C0KRPB\nHQXj+TLxKGdhMhSySn8TT9ohgG0w0vvm1zZpu8ZB8efpZTNhA8CURbGLkqTd7/cDcrCrZLwes8FW\n6ZtG2GEgbul8MF8jB4c0Gg1D2CcnJzg5OcHp6alxSCRpy+qsfD6P7e1t7O7umvI+yiWviodNeNJe\ncZBspbGQtGVHJb1ilmuRtCVZE9LTBmDCJCwblJ72YDAw8Ts+36rX73o8h50rsXdM9tgxm9xXCdN2\ni7RZDss+OzszpH18fIyTkxMzUIShQcoUM5Ytx//l83kzi5XCWaukyeNJOwSQxC0Nh4Q9Ho+dnrb9\n+wQNHnheItXr9Yw3Qk+bNa7y2p6wwwfbbq7ztleFmK6DTDpKOVrqiEtP+/j4GMfHxxN17CRthkfo\naZfLZVNJY4dHVuVv4kk7JLA9bVv/ejAYOMMjkrBlezqTjIPBwNysDI9IdTh64/TGpiWFPFYfLwqL\nrEri8TrI5Kr0tKkh7iLtp0+fBrTmZYs6wyOFQsGER1gKyfvJFdP2iUgP52ADehOyVVnW38rzer0e\nyJDTQ7Y9bMK+WRnjY1WBHAgsR5StirfhMRtY1iarRNi6zsk0Upb3vuG6D6izYq9qtWpmYZKw5RBj\nlrByfJpcqVQKxWIRL730Evb29lAqlQJt6i7nZ1XgSXtF4GpJZqzZXq4hCI1GAycnJ6hWq2i324a0\n+dwS9LjsulOOmJLDXknenrTDCXusGBtppKLfKiWYeQ/Ie4EetT0MRJL2+fk5zs/PUa/X0el0jHAa\n56NSM1yOVSsUCqZ5Znt7G9ls1uiurIKa3zR40l4h2B61zI7LJUuceE7SrtVqpnnG9lgIGQ+XnrWs\n1yVx0xOjt70qN7fHbLBJm5USFIdatZZ1e4fJEJ5MNnK5SLvVapnehEgkYsb2sWGGx2KxaBKOPJK0\npSOzio6KJ+0VgTRWetCUXpWC9p1Ox8SaeeT0dulpM0vuikFLT5setYzjSdK2p3GvmgF7XA+2akvS\nLhQKK+tpAwh42LLDsdFooNFooF6vo9FoTBD2+fk5ut2ucUZisRiSySRisZjxqMvlcqA1XYZL7AYj\n72l7XAuXzCZbz9nhRTU/VnewI/Li4gKtVgvn5+fG05bhERdYdy01KGzCphj+KpeBeVwPfjjbpC09\n7VUibdt54W6SWiIs6ZOELYmbM1AzmYyJ42cyGezs7ODRo0fY3983S7amy0HXrrr2VYIn7QXCVR8N\nIFB+xHMZ7mDIg80CLGei+A3rqOXQ3na7bX5GJiJdkJ621KGQIRGGRVZRntXjZpAlfjLUdZ95Cle7\nPQd72LKq7XYb5+fnqFQqpi29UqkYTZFGo2GcGd5PsVgMqVTKhEN2d3exu7uLvb097O3t4dGjR8hk\nMkt/3/OAvyMXjGmGaS9XooUetJRhbbfbpoba/ll+T85/dMElICTn3/mkY/hgVwoRcsJRp9NBo9Ew\nSedsNoter4fRaLT0Uk6WndrLnrrDXaYUfuKiNrZSysgIR6NRbG9vmyU1sSmvmkgkVjLsMSs8aS8Q\nrooQJhfZccijbaj2UZ7TA5GeOXVEJGlPi2fbpM0widwqruK20CMI1//XfkySdrvdNqGQTCZj8iOc\n5LJssCpELu4mubuUO0wZImw2myZZGYlETPliIpEIxK13dnaws7NjxJ9I2mFuEvOkvWDYI71I2jIL\nLr1oetZSK4FkzCXL/uSSW0p2PErIbrdZPG2PcMAVhpNVQ8yNkLC11igUCsYBuC9PW3Y0cnGQB2PU\n1WoV9XrdOftSNsywVC+TyaBcLptwCFc6nTaLyfWwwpP2AiE9bZlYYXJRZsJ5lOetVmvCE+n3+4Em\nG7lc09YJu43ZJdPpG2nWCzIc1+v1oJQytsjcCD3tZQ+84K5TJhlbrVZAN4THWq1mbJ+OSb/fN3Xn\nUl61WCyiXC6buDVj2CxtlHYeVnjSXjBcVSH0tFm2RI+CXgaPzWZzoh6bW9nrNIZtuPSwbU9btvj6\n8r71AUmbhE1bkqR9nzFthvXk/XBycoInT57gyZMnePr0KarV6kQzGR0XDp9OJBIB/RAmHVkpIj3r\nsNu1J+0bwtUWLr1pSdBMMHIx1GGTc61WQ71eN3E8ZsI5OVp2QF6XYLQnS7OxwtZd4GQaNhTwSFlK\nlkt54g4/pG1yOHQkEjFEKTU7RqNRoLKEC5j84LeT6/aOUvYb2OPPKFImq6K4y2R1CLWv5cACSi3w\n+hwLxikzXBxawDj3ujWFzTIj8mUAXwSwi2dz8z6ntf73SqkCgN8G8CqezdL7uNa6scDXujJwaYS4\nKkLkcFEe6VHYoRDGtpl0lIQtPwymQXY5yo4uW3MhmUwGxkvJI42enWFh3kLOgnW3bUmwJFSbtBuN\nBqrVKk5PTzEcDs00Gx4pAWwrBcpaatnBa4cwuEO0pRf6/X4gh8Nz3hcMDQIwHZt20wvtVVaLFItF\n5HK5lRjAuyjM8o5GAH5aa/11pdQWgD9TSn0FwI8A+GOt9S8rpT4F4OcA/OwCX+tKwfY05KBcHunJ\n2EsmG6Ucql3yNxwOJ0SipkGStrzxbGJm4wHlJ+VQ12w2a1YymVx70saa2LZLgU4+JkmbIGnX63Wc\nn59ja2sLo9HIdAZSswPAhM52JBIx5MtFR8VOFnLHyConWe1kzzmVfQcsa9Vam+omKbsQi8UmCJsl\nfnKAwYMkba31MYDjq/O2UuqbAF4G8DEA33f1Y18AcIgVNux5wiZshkJYusdVr9cnQiEUtLFL+mTM\nWgpDye3nrKTNRplkMhnYQnJR4H1rayuQVefv8PfDnGGfBWG3bdeW336M9sNzLplXqVarSCaTGI/H\n5gNdaz0RbiNhSzEzWXoqS/ZkeR5J2NWHYFdH2cl0krZsAKONkqRlLXahUAjY84MkbQml1HcB+BCA\nPwGwq7U+AZ4Zv1KqPPdXt4KwCZuL8WuZCee2kzPrOKvOZcCSoG3Peppanw2SNidypNNp5PN5U7O6\nu7uLcrmMQqGAVCo1sabFMx8Cwm7b19XVu5TzZHiE+hy0Y9Y+06uVxE3bl/MZGQqhPojdBGM7KXLI\nhl0VQudDtpXbNk1nw+Vp53K5QHfvOtrwzKR9tX38HQA/deWV2AwylVEeP35szg8ODnBwcHCzV7lg\nvKgSQx6lLog0XOld8JyttzZpy3pqOR3GBZcOgku8Ph6Pm3AHDXtraytQs8rzQqEQ8Fh4DEOy5vDw\nEIeHh3N9znW2bSBov/wfS5IlMVOcScagh8OhM57sUqCUddaStBn+k8TN8J8dYrElFORIMBney2az\nKBaLJo6dz+fN98Na2jerbatZSn2UUhsAfh/AH2itP3v12DcBHGitT5RSewD+h9b6g47f1ffRbXUT\nuMh5Wnacw3ClxyC9a5lcYXiE1SHUSXAlLaf9jWwxG9kMI3VCmGCUnsjW1paZfSfDI9ls1ij4yaRT\nGEjbxlVS7NYvPIy2PRwOnd7rm2++ada3vvUtvPnmm6jX63yt5vcjkQjy+TwKhYLpFCwUCoFqIp5n\nMplA7f51pH1xceHM35Ck5Q7THtLBcxkG4Uqn087XZkurMuwnK6ZI/utk27N62r8O4K9p1Ff4MoBP\nAvglAJ8A8KW7vsj7hE3Udn01j3IenSxVsrV+qZkgiZyxa5lJvy7ByK2iPVGG4Yx0Om2O0xa9Ei6W\nQq260PsSsfa2DQS9bNklyRFyTCJyt1ir1cyHv03a0WjU5HDs5fowkSEQLlleKJ+XOz+ZEN3a2kI+\nnw+QNMv65O5Slvitc4PYLCV/HwHwLwF8Qyn1F3i2Vfw0nhn0f1ZK/SiAtwF8fJEvdBlwJRhlQT8F\nbWQ9KWUhbZ0QKvFJ74JxPPkhcNPkYiKRCHgcXJlMJkDiPMobQM69k577QyXth2TbQDAMSNKWBN5u\nt429cLGKyCbYaQM6XB28LtkFJjoZ2uNuTzoktONMJmMGGMjlkhKWjsiDJW2t9f8CMC0w9P3zfTn3\nh2ketoxfj0YjdDod1Ot1VCoVHB0d4ejoCCcnJ4FQCZcdH5QGK8Mu10GSNo2ZIQ+ZOc/n887kIuOD\nsrlGTuQIy0DXReCh2LYNEjXwnLxbrZYhTltPXYbnrvO0qWNiOzouyQUOZyBp8wNCes9c7CGQjTTF\nYnFC5GyaHva6Yf3qYQRIiGwGuO7nriNsWY4nSfvp06d455138OTJkwlRJw4WvQvkdBnG9uRsO6kP\nXCwWA94Rjy5CXkdDfqi47n953ffoQEi4ZA2kHo0k7tFoNFEFNRgMZn7dGxsb0Fqb8B89antxV2lL\nrZZKpVAlGeeJtSVtSdKSPO06aJ7LZgGe2/Knw+EQ1WoVx8fHOD4+xvn5OZrNpgl7yGaY62AndeRA\nAtnNuLm56fQ86G3Qw2acmh1sqzrbzmN+kNox8Xgcl5eXJh4scxqU6bU9XRfs0CDtR4YKabOs06Yn\nLe8328u1E+nRaBSbm5sTIT52MrrCI3bDzEO27bUlbcLOUFNVTIYzZLut3TBgr2azGZhJ12q1jJfB\nBMssyUVbpMluHJAt57IahJl0O57NUAi3jA815PEQYE8eor3JeDQ/4GWITpKsCy7CZpmrXWIqSVvm\nZvja5JFhF7k4VUZWsbB/QOZvZDJynRtmboK1fvd2NQjwrD6VsqhyfJedPHHVUrO21a7JZicX1yye\nthzztbm5aUIfbCnn0fY67OoQmTlf9SnSHvOBVGmMxWLGtuWQWtqGtGkAL7RNEjftWBK2JGIZQpS7\nS3rYchdpVzzxtTE2LY+Mo8vFWY90aLyn/QAgiZsz56i5wBCHbKedlgkfDAbOjHmv15soD5zF02Zy\nMZVKIZfLTWTHs9msM7loey2stV71KdIe8wNDDSzlozypTYy9Xs/YAjsep8FueWfC0LXs/I/tacsy\nPnrWsvSUMqo7OzsmXr2zsxNIfMqQitQe8Z72GsNVd01Pm8nE4+NjM29uWn2pXb5k125LbZBZmi2Y\nfGFykU0wHI3EVSwWA6V6TC7Sk3YNaH1RxvwheyjrAulpA8/j2zI8IkkbeE7YL/owZziE15HXtB+z\nQ4/ytcmYuyRtlu6VSiWUy+WJZasKyud76NVORKhIe1rnoqukSEpGyvPT01PnZAy7XI8le3a45CYV\nIdJ4pb611LDmKpVKE6TNdnN70WjXvbTJYzq4Y5PEJrsHi8Uiut0ulFLGbuRIOfv+cCUor3NAZALd\nnoIk28jj8bghay5OSKeXXSqVTHz7oXvRsyB0fyFb+IYhD5f8o10hMhqNjAaIbI5heMSuOZWVJbOQ\nte2NMG4tEzGcsGEnE+3QSCaTMR1eMrkor+GJ+mHC5YkCQCKRMI0oHHiQTqedreWUD5ZSwjcBwx4u\n+5ZNL3xN9tAN2jztnB62x4sRKtKmZ217CXZog2OUXElFKWRDPRC2l7sIf5Z2cwl5M8VisYkSrEwm\nY9pwmTmn2A0TkExCynZz1svanrU39IcLSdj0qLPZrCFs6qnLqeY8SnkFAKY0cFaQtG1532lJc1fp\nKu2ddu4xG0JF2kCwZpSkyooOaZRykrlMMtoDCFiyZ3ct2lM5XuRpu7wfTo6R3gWJWm4TqQEsM+Q8\n2lKpnrA9gEkBKOC5pw0A8XjcTCjihKR6vT5ROsed6k3jxNQJofyvLTIlFfmkjALP7RJX72nPjtCR\ntixJkmO9OOuOQ3Ip4mR74CRvOSGDyUV7uZT+roMkbOlpc8sqRdtlqVOxWDSGa9dvu+pevXF72B/c\nWmsjr0vCzufzplJK7uCohAc887A7nc6tSJueNitBOJ/RXjLMJ8eYyRLVh17GdxOEirRl8b9sMac6\nWbVaRaVSMclFbgG5DWy32xMdjoPBYCYvWh5dsAfqRiIRU3/NxBD1rKVmCM+pw+CTix4vwjR7JBmm\nUinjeHS7XSMmxrBFKpUCENQdicVizjZ0qQ4owdrrbDYbkFWw9UGKxeKEQNlDG7Axb4SOtDkUVIY8\npG51tVo1Iuyy85Gi67IrTJK1TbhKqQmRnGn1z/SqZdY8Foshl8sFyJlHdjEylucbYTzmAVeIjh5x\nOp02MetoNIrLy0sT4uBOsNfrzbSbBIBMJjMx7ouj7KhrzRCMrbrnbf1uCCVp07umbjXjdXImY71e\nn8iQs81czp+jJ2GL4rCgX27p4vG4k2BZbmXH6Zh0lPFs6isw0Sif0xu0x10hCVuL+Yoc1MsORZuw\ny+XyhKcta69tsCFMLibT5YxGKTjlnZP5IHSkPR6P0e/3jf5vs9kMELasDLEV+hgKsZOLNHI7puzS\nomYdqdyiRiIRZ2u561wmYmxP2ycYPe4C2o2c5UjSlop60qkoFovG+bmuW9IGvXdbw106LzIs4r3s\n+SF0pM3wCD1tljNJL5uJSFkBwji4TCpy2YpprDl1lScxgWOTtmyW4fm0lnP7w8Guv/bwuAukLdHJ\noIedTCYxHA6RyWQCzWMczjHr89s7US456stWnPS7yfkglKTNOY0kbdvTrlaraDabU2c+yucDnneX\nSdKmILu9Bdzc3DS/Q+OLRqMTJXzsZnQJyNtVJg+9LddjfrA//KXWjUs33q6Wusl1XO3ldpWTrCP3\nmA9CRdpAkGCZ+GMDSzabNTKpJNdZYI/zYuhCes08xuPxiVBGNBoNyEzy6IpXe4L2WBRcxCgdC4/1\nwCwzIl8G8EUAuwAuAfya1vpXlFKvA/hxAKdXP/pprfUfLuyVIrjFy2QyuLy8NKGNRCIREF66uLiY\n+XmZYZcJR8bsbC1r2QQgbwjWwcrhon5buNpYJdv28JgVs3jaIwA/rbX+ulJqC8CfKaX+6Op7n9Fa\nf2ZxLy8I2bBCwqZanhzDRW2FWTFtzJJrKAE9FlsDxJ7L6BMvocDK2LaHx6yYZbDvMYDjq/O2Uuqb\nAF66+vZSGYkkLQmbUy0KhUKgZd2ef3cdXCV/jAPKBAvJWP4ej9JLp6ftK0JWG6tk2x4es0LdMPnw\nXQAOAfxDAD8D4JMAGgD+FMDPaK0bjt/RN7nGdbB1R2wNkptMj7FeozOBIiUneT4tbmj/nE3a8mc9\n5gelFLTWd/6j3rdte3jYmGbbMycir7aPvwPgp668kl8F8PNaa62U+gUAnwHwY67fffz4sTk/ODjA\nwcHBzV79FaQmtcfDxOHhIQ4PD+f6nKtg2x4es9r2TJ62UmoDwO8D+AOt9Wcd338VwO9prb/H8T3v\njXgsDHf1tL1te6wqptn2rPVnvw7gr6VRK6X2xPd/EMBf3e0lenjcC7xte4QKL/S0lVIfAfA/AXwD\ngL5anwbwwwA+hGelUm8B+Amt9Ynj97034rEw3MXT9rbtscqYZts3SkTe8sLesD0WhnklIm95bW/b\nHgvDXcMjHh4eHh4rAE/aHh4eHiGCJ20PDw+PEMGTtoeHh0eI4Enbw8PDI0RYKmnPu5PNX2t9rnUf\n15sX/P/EX2uZ1/Kk7a+1Ete6j+vNC/5/4q+1zGv58IiHh4dHiOBJ28PDwyNEWEpH5EIv4PHgcZ8d\nkfdxXY+Hg3tpY/fw8PDwmB98eMTDw8MjRPCk7eHh4REieNL28PDwCBGWQtpKqY8qpf5GKfV3SqlP\nLfhabyml/q9S6i+UUv9nAc//eaXUiVLqL8VjBaXUV5RSf6uU+m9KqdwCr/W6Uuo9pdSfX62Pzula\nLyul/rtS6v8ppb6hlPq3V4/P/b05rvVvrh5fyHtbJNbFtpdp19dcb+7//2Xa9ZTrzd+2tdYLXXj2\nwfAmgFcBxAB8HcAHFni9bwMoLPD5vxfPBPL/Ujz2SwD+3dX5pwD84gKv9TqAn17A+9oD8KGr8y0A\nfwvgA4t4b9dcayHvbYG2sDa2vUy7vuZ6c///L9OuX3C9ub23ZXjaHwbw91rrt7XWQwC/BeBjC7ye\nwgJ3EFrrrwKoWQ9/DMAXrs6/AOBfLPBawLP3OFdorY+11l+/Om8D+CaAl7GA9zblWi9dfTtM4+rX\nxraXadfXXA+Y8/9/mXZ9zfXmatvLIO2XALwrvn4Pz9/EIqAB/JFS6mtKqR9f4HUkyvpqHJXW+hhA\necHX+0ml1NeVUv9hnltWQin1XXjmBf0JgN1Fvjdxrf999dBC39ucse62vWy7Bhb4/1+mXVvXm6tt\nr2Mi8iNa638E4J8D+NdKqe+9h9ewyOL3XwXwD7TWHwJwDOAz83xypdQWgN8B8FNXnoL9Xub23hzX\nWuh7WwPct20vuqljYf//Zdr1lOvN7b0tg7SfAHhFfP3y1WMLgdb66OpYAfC7eLaFXTROlFK7gJnk\nfbqoC2mtK/oqYAbgcwD+8byeWym1gWeG9h+11l+6engh7811rUW+twVh3W17aXYNLO7/v0y7nna9\neb63ZZD21wB8t1LqVaVUHMAPAfjyIi6klEpdfcJBKZUG8M8A/NUiLoVgfOrLAD55df4JAF+yf2Fe\n17oyMOIHMd/39+sA/lpr/Vnx2KLe28S1FvzeFoF1s+1l2vXE9Rb4/1+mXTuvN9f3Ns9M7TUZ1Y/i\nWRb17wH87AKv8348y+D/BYBvLOJaAH4TwFMAfQDvAPgRAAUAf3z1Hr8CIL/Aa30RwF9evc//imex\nuXlc6yMAxuLv9+dX/7fivN/bNddayHtb5FoX216mXV9zvbn//5dp1y+43tzem9ce8fDw8AgR1jER\n6eHh4bG28KTt4eHhESJ40vbw8PAIETxpe3h4eIQInrQ9PDw8QgRP2h4eHh4hgidtDw8PjxDh/wPY\nbj/C7XFdaAAAAABJRU5ErkJggg==\n", + "text/plain": [ + "<matplotlib.figure.Figure at 0x7f22684f2890>" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "print 'Training data shape', train_data.shape\n", + "_, (ax1, ax2) = plt.subplots(1, 2)\n", + "ax1.imshow(train_data[0].reshape(28, 28), cmap=plt.cm.Greys);\n", + "ax2.imshow(train_data[1].reshape(28, 28), cmap=plt.cm.Greys);" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "cwBhQ3ouTQcW" + }, + "source": [ + "Looks good. Now we know how to index our full set of training and test images." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "PBCB9aYxRvBi" + }, + "source": [ + "### Label data\n", + "\n", + "Let's move on to loading the full set of labels. As is typical in classification problems, we'll convert our input labels into a [1-hot](https://en.wikipedia.org/wiki/One-hot) encoding over a length 10 vector corresponding to 10 digits. The vector [0, 1, 0, 0, 0, 0, 0, 0, 0, 0], for example, would correspond to the digit 1." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 191, + "status": "ok", + "timestamp": 1446749131421, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "716a6ad5e180d821", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "9pK1j2WlRwY9", + "outputId": "1ca31655-e14f-405a-b266-6a6c78827af5" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Extracting /tmp/mnist-data/train-labels-idx1-ubyte.gz\n", + "Extracting /tmp/mnist-data/t10k-labels-idx1-ubyte.gz\n" + ] + } + ], + "source": [ + "NUM_LABELS = 10\n", + "\n", + "def extract_labels(filename, num_images):\n", + " \"\"\"Extract the labels into a 1-hot matrix [image index, label index].\"\"\"\n", + " print 'Extracting', filename\n", + " with gzip.open(filename) as bytestream:\n", + " # Skip the magic number and count; we know these values.\n", + " bytestream.read(8)\n", + " \n", + " buf = bytestream.read(1 * num_images)\n", + " labels = numpy.frombuffer(buf, dtype=numpy.uint8)\n", + " # Convert to dense 1-hot representation.\n", + " return (numpy.arange(NUM_LABELS) == labels[:, None]).astype(numpy.float32)\n", + "\n", + "train_labels = extract_labels(train_labels_filename, 60000)\n", + "test_labels = extract_labels(test_labels_filename, 10000)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "hb3Vaq72UUxW" + }, + "source": [ + "As with our image data, we'll double-check that our 1-hot encoding of the first few values matches our expectations." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 127, + "status": "ok", + "timestamp": 1446749132853, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "716a6ad5e180d821", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "uEBID71nUVj1", + "outputId": "3f318310-18dd-49ed-9943-47b4aae7ee69" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Training labels shape (60000, 10)\n", + "First label vector [ 0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]\n", + "Second label vector [ 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n" + ] + } + ], + "source": [ + "print 'Training labels shape', train_labels.shape\n", + "print 'First label vector', train_labels[0]\n", + "print 'Second label vector', train_labels[1]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "5EwtEhxRUneF" + }, + "source": [ + "The 1-hot encoding looks reasonable.\n", + "\n", + "### Segmenting data into training, test, and validation\n", + "\n", + "The final step in preparing our data is to split it into three sets: training, test, and validation. This isn't the format of the original data set, so we'll take a small slice of the training data and treat that as our validation set." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 176, + "status": "ok", + "timestamp": 1446749134110, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "716a6ad5e180d821", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "e7aBYBtIVxHE", + "outputId": "bdeae1a8-daff-4743-e594-f1d2229c0f4e" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Validation shape (5000, 28, 28, 1)\n", + "Train size 55000\n" + ] + } + ], + "source": [ + "VALIDATION_SIZE = 5000\n", + "\n", + "validation_data = train_data[:VALIDATION_SIZE, :, :, :]\n", + "validation_labels = train_labels[:VALIDATION_SIZE]\n", + "train_data = train_data[VALIDATION_SIZE:, :, :, :]\n", + "train_labels = train_labels[VALIDATION_SIZE:]\n", + "\n", + "train_size = train_labels.shape[0]\n", + "\n", + "print 'Validation shape', validation_data.shape\n", + "print 'Train size', train_size" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "1JFhEH8EVj4O" + }, + "source": [ + "# Defining the model\n", + "\n", + "Now that we've prepared our data, we're ready to define our model.\n", + "\n", + "The comments describe the architecture, which fairly typical of models that process image data. The raw input passes through several [convolution](https://en.wikipedia.org/wiki/Convolutional_neural_network#Convolutional_layer) and [max pooling](https://en.wikipedia.org/wiki/Convolutional_neural_network#Pooling_layer) layers with [rectified linear](https://en.wikipedia.org/wiki/Convolutional_neural_network#ReLU_layer) activations before several fully connected layers and a [softmax](https://en.wikipedia.org/wiki/Convolutional_neural_network#Loss_layer) loss for predicting the output class. During training, we use [dropout](https://en.wikipedia.org/wiki/Convolutional_neural_network#Dropout_method).\n", + "\n", + "We'll separate our model definition into three steps:\n", + "\n", + "1. Defining the variables that will hold the trainable weights.\n", + "1. Defining the basic model graph structure described above. And,\n", + "1. Stamping out several copies of the model graph for training, testing, and validation.\n", + "\n", + "We'll start with the variables." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 2081, + "status": "ok", + "timestamp": 1446749138298, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "716a6ad5e180d821", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "Q1VfiAzjzuK8", + "outputId": "f53a39c9-3a52-47ca-d7a3-9f9d84eccf63" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Done\n" + ] + } + ], + "source": [ + "import tensorflow as tf\n", + "\n", + "# We'll bundle groups of examples during training for efficiency.\n", + "# This defines the size of the batch.\n", + "BATCH_SIZE = 60\n", + "# We have only one channel in our grayscale images.\n", + "NUM_CHANNELS = 1\n", + "# The random seed that defines initialization.\n", + "SEED = 42\n", + "\n", + "# This is where training samples and labels are fed to the graph.\n", + "# These placeholder nodes will be fed a batch of training data at each\n", + "# training step, which we'll write once we define the graph structure.\n", + "train_data_node = tf.placeholder(\n", + " tf.float32,\n", + " shape=(BATCH_SIZE, IMAGE_SIZE, IMAGE_SIZE, NUM_CHANNELS))\n", + "train_labels_node = tf.placeholder(tf.float32,\n", + " shape=(BATCH_SIZE, NUM_LABELS))\n", + "\n", + "# For the validation and test data, we'll just hold the entire dataset in\n", + "# one constant node.\n", + "validation_data_node = tf.constant(validation_data)\n", + "test_data_node = tf.constant(test_data)\n", + "\n", + "# The variables below hold all the trainable weights. For each, the\n", + "# parameter defines how the variables will be initialized.\n", + "conv1_weights = tf.Variable(\n", + " tf.truncated_normal([5, 5, NUM_CHANNELS, 32], # 5x5 filter, depth 32.\n", + " stddev=0.1,\n", + " seed=SEED))\n", + "conv1_biases = tf.Variable(tf.zeros([32]))\n", + "conv2_weights = tf.Variable(\n", + " tf.truncated_normal([5, 5, 32, 64],\n", + " stddev=0.1,\n", + " seed=SEED))\n", + "conv2_biases = tf.Variable(tf.constant(0.1, shape=[64]))\n", + "fc1_weights = tf.Variable( # fully connected, depth 512.\n", + " tf.truncated_normal([IMAGE_SIZE / 4 * IMAGE_SIZE / 4 * 64, 512],\n", + " stddev=0.1,\n", + " seed=SEED))\n", + "fc1_biases = tf.Variable(tf.constant(0.1, shape=[512]))\n", + "fc2_weights = tf.Variable(\n", + " tf.truncated_normal([512, NUM_LABELS],\n", + " stddev=0.1,\n", + " seed=SEED))\n", + "fc2_biases = tf.Variable(tf.constant(0.1, shape=[NUM_LABELS]))\n", + "\n", + "print 'Done'" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "QHB_u04Z4HO6" + }, + "source": [ + "Now that we've defined the variables to be trained, we're ready to wire them together into a TensorFlow graph.\n", + "\n", + "We'll define a helper to do this, `model`, which will return copies of the graph suitable for training and testing. Note the `train` argument, which controls whether or not dropout is used in the hidden layer. (We want to use dropout only during training.)" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 772, + "status": "ok", + "timestamp": 1446749138306, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "716a6ad5e180d821", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "V85_B9QF3uBp", + "outputId": "457d3e49-73ad-4451-c196-421dd4681efc" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Done\n" + ] + } + ], + "source": [ + "def model(data, train=False):\n", + " \"\"\"The Model definition.\"\"\"\n", + " # 2D convolution, with 'SAME' padding (i.e. the output feature map has\n", + " # the same size as the input). Note that {strides} is a 4D array whose\n", + " # shape matches the data layout: [image index, y, x, depth].\n", + " conv = tf.nn.conv2d(data,\n", + " conv1_weights,\n", + " strides=[1, 1, 1, 1],\n", + " padding='SAME')\n", + "\n", + " # Bias and rectified linear non-linearity.\n", + " relu = tf.nn.relu(tf.nn.bias_add(conv, conv1_biases))\n", + " \n", + " # Max pooling. The kernel size spec ksize also follows the layout of\n", + " # the data. Here we have a pooling window of 2, and a stride of 2.\n", + " pool = tf.nn.max_pool(relu,\n", + " ksize=[1, 2, 2, 1],\n", + " strides=[1, 2, 2, 1],\n", + " padding='SAME')\n", + " conv = tf.nn.conv2d(pool,\n", + " conv2_weights,\n", + " strides=[1, 1, 1, 1],\n", + " padding='SAME')\n", + " relu = tf.nn.relu(tf.nn.bias_add(conv, conv2_biases))\n", + " pool = tf.nn.max_pool(relu,\n", + " ksize=[1, 2, 2, 1],\n", + " strides=[1, 2, 2, 1],\n", + " padding='SAME')\n", + " \n", + " # Reshape the feature map cuboid into a 2D matrix to feed it to the\n", + " # fully connected layers.\n", + " pool_shape = pool.get_shape().as_list()\n", + " reshape = tf.reshape(\n", + " pool,\n", + " [pool_shape[0], pool_shape[1] * pool_shape[2] * pool_shape[3]])\n", + " \n", + " # Fully connected layer. Note that the '+' operation automatically\n", + " # broadcasts the biases.\n", + " hidden = tf.nn.relu(tf.matmul(reshape, fc1_weights) + fc1_biases)\n", + " \n", + " # Add a 50% dropout during training only. Dropout also scales\n", + " # activations such that no rescaling is needed at evaluation time.\n", + " if train:\n", + " hidden = tf.nn.dropout(hidden, 0.5, seed=SEED)\n", + " return tf.matmul(hidden, fc2_weights) + fc2_biases\n", + "\n", + "print 'Done'" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "7bvEtt8C4fLC" + }, + "source": [ + "Having defined the basic structure of the graph, we're ready to stamp out multiple copies for training, testing, and validation.\n", + "\n", + "Here, we'll do some customizations depending on which graph we're constructing. `train_prediction` holds the training graph, for which we use cross-entropy loss and weight regularization. We'll adjust the learning rate during training -- that's handled by the `exponential_decay` operation, which is itself an argument to the `MomentumOptimizer` that performs the actual training.\n", + "\n", + "The vaildation and prediction graphs are much simpler the generate -- we need only create copies of the model with the validation and test inputs and a softmax classifier as the output." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 269, + "status": "ok", + "timestamp": 1446749139596, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "716a6ad5e180d821", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "9pR1EBNT3sCv", + "outputId": "570681b1-f33e-4618-b742-48e12aa58132" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Done\n" + ] + } + ], + "source": [ + "# Training computation: logits + cross-entropy loss.\n", + "logits = model(train_data_node, True)\n", + "loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(\n", + " logits, train_labels_node))\n", + "\n", + "# L2 regularization for the fully connected parameters.\n", + "regularizers = (tf.nn.l2_loss(fc1_weights) + tf.nn.l2_loss(fc1_biases) +\n", + " tf.nn.l2_loss(fc2_weights) + tf.nn.l2_loss(fc2_biases))\n", + "# Add the regularization term to the loss.\n", + "loss += 5e-4 * regularizers\n", + "\n", + "# Optimizer: set up a variable that's incremented once per batch and\n", + "# controls the learning rate decay.\n", + "batch = tf.Variable(0)\n", + "# Decay once per epoch, using an exponential schedule starting at 0.01.\n", + "learning_rate = tf.train.exponential_decay(\n", + " 0.01, # Base learning rate.\n", + " batch * BATCH_SIZE, # Current index into the dataset.\n", + " train_size, # Decay step.\n", + " 0.95, # Decay rate.\n", + " staircase=True)\n", + "# Use simple momentum for the optimization.\n", + "optimizer = tf.train.MomentumOptimizer(learning_rate,\n", + " 0.9).minimize(loss,\n", + " global_step=batch)\n", + "\n", + "# Predictions for the minibatch, validation set and test set.\n", + "train_prediction = tf.nn.softmax(logits)\n", + "# We'll compute them only once in a while by calling their {eval()} method.\n", + "validation_prediction = tf.nn.softmax(model(validation_data_node))\n", + "test_prediction = tf.nn.softmax(model(test_data_node))\n", + "\n", + "print 'Done'" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "4T21uZJq5UfH" + }, + "source": [ + "# Training and visualizing results\n", + "\n", + "Now that we have the training, test, and validation graphs, we're ready to actually go through the training loop and periodically evaluate loss and error.\n", + "\n", + "All of these operations take place in the context of a session. In Python, we'd write something like:\n", + "\n", + " with tf.Session() as s:\n", + " ...training / test / evaluation loop...\n", + " \n", + "But, here, we'll want to keep the session open so we can poke at values as we work out the details of training. The TensorFlow API includes a function for this, `InteractiveSession`.\n", + "\n", + "We'll start by creating a session and initializing the varibles we defined above." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [] + }, + "colab_type": "code", + "collapsed": true, + "executionInfo": { + "elapsed": 219, + "status": "ok", + "timestamp": 1446749385874, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "716a6ad5e180d821", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "z6Kc5iql6qxV", + "outputId": "51512ffa-8eda-4579-bbf8-d684638df2aa" + }, + "outputs": [], + "source": [ + "# Create a new interactive session that we'll use in\n", + "# subsequent code cells.\n", + "s = tf.InteractiveSession()\n", + "\n", + "# Use our newly created session as the default for \n", + "# subsequent operations.\n", + "s.as_default()\n", + "\n", + "# Initialize all the variables we defined above.\n", + "tf.initialize_all_variables().run()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "hcG8H-Ka6_mw" + }, + "source": [ + "Now we're ready to perform operations on the graph. Let's start with one round of training. We're going to organize our training steps into batches for efficiency; i.e., training using a small set of examples at each step rather than a single example." + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 386, + "status": "ok", + "timestamp": 1446749389138, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "716a6ad5e180d821", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "LYVxeEox71Pg", + "outputId": "9184b5df-009a-4b1b-e312-5be94351351f" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Done\n" + ] + } + ], + "source": [ + "BATCH_SIZE = 60\n", + "\n", + "# Grab the first BATCH_SIZE examples and labels.\n", + "batch_data = train_data[:BATCH_SIZE, :, :, :]\n", + "batch_labels = train_labels[:BATCH_SIZE]\n", + "\n", + "# This dictionary maps the batch data (as a numpy array) to the\n", + "# node in the graph is should be fed to.\n", + "feed_dict = {train_data_node: batch_data,\n", + " train_labels_node: batch_labels}\n", + "\n", + "# Run the graph and fetch some of the nodes.\n", + "_, l, lr, predictions = s.run(\n", + " [optimizer, loss, learning_rate, train_prediction],\n", + " feed_dict=feed_dict)\n", + "\n", + "print 'Done'" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "7bL4-RNm_K-B" + }, + "source": [ + "Let's take a look at the predictions. How did we do? Recall that the output will be probabilities over the possible classes, so let's look at those probabilities." + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 160, + "status": "ok", + "timestamp": 1446749519023, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "716a6ad5e180d821", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "2eNitV_4_ZUL", + "outputId": "f1340dd1-255b-4523-bf62-7e3ebb361333" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[ 2.25393465e-04 4.76219648e-05 1.66868104e-03 5.67830284e-05\n", + " 6.03432834e-01 4.34970111e-02 2.19316153e-05 1.41285171e-04\n", + " 1.54902827e-05 3.50893021e-01]\n" + ] + } + ], + "source": [ + "print predictions[0]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "X5MgraJb_eQZ" + }, + "source": [ + "As expected without training, the predictions are all noise. Let's write a scoring function that picks the class with the maximum probability and compares with the example's label. We'll start by converting the probability vectors returned by the softmax into predictions we can match against the labels." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 220, + "status": "ok", + "timestamp": 1446750411574, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "716a6ad5e180d821", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "wMMlUf5rCKgT", + "outputId": "2c10e96d-52b6-47b0-b6eb-969ad462d46b" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "First prediction 4\n", + "(60, 10)\n", + "All predictions [4 4 2 7 7 7 7 7 7 7 7 7 0 8 9 0 7 7 0 7 4 0 5 0 9 9 7 0 7 4 7 7 7 0 7 7 9\n", + " 7 9 9 0 7 7 7 2 7 0 7 2 9 9 9 9 9 0 7 9 4 8 7]\n" + ] + } + ], + "source": [ + "# The highest probability in the first entry.\n", + "print 'First prediction', numpy.argmax(predictions[0])\n", + "\n", + "# But, predictions is actually a list of BATCH_SIZE probability vectors.\n", + "print predictions.shape\n", + "\n", + "# So, we'll take the highest probability for each vector.\n", + "print 'All predictions', numpy.argmax(predictions, 1)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "8pMCIZ3_C2ni" + }, + "source": [ + "Next, we can do the same thing for our labels -- using `argmax` to convert our 1-hot encoding into a digit class." + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 232, + "status": "ok", + "timestamp": 1446750498351, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "716a6ad5e180d821", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "kZWp4T0JDDUe", + "outputId": "47b588cd-bc82-45c3-a5d0-8d84dc27a3be" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Batch labels [7 3 4 6 1 8 1 0 9 8 0 3 1 2 7 0 2 9 6 0 1 6 7 1 9 7 6 5 5 8 8 3 4 4 8 7 3\n", + " 6 4 6 6 3 8 8 9 9 4 4 0 7 8 1 0 0 1 8 5 7 1 7]\n" + ] + } + ], + "source": [ + "print 'Batch labels', numpy.argmax(batch_labels, 1)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "bi5Z6whtDiht" + }, + "source": [ + "Now we can compare the predicted and label classes to compute the error rate and confusion matrix for this batch." + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + }, + { + "item_id": 2 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 330, + "status": "ok", + "timestamp": 1446751307304, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "716a6ad5e180d821", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "U4hrLW4CDtQB", + "outputId": "720494a3-cbf9-4687-9d94-e64a33fdd78f" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "0.0666666666667\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPcAAAD7CAYAAAC2TgIoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADpNJREFUeJzt3X+MXWWdx/H3p7TUli6UXYnrUtuxMV2RhECDSKzQ64Ju\nxQ1m12wEJGTJhv1DmxJNDMo/vf1jN9kYY0jWf4jYFVp0Q6GBRJcUJVMDG1nsT6QtdWUHCkKDsdOG\nlGBLv/vHPe1O6S3z3DnPM515+nklN3Nmcvqdb+6dT89zz7n3exURmFl9ZpzpBsysDIfbrFIOt1ml\nHG6zSjncZpVyuM0qNTNXIUm+pmZ2hkSE3v2zbOEGYH5ivt/qwpxu0q7LDzye/OtHuusY6t6avP9m\n/TJxz2Ggk1x3MAPWXtdN2+/hLnwxcV+AlYn7DfDYQe2PX6m6g9Ze0/enXpabVcrhNqvUmQn3zE6R\nsvM7lxWpC0OF6hasfUmnTN1Cjx1Mx8evVN08tc9MuGd1ipSdfn8cBWt/rFOmbqHHDqbj41eqbp7a\nSeGWtELSHkl7Jd3V+reaWXHjhlvSDODfgL8GLgVulvTR0o2ZWTspR+6rgN9ExEsRcQT4MfCFsm2Z\nWVsp4b4Y2Dfm+1ean5nZFOZLYWaVSnmF2qvAwjHfL2h+dqq3umMqd4qeWTU7e400t/eWEu5ngY9I\nWgS8BtwE3Nx3zwFelmhmEzXEyZfKNvfda9xwR8Q7klYCm+gt4++LiN3tGzSzkpLeOBIRjwN/WbgX\nM8vIJ9TMKuVwm1XK4TarlMNtVimH26xSDrdZpZTrs8J6AxJXZ6llZoNY03dAoo/cZpVyuM0q5XCb\nVcrhNquUw21WKYfbrFIOt1mlUqaf3idpv6Sdk9GQmeWRcuReS2+ssZlNI+OGOyKeAg5MQi9mlpGf\nc5tVKu/nczM8ZnuIsp+lZHa2GiHX9NMBdPKWM7M+hkiZfpq6LFdzM7NpIuVS2IPAfwFLJL0s6fby\nbZlZWylzy2+ZjEbMLC+fLTerlMNtVimH26xSDrdZpRxus0o53GaVyjvaeH6eWicZ7eavOZ3N75ap\n6/t5cpR4/Ebl0cZmZxOH26xSDrdZpRxus0o53GaVcrjNKpXyls8Fkp6U9Lyk5yStmozGzKydlEks\nR4GvR8R2SfOALZI2RcSewr2ZWQsp009fj4jtzfabwG7g4tKNmVk7Az3nljQEXA48U6IZM8sneUBi\nsyTfANzZHMFP9VZ3TOUOzOq06c3M+jkyDEeHx90tKdySZtIL9gMR8ehpd5zTTerNzFqY1Tn5wPn2\nmr67pS7LfwDsioh7WrZlZpMk5VLYMuDLwF9J2iZpq6QV5VszszZSpp8+DZwzCb2YWUZ+hZpZpRxu\ns0o53GaVcrjNKuVwm1XK4TarVN7pp6zOUsvMBrHG00/NziYOt1mlHG6zSjncZpVyuM0q5XCbVWrc\nd4VJmg38Aji3uT0aEXeXbszM2kl5y+fbkj4dEYclnQM8LWlZ81ZQM5uikpblEXG42Zzd/JsDxToy\nsyySwi1phqRtwOvAcETsKtuWmbWVeuQ+FhFXAAuAayUtL9uWmbWVPNoYICIOSfoJcCWw+dQ9hsds\nDzU3M8trpLm9t5Sz5e8HjkTEQUlzgM8A/Wep0knvz8wmaIiTD5x9jrOkHbk/CPxQkugt4x+IiJ+3\n7M7MCku5FPYcsHQSejGzjPwKNbNKOdxmlXK4zSrlcJtVyuE2q5TDbVYph9usUgO9/PSMmN8tV3u0\nUO3p2PN0tK5bpu7KMmWBSX38fOQ2q5TDbVYph9usUg63WaUcbrNKOdxmlUoOdzNHbaukx0o2ZGZ5\nDHLkvhPwYESzaSJ1+ukC4Abg+2XbMbNcUo/c3wW+AUTBXswso5QBiZ8H9kfEdkkdQKffe3jM9hCe\nfmpWwghZpp8Cy4AbJd0AzAH+RNL9EXHbqbt2BmjQzCZmiJTpp+MuyyPi7ohYGBGLgZuAJ/sH28ym\nEl/nNqvUoJ84spnTrQHMbErxkdusUg63WaUcbrNKOdxmlXK4zSrlcJtVaupPP52O0z6nY8/TUakp\npZU8fj5ym1XK4TarlMNtVimH26xSDrdZpRxus0olXQqTNAIcBI4BRyLiqpJNmVl7qde5jwGdiDhQ\nshkzyyd1Wa4B9jWzKSA1sAE8IelZSXeUbMjM8khdli+LiNckXUQv5Lsj4qlTdxsesz2Ep5+alTBC\nrumnRMRrzdc3JG0ErgL6hLuT3J6ZTdQQWaafSporaV6zfR7wWeDXrfszs6JSjtwfADZKimb/9RGx\nqWxbZtbWuOGOiP8FLp+EXswsI1/eMquUw21WKYfbrFIOt1mlHG6zSjncZpWa+tNPzU6nkimlpfjI\nbVYph9usUg63WaUcbrNKOdxmlXK4zSqVFG5JF0h6SNJuSc9L+kTpxsysndTr3PcAP42Iv5c0E5hb\nsCczy2DccEs6H7gmIv4BICKOAocK92VmLaUsyz8M/F7SWklbJd0raU7pxsysnZRwzwSWAt+LiKXA\nYeCbRbsys9ZSnnO/AuyLiF81328A7uq/6/CY7SE82tishBGyjDaOiP2S9klaEhF7geuAXf337gzQ\noJlNzBApo41Tz5avAtZLmgW8CNzeojMzmwSpH0qwA/h44V7MLCO/Qs2sUg63WaUcbrNKOdxmlXK4\nzSrlcJtVyuE2q1TW0cbL4+qc5QDYvH5F9ponrCxTdvmBx8sUpuD94fvihNW3qkhdgOH4z+w1N5+m\nXR+5zSrlcJtVyuE2q5TDbVYph9usUg63WaXGDbekJZK2NfPTtkk6KGnVZDRnZhOXMollL3AFgKQZ\n9MYubSzcl5m1NOiy/HrgtxGxr0QzZpbPoOH+EvCjEo2YWV7JLz9t5qfdyHuMNR7prjuxPb9zGfM7\nl7VqzsxONTq8k9HhnePuN8hryz8HbImIN063w1D31gHKmdlEvPvA+dKa9X33G2RZfjNekptNG6mf\n8jmX3sm0R8q2Y2a5pI42PgxcVLgXM8vIr1Azq5TDbVYph9usUg63WaUcbrNKOdxmlVJE5CkkBazO\nUusk87v5a5Y22j3THdhZZQ0RccoMVB+5zSrlcJtVyuE2q5TDbVYph9usUg63WaVS3/L5LUnPS9op\nab2kc0s3ZmbtpIw2XgTcAVwREZfRe5voTaUbM7N2Ut7PfQj4I3CepGPAXOB3Rbsys9bGPXJHxAHg\nO8DLwKvAaET8rHRjZtbOuEduSYuBrwGLgIPABkm3RMSDp+49PGZ7qLmZWV4jze29pSzLrwSejog/\nAEh6BPgk0CfcneT2zGyihjj5wLm5714pZ8tfAK6W9D5JAq4DdrfszswKS3nOvQO4H9gC7AAE3Fu4\nLzNrKXX66beBbxfuxcwy8ivUzCrlcJtVyuE2q5TDbVYph9usUg63WaUcbrNK5R1tvC5PrZOszF/y\nhFIjiKfjOOZpaPmBx4vU3XzhiiJ1ixmVRxubnU0cbrNKOdxmlXK4zSrlcJtVKnX66Z2Snmtuq0o3\nZWbtpUw/vRT4R3oTWS4H/qYZvWRmU1jKkfsS4JmIeDsi3gF+Afxd2bbMrK2UcP8auEbShZLmAjcA\nHyrblpm1Ne4klojYI+lfgSeAN4FtwDulGzOzdlLHLK0F1gJI+mdgX98dH+7+//YlHfhYp113Znaq\nI8NwdHjc3ZLCLemiiHhD0kLgb4Gr++74xW5yf2Y2QbM6vdtxb6/pu1tSuIGHJf0pcAT4SkQcated\nmZWWuiy/tnQjZpaXX6FmVimH26xSDrdZpc5MuHcNl6l7pFDdhE9UnLBSPU+3ugVrjw7vLFJ3qt8X\nZybcu4fL1E249jcxI4XqUq7n6Va3YO1i4Z7i94WX5WaVcrjNKpV3+qmZnRH9pp9mC7eZTS1elptV\nyuE2q9SkhlvSCkl7JO2VdFfGuvdJ2i8p6zUPSQskPSnp+Zzz4yTNlvSMpG1N7X/JUXdM/RmStkp6\nLHPdEUk7mr7/O2PdCyQ9JGl3c398IlPdJU2vW5uvBzM+ht9qet0pab2kczPVzTevMCIm5UbvP5L/\nARYBs4DtwEcz1f4UvfluOzP3/OfA5c32POCFjD3Pbb6eA/wSWJax768B64DHMt8fLwIXFvjb+Hfg\n9mZ7JnB+gd8xA/gd8KEMtRY198W5zff/AdyWoe6lwE5gdvN3sQlYPNF6k3nkvgr4TUS8FBFHgB8D\nX8hROCKeAg7kqPWuuq9HxPZm+01gN3BxptqHm83Z9P7wsvQvaQG9UVjfz1Hv3eXJvNqTdD5wTfQG\nghARR6PMW4qvB34bEf0HjQzmEPBH4DxJM4G59P7jaCvrvMLJDPfFnDzB5RUyBWUySBqitzp4JlO9\nGZK2Aa8DwxGxK0dd4LvAN4ASl0ECeELSs5LuyFTzw8DvJa1tls/3SpqTqfZYXwJ+lKNQRBwAvgO8\nDLwKjEbEzzKUzjqv0CfUEkiaB2wA7myO4K1FxLGIuAJYAFwraXnbmpI+D+xvVhtqbjkti4il9P7o\nvirpUxlqzgSWAt9rah8Gvpmh7gmSZgE3Ag9lqreY3lOfRcBfAPMk3dK2bkTsAY7PK/wpLecVTma4\nXwUWjvl+QfOzKa1Zdm0AHoiIR3PXb5agP6E3F76tZcCNkl6kd5T6tKT7M9QFICJea76+AWyk91Sr\nrVeAfRHxq+b7DfTCntPngC1N3zlcCTwdEX9ols+PAJ/MUTgi1kbElRHRAUaBvROtNZnhfhb4iKRF\nzZnFm4CcZ3NLHKkAfgDsioh7chWU9H5JFzTbc4DP0DvB2EpE3B0RCyNiMb3798mIuK1tXQBJc5sV\nDJLOAz5LbxnZSkTsB/ZJWtL86Dog11OU424m05K88QJwtaT3SRK9nnfnKCzpoubr8XmFD060VuoM\ntdYi4h1JK+mdAZwB3BcRue6QB4EO8GeSXgZWHz9B07LuMuDLwHPN8+MA7o6Itp/6/kHgh80fxgx6\nq4Kft6xZ2geAjc3LjGcC6yNiU6baq4D1zfL5ReD2THVpnrteD/xTrpoRsaNZEW2ht2zeBtybqXy2\neYV++alZpXxCzaxSDrdZpRxus0o53GaVcrjNKuVwm1XK4TarlMNtVqn/A4ohGJAOEeBVAAAAAElF\nTkSuQmCC\n", + "text/plain": [ + "<matplotlib.figure.Figure at 0x7f226836c350>" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "correct = numpy.sum(numpy.argmax(predictions, 1) == numpy.argmax(batch_labels, 1))\n", + "total = predictions.shape[0]\n", + "\n", + "print float(correct) / float(total)\n", + "\n", + "confusions = numpy.zeros([10, 10], numpy.float32)\n", + "bundled = zip(numpy.argmax(predictions, 1), numpy.argmax(batch_labels, 1))\n", + "for predicted, actual in bundled:\n", + " confusions[predicted, actual] += 1\n", + "\n", + "plt.grid(False)\n", + "plt.xticks(numpy.arange(NUM_LABELS))\n", + "plt.yticks(numpy.arange(NUM_LABELS))\n", + "plt.imshow(confusions, cmap=plt.cm.jet, interpolation='nearest');" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "iZmx_9DiDXQ3" + }, + "source": [ + "Now let's wrap this up into our scoring function." + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 178, + "status": "ok", + "timestamp": 1446751995007, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "716a6ad5e180d821", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "DPJie7bPDaLa", + "outputId": "a06c64ed-f95f-416f-a621-44cccdaba0f8" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Done\n" + ] + } + ], + "source": [ + "def error_rate(predictions, labels):\n", + " \"\"\"Return the error rate and confusions.\"\"\"\n", + " correct = numpy.sum(numpy.argmax(predictions, 1) == numpy.argmax(labels, 1))\n", + " total = predictions.shape[0]\n", + "\n", + " error = 100.0 - (100 * float(correct) / float(total))\n", + "\n", + " confusions = numpy.zeros([10, 10], numpy.float32)\n", + " bundled = zip(numpy.argmax(predictions, 1), numpy.argmax(labels, 1))\n", + " for predicted, actual in bundled:\n", + " confusions[predicted, actual] += 1\n", + " \n", + " return error, confusions\n", + "\n", + "print 'Done'" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "sLv22cjeB5Rd" + }, + "source": [ + "We'll need to train for some time to actually see useful predicted values. Let's define a loop that will go through our data. We'll print the loss and error periodically.\n", + "\n", + "Here, we want to iterate over the entire data set rather than just the first batch, so we'll need to slice the data to that end.\n", + "\n", + "(One pass through our training set will take some time on a CPU, so be patient if you are executing this notebook.)" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 2 + }, + { + "item_id": 3 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 12621, + "status": "error", + "timestamp": 1446752161966, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "716a6ad5e180d821", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "4cgKJrS1_vej", + "outputId": "df1be05c-a41b-4eff-e91a-5cd24012d140" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Step 0 of 916\n", + "Mini-batch loss: 3.01724 Error: 6.66667 Learning rate: 0.00950\n", + "Validation error: 2.1%\n", + "Step 100 of 916\n", + "Mini-batch loss: 2.91099 Error: 3.33333 Learning rate: 0.00950\n", + "Validation error: 1.8%\n", + "Step 200 of 916\n", + "Mini-batch loss: 2.85178 Error: 1.66667 Learning rate: 0.00950\n", + "Validation error: 1.6%\n", + "Step 300 of 916\n", + "Mini-batch loss: 2.81923 Error: 3.33333 Learning rate: 0.00950\n", + "Validation error: 1.6%\n", + "Step 400 of 916\n", + "Mini-batch loss: 2.82241 Error: 3.33333 Learning rate: 0.00950\n", + "Validation error: 1.4%\n", + "Step 500 of 916\n", + "Mini-batch loss: 2.73972 Error: 0.00000 Learning rate: 0.00950\n", + "Validation error: 1.4%\n", + "Step 600 of 916\n", + "Mini-batch loss: 2.74712 Error: 1.66667 Learning rate: 0.00950\n", + "Validation error: 1.5%\n", + "Step 700 of 916\n", + "Mini-batch loss: 2.82140 Error: 5.00000 Learning rate: 0.00950\n", + "Validation error: 1.3%\n", + "Step 800 of 916\n", + "Mini-batch loss: 2.68445 Error: 1.66667 Learning rate: 0.00950\n", + "Validation error: 1.3%\n", + "Step 900 of 916\n", + "Mini-batch loss: 2.61466 Error: 0.00000 Learning rate: 0.00950\n", + "Validation error: 1.2%\n" + ] + } + ], + "source": [ + "# Train over the first 1/4th of our training set.\n", + "steps = int(train_size / BATCH_SIZE)\n", + "for step in xrange(steps):\n", + " # Compute the offset of the current minibatch in the data.\n", + " # Note that we could use better randomization across epochs.\n", + " offset = (step * BATCH_SIZE) % (train_size - BATCH_SIZE)\n", + " batch_data = train_data[offset:(offset + BATCH_SIZE), :, :, :]\n", + " batch_labels = train_labels[offset:(offset + BATCH_SIZE)]\n", + " # This dictionary maps the batch data (as a numpy array) to the\n", + " # node in the graph is should be fed to.\n", + " feed_dict = {train_data_node: batch_data,\n", + " train_labels_node: batch_labels}\n", + " # Run the graph and fetch some of the nodes.\n", + " _, l, lr, predictions = s.run(\n", + " [optimizer, loss, learning_rate, train_prediction],\n", + " feed_dict=feed_dict)\n", + " \n", + " # Print out the loss periodically.\n", + " if step % 100 == 0:\n", + " error, _ = error_rate(predictions, batch_labels)\n", + " print 'Step %d of %d' % (step, steps)\n", + " print 'Mini-batch loss: %.5f Error: %.5f Learning rate: %.5f' % (l, error, lr)\n", + " print 'Validation error: %.1f%%' % error_rate(\n", + " validation_prediction.eval(), validation_labels)[0]\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "J4LskgGXIDAm" + }, + "source": [ + "The error seems to have gone down. Let's evaluate the results using the test set.\n", + "\n", + "To help identify rare mispredictions, we'll include the raw count of each (prediction, label) pair in the confusion matrix." + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + }, + { + "item_id": 2 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 436, + "status": "ok", + "timestamp": 1446752934104, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "716a6ad5e180d821", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "6Yh1jGFuIKc_", + "outputId": "4e411de4-0fe2-451b-e4ca-8a4854f0db89" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Test error: 1.4%\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQYAAAEKCAYAAADw9/tHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4VOXZx/HvTSYkhCUsokBQAgooQkBZFBFEYhRcK1CV\nRcGoVUsRK68o1L4oVdGqUCv1bVRKhYIoVlwQEAyLLKKCIASNWJDKEiNhCxAJCTzvH+ckBCchk8x5\nMjNwf65rrpycnLnPk8lwc7b5HTHGoJRSJVUL9QCUUuFHG4NSyo82BqWUH20MSik/2hiUUn60MSil\n/PhCPQAAEdFzpkqFiDFGfjkvLBoDgGkb2HKPZ8PjZwW2rGSMrcAIlgA9K7B8qOvarB0udaMrsGw6\nkFyB5QsCXG4J4fFa2Kr9RKlzdVdCKeVHG4NSyk/ENYaeNW1VToywujZrR1pdgOaW6iZGWF1vaks4\nfFZCREygxxgqVLdCxxhU+KnIMYaKCvQYw6nuiVIPPlrfYhCR3iKSKSKbROQR2+tTSgXPamMQkWrA\nJOAa4EJggIicb3OdSqng2d5i6AJ8Z4z5rzGmAJgJ3GR5nUqpINluDAnAthLfb3fnle/+sTB1ObyW\nDi3bQp/bYPIi5/t3N8ILbznL1akLL70PU5bAI3/xePiVM2/eYLKzH2b06O6hHkrI2XwtDh0aRXr6\nINLTBzF0aJInNW2O11btWrWqs3x5KunpQ1i+PJXLLjs76Jphc4HTCVolQdvOcMflcFYCPDUV7k6G\neTOdn/9hEnyxxJm+cxTMnwkfzoAnXoOuKfDpwpANHSA19T2uuqoFTZvWCek4woHN12L79lySk6d7\nWtPmeG3VPnjwCN27T8EYQ2JiXaZP70e3bpODqml7i2EHcE6J75u68/w8nn38sbFRK/h6jfOD7B2Q\n0Bx8bg+LioLL+8Di953vO10BS+c400s/cL4PsaysA4jfcd7Tk83XolGjWixaNIhZs/pyzjne/GOz\nOV6btYvOLsbHx7J7d95JltyKc2Vk0aN0trcYvgDOE5FmQBZwGzCgtAVPuMx5WwakDneaQYs2zlZD\nnXqwZ5fTFFYvhYIjzrJ16sHBXGc6dx/E17f466hwkpg4ib17D5OS0pzJk68nJWVGqIcUMo0b1+bN\nN/vTuvUZ3HTTGydZMpETr3NYWupSVrcYjDFHgd8BC4CNwExjzDflPvH7TGfXIG0BDBwO/9noNAWA\n6wfDnH8dXzZ3L9Ss7UzXjof9ezz+LVS42rv3MAALF35Ps2bxIR5NaGVlHaBHjyl06vQKaWk3BF3P\n+nUMxpj5xpjWxpiWxphnAn7irDS4qxdMmwjfbXDm1awNF1wMny06vtzqpdDjOme6+7XO92FCdyeO\n8/q1iIuLLq7Zrt2Z7Np1ss3nirP5t/O6dnR0VPH0wYNHqF496iRLByY8Dz4C/H0+RPlgXw48NcyZ\nl9IPFr174nJTnoOnp8It98Gm9SE/8AiQlnYDXbs2JSbGR8eOTejX781QDylkbL0WbdqcQVrateTm\n5mOM4d5753pS1+bfzlbttm3PZOLEaygsPEZMjI8RI+YFXVMviVZhTC+Jti9El0QrpSKPNgallB9t\nDEopP9oYlFJ+wuashI0DhWPLyLPzwhPogU379ABhqOgWg1LKjzYGpZQfbQxKKT/aGJRSfrQxKKX8\naGNQSvnRxqCU8mM7JXqyiGSLyHqb61FKecv2FsMUnOh4pVQEsZ3gtBzYa3MdgRo8bx4PZ2fTffRo\nAOo1b85vvviC0fv3c3bXrsXLXTNhAnetXMldK1bQbdSo4vltb7uNu1etInX5clKefbbM9bRv34hl\ny1JZvHgoCxfeQbNmde39Uqe5li0bkJ//R7p2DT4VuYiNxGXbbLznrOcxuHmPHxhjysz3FhGDhUuM\nS14SXbtxY1pcdRV1mjZl2fjx+GJi8MXGcs2ECXz52mts+/RTAOq1aMHeLVsAuGvFCv49aBD7tm7l\nge++4+V27Sg8fJghixbR+74MNm3a7bfOhg1rcujQEfLyCujd+zwGDGjHkCGzPf/dFLz++s00alSL\nxx9fwqefbiv/CQESEU8Tl20L7j1Xeh5D2HxW4sTE2kS8vunngaysEzK1CvPzKczP98vZKmoKAMcK\nCzFHjwLw8549xMbH8/PRo0RFR7Nv3+FS17Nr16Hi6fz8oxQUHPXy11Cuzp0TyMo6QGHhMc9rB564\nHB4q9p7b6j5OLowaQ89QD+AE7QYOZM/mzezf5vxPtGz8eO5bt46CvDw2vvUWP/1UemMoEhcXzZNP\n9iI19b2qGO5pZ8yY7tx557tMmOD9IazAE5fDS2DvuURCnhLtEvcRMVokJ9NhyBDm3HsvAL6YGK4c\nN46/nnceL557Lg0vvJCOHZuU+fyoqGrMnNmf8eOX8e23OVU17NNGnz4tWb16Z5lbbcHyOnG5Knj9\nnrN9unIGsBJoJSI/iMidNtcXkNIiekvMS+jShSvHjeOt/v05WuB87Leaz0e1qCiOHHI22Q7v3Uu9\nerFlrmL69L7Mnp3JnDmbvB27AqBDh0b07JnI3LmDSEk5l+efv9qzuzvZSFyuCl6/58ImDNb2wccb\n0tJo2rUrvpgYfsrIYPaQIdz6zjs0vOACcnfs4Lu5c1k6bhz3r18PxpC3ezcYw0cjR/LjunVc8sAD\nJA0aRGF+Pnu++46L7/qh1HX27XsBU6b8itWrdyIC69dn8+CD8z3/3ZRj8uSbePXVNaxatd2Tehdd\n1PiExOU//WkpCxZs9qS2LcG950o/+HjaNAavaVCLOjVoSrRSKkDaGJRSfrQxKKX8aGNQSvnRxqCU\n8hNGVz56z+aZg5woO2c8zjiqZzuqRg1LdX+2VLdq6RaDUsqPNgallB9tDEopP9oYlFJ+tDEopfxo\nY1BK+bH9seumIrJIRDaKyAYRecDm+pRS3rB9HUMh8JAxZp2I1ALWiMgCY0ym5fUqpYJgOyX6R2PM\nOnf6IPANkFCRGvPmDSY7+2FGj+7u6di8qlv7w3nU25lNjUdHF8+Lm/gidRYvpfbs95D4eGfeCxOo\ns3wldZatIPZhN306Joba8z6izuKl1Fm2guhrNGk/HIwd243lyweRnn4bF154hmd1bb2XbaiyKx9F\nJBHoAHxWkeelpr7HVVe18Cyhx+u6B+9OJTr5KqKaNgUg+uqrkRo1yL3yCqoPGkyNUY+Q94cxHP7b\nJI6NfAiAOstWcGTWWxzbto1Dv7mbY9u2IfXrE//JCpg7M+jfTVVeUlJDOnduzOWXTychoTZTp15H\ncrI3fxNb72UbqqQxuLsRbwMj3C2HUiwpMZ1IUWBlVtaBUtPYguVVXZOVhZQo5OtxBUc+nANAwZwP\niL3vfgCOlUifprAQjh6Fo0c55obNmsOHixOpVei0alWfNWt+BGDHjgM0bx6Pz1fNkzRqW+/litlK\nWKREi4gPpylMM8acJL62p+2hVIlq9Rtg9jn32DH79yPxJ978o/qAgRzdvLm4IRSp+fwEDj//Z6B5\nVQ1VlSIjI4fhwzvi81WjTZsGJCTUpl69WHbtCv8Y+cAkEkhKdFVsMfwD+NoY82IVrCvkzN49xc1A\n6tQpbhIA0cnJxNwxhAM3Xn/Cc2qM+QMmdz/506ZiI+JOBS4zczczZnzNggW3sHnzPjZuzDmFmkLg\nbJ+u7AYMAnqJyFoR+VJEeleulrdj87yuW6jgk6VU73MtANHXXkfBJ05H9nXpQo3Hx3Hwlv7gpk8D\nxP52GNXOPY+8Rx/xaCAqWGlp6+jVayYTJ65mw4ZdntcP/e5E+cI+DDYt7Qa6dm1KTIyPjIyf6Nfv\nTU/WGWzdoo9d1/y/NHyXdkViYijcmMHBX/ej5osvEZWUhNm/n4ND78Ds20f8Wid92uxx0qcPPTyS\nY9u3U297FoWfrnSOORhD9V6fePL7qfKU/bHr+fNvwecTcnJ+ZtiwhezeXZGPUpe9rK33cnBOw5Ro\nmzSPIdJpHoNDU6KVUgHSxqCU8qONQSnlRxuDUsqPNgallJ9TOiXaJltnD8yZ9u63KT/pGY/jIu3s\nQdXSLQallB9tDEopP9oYlFJ+tDEopfxoY1BK+dHGoJTyY/V0pYjEAJ8A1d3He8aYMTbXqZQKntXG\nYIzJF5ErjTF5IhIFrBCRbsaYFTbXq5QKjvVdCWNMUfxNjLu+vSdZXCkVBqw3BhGpJiJrgR+BJcaY\nr22vMxC2orzbt2/EsmWpLF48lIUL76BZs7rlP6k8/zMWPlgO/06H8y+E2Fh49S14ZxFMfhtq1XaW\na36eM+/f6TD2ueDXG6RataqzfHkq6elDWL48lcsuOzvUQyqXzTGH+60QSqqyoBYRqQMsAB4xxiz9\nxc+qPKilcePaxVHe48cv86xuw4Y1OXToCHl5BfTufR4DBrRjyJDZAT/f75LoNkkw5mkYfD00ToBJ\nU2H+e1A9Bv72HNz4a2eZZ/4IU96BF5+Gdath/CSY+w4sW1RcKhSXRIsIxhgSE+syfXo/unWbXOVj\nqChbY7b1nguuboiDWowxucCHQKfSl1hS4rHV+nhsRXnv2nWIvDwn0zE//ygFBUFGwp/bCtavcaaz\ndsA5zaFFK/hqtTNv7edwWU9nukUr+Mpddt0X0O3K4NbtgaL/eOLjY9m9OzJCVW2NOTxuhbCVE/+t\nlc72WYkzgAJjzH4RqQGkAGV8SqinzaFUubi4aJ58shepqSdJzA9EZgbcNRx8PmjVBho3hR0/QK8+\nsHwxXHUd1K3vLPvNeujVG9LnQXIf2LM7+F8kSI0b1+bNN/vTuvUZ3HTTG6EeTkAiccyBSySQ+Hjb\nWwyNgcXuMYZVwPvGmHTL6wy5qKhqzJzZn/Hjl/HttznBFfsuE96ZAW8tgHsecBrF3yc4xxne/hga\nNYHsnc6yj/8PDLob3vzIaQpF80MoK+sAPXpMoVOnV0hLuyHUwwlIJI7Za7ZPV24ALra5jmDZ2LSb\nPr0vs2dnMmfOJm8KTk1zHq3bwO8ece5kNca9cfjgu2GHe/OaH3dCaj9n+qXX4cN3vFl/JUVHRxXv\nSh08eITq1aNCOp5AVMWYw/5WCJzGeQwlo7w7dmziWZR3374X0KdPSxo2rMnttyexfn02Dz44P7ii\nM+c7uxJ7cuDRYdDyfHj2ZadBfL0ennjYWe7m22DwPXDsGMyaBpu+Cf4XCkLbtmcyceI1FBYeIybG\nx4gR80I6nkDYHLOt95yNuhofH2Y0qEVVLY2PV0oFSBuDUsqPNgallB9tDEopP9oYlFJ+TtvTleHK\n5pkD09bOGQ/J0LMdVSO6ytZ00sYgIg+d7OfGmAneDkcpFQ7K22JwP89La6Az8L77/Q3A57YGpZQK\nrZM2BmPMEwAi8glwsTHmgPv94ziflFRKnYICPfh4FnCkxPdH3HlKqVNQoAcfpwKfi0hR4sivgNft\nDEkpFWoBNQZjzFMiMg8oyo660xiz1t6wlFKhVJHrGOKAXGPMi8B2EWke6BPd3McvReT98pdWSoVa\nQI1BRMYCjwCj3VnRwL8qsJ4RQFiEwCqlyhfoFsPNwI3AIQBjzE6On8o8KRFpClwLvFaZASqL7h8L\nU5fDa+nQsi30uQ0mL3K+f3cjvPCWs9yoCfCvlTBtBdz5cGjH7IqkxOWqqH3RRY2YP/82Pv54IOPH\nB5/1GejBxyPGGOPkJoCI1KzAOiYCDwPxFR2csqhVErTtDHdcDmclwFNT4e5kmDfT+fkfJsFqNw9w\nxiT4s3ut27QVsGAW7NgakmEXSU19rzgZORLq2qzt81XjmWeu5Oab3y4OIg5WoFsMb4lIGlBXRO4B\nPiaALQARuQ7INsasA8R9lGEJVZkSfdpLbAVfu4nS2TsgobmTEgUQFQWX94FFbpDt9i3Hn3e0EI4G\nmXztgfBIXA6P2l27JnDw4BHeeONXLFw4kG7dmp5k6S1AeolH6QI9K/G8iKQAuThXQf6vMWZhAE/t\nBtwoItcCNYDaIjLVGHOH/6I9AxmK8sp3GTDQTZ9u0cbZaqhTD/bscprC6qVQcOTE51w3ELZthh+3\nhWbMqlRNmtQmKelM2rd/jfj4GNLTB9GmTVoZS7dwH0UWl7pUQI1BRJ41xjwCLCxlXpncG9iOcZe/\nAhhZelNQVe77TPhwBqQtcP6x/2ej0xQArh8Mb79y4vKXJsONQ2DY9VU/VnVSe/b8zMqV28nLKyAv\nr4CcnDwaNKjB7t0/V7pmoLsSKaXM61PptarwMCsN7uoF0ybCdxuceTVrwwUXw2fH72BFuy4wbBw8\n1B8KvdmH9UokJC7brv3ZZztp1aoBIs4t9ho2jAuqKUD5n668H/gtcK6IrC/xo9rAyoqsyL0tXel3\nt1Ch8ff5EOWDfTnw1DBnXko/WPTuics98RoYA399z/n6/EjIXFf14y0hkhKXbdfOzc3npZe+YOnS\n2/H5qjFq1KLyn1SOk6ZEi0g8UA8YDzxa4kcHjDF7gl778fVoSnQV0DyGSGcjj+GxiqdEG2P2G2O2\nAi8Ce4wx/zXG/BcoFJFLLIxSKRUGAj3G8H/AwRLfH3TnKaVOQYE2BjEl9jmMMcfQWDilTlmBNoYt\nIvKAiES7jxE4V0oopU5BgTaG+4DLgB3AduAS4De2BqWUCi29d2XYsZkEbOcaBDPS0tmOF560UtcR\nXtdjhE7p964s7zqGUcaYP4vIS4BfBzHGPODhCJVSYaK8A4hF91FfbXsgSqnwUV5K9AfuV813VOo0\nUt6uxAeUsgtRxBhzo+cjUkqFXHm7Es+7X/sCjTge5zYAyLY1KKVUaJW3K7EUQEReMMZ0KvGjD0RE\njzsodYoK9OrFmiLSwhizBcBNiA4o3k1EtgL7gWNAgTGmS2UGqpSqOoE2ht8DS0RkC048WzPg3gCf\newzoaYzZW4nxKaVCINBot/ki0hI4352VaYzJD3AdQsXuX3GC9u0bMWnStRQWHqOw8Bh33/0+//3v\nvsqWs17XtkOHRrFq1Q4Apk3bwD//ub6cZ4TA9c9B4mVQLQqWToDMuTDoDaheE6r54J374ceNcN0z\ncHZnQKBha0h/Cla+HNAqzj+/AS+/3BtjIDbWR8uW9TjzzL8EPfR58wZz8cWN+ctfVjF+/LKg6xWp\nVas68+cPJj//KDExUYwatZCVK72LyGvZsgEZGb+lZ89/8umnwdcNNNotDngIaGaMuUdEWopIa2PM\nnACeboCFInIUeMUY82pFBrhz5wGuuWYaeXkF9O59HuPGXcmQIbPLf2KI6tq2fXsuycnTQz2MsjVs\nDQkXwaRuTiN4aB3E1YMfPoOPn4QWPSD5MZg+AD4sEfHx0DrY8O+AV5OZuZtevZzXoX//87nyykRP\nhm8ryfngwSN07z4FYwyJiXWZPr0f3bpN9qz+Y4/1YMmSrZ7VC3RXYgqwBujqfr8DmAUE0hi6GWOy\nRKQhToP4xhiz3H+xJSWmE90H7Np1qHhufv5RCgq8SSi2Vde2Ro1qsWjRIHbv/pmRIz/mhx9yQz2k\nEx3eD1HVna2F2DqQtxsO5cAZrZyfx9WHg784odWkAxzIdh6VMHhwO559tkKBYmWymRJd9PGD+PhY\ndu/O86xu584JZGUdoLDwWABLbyWQFPZAG8O5xphbRWQAgDEmTySwl88Yk+V+3eXeFLcLUEpj6HnS\nOnFx0Tz5ZC9SU98LcMiBsVXXlsTESezde5iUlOZMnnw9KSkzQj2kEx340dk6eGQTRMfB2/dA5nzo\n8RCMXA+x8fC3y098TsfBsLZyW0H16sXSunV9Pv10hweDt6tx49q8+WZ/Wrc+g5tuesOzumPGdOfO\nO99lwoRrAlg6kaL/dB2lpy0Guu9/RERq4F7sJCLnAuUeYxCROBGp5U7XBK4GMgJcZ7GoqGrMnNmf\n8eOX8e23ORV9epXXtWnv3sMALFz4Pc2aheE9fJp1hfrNYfy58Ofz4drxkDwavnoLXkiCab+GviWO\nI4jAhTfB+sB3I0q69dY2zJqV6dHg7crKOkCPHlPo1OkV0tJu8KRmnz4tWb16J/v2HfakXpFAtxjG\nAvOBs0VkOs79IoYG8LyzgNnuHax8wHRjzIKKDnL69L7Mnp3JnDmbKvrUkNS1JS4ump9/LsAYaNfu\nTHbt8m5z1DM16sLP7gmoI4ec3YrqNWH39868QzlQo97x5VteBdu+cJathEGD2nLXXYHs0VaM17sT\n0dFRxburBw8eoXr1KE/qdujQiJ49E+nW7WzatTuL1q3P4NZbZ7F9e3C7mOU2BneXIRPn6sdLcc4y\njDDGlPtfrDHme6BDMAPs2/cC+vRpScOGNbn99iTWr8/mwQfnB1PSal2b2rQ5g7S0a8nNzccYw733\nzg31kPx9+xF0uA1++wn4qsOyFyHjXRg4Dbqkgi8W5pa4HcnFg+DLitwf+bjExHiqV49i0ybPcomt\nJTm3bXsmEydeQ2HhMWJifIwYMc+TuuPHLys+ezJ58k28+uqaoJsCBJjHICIbjDHtgl5b2fU1j6GY\n5jEU0TyGqlB6HkOgxxi+FJHOHo9IKRWmAj3GcAkw2L28+RDO7oQxxiTZGphSKnQCbQyBnAdRSp0i\nystjiMUJgj0P2ABMNsYUVsXAlFKhU94xhteBTjhNoQ/wgvURKaVCrrx7VxafjRARH/C5MeZizweh\nZyVUKcwNds52AMgH+n5zVO6sRPE5Hd2FUOr0Ud7Bx/YiUnS1hAA13O+Lzkp4+xE0pVRYKC/azZvr\nNpVSEaXSASpKqVOXNgallB9tDEopP9Ybg4jEi8gsEflGRDaKyCW216mUCk6gl0QH40VgrjHm1+61\nEHFVsE6lVBCsNgYRqQN0N8YMheJrIcIspFAp9Uu2txiaAzkiMgVoj3PX7BHGmJ8DLRCpMe+RxOZr\n7Gkc+4Cx0CEFCvLh1RFQqx7c/jQUFkDhEZh4O+zfBXdNgNaXgjHw2bvwznOe/C7hykbkfUBBLZUu\nLtIRWAV0NcasFpG/APuNMWN/sVyZl0Q3bFiTQ4eOFMe8DxjQLiJi3iOJzde4cePaxXHsFX3TnnBJ\ndGKS0wT+dD00SIDfT4X/vRqOueneyUOh8Xnwr8egUQv4cYsz/9kVMGEQZG89ofapdEl0MK9xWZdE\n295i2A5sM8YU3efybeCR0hddUmI6Edvx8eo4m6+xZ3HsCa1g8xpnevcOOKv5icGMcXXgoBvxVtQU\nAI4VwtFT+z1Tsdd4K17Gx1eKMSZbRLaJSCtjzCYgGfi69KV7nrRWpMW8R6Kwfo3/mwHXD4coH5zd\nBuonOLsS53WCgU9Azbow8hchY1cMhKzNkOPdHZ8iXyKBxMdXxVmJB4DpIhINbAHurGiBSIx5jzRh\n/xpvz4SlM+CJBfDjZti20TmesGae87isH9z5PEy6x1m+fTL0GuLseqgKs94YjDFfAUHlRUZazHsk\nsv0ae7I7MT/NeZzdBvo9Ar5o58AjQN5+qB7rTLfqAgPHweO9j//8NOBl5H1VbDEEJRJj3iONzdfY\n0zj2x+c7uxK5OfD3YXDl7dDzdjDH4GghvHyfs9zvXgMM/OE95+s/RsKWdV78OmHJRuS91bMSAQ9C\ng1pUKTSopSoEFx+vlDqNaGNQSvnRxqCU8qONQSnlRxuDUspP2J+uVKcvm2cOzJmWbsT7k82zHTZv\neHwi3WJQSvnRxqCU8qONQSnlRxuDUsqPNgallB9tDEopP1Ybg4i0EpG1IvKl+3W/iDxgc51KqeDZ\nTnDaBFwEICLVcKLeNLBRqTBXlbsSVwGbjTEVytmaN28w2dkPM3p0d08HY6uuzdo2x3zaemMebMyG\nEaOPz3vqRXh3KUx9D+rEO/Pi68K092H2EnjyL8eXbdsB3l/mPG6546Srsvn3O3RoFOnpg0hPH8TQ\noUlB16vKKx9vBd6o6JNSU98rTsD1kq26NmvbHPNp68FU6HEVNGnqfN/zaoitAb+6AvoPhuGPwFNj\n4Hej4N2Z8O8ZMPE1uCIFli6Ep1+C+wdCdhbMXUWdqQvIzc0vdVU2/37bt+eSnDzds3pVssXg5j3e\nCMwqe6klJR5bi+d6ljL8C7bq2qxtc8ynreysEzPRLrsCFs5xphd8AJf2cKa7XgELSsy/7AqIjoYa\ncbBjGxQWwqpP6NIlocxV2fz7NWpUi0WLBjFrVl/OOedkjWcLkF7iUbqq2mLoA6wxxuwqe5GeVTQU\npU6iXgPYt9eZzt0P8fWc6br14YB7E7X9+5zv6zWA3BI35sndT/36Tat2vK7ExEns3XuYlJTmTJ58\nPSkpM8pYsoX7KLK41KWq6hjDACqxG6FUldu3xzmeAFC7Dux3m8T+vVCrtjNdJ95Zbt+e443Dnb9n\nT8A3WfPU3r2HAVi48HuaNYsPul5V3O06DufA4zvB1fFmPFVV12Zt3Z2woOhFXbkUkq91plOug0/d\n+y6sXOJ8D87PVy6FI0fg0EFonAA+H3Tpxuef7wh4VV6Ji4surtmu3Zns2pUXdM2qiI/PAxpW9vk2\nEnBt1rVZ2+aYT1vPp0GnrlA9BpI6Qmo/SLneOStxYD/8zj3T8LfnYNJUuOM++Ga9c+AR4I8PQtpM\nZ/offyM3t3mZq7L192vT5gzS0q4lNzcfYwz33js36JqaEq1OS5rHUOQxTYlWSgVGG4NSyo82BqWU\nH20MSik/2hiUUn40JVqFsRrWKts6e2A6Wbzf5uonrdX+Jd1iUEr50caglPKjjUEp5Ucbg1LKjzYG\npZQfbQxKKT9V8bHr0SKyUUTWi8h0Ealue51KqeDYjo9vBtwDXGSMScK5buI2m+tUSgXP9hZDLnAE\nqCkiPiAO2FnRIi1bNiA//4907Xq2ZwNr374Ry5alsnjxUBYuvINmzep6VhvsjNlWXduvhS1jx3Zj\n+fJBpKffxoUXnuFJTc+TnO8ZC68th5fT4dwLnXl9BsPfFsLLH0PKrScuP/af8NJHFV5NRKVEG2P2\nisgLwA9AHrDAGPNxRes89lgPlizZ6unYdu48wDXXTCMvr4Devc9j3LgrGTLEu1te2Bizrbq2Xwsb\nkpIa0rlzYy6/fDoJCbWZOvU6kpNnBl3X0yTnlknQpjPcfTmcmQCPT4XnfgddroJhKf7Ln9sWalUu\nli2iUqIC38bxAAALk0lEQVRFpAXwe6AZ0ASoJSIDS196CaWlRHfunEBW1gG2b8/1dGy7dh0iL68A\ngPz8oxQUHPWstq0xR+JrYUurVvVZs+ZHAHbsOEDz5vH4fMG/nT1Ncj6nFWSucaZ/2gFNmkOvfnA4\nz9kqePZtaNjk+PJ3/RGmPFWpVXmdEm17V6ITsMIYs8cYcxQn9/Gy0hftWeKRWDx3zJjuPPPMcms5\nh3Fx0Tz5ZC+ee26lZzVtjTkSXwtbMjJy6NnzHHy+aiQlNSQhoTb16sWGelgn2pwBF/eEKJ+z9XBW\nUzijCcTXh+HXwPv/gBHPO8te3AN++Bb2/FSpVSUmTqJXr+m88spaJk++/iRLtgCSSzxKZ7sxfAtc\nKiKxIiLuSL4J9Ml9+rRk9eqd7Nt32MrgoqKqMXNmf8aPX8a33+Z4UtPWmCPxtbApM3M3M2Z8zYIF\ntzB8eEc2bszxJATVU1sz4aMZMGkB3DLcaRS5e+BT9xjCqo+c3QeAIY/CtOecpNhKdP6ISok2xnwF\nTAXWAF8BArwS6PM7dGhEz56JzJ07iJSUc3n++as9vYvP9Ol9mT07kzlzNnlW09aYI/G1sC0tbR29\nes1k4sTVbNhwkluWVIJnW2XvpMH9veCNifCfDbBmiXPcAeCCTrB9M9SoCfXPgqdmOgcfW3WAoY8G\nvAobKdEREwY7efJNvPrqGlat2u7JOvv2vYApU37F6tU7EYH167N58MH5ntQu4vWYbdWtiteick7+\nsev582/B5xNycn5m2LCF7N5dkXs6lL5sySTnjIyfKpzk7Pex67/Od6Ll9+XAs8Ng/2548AVofZHT\nfZ7+Dfzw3fHlG50Df3jV2dX4hbI+dt2pU+MTUqIfeGABGRmBNsrSw2AjpjGo05G9PIayGkOwIi+P\nQVOilVIB0saglPKjjUEp5Ucbg1LKjzYGpZQfbQxKKT96uvK0YuOmqAAFluraZOtUqJ3ToACmpfen\nQuU79HSlUiow2hiUUn60MSil/GhjUEr50caglPJTFSnRI0Rkg/t4wPb6lFLBsx3tdiFwF06SUwfg\nejfuTSkVxmxvMVwAfGaMyXej3T4B+lpep1IqSLYbQwbQXUTqiUgccC3gbZ76acTzaPMSLrqoEfPn\n38bHHw9k/PgrPa8fSWzE0nse0T98LLyxHF5Ph5YXQs1akPYBTF0EMz5x5gGMngBvroSZK+DuhwMu\nbzs+PlNEngUWAgeBtUD4RxCHKU+jzUvw+arxzDNXcvPNbxenRZ+ubMXSexrRf34StOsMAy6HsxLg\nz1Nh3lvw1Wfw8pPQuQf89jH4/QD41yQY/5DzvJkrYP4s2L613FVYbQwAxpgpwBQAEXkK2Fb6kktK\nTCdSMilaOTyNNi+ha9cEDh48whtv/Iq4uGgef/wTVqzwNo4uUpQVS19YeCyourt2HSqeDjqiP7EV\nZLix9Nk7oGlz2Lcbmrdy5tWtDznZzvS2Lcefd7SQTw8e5aPd5a/CemMQkYbGmF0icg5wM3Bp6Uv2\ntD0UVYYmTWqTlHQm7du/Rnx8DOnpg2jTJi3UwwqJjIwchg/viM9XjTZtGhTH0nuVQF0U0Z+a+l7l\ni3yXAbcPd7Ikz2vjbDWs/RSG/h4+WA+1452tiZJuGAg/bKbrvm10bXB89hN7Sl+F9cYA/FtE6uN8\n0ua3xhhv75aigrZnz8+sXLmdvLwC8vIKyMnJo0GDGhUMVz01lIyl37x5n6ex9J5F9G/OhA9mwD8W\nwLbN8J+NcPMdzu7E6y9CUhd4/GW49wZn+cuS4eYhcO/J7jdxIuvXMRhjehhj2hpjLjLGLLG9vtOB\n17sTn322k1atGiACtWpVp2HDuNOyKRSxFUvvaUT/zDS4oxdMmQibNji3ttvn7iPszYE69ZzppC7w\nwDgY3h8KAj9+VBVbDMojJaPNO3ZsUuFo87Lk5ubz0ktfsHTp7fh81Rg1apEndSPVL2PpvdC37wX0\n6dOShg1rcvvtScFH9E92Y+n35sATw6B6rHMQsl8qxMTCc6Oc5Z56DYyB/3vP+frMSPhmXbnlNY/h\ntKJ5DMdpHgNoHoNSqgIisDFs1brWa28pf5FK2Wqprs3amy3V3WqpLizx4FipNoaIrWuz9veW6m61\nVNdm7chrkks82JuJwMaglLJNG4NSyk8YnZVQSoVC2N7tWikVXnRXQinlRxuDUspPxDQGEektIpki\nsklEHvGw7mQRyRaR9V7VdOs2FZFFIrLRy7xLEYkRkc9EZK1b+2kv6paoX01EvhSR9z2uu1VEvnLH\n/bmHdeNFZJaIfOO+Hpd4VLeVO9Yv3a/7PfwbjnbHul5EpotIdY/qepevaowJ+wdOA/sP0Aznut51\nwPke1b4cJ49yvcdjbgR0cKdrAd96OOY492sUsAro5uG4fw/8C3jf49djC1DPwnvjn8Cd7rQPqGNh\nHdWAncDZHtRq5r4W1d3v3wTu8KDuhcB6IMZ9XywAWlS2XqRsMXQBvjPG/NcYUwDMBG7yorAxZjmw\n14tav6j7ozFmnTt9EPgGSPCodtG1bTE4b1pPxi8iTXHi917zot4vy+PxFqqI1AG6GycMCGNMobHz\nsf6rgM3GmDJChiokFzgC1BQRHxCH03SC5Wm+aqQ0hgROTH7ajkf/yKqCiCTibJV85lG9aiKyFvgR\nWGKM+dqLusBE4GHAxqkqAywUkS9E5B6PajYHckRkirvJ/4qI2Ph01K3AG14UMsbsBV4AfgB2APuM\nMR97UNrTfNVIaQwRS0RqAW8DI9wth6AZY44ZYy4CmgI9ROSKYGuKyHVAtruVI+7DS92MMRfjvGGH\nicjl5T0hAD7gYuBvbu084FEP6hYTkWjgRmCWR/Va4OyuNQOaALVEZGCwdY0xmUBRvupcgsxXjZTG\nsAM4p8T3Td15Yc3dVHwbmGaMCSLLq3TuZvOHOPftCFY34EYR2YLzv+OVIjLVg7oAGGOy3K+7gNk4\nu4fB2g5sM8asdr9/G6dReKkPsMYdtxc6ASuMMXvcTf53gMu8KGyMmWKM6WSM6QnsAyqdCBMpjeEL\n4DwRaeYewb0N8PKouY3/IQH+AXxtjHnRq4IicoaIxLvTNYAUnIOxQTHGjDHGnGOMaYHz+i4yxtwR\nbF0AEYlzt5wQkZrA1TibvkExxmQD20TETUElGfBqt6rIADzajXB9C1wqIrEiIjhj/saLwiLS0P1a\nlK86o7K1IiLByRhzVER+h3OktRow2Rjj1Ys5AyeJtoGI/ACMLTqYFWTdbsAgYIN7PMAAY4wxQcT2\nANAYeN19U1XD2RpJD7KmbWcBs91L333AdGPMAo9qPwBMdzf5twB3elQXd1/9KuA3XtU0xnzlbomt\nwdnUXwu84lF5z/JV9ZJopZSfSNmVUEpVIW0MSik/2hiUUn60MSil/GhjUEr50caglPKjjUEBICK/\nEpFjJS4WKmu5ISLSKIj1XCEiH1T2+apqaGNQRW4D5uBc6XcyQwn+A2x68UyY08agii5TvgQYhtMg\niuY/4oaJrBWRp0WkH861/v9yP80YKyLfu1fbISIdRWSxO91ZRFaKyBoRWS4iLUPwq6lKiohLopV1\nNwEfGWO2ichPInIRzmXMNwCdjTH5IlLXGLNPRIYBI40xa6HUhO+i778BLjfGHBORZGA80L9qfh0V\nLG0MCpzdh4nu9CxgIM6HyqYYY/IBjDH73J//8gNnZX34rC4w1d1SKPqMhIoQ+sc6zYlIPaAX0Nb9\n3z8K5x/yLAL7xGkhx3dJY0vM/xPOJzT7ikgzYLF3o1a26TEG9WtgqjGmuTGmhTGmGc7NK3OBoUWJ\nSG4DwZ1fp8Tzvwc6utP9SsyP53hmhmefeFRVQxuDuhUnOKWkf+OE2b4PrBaRL4GR7s9eB/7uHnyM\nAcYBf3WTnwtL1Pgz8IyIrEHfZxFHP3atlPKjnVwp5Ucbg1LKjzYGpZQfbQxKKT/aGJRSfrQxKKX8\naGNQSvnRxqCU8vP/Jbv/zp0OVTQAAAAASUVORK5CYII=\n", + "text/plain": [ + "<matplotlib.figure.Figure at 0x7f22684b17d0>" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "test_error, confusions = error_rate(test_prediction.eval(), test_labels)\n", + "print 'Test error: %.1f%%' % test_error\n", + "\n", + "plt.xlabel('Actual')\n", + "plt.ylabel('Predicted')\n", + "plt.grid(False)\n", + "plt.xticks(numpy.arange(NUM_LABELS))\n", + "plt.yticks(numpy.arange(NUM_LABELS))\n", + "plt.imshow(confusions, cmap=plt.cm.jet, interpolation='nearest');\n", + "\n", + "for i, cas in enumerate(confusions):\n", + " for j, count in enumerate(cas):\n", + " if count > 0:\n", + " xoff = .07 * len(str(count))\n", + " plt.text(j-xoff, i+.2, int(count), fontsize=9, color='white')" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "yLnS4dGiMwI1" + }, + "source": [ + "We can see here that we're mostly accurate, with some errors you might expect, e.g., '9' is often confused as '4'.\n", + "\n", + "Let's do another sanity check to make sure this matches roughly the distribution of our test set, e.g., it seems like we have fewer '5' values." + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "metadata": { + "cellView": "both", + "colab": { + "autoexec": { + "startup": false, + "wait_interval": 0 + }, + "output_extras": [ + { + "item_id": 1 + } + ] + }, + "colab_type": "code", + "collapsed": false, + "executionInfo": { + "elapsed": 352, + "status": "ok", + "timestamp": 1446753006584, + "user": { + "color": "#1FA15D", + "displayName": "Michael Piatek", + "isAnonymous": false, + "isMe": true, + "permissionId": "00327059602783983041", + "photoUrl": "//lh6.googleusercontent.com/-wKJwK_OPl34/AAAAAAAAAAI/AAAAAAAAAlk/Rh3u6O2Z7ns/s50-c-k-no/photo.jpg", + "sessionId": "716a6ad5e180d821", + "userId": "106975671469698476657" + }, + "user_tz": 480 + }, + "id": "x5KOv1AJMgzV", + "outputId": "2acdf737-bab6-408f-8b3c-05fa66d04fe6" + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEACAYAAABfxaZOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEqxJREFUeJzt3W+MZXV9x/H3B1dUVMjWlt2WRf4EQTC2SgzaUttpbVE0\nXWhNEDX1D8Y+gAZjE+MufcD6pEqTRm1aTIwWF4vSRTGskchCN5NGUwQVXMuuuK0Blq071EgxjYnu\n6rcP7tnudTqzM3Nnds5Zfu9XcrPn/u7vzv3M3ZnP/d1z7r2TqkKS1IYT+g4gSVo9lr4kNcTSl6SG\nWPqS1BBLX5IaYulLUkMWLP0kn0wyk2TX2NhfJ9mT5MEkn09y8thlm5Ps7S6/ZGz8wiS7knw3yUdW\n/luRJC1kMSv9m4DXzhrbAbykql4G7AU2AyS5ALgCOB+4FLgxSbrrfAx4V1WdC5ybZPbXlCQdYwuW\nflV9BXhy1tg9VfXz7uy9wIZueyNwa1UdqqpHGD0gXJRkPfD8qrq/m3czcPkK5JckLcFK7NO/Criz\n2z4N2Dd22f5u7DTg8bHxx7sxSdIqWlbpJ/lL4GBVfXaF8kiSjqE1k14xyTuA1wO/Pza8Hzh97PyG\nbmy+8fm+th8IJEkTqKoc7fLFrvTTnUZnktcB7wM2VtVPxuZtB65McmKSs4BzgPuq6gDwVJKLugO7\nbwPuWCD4oE7XX3997xnM9PTKZSYzrXSmxVhwpZ/kM8AU8IIkjwHXA9cBJwJ3dy/Oubeqrq6q3Um2\nAbuBg8DVVXV41X4N8Cng2cCdVfXlRSWUJK2YBUu/qt4yx/BNR5n/QeCDc4x/A3jpktJJklaU78hd\npKmpqb4j/D9mWrwh5jLT4phpcRabKUf2vgxHkhpiLkkasiTUCh3IlSQ9DVj6ktQQS1+SGmLpS1JD\nLH1JaoilL0kNsfTnsX79mSTp/bR+/Zl93xWSnkZ8nf78GYAh3Deh7/tC0vHB1+lLkn6BpS9JDbH0\nJakhlr4kNcTSl6SGWPqS1BBLX5IaYulLUkMsfUlqiKUvSQ2x9CWpIZa+JDXE0pekhlj6ktQQS1+S\nGmLpS1JDLH1JasiavgNoIc/q/opXf9atO4MDBx7pNYOklbHgSj/JJ5PMJNk1NrY2yY4kDye5K8kp\nY5dtTrI3yZ4kl4yNX5hkV5LvJvnIyn8rT1c/YfRnG/s7zcw8euy/TR13/DvSx6fF7N65CXjtrLFN\nwD1VdR6wE9gMkOQC4ArgfOBS4MYcWaZ+DHhXVZ0LnJtk9teUdBwZLQb6XZC4KFm6BUu/qr4CPDlr\n+DJga7e9Fbi8294I3FpVh6rqEWAvcFGS9cDzq+r+bt7NY9eRJK2SSffpn1pVMwBVdSDJqd34acC/\njs3b340dAh4fG3+8G9dxof/jCuCxBQ3X+vVnHjfPOFbqQG6t0NfRIB0+rtCvmZn+H3ikuRzZ1dW3\nhX9HJi39mSTrqmqm23XzRDe+Hzh9bN6Gbmy+8Xlt2bLl/7anpqaYmpqaMKokPV1Nd6fFS9XCj05J\nzgS+WFUv7c7fAPywqm5I8n5gbVVt6g7k3gK8ktHum7uBF1VVJbkXuBa4H/gS8LdV9eV5bq8Wk+tY\nGu3OGMojd985hpABIPT9c6EjhvQ70vfPxcDui6Mu9xdc6Sf5DDAFvCDJY8D1wIeA25JcBTzK6BU7\nVNXuJNuA3cBB4Oqx9r4G+BTwbODO+QpfknTsLGqlv9qS1Akn9Pu+sZ///BBDeeTuP8cQMsBovfCT\nXhN4MPmIga1u+00wrPviqCv9wZY+/LTHBF8A3sRQ/hP7zzGEDDCMHP0XzFAMrOj6TTCs+2J5u3f6\n88web3vAd4ukWYbxkuLjhe0m6Tg3hJcUHz8POn7KpiQ1xJW+dBw6nt4BqmGx9KXj0DDeAXr87NLQ\nEe7ekaSGWPqS1BBLX5IaYulLUkMsfUlqiKUvSQ2x9CWpIZa+JDXE0pekhlj6ktQQP4ZBWhI/xlfH\nN0tfWpIhfIwv+Lk3mpS7dySpIZa+JDXE0pekhlj6ktQQS1+SGmLpS1JDLH1JaoilL0kNsfQlqSGW\nviQ1xNKXpIYsq/STbE7yUJJdSW5JcmKStUl2JHk4yV1JTpk1f2+SPUkuWX58SdJSTFz6Sc4A3g28\nvKp+ndGHt70Z2ATcU1XnATuBzd38C4ArgPOBS4Eb48cVStKqWs5K/0fAT4HnJlkDPAfYD1wGbO3m\nbAUu77Y3ArdW1aGqegTYC1y0jNuXJC3RxKVfVU8CfwM8xqjsn6qqe4B1VTXTzTkAnNpd5TRg39iX\n2N+NSZJWycSfp5/kbOC9wBnAU8BtSd7K//+w8Qk/fHzL2PZUd5IkHTHdnRZvOX9E5RXAV6vqhwBJ\nvgD8FjCTZF1VzSRZDzzRzd8PnD52/Q3d2Dy2LCOaJLVgil9cEH9gwWssZ5/+w8Crkjy7OyD7GmA3\nsB14Rzfn7cAd3fZ24MruFT5nAecA9y3j9iVJSzTxSr+qvpXkZuAbwM+AB4CPA88HtiW5CniU0St2\nqKrdSbYxemA4CFxdVUP4u3OS1IwMsXeTVL9/h/R24I0M52+h9p1jCBlgGDmGkAGGkWMIGWAYOYaQ\nASBU1VFfCu87ciWpIZa+JDXE0pekhlj6ktQQS1+SGmLpS1JDLH1JaoilL0kNsfQlqSGWviQ1xNKX\npIZY+pLUEEtfkhpi6UtSQyx9SWqIpS9JDbH0Jakhlr4kNcTSl6SGWPqS1BBLX5IaYulLUkMsfUlq\niKUvSQ2x9CWpIZa+JDXE0pekhlj6ktSQZZV+klOS3JZkT5KHkrwyydokO5I8nOSuJKeMzd+cZG83\n/5Llx5ckLcVyV/ofBe6sqvOB3wC+A2wC7qmq84CdwGaAJBcAVwDnA5cCNybJMm9fkrQEE5d+kpOB\nV1fVTQBVdaiqngIuA7Z207YCl3fbG4Fbu3mPAHuBiya9fUnS0i1npX8W8IMkNyX5ZpKPJzkJWFdV\nMwBVdQA4tZt/GrBv7Pr7uzFJ0ipZs8zrXghcU1VfT/JhRrt2ata82ecXacvY9lR3kiQdMd2dFm85\npf84sK+qvt6d/zyj0p9Jsq6qZpKsB57oLt8PnD52/Q3d2Dy2LCOaJLVgil9cEH9gwWtMvHun24Wz\nL8m53dBrgIeA7cA7urG3A3d029uBK5OcmOQs4BzgvklvX5K0dMtZ6QNcC9yS5JnA94B3As8AtiW5\nCniU0St2qKrdSbYBu4GDwNVVNeGuH0nSJDLE3k1SEx8KWBG3A2+k3wyHhf5zDCEDDCPHEDLAMHIM\nIQMMI8cQMgCEqjrqS+F9R64kNcTSl6SGWPqS1BBLX5IaYulLUkMsfUlqiKUvSQ2x9CWpIZa+JDXE\n0pekhlj6ktQQS1+SGmLpS1JDLH1JaoilL0kNsfQlqSGWviQ1xNKXpIZY+pLUEEtfkhpi6UtSQyx9\nSWqIpS9JDbH0Jakhlr4kNcTSl6SGWPqS1BBLX5IasuzST3JCkm8m2d6dX5tkR5KHk9yV5JSxuZuT\n7E2yJ8kly71tSdLSrMRK/z3A7rHzm4B7quo8YCewGSDJBcAVwPnApcCNSbICty9JWqRllX6SDcDr\ngU+MDV8GbO22twKXd9sbgVur6lBVPQLsBS5azu1LkpZmuSv9DwPvA2psbF1VzQBU1QHg1G78NGDf\n2Lz93ZgkaZWsmfSKSd4AzFTVg0mmjjK1jnLZUWwZ257qTpKkI6a70+JNXPrAxcDGJK8HngM8P8mn\ngQNJ1lXVTJL1wBPd/P3A6WPX39CNzWPLMqJJUgum+MUF8QcWvMbEu3eq6rqqemFVnQ1cCeysqj8F\nvgi8o5v2duCObns7cGWSE5OcBZwD3Dfp7UuSlm45K/35fAjYluQq4FFGr9ihqnYn2cbolT4Hgaur\nasJdP5KkSWSIvZukJj4UsCJuB95IvxkOC/3nGEIGGEaOIWSAYeQYQgYYRo4hZAAIVXXUl8L7jlxJ\naoilL0kNsfQlqSGWviQ1xNKXpIZY+pLUEEtfkhpi6UtSQyx9SWqIpS9JDbH0Jakhlr4kNcTSl6SG\nWPqS1BBLX5IaYulLUkMsfUlqiKUvSQ2x9CWpIZa+JDXE0pekhlj6ktQQS1+SGmLpS1JDLH1Jaoil\nL0kNsfQlqSGWviQ1ZOLST7Ihyc4kDyX5dpJru/G1SXYkeTjJXUlOGbvO5iR7k+xJcslKfAOSpMVb\nzkr/EPAXVfUS4DeBa5K8GNgE3FNV5wE7gc0ASS4ArgDOBy4FbkyS5YSXJC3NxKVfVQeq6sFu+3+A\nPcAG4DJgazdtK3B5t70RuLWqDlXVI8Be4KJJb1+StHQrsk8/yZnAy4B7gXVVNQOjBwbg1G7aacC+\nsavt78YkSatk2aWf5HnA54D3dCv+mjVl9nlJUk/WLOfKSdYwKvxPV9Ud3fBMknVVNZNkPfBEN74f\nOH3s6hu6sXlsGdue6k6SpCOmu9PipWryhXiSm4EfVNVfjI3dAPywqm5I8n5gbVVt6g7k3gK8ktFu\nnbuBF9UcAZJUv08QbgfeyDCepIT+cwwhAwwjxxAywDByDCEDDCPHEDIAhKo66gtkJl7pJ7kYeCvw\n7SQPMPqOrwNuALYluQp4lNErdqiq3Um2AbuBg8DVcxW+JOnYWdZK/1hxpT9uCCuIIWSAYeQYQgYY\nRo4hZIBh5BhCBljMSt935EpSQyx9SWqIpS9JDbH0Jakhlr4kNcTSl6SGWPqS1BBLX5IaYulLUkMs\nfUlqiKUvSQ2x9CWpIZa+JDXE0pekhlj6ktQQS1+SGmLpS1JDLH1JaoilL0kNsfQlqSGWviQ1xNKX\npIZY+pLUEEtfkhpi6UtSQyx9SWqIpS9JDbH0Jakhq176SV6X5DtJvpvk/at9+5LUslUt/SQnAH8H\nvBZ4CfDmJC9ezQyTm+47wBym+w4wh+m+A8xjuu8Ac5juO8AcpvsOMIfpvgPMYbrvAHOYXtSs1V7p\nXwTsrapHq+ogcCtw2SpnmNB03wHmMN13gDlM9x1gHtN9B5jDdN8B5jDdd4A5TPcdYA7TfQeYw/Si\nZq126Z8G7Bs7/3g3JklaBWv6DjCfk0/+o95u+9Ch7/PjH/d285J0zKSqVu/GklcBW6rqdd35TUBV\n1Q2z5q1eKEl6GqmqHO3y1S79ZwAPA68Bvg/cB7y5qvasWghJatiq7t6pqp8l+XNgB6PjCZ+08CVp\n9azqSl+S1K9BvSN3iG/cSvLJJDNJdvWd5bAkG5LsTPJQkm8nuXYAmZ6V5GtJHuhy/VXfmQ5LckKS\nbybZ3ncWgCSPJPlWd1/d13cegCSnJLktyZ7u/++VA8h0bncffbP796mB/Kxv7u6jXUluSXLiADK9\np+uChfugqgZxYvQA9O/AGcAzgQeBFw8g128DLwN29Z1lLNN64GXd9vMYHScZwn11UvfvM4B7gYv7\nztTleS/wj8D2vrN0eb4HrO07x6xMnwLe2W2vAU7uO9OsfCcA/wmc3nOOM7r/vxO78/8EvK3nTC8B\ndgHP6n73dgBnzzd/SCv9Qb5xq6q+AjzZd45xVXWgqh7stv8H2MMA3u9QVYdf6PosRr+kvd9vSTYA\nrwc+0XeWMWFAz7KTnAy8uqpuAqiqQ1X1o55jzfYHwH9U1b4FZx5bPwJ+Cjw3yRrgJEYPRn06H/ha\nVf2kqn4G/AvwJ/NNHswPHr5xayJJzmT0TORr/Sb5v90oDwAHgOmq2t13JuDDwPuAIR28KuDuJPcn\neXffYYCzgB8kuanblfLxJM/pO9QsbwI+23eIqnoS+BvgMWA/8N9VdU+/qfg34NVJ1iY5idEi5/T5\nJg+p9LVESZ4HfA54T7fi71VV/byqXg5sAH4nye/2mSfJG4CZ7llRutMQXFxVFzL65bwmyW/3nGcN\ncCHw912uHwOb+o10RJJnAhuB2waQ5WxGuwvPAH4NeF6St/SZqaq+A9wA3A3cCTwA/Gy++UMq/f3A\nC8fOb+jGNIfuqeXngE9X1R195xnX7Rr4EvCKnqNcDGxM8j1Gq8TfS3Jzz5moqu93//4X8AVGuzb7\n9Diwr6q+3p3/HKMHgaG4FPhGd3/17RXAV6vqh92ulNuB3+o5E1V1U1W9oqqmgP8Gvjvf3CGV/v3A\nOUnO6I6GXwkM4tUWDGuVeNg/ALur6qN9BwFI8stJTum2nwP8IaOD8b2pquuq6oVVdTajn6edVfW2\nPjMlOal7hkaS5wKXMHp63puqmgH2JTm3G3oNMIRdc4e9mQHs2uk8DLwqybOThNF91ft7jZL8Svfv\nC4E/Bj4z39zBfPZODfSNW0k+A0wBL0jyGHD94QNePWa6GHgr8O1uH3oB11XVl3uM9avA1u4X4QRG\nz0D+ucc8Q7UO+EL3USNrgFuqakfPmQCuBW7pdqV8D3hnz3mA0YMko4O4f9Z3FoCq+lb3bPEbjHah\nPAB8vN9UAHw+yS8BB4Grj3Yg3jdnSVJDhrR7R5J0jFn6ktQQS1+SGmLpS1JDLH1JaoilL0kNsfQl\nqSGWviQ15H8Bc9sQZEMpbW0AAAAASUVORK5CYII=\n", + "text/plain": [ + "<matplotlib.figure.Figure at 0x7f2265a72d50>" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "plt.xticks(numpy.arange(NUM_LABELS))\n", + "plt.hist(numpy.argmax(test_labels, 1));" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "E6DzLSK5M1ju" + }, + "source": [ + "Indeed, we appear to have fewer 5 labels in the test set. So, on the whole, it seems like our model is learning and our early results are sensible.\n", + "\n", + "But, we've only done one round of training. We can greatly improve accuracy by training for longer. To try this out, just re-execute the training cell above." + ] + } + ], + "metadata": { + "colabVersion": "0.3.2", + "colab_default_view": {}, + "colab_views": {}, + "kernelspec": { + "display_name": "Python 2", + "language": "python", + "name": "python2" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 2 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython2", + "version": "2.7.6" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/tensorflow/tools/docker/notebooks/LICENSE b/tensorflow/tools/docker/notebooks/LICENSE new file mode 100644 index 0000000000..28711d7885 --- /dev/null +++ b/tensorflow/tools/docker/notebooks/LICENSE @@ -0,0 +1,13 @@ +Copyright 2015 The TensorFlow Authors. All rights reserved. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. |