diff options
| author |  A. Unique TensorFlower <gardener@tensorflow.org> | 2018-10-08 13:31:52 -0700 |
|---|---|---|
| committer | TensorFlower Gardener <gardener@tensorflow.org> | 2018-10-08 13:38:33 -0700 |
| commit | 76ab96c8a5b2d77dfc191c94ff54fd5e52c561f2 (patch) | |
| tree | c3b929dd45c2b56509c957c208e80105b3ab4535 | |
| parent | 9b558126e31d25ec4e82cb4f50033d6eca44349a (diff) | |
Changed Adam algorithm variant formula from sqrt(max(v, epsilon**2)) to sqrt(v + epsilon**2) and changed flag name accordingly.
PiperOrigin-RevId: 216240045
| -rw-r--r-- | tensorflow/contrib/tpu/proto/optimization_parameters.proto | 6 |
1 files changed, 3 insertions, 3 deletions
diff --git a/tensorflow/contrib/tpu/proto/optimization_parameters.proto b/tensorflow/contrib/tpu/proto/optimization_parameters.proto index 8529b48c15..c2e3be03db 100644 --- a/tensorflow/contrib/tpu/proto/optimization_parameters.proto +++ b/tensorflow/contrib/tpu/proto/optimization_parameters.proto @@ -62,9 +62,9 @@ message FtrlParameters { // (https://www.tensorflow.org/api_docs/python/tf/contrib/opt/AdamOptimizer). If // use_non_lazy_adam is enabled, use_gradient_accumulation is also required in // order to get correct results; a warning will be printed otherwise (which may -// change to an error in the future). If use_max_with_epsilon is set, the Adam +// change to an error in the future). If use_sum_inside_sqrt is set, the Adam // variable update formula will be changed from m / (sqrt(v) + epsilon) to -// m / max(sqrt(v), abs(epsilon)); this option improves the performance of TPU +// m / sqrt(v + epsilon**2); this option improves the performance of TPU // training and is not expected to harm model quality. message AdamParameters { float beta1 = 3; @@ -73,7 +73,7 @@ message AdamParameters { float initial_m = 6; float initial_v = 7; bool use_non_lazy_adam = 8; - bool use_max_with_epsilon = 9; + bool use_sum_inside_sqrt = 10; } // https://www.tensorflow.org/api_docs/python/tf/train/MomentumOptimizer |