diff options
| -rw-r--r-- | tensorflow/contrib/lite/g3doc/performance.md | 186 | ||||
| -rw-r--r-- | tensorflow/contrib/lite/g3doc/performance_benchmarks.md | 174 |
2 files changed, 199 insertions, 161 deletions
diff --git a/tensorflow/contrib/lite/g3doc/performance.md b/tensorflow/contrib/lite/g3doc/performance.md index 28cb6aba6e..0ae9400068 100644 --- a/tensorflow/contrib/lite/g3doc/performance.md +++ b/tensorflow/contrib/lite/g3doc/performance.md @@ -1,174 +1,38 @@ -# Performance +# Performance best practices -This document lists TensorFlow Lite performance benchmarks when running well -known models on some Android and iOS devices. +Mobile and embedded devices have limited computational resources and it is important to keep your application resource efficient. We have compiled a list of best practices and strategies you can use to optimize your model and application when using Tensorflow Lite. -These performance benchmark numbers were generated with the -[Android TFLite benchmark binary](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/lite/tools/benchmark) -and the [iOS benchmark app](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/lite/tools/benchmark/ios). +## Choose the most efficient model for the problem +Some models may be too large to run on embedded devices. Instead of large models it is better to use a slightly less precise but smaller model for embedded devices. Smaller models not only use less disk space and memory but are generally faster and more energy efficient. One example of models optimized for mobile devices are [MobileNets](https://arxiv.org/abs/1704.04861), which are optimized for mobile vision applications. Tensorflow Lite [models page](models.md) lists several other models that have been optimized specifically for mobile and embedded devices. -# Android performance benchmarks +You can retrain the listed models on your own dataset by using transfer learning. Check out our transfer learning tutorial for +[image classification] (https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/#0) and + [object detection](https://medium.com/tensorflow/training-and-serving-a-realtime-mobile-object-detector-in-30-minutes-with-cloud-tpus-b78971cf1193). -For Android benchmarks, the CPU affinity is set to use big cores on the device to -reduce variance (see [details](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/lite/tools/benchmark#reducing-variance-between-runs-on-android)). -It assumes that models were download and unzipped to the -`/data/local/tmp/tflite_models` directory. The benchmark binary is built -using [these instructions](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/lite/tools/benchmark#on-android) -and assumed in the `/data/local/tmp` directory. +## Profile your model +Before starting any optimization, it is a good practice to profile and benchmark your model. Tensorflow Lite [benchmarking tool](../tools/benchmark) has a built-in profiler that shows per operator profiling statistics. This can help in understanding performance bottlenecks and which operators dominate the computation time. -To run the benchmark: +## Profile and optimize operators in the graph +If a particular operator appears frequently in the model and based on profiling you find the operator consuming the most amount of time, you can look into optimizing the operator. + This scenario should be rare as Tensorflow Lite has optimized versions for most ops. However you may be able to write a faster version of a custom op, if you know the constraints in which the operator is executed. Check out our [custom operator documentation](custom_operators.md). -``` -adb shell taskset ${CPU_MASK} /data/local/tmp/benchmark_model \ - --num_threads=1 \ - --graph=/data/local/tmp/tflite_models/${GRAPH} \ - --warmup_runs=1 \ - --num_runs=50 \ - --use_nnapi=false -``` +## Quantize your model +If your model uses floating point weights or activations then it may be possible to reduce the size of model up to ~4x by using quantization and other model optimizations. Check out our [model optimization toolkit](https://www.tensorflow.org/performance/model_optimization) for details about optimizing your model. Fully quantized models can be remarkably power efficient as well. -Here, `${GRAPH}` is the name of model and `${CPU_MASK}` is the CPU affinity -chosen according to the following table: +## Tweak the number of threads +Tensorflow Lite supports multi-threaded kernels for many operators. You can increase the number of threads and speed up execution of operators. Increasing the number of threads will however make your model use more resources and power. For some applications latency may be more important than energy efficiency. You can increase the number of threads by setting the number of [interpreter](../interpreter.h) threads. -Device | CPU_MASK | --------| ---------- -Pixel 2 | f0 | -Pixel xl | 0c | +## Eliminate redundant copies +Tensorflow Lite is optimized to reduce redundant copies. The APIs allow user to [mmap a model file](https://github.com/tensorflow/tensorflow/blob/9982fd6c8831cbd2f58954f79ea71f26660393bc/tensorflow/contrib/lite/model.h#L152) and avoid copies. If your application is not careful, there can be redundant copies when feeding the input to the model and reading output from the model. Make sure to eliminate redundant copies. If you are using higher level APIs like Java API, make sure to carefully check the documentation for performance caveats. For example, the Java API is a lot faster if ByteBuffers are used as [inputs](https://github.com/tensorflow/tensorflow/blob/6305a6d83552ba6a472cd72398b60d9241467f1f/tensorflow/contrib/lite/java/src/main/java/org/tensorflow/lite/Interpreter.java#L151). -<table> - <thead> - <tr> - <th>Model Name</th> - <th>Device </th> - <th>Mean inference time (std dev)</th> - </tr> - </thead> - <tr> - <td rowspan = 2> - <a href="http://download.tensorflow.org/models/mobilenet_v1_2018_08_02/mobilenet_v1_1.0_224.tgz">Mobilenet_1.0_224(float)</a> - </td> - <td>Pixel 2 </td> - <td>166.5 ms (2.6 ms)</td> - </tr> - <tr> - <td>Pixel xl </td> - <td>122.9 ms (1.8 ms) </td> - </tr> - <tr> - <td rowspan = 2> - <a href="http://download.tensorflow.org/models/mobilenet_v1_2018_08_02/mobilenet_v1_1.0_224_quant.tgz">Mobilenet_1.0_224 (quant)</a> - </td> - <td>Pixel 2 </td> - <td>69.5 ms (0.9 ms)</td> - </tr> - <tr> - <td>Pixel xl </td> - <td>78.9 ms (2.2 ms) </td> - </tr> - <tr> - <td rowspan = 2> - <a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/model_zoo/upload_20180427/nasnet_mobile_2018_04_27.tgz">NASNet mobile</a> - </td> - <td>Pixel 2 </td> - <td>273.8 ms (3.5 ms)</td> - </tr> - <tr> - <td>Pixel xl </td> - <td>210.8 ms (4.2 ms)</td> - </tr> - <tr> - <td rowspan = 2> - <a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/model_zoo/upload_20180427/squeezenet_2018_04_27.tgz">SqueezeNet</a> - </td> - <td>Pixel 2 </td> - <td>234.0 ms (2.1 ms)</td> - </tr> - <tr> - <td>Pixel xl </td> - <td>158.0 ms (2.1 ms)</td> - </tr> - <tr> - <td rowspan = 2> - <a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/model_zoo/upload_20180427/inception_resnet_v2_2018_04_27.tgz">Inception_ResNet_V2</a> - </td> - <td>Pixel 2 </td> - <td>2846.0 ms (15.0 ms)</td> - </tr> - <tr> - <td>Pixel xl </td> - <td>1973.0 ms (15.0 ms) </td> - </tr> - <tr> - <td rowspan = 2> - <a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/model_zoo/upload_20180427/inception_v4_2018_04_27.tgz">Inception_V4</a> - </td> - <td>Pixel 2 </td> - <td>3180.0 ms (11.7 ms)</td> - </tr> - <tr> - <td>Pixel xl </td> - <td>2262.0 ms (21.0 ms) </td> - </tr> +## Profile your application with platform specific tools +Platform specific tools like [Android profiler](https://developer.android.com/studio/profile/android-profiler) and [Instruments](https://help.apple.com/instruments/mac/current/) provide a wealth of profiling information that can be used to debug your app. Sometimes the performance bug may be not in the model but in parts of application code that interact with the model. Make sure to familiarize yourself with platform specific profiling tools and best practices for your platform. - </table> +## Use hardware accelerators available on the device +Tensorflow Lite is working on adding support for accelerators like GPU and provides acceleration through [NNAPI](https://developer.android.com/ndk/guides/neuralnetworks/) on Android. +You can utilize these hardware accelerator backends to improve the speed and efficiency of your model. To enable NNAPI call [UseNNAPI](https://github.com/tensorflow/tensorflow/blob/6305a6d83552ba6a472cd72398b60d9241467f1f/tensorflow/contrib/lite/interpreter.h#L334) on the interpreter instance. -# iOS benchmarks - -To run iOS benchmarks, the [benchmark -app](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/lite/tools/benchmark/ios) -was modified to include the appropriate model and `benchmark_params.json` was -modified to set `num_threads` to 1. - -<table> - <thead> - <tr> - <th>Model Name</th> - <th>Device </th> - <th>Mean inference time (std dev)</th> - </tr> - </thead> - <tr> - <td> - <a href="http://download.tensorflow.org/models/mobilenet_v1_2018_08_02/mobilenet_v1_1.0_224.tgz">Mobilenet_1.0_224(float)</a> - </td> - <td>iPhone 8 </td> - <td>32.2 ms (0.8 ms)</td> - </tr> - <tr> - <td> - <a href="http://download.tensorflow.org/models/mobilenet_v1_2018_08_02/mobilenet_v1_1.0_224_quant.tgz)">Mobilenet_1.0_224 (quant)</a> - </td> - <td>iPhone 8 </td> - <td>24.4 ms (0.8 ms)</td> - </tr> - <tr> - <td> - <a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/model_zoo/upload_20180427/nasnet_mobile_2018_04_27.tgz">NASNet mobile</a> - </td> - <td>iPhone 8 </td> - <td>60.3 ms (0.6 ms)</td> - </tr> - <tr> - <td> - <a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/model_zoo/upload_20180427/squeezenet_2018_04_27.tgz">SqueezeNet</a> - </td> - <td>iPhone 8 </td> - <td>44.3 (0.7 ms)</td> - </tr> - <tr> - <td> - <a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/model_zoo/upload_20180427/inception_resnet_v2_2018_04_27.tgz">Inception_ResNet_V2</a> - </td> - <td>iPhone 8</td> - <td>562.4 ms (18.2 ms)</td> - </tr> - <tr> - <td> - <a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/model_zoo/upload_20180427/inception_v4_2018_04_27.tgz">Inception_V4</a> - </td> - <td>iPhone 8 </td> - <td>661.0 ms (29.2 ms)</td> - </tr> - </table> +## Need more help +The Tensorflow team is happy to help diagnose and address specific performance issues you may be facing. Please file a bug on [github](https://github.com/tensorflow/tensorflow/issues) with details of the issue. diff --git a/tensorflow/contrib/lite/g3doc/performance_benchmarks.md b/tensorflow/contrib/lite/g3doc/performance_benchmarks.md new file mode 100644 index 0000000000..28cb6aba6e --- /dev/null +++ b/tensorflow/contrib/lite/g3doc/performance_benchmarks.md @@ -0,0 +1,174 @@ + +# Performance + +This document lists TensorFlow Lite performance benchmarks when running well +known models on some Android and iOS devices. + +These performance benchmark numbers were generated with the +[Android TFLite benchmark binary](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/lite/tools/benchmark) +and the [iOS benchmark app](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/lite/tools/benchmark/ios). + +# Android performance benchmarks + +For Android benchmarks, the CPU affinity is set to use big cores on the device to +reduce variance (see [details](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/lite/tools/benchmark#reducing-variance-between-runs-on-android)). + +It assumes that models were download and unzipped to the +`/data/local/tmp/tflite_models` directory. The benchmark binary is built +using [these instructions](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/lite/tools/benchmark#on-android) +and assumed in the `/data/local/tmp` directory. + +To run the benchmark: + +``` +adb shell taskset ${CPU_MASK} /data/local/tmp/benchmark_model \ + --num_threads=1 \ + --graph=/data/local/tmp/tflite_models/${GRAPH} \ + --warmup_runs=1 \ + --num_runs=50 \ + --use_nnapi=false +``` + +Here, `${GRAPH}` is the name of model and `${CPU_MASK}` is the CPU affinity +chosen according to the following table: + +Device | CPU_MASK | +-------| ---------- +Pixel 2 | f0 | +Pixel xl | 0c | + +<table> + <thead> + <tr> + <th>Model Name</th> + <th>Device </th> + <th>Mean inference time (std dev)</th> + </tr> + </thead> + <tr> + <td rowspan = 2> + <a href="http://download.tensorflow.org/models/mobilenet_v1_2018_08_02/mobilenet_v1_1.0_224.tgz">Mobilenet_1.0_224(float)</a> + </td> + <td>Pixel 2 </td> + <td>166.5 ms (2.6 ms)</td> + </tr> + <tr> + <td>Pixel xl </td> + <td>122.9 ms (1.8 ms) </td> + </tr> + <tr> + <td rowspan = 2> + <a href="http://download.tensorflow.org/models/mobilenet_v1_2018_08_02/mobilenet_v1_1.0_224_quant.tgz">Mobilenet_1.0_224 (quant)</a> + </td> + <td>Pixel 2 </td> + <td>69.5 ms (0.9 ms)</td> + </tr> + <tr> + <td>Pixel xl </td> + <td>78.9 ms (2.2 ms) </td> + </tr> + <tr> + <td rowspan = 2> + <a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/model_zoo/upload_20180427/nasnet_mobile_2018_04_27.tgz">NASNet mobile</a> + </td> + <td>Pixel 2 </td> + <td>273.8 ms (3.5 ms)</td> + </tr> + <tr> + <td>Pixel xl </td> + <td>210.8 ms (4.2 ms)</td> + </tr> + <tr> + <td rowspan = 2> + <a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/model_zoo/upload_20180427/squeezenet_2018_04_27.tgz">SqueezeNet</a> + </td> + <td>Pixel 2 </td> + <td>234.0 ms (2.1 ms)</td> + </tr> + <tr> + <td>Pixel xl </td> + <td>158.0 ms (2.1 ms)</td> + </tr> + <tr> + <td rowspan = 2> + <a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/model_zoo/upload_20180427/inception_resnet_v2_2018_04_27.tgz">Inception_ResNet_V2</a> + </td> + <td>Pixel 2 </td> + <td>2846.0 ms (15.0 ms)</td> + </tr> + <tr> + <td>Pixel xl </td> + <td>1973.0 ms (15.0 ms) </td> + </tr> + <tr> + <td rowspan = 2> + <a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/model_zoo/upload_20180427/inception_v4_2018_04_27.tgz">Inception_V4</a> + </td> + <td>Pixel 2 </td> + <td>3180.0 ms (11.7 ms)</td> + </tr> + <tr> + <td>Pixel xl </td> + <td>2262.0 ms (21.0 ms) </td> + </tr> + + </table> + +# iOS benchmarks + +To run iOS benchmarks, the [benchmark +app](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/lite/tools/benchmark/ios) +was modified to include the appropriate model and `benchmark_params.json` was +modified to set `num_threads` to 1. + +<table> + <thead> + <tr> + <th>Model Name</th> + <th>Device </th> + <th>Mean inference time (std dev)</th> + </tr> + </thead> + <tr> + <td> + <a href="http://download.tensorflow.org/models/mobilenet_v1_2018_08_02/mobilenet_v1_1.0_224.tgz">Mobilenet_1.0_224(float)</a> + </td> + <td>iPhone 8 </td> + <td>32.2 ms (0.8 ms)</td> + </tr> + <tr> + <td> + <a href="http://download.tensorflow.org/models/mobilenet_v1_2018_08_02/mobilenet_v1_1.0_224_quant.tgz)">Mobilenet_1.0_224 (quant)</a> + </td> + <td>iPhone 8 </td> + <td>24.4 ms (0.8 ms)</td> + </tr> + <tr> + <td> + <a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/model_zoo/upload_20180427/nasnet_mobile_2018_04_27.tgz">NASNet mobile</a> + </td> + <td>iPhone 8 </td> + <td>60.3 ms (0.6 ms)</td> + </tr> + <tr> + <td> + <a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/model_zoo/upload_20180427/squeezenet_2018_04_27.tgz">SqueezeNet</a> + </td> + <td>iPhone 8 </td> + <td>44.3 (0.7 ms)</td> + </tr> + <tr> + <td> + <a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/model_zoo/upload_20180427/inception_resnet_v2_2018_04_27.tgz">Inception_ResNet_V2</a> + </td> + <td>iPhone 8</td> + <td>562.4 ms (18.2 ms)</td> + </tr> + <tr> + <td> + <a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/model_zoo/upload_20180427/inception_v4_2018_04_27.tgz">Inception_V4</a> + </td> + <td>iPhone 8 </td> + <td>661.0 ms (29.2 ms)</td> + </tr> + </table> |