From 2f963277f8730be27ae1019145db30519023095b Mon Sep 17 00:00:00 2001 From: jingwen Date: Wed, 11 Apr 2018 20:18:44 -0700 Subject: Documentation for android_instrumentation_test on docs.bazel.build This document is targeted at intermediate/advanced Android + Bazel users, so it assumes that the user has a grasp of android_* rules, external dependency management and Android testing. RELNOTES: None. PiperOrigin-RevId: 192550803 --- site/docs/android-instrumentation-test.md | 538 ++++++++++++++++++++++++++++++ 1 file changed, 538 insertions(+) create mode 100644 site/docs/android-instrumentation-test.md (limited to 'site/docs/android-instrumentation-test.md') diff --git a/site/docs/android-instrumentation-test.md b/site/docs/android-instrumentation-test.md new file mode 100644 index 0000000000..b331b85226 --- /dev/null +++ b/site/docs/android-instrumentation-test.md @@ -0,0 +1,538 @@ +--- +layout: documentation +title: Android Instrumentation Tests +--- + +# Android Instrumentation Tests + +_If you're new to Bazel, please start with the [Building Android with +Bazel](https://docs.bazel.build/versions/master/tutorial/android-app.html) +tutorial._ + +![Running Android instrumentation tests in parallel](/assets/android_test.gif) + +[`android_instrumentation_test`](https://docs.bazel.build/versions/master/be/android.html#android_instrumentation_test) +allows developers to test their apps on Android emulators and devices. +It utilizes real Android framework APIs and the Android Test Library. + +For hermeticity and reproducibility, Bazel creates and launches Android +emulators in a sandbox, ensuring that tests always run from a clean state. Each +test gets an isolated emulator instance, allowing tests to run in parallel +without passing states between them. + +For more information on Android instrumentation tests, check out the [Android +developer +documentation](https://developer.android.com/training/testing/unit-testing/instrumented-unit-tests.html). + +The current state is __experimental__ as of Bazel 0.12.0. Please file issues in +the [GitHub issue tracker](https://github.com/bazelbuild/bazel/issues). + +**Table of Contents** + +- [How it works](#how-it-works) +- [Prerequisites](#prerequisites) +- [Getting started](#getting-started) + - [`BUILD` file](#build-file) + - [`WORKSPACE` dependencies](#workspace-dependencies) +- [Maven dependencies](#maven-dependencies) +- [Choosing an `android_device` target](#choosing-an-android_device-target) +- [Running tests](#running-tests) + - [Headless testing](#headless-testing) + - [GUI testing](#gui-testing) + - [Testing with a local emulator or device](#testing-with-a-local-emulator-or-device) +- [Sample projects](#sample-projects) +- [Tips](#tips) + - [Reading test logs](#reading-test-logs) + - [Testing against multiple API levels](#testing-against-multiple-api-levels) +- [Known issues](#known-issues) +- [Planned features](#planned-features) + +# How it works + +When you run `bazel test` on an `android_instrumentation_test` target for the +first time, Bazel performs the following steps: + +1. Builds the test APK, APK under test, and their transitive dependencies +2. Creates, boots, and caches clean emulator states +3. Starts the emulator +4. Installs the APKs +5. Runs tests utilizing the [Android Test Orchestrator](https://developer.android.com/training/testing/junit-runner.html#using-android-test-orchestrator) +6. Shuts down the emulator +7. Reports the results + +In subsequent test runs, Bazel boots the emulator from the clean, cached state +created in step 2, so there are no leftover states from previous runs. Caching +emulator state also speeds up test runs. + +# Prerequisites + +Ensure your enivornment satisfies the following prerequisites: + +- **Linux**. Tested on Ubuntu 14.04 and 16.04. + +- **Bazel 0.12.0** or later. Verify the version by running `bazel info release`. + +``` +$ bazel info release +release 0.12.0 +``` + +- **KVM**. Bazel requires emulators to have [hardware + acceleration](https://developer.android.com/studio/run/emulator-acceleration.html#accel-check) + with KVM on Linux. You can follow these + [installation instructions](https://help.ubuntu.com/community/KVM/Installation) + for Ubuntu. Run `apt-get install cpu-checker && kvm-ok` to verify that KVM has + the correct configuration. If it prints the following message, you're good to + go: + +``` +$ kvm-ok +INFO: /dev/kvm exists +KVM acceleration can be used +``` + +- **Xvfb**. To run headless tests (for example, on CI servers), Bazel requires + the [X virtual framebuffer](https://www.x.org/archive/X11R7.6/doc/man/man1/Xvfb.1.xhtml). + Install it by running `apt-get install xvfb`. Verify that `Xvfb` is installed + correctly by running `which Xvfb` and ensure that it's installed at + `/usr/bin/Xvfb`: + +``` +$ which Xvfb +/usr/bin/Xvfb +``` + +- **Maven**. Bazel uses `maven` to download JARs and AARs from Maven + repositories such as [Google Maven](https://maven.google.com). Install it by + running `apt-get install maven`. Run `which mvn` and ensure that it's + installed at `/usr/bin/mvn`: + +``` +$ which mvn +/usr/bin/mvn +``` + +# Getting started + +Here is a typical target dependency graph of an `android_instrumentation_test`: + +![The target dependency graph on an Android instrumentation test](/assets/android_instrumentation_test.png) + +## `BUILD` file + +The graph translates into a `BUILD` file like this: + +```python +load("@gmaven_rules//:defs.bzl", "gmaven_artifact") + +android_instrumentation_test( + name = "my_test", + test_app = ":my_test_app", + target_device = "@android_test_support//tools/android/emulated_devices/generic_phone:android_23_x86_qemu2", +) + +# Test app and library +android_binary( + name = "my_test_app", + instruments = ":my_app", + manifest = "AndroidTestManifest.xml", + deps = [":my_test_lib"], + # ... +) + +android_library( + name = "my_test_lib", + srcs = glob(["javatest/**/*.java"]), + deps = [ + ":my_app_lib", + gmaven_artifact("com.android.support.test.espresso:espresso_core:aar:3.0.1"), + ], + # ... +) + +# Target app and library under test +android_binary( + name = "my_app", + manifest = "AndroidManifest.xml", + deps = [":my_app_lib"], + # ... +) + +android_library( + name = "my_app_lib", + srcs = glob(["java/**/*.java"]), + deps = [ + gmaven_artifact("com.android.support:design:aar:27.0.2"), + gmaven_artifact("com.android.support:support_annotations:jar:27.0.2"), + ] + # ... +) +``` + +The main attributes of the rule `android_instrumentation_test` are: + +- `test_app`: An `android_binary` target. This target contains test code and + dependencies like Espresso and UIAutomator. The selected `android_binary` + target is required to specify an `instruments` attribute pointing to another + `android_binary`, which is the app under test. + +- `target_device`: An `android_device` target. This target describes the + specifications of the Android emulator which Bazel uses to create, launch and + run the tests. See the [section on choosing an Android + device](#choosing-an-android_device) for more information. + +## `WORKSPACE` dependencies + +In order to use this rule, your project needs to depend on these external +repositories: + +- `@androidsdk`: The Android SDK. Download this through Android Studio. + +- `@android_test_support`: Hosts the test runner, emulator launcher, and + `android_device` targets. + +- `@gmaven_rules`: Defines the `maven_jar` and `maven_aar` targets available on + the [Google Maven repository](https://maven.google.com). + +You can enable these dependencies by adding the following lines to your +`WORKSPACE` file: + +```python +# Android SDK +android_sdk_repository( + name = "androidsdk", + path = "/path/to/sdk", # or set ANDROID_HOME +) + +# Android Test Support +ATS_COMMIT = "$COMMIT_HASH" +http_archive( + name = "android_test_support", + strip_prefix = "android-testing-support-library-%s" % ATS_COMMIT", + urls = ["https://github.com/google/android-testing-support-library/archive/%s.tar.gz" % ATS_COMMIT], +) +load("@android_test_support//:repo.bzl", "android_test_repositories") +android_test_repositories() + +# Google Maven Repository +GMAVEN_COMMIT = "$COMMIT_HASH" +http_archive( + name = "gmaven_rules", + strip_prefix = "gmaven_rules-%s" % GMAVEN_COMMIT, + urls = ["https://github.com/bazelbuild/gmaven_rules/archive/%s.tar.gz" % GMAVEN_COMMIT], +) +load("@gmaven_rules//:gmaven.bzl", "gmaven_rules") +gmaven_rules() +``` + +# Maven dependencies + +Use the +[maven_jar](https://docs.bazel.build/versions/master/be/workspace.html#maven_jar) +repository rule for Maven dependencies not hosted on Google Maven. For example, +to use JUnit 4.12 and Hamcrest 2, add the following lines to your `WORKSPACE`: + +``` +maven_jar( + name = "junit_junit", + artifact = "junit:junit:4.12", +) + +maven_jar( + name = "org_hamcrest_java_hamcrest", + artifact = "org.hamcrest:java-hamcrest:2.0.0.0", +) +``` + +Then, you can depend on them in your `BUILD` files: + +```python +java_library( + name = "test_deps", + visibility = ["//visibility:public"], + exports = [ + "@junit_junit//jar", + "@org_hamcrest_java_hamcrest//jar", + ], +) + +android_library( + name = "my_test_lib", + srcs = [..], + deps = [":test_deps"], +) +``` + +[`bazel-deps`](https://github.com/johnynek/bazel-deps) is another useful tool +for managing Maven dependencies using [a `YAML` +file](https://github.com/johnynek/bazel-deps/blob/master/dependencies.yaml). + +For dependencies hosted on on [Google's Maven +repository](https://maven.google.com), [`@gmaven_rules`](https://github.com/bazelbuild/gmaven_rules) +provides a simple way to fetch dependencies hosted with `gmaven_artifact`. + +`gmaven_artifact` is a macro that maps an artifact's coordinate to the actual +generated target in +[`gmaven.bzl`](https://raw.githubusercontent.com/bazelbuild/gmaven_rules/master/gmaven.bzl) +(warning: big file!). The packaging type defaults to `jar` if it isn't +specified. + +Load the `gmaven_artifact` macro at the beginning of your `BUILD` file to use +it: + +```python +load("@gmaven_rules//:defs.bzl", "gmaven_artifact") + +android_library( + name = "my_app_lib", + srcs = glob(["java/**/*.java"]), + deps = [ + gmaven_artifact("com.android.support:design:aar:27.0.2"), + gmaven_artifact("com.android.support:support_annotations:jar:27.0.2"), + ] + # ... +) +``` + +# Choosing an android_device target + +`android_instrumentation_test.target_device` specifies which Android device to +run the tests on. These `android_device` targets are defined in +[`@android_test_support`](https://github.com/google/android-testing-support-library/tree/master/tools/android/emulated_devices). + +```python +$ bazel query --output=build @android_test_support//tools/android/emulated_devices/generic_phone:android_23_x86_qemu2 +# .../external/android_test_support/tools/android/emulated_devices/generic_phone/BUILD:43:1 +android_device( + name = "android_23_x86_qemu2", + visibility = ["//visibility:public"], + tags = ["requires-kvm"], + generator_name = "generic_phone", + generator_function = "make_device", + generator_location = "tools/android/emulated_devices/generic_phone/BUILD:43", + vertical_resolution = 800, + horizontal_resolution = 480, + ram = 2048, + screen_density = 240, + cache = 32, + vm_heap = 256, + system_image = "@android_test_support//tools/android/emulated_devices/generic_phone:android_23_x86_qemu2_images", + default_properties = "@android_test_support//tools/android/emulated_devices/generic_phone:_android_23_x86_qemu2_props", +) +``` + +The device target names use this template: + +``` +@android_test_support//tools/android/emulated_devices/${device_type}:${system}_${api_level}_x86_qemu2 +``` + +In order to launch an `android_device`, the `system_image` for the selected API +level is required. To download the system image, use Android SDK's +`tools/bin/sdkmanager`. For example, to download the system image for +`generic_phone:android_23_x86_qemu2`, run `$sdk/tools/bin/sdkmanager +"system-images;android-23;default;x86"`. + +To see the full list of supported `android_device` targets in +`@android_test_support`, run the following command: + +``` +bazel query 'filter("x86_qemu2$", kind(android_device, @android_test_support//tools/android/emulated_devices/...:*))' +``` + +Bazel currently supports x86-based emulators only. For better performance, +we also recommend using `QEMU2` `android_device` targets instead of `QEMU` ones. + +# Running tests + +To run tests, add these lines to your project's `tools/bazel.rc` file. + +``` +# Configurations for testing with Bazel +# Select a configuration by running +# `bazel test //my:target --config={headless, gui, local_device}` + +# Headless instrumentation tests +test:headless --spawn_strategy=local +test:headless --test_arg=--enable_display=false + +# Graphical instrumentation tests. Ensure that $DISPLAY is set. +test:gui --test_env=DISPLAY +test:gui --test_arg=--enable_display=true + +# Testing with a local emulator or device. Ensure that `adb devices` lists the +# device. +# Run tests serially. +test:local_device --test_strategy=exclusive +# Use the local device broker type, as opposed to WRAPPED_EMULATOR. +test:local_device --test_arg=--device_broker_type=LOCAL_ADB_SERVER +# Uncomment and set $device_id if there is more than one connected device. +# test:local_device --test_arg=--device_serial_number=$device_id +``` + +Then, use one of the configurations to run tests: + +- `bazel test //my/test:target --config=headless` +- `bazel test //my/test:target --config=gui` +- `bazel test //my/test:target --config=local_device` + +Use __only one configuration__ or tests will fail. + +## Headless testing + +With `Xvfb`, it is possible to run headless emulators. However, due to a known +issue between Xvfb and Bazel's sandbox, sandboxing fails with headless tests. +Pass the flag `--spawn_strategy=local` to disable sandboxing for the testing to +work. + +## GUI testing + +If the `$DISPLAY` environment variable is set, it's also possible to enable the +graphical interface of the emulator while the test is running. Pass the variable +to the test environment using the flag `--test_env=DISPLAY`. + +## Testing with a local emulator or device + +Bazel also supports testing directly on a locally launched emulator or connected +device. Pass the flags +`--test_strategy=exclusive` and +`--test_arg=--device_broker_type=LOCAL_ADB_SERVER` to enable local testing mode. +If there is more than one connected device, pass the flag +`--test_arg=--device_serial_number=$device_id` where `$device_id` is the id of +the device/emulator listed in `adb devices`. + +# Sample projects + +If you are looking for canonical project samples, see the [Android testing +samples](https://github.com/googlesamples/android-testing#experimental-bazel-support) +for projects using Espresso and UIAutomator. + +``` +$ git clone https://github.com/googlesamples/android-testing && cd android-testing +# Set path to Android SDK in WORKSPACE +$ bazel test //ui/... --config=headless +INFO: Analysed 45 targets (1 packages loaded). +INFO: Found 36 targets and 9 test targets... + +... + +INFO: Elapsed time: 195.665s, Critical Path: 195.22s +INFO: Build completed successfully, 417 total actions +//ui/espresso/BasicSample:BasicSampleInstrumentationTest PASSED in 103.7s +//ui/espresso/CustomMatcherSample:CustomMatcherSampleInstrumentationTest PASSED in 113.2s +//ui/espresso/DataAdapterSample:DataAdapterSampleInstrumentationTest PASSED in 110.2s +//ui/espresso/IdlingResourceSample:IdlingResourceSampleInstrumentationTest PASSED in 102.3s +//ui/espresso/IntentsAdvancedSample:IntentsAdvancedSampleInstrumentationTest PASSED in 98.3s +//ui/espresso/IntentsBasicSample:IntentsBasicSampleInstrumentationTest PASSED in 103.3s +//ui/espresso/MultiWindowSample:MultiWindowSampleInstrumentationTest PASSED in 108.3s +//ui/espresso/RecyclerViewSample:RecyclerViewSampleInstrumentationTest PASSED in 102.9s +//ui/uiautomator/BasicSample:BasicSampleInstrumentationTest PASSED in 122.6s +``` + +# Tips + +## Reading test logs + +Use `--test_output=errors` to print logs for failing tests, or +`--test_output=all` to print all test output. If you're looking for an +individual test log, go to +`$PROJECT_ROOT/bazel-testlogs/path/to/InstrumentationTestTargetName`. + +For example, the test logs for `BasicSample` canonical project are in +`bazel-testlogs/ui/espresso/BasicSample/BasicSampleInstrumentationTest`: + +``` +$ tree bazel-testlogs/ui/espresso/BasicSample/BasicSampleInstrumentationTest +. +├── adb.409923.log +├── broker_logs +│   ├── aapt_binary.10.ok.txt +│   ├── aapt_binary.11.ok.txt +│   ├── adb.12.ok.txt +│   ├── adb.13.ok.txt +│   ├── adb.14.ok.txt +│   ├── adb.15.fail.txt +│   ├── adb.16.ok.txt +│   ├── adb.17.fail.txt +│   ├── adb.18.ok.txt +│   ├── adb.19.fail.txt +│   ├── adb.20.ok.txt +│   ├── adb.21.ok.txt +│   ├── adb.22.ok.txt +│   ├── adb.23.ok.txt +│   ├── adb.24.fail.txt +│   ├── adb.25.ok.txt +│   ├── adb.26.fail.txt +│   ├── adb.27.ok.txt +│   ├── adb.28.fail.txt +│   ├── adb.29.ok.txt +│   ├── adb.2.ok.txt +│   ├── adb.30.ok.txt +│   ├── adb.3.ok.txt +│   ├── adb.4.ok.txt +│   ├── adb.5.ok.txt +│   ├── adb.6.ok.txt +│   ├── adb.7.ok.txt +│   ├── adb.8.ok.txt +│   ├── adb.9.ok.txt +│   ├── android_23_x86_qemu2.1.ok.txt +│   └── exec-1 +│   ├── adb-2.txt +│   ├── emulator-2.txt +│   └── mksdcard-1.txt +├── device_logcat +│   └── logcat1635880625641751077.txt +├── emulator_itCqtc.log +├── outputs.zip +├── pipe.log.txt +├── telnet_pipe.log.txt +└── tmpuRh4cy + ├── watchdog.err + └── watchdog.out + +4 directories, 41 files +``` + +## Testing against multiple API levels + +If you would like to test against multiple API levels, you can use a list +comprehension to create test targets for each API level. For example: + +```python +API_LEVELS = [ + "19", + "20", + "21", + "22", +] + +[android_instrumentation_test( + name = "my_test_%s" % API_LEVEL, + test_app = ":my_test_app", + target_device = "@android_test_support//tools/android/emulated_devices/generic_phone:android_%s_x86_qemu2" % API_LEVEL, +) for API_LEVEL in API_LEVELS] +``` + +# Known issues + +- [Forked adb server processes are not terminated after + tests](https://github.com/bazelbuild/bazel/issues/4853) +- Headless mode is unstable in sandboxed mode due to Xvfb issues, use + `--spawn_strategy=local` as a workaround. +- While APK building works on all platforms (Linux, macOS, Windows), testing + only works on Linux. +- Even with `--config=local_adb`, users still need to specify + `android_instrumentation_test.target_device`. +- If using a local device or emulator, Bazel does not uninstall the APKs after + the test. Clean the packages by running this command: `adb shell pm list + packages com.example.android.testing | cut -d ':' -f 2 | tr -d '\r' | xargs + -L1 -t adb uninstall` + +# Planned features + +- Fully sandboxed headless testing +- Code coverage collection +- macOS support +- Windows support +- Improved external dependency management +- Remote test caching and execution -- cgit v1.2.3