From 53179466fa10aad99c0660b68544561a2b6faade Mon Sep 17 00:00:00 2001 From: spomorski Date: Wed, 13 Jun 2018 14:15:08 -0700 Subject: Document troubleshooting Bazel remote execution using the Docker sandbox feature. PiperOrigin-RevId: 200451804 --- site/_layouts/documentation.html | 3 +- site/docs/remex-rules-guidelines.md | 168 --------------------- site/docs/remote-caching.md | 3 +- site/docs/remote-execution-rules.md | 172 +++++++++++++++++++++ site/docs/remote-execution-sandbox.md | 272 ++++++++++++++++++++++++++++++++++ 5 files changed, 448 insertions(+), 170 deletions(-) delete mode 100644 site/docs/remex-rules-guidelines.md create mode 100644 site/docs/remote-execution-rules.md create mode 100644 site/docs/remote-execution-sandbox.md (limited to 'site') diff --git a/site/_layouts/documentation.html b/site/_layouts/documentation.html index f41ae43811..b55ad03236 100644 --- a/site/_layouts/documentation.html +++ b/site/_layouts/documentation.html @@ -230,7 +230,8 @@ nav: docs
  • Linter
  • Optimizing Performance
  • Deploying Rules
    • -
  • Guidelines for Remote Execution
    • +
  • Guidelines for Remote Execution
    • +
  • Troubleshooting Remote Execution
    • diff --git a/site/docs/remex-rules-guidelines.md b/site/docs/remex-rules-guidelines.md deleted file mode 100644 index eac255dddf..0000000000 --- a/site/docs/remex-rules-guidelines.md +++ /dev/null @@ -1,168 +0,0 @@ ---- -layout: documentation -title: Adapting Bazel Rules for Remote Execution ---- - -# Adapting Bazel Rules for Remote Execution - -Remote execution allows Bazel to execute actions on a separate platform, such as -a datacenter. A [gRPC protocol](https://github.com/googleapis/googleapis/blob/master/google/devtools/remoteexecution/v1test/remote_execution.proto) -is currently in development. You can try remote execution with [bazel-buildfarm](https://github.com/bazelbuild/bazel-buildfarm), -an open-source project that aims to provide a distributed remote execution -platform. This document is intended for Bazel users writing custom build and -test rules in Skylark who want to understand the requirements for Bazel rules in -the context of remote execution. - -This document uses the following terminology when referring to different -environment types or *platforms*: - -* **Host platform** - where Bazel runs. -* **Execution platform** - where Bazel actions run. -* **Target platform** - where the build outputs (and some actions) run. - -## Overview - -When configuring a Bazel build for remote execution, you must follow the -guidelines described in this document to ensure the build executes remotely -error-free. This is due to the nature of remote execution, namely: - -* **Isolated build actions.** Build tools do not retain state and dependencies - cannot leak between them. - -* **Diverse execution environments.** Local build configuration is not always - suitable for remote execution environments. - -This document describes the issues that can arise when implementing custom Bazel -build and test rules for remote execution and how to avoid them. It covers the -following topics: - -* [Invoking build tools through toolchain rules](#invoking-build-tools-through-toolchain-rules) -* [Managing dependencies](#managing-dependencies) -* [Managing platform-dependent binaries](#managing-platform-dependent-binaries) -* [Managing configure-style WORKSPACE rules](#managing-configure-style-workspace-rules) - -## Invoking build tools through toolchain rules - -A Bazel toolchain rule is a configuration provider that tells a build rule what -build tools, such as compilers and linkers, to use and how to configure them -using parameters defined by the rule's creator. A toolchain rule allows build -and test rules to invoke build tools in a predictable, preconfigured manner -that's compatible with remote execution. For example, use a toolchain rule -instead of invoking build tools via the `PATH`, `JAVA_HOME`, or other local -variables that may not be set to equivalent values (or at all) in the remote -execution environment. - -Toolchain rules currently exist for Bazel build and test rules for -[Scala](https://github.com/bazelbuild/rules_scala/blob/master/scala/scala_toolch -ain.bzl), -[Rust](https://github.com/bazelbuild/rules_rust/blob/master/rust/toolchain.bzl), -and [Go](https://github.com/bazelbuild/rules_go/blob/master/go/toolchains.rst), -and new toolchain rules are under way for other languages and tools such as -[bash](https://docs.google.com/document/d/e/2PACX-1vRCSB_n3vctL6bKiPkIa_RN_ybzoAccSe0ic8mxdFNZGNBJ3QGhcKjsL7YKf-ngVyjRZwCmhi_5KhcX/pub). -If a toolchain rule does not exist for the tool your rule uses, consider [creating a custom toolchain rule](https://docs.bazel.build/versions/master/toolchains.html#creating-a-toolchain-rule). - -## Managing implicit dependencies - -If a build tool can access dependencies across build actions, those actions will -fail when remotely executed because each remote build action is executed -separately from others. Some build tools retain state across build actions and -access dependencies that have not been explicitly included in the tool -invocation, which will cause remotely executed build actions to fail. - -For example, when Bazel instructs a stateful compiler to locally build _foo_, -the compiler retains references to foo's build outputs. When Bazel then -instructs the compiler to build _bar_, which depends on _foo_, without -explicitly stating that dependency in the BUILD file for inclusion in the -compiler invocation, the action executes successfully as long as the same -compiler instance executes for both actions (as is typical for local execution). -However, since in a remote execution scenario each build action executes a -separate compiler instance, compiler state and _bar_'s implicit dependency on -_foo_ will be lost and the build will fail. - -## Managing platform-dependent binaries - -Typically, a binary built on the host platform cannot safely execute on an -arbitrary remote execution platform due to potentially mismatched dependencies. -For example, the SingleJar binary supplied with Bazel targets the host platform. -However, for remote execution, SingleJar must be compiled as part of the process -of building your code so that it targets the remote execution platform. (See the -[target selection -logic](https://github.com/bazelbuild/bazel/blob/130aeadfd660336572c3da397f1f107f -0c89aa8d/tools/jdk/BUILD#L115).) - -Do not ship binaries of build tools required by your build with your source code -unless you are sure they will safely run in your execution platform. Instead, do -one of the following: - -* Ship or externally reference the source code for the tool so that it can be - built for the remote execution platform. - -* Pre-install the tool into the remote execution environment (for example, a - toolchain container) if it's stable enough and use toolchain rules to run it - in your build. - -## Managing `configure`-style WORKSPACE rules - -Bazel's `WORKSPACE` rules can be used for probing the host platform for tools -and libraries required by the build, which, for local builds, is also Bazel's -execution platform. If the build explicitly depends on local build tools and -artifacts, it will fail during remote execution if the remote execution platform -is not identical to the host platform. - -The following actions performed by `WORKSPACE` rules are not compatible with -remote execution: - -* **Building binaries.** Executing compilation actions in `WORKSPACE` rules - results in binaries that are incompatible with the remote execution platform - if different from the host platform. - -* **Installing `pip` packages.** `pip` packages installed via `WORKSPACE` - rules require that their dependencies be pre-installed on the host platform. - Such packages, built specifically for the host platform, will be - incompatible with the remote execution platform if different from the host - platform. - -* **Symlinking to local tools or artifacts.** Symlinks to tools or libraries - installed on the host platform created via `WORKSPACE` rules will cause the - build to fail on the remote execution platform as Bazel will not be able to - locate them. Instead, create symlinks using standard build actions so that - the symlinked tools and libraries are accessible from Bazel's `runfiles` - tree. Do not use [`repository_ctx.symlink`](https://docs.bazel.build/versions/master/skylark/lib/repository_ctx.html#symlink) - to symlink target files outside of the external repo directory. - -* **Mutating the host platform.** Avoid creating files outside of the Bazel - `runfiles` tree, creating environment variables, and similar actions, as - they may behave unexpectedly on the remote execution platform. - -If an external dependency executes specific operations dependent on the host -platform, we recommend splitting those operations between `WORKSPACE` and build -rules as follows: - -* **Platform inspection and dependency enumeration.** These operations are - safe to execute locally via `WORKSPACE` rules, which can check which - libraries are installed, download packages that must be built, and prepare - required artifacts for compilation. For remote execution, these rules must - also support using pre-checked artifacts to provide the information that - would normally be obtained during host platform inspection. Pre-checked - artifacts allow Bazel to describe dependencies as if they were local. Use - conditional statements or the `--override_repository` flag for this. - -* **Generating or compiling target-specific artifacts and platform mutation**. - These operations must be executed via regular build rules. Actions that - produce target-specific artifacts for external dependencies must execute - during the build. - -To more easily generate pre-checked artifacts for remote execution, you can use -`WORKSPACE` rules to emit generated files. You can run those rules on each new -execution environment, such as inside each toolchain container, and check the -outputs of your remote execution build in to your source repo to reference. - -For example, for Tensorflow's rules for [`cuda`](https://github.com/tensorflow/tensorflow/blob/master/third_party/gpus/cuda_configure.bzl) -and [`python`](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl), -the `WORKSPACE` rules produce the following [`BUILD files`](https://github.com/tensorflow/tensorflow/tree/master/third_party/toolchains/cpus/py). -For local execution, files produced by checking the host environment are used. -For remote execution, a [conditional statement](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl#L304) -on an environment variable allows the rule to use files that are checked into -the repo. The `BUILD` files declare [`genrules`](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl#L84\) -that can run both locally and remotely, and perform the necessary processing -that was previously done via `repository_ctx.symlink` as shown [here](https://github.com/tensorflow/tensorflow/blob/d1ba01f81d8fa1d0171ba9ce871599063d5c7eb9/third_party/gpus/cuda_configure.bzl#L730). diff --git a/site/docs/remote-caching.md b/site/docs/remote-caching.md index 7900f573d4..416550f63c 100644 --- a/site/docs/remote-caching.md +++ b/site/docs/remote-caching.md @@ -374,7 +374,8 @@ separate platform, such as a datacenter. You can try remote execution with [Buildfarm], an open source project that aims to provide a distributed remote execution platform. -[Adapting Bazel Rules for Remote Execution](/remex-rules-guidelines.html) +[Adapting Rules for Remote Execution](https://docs.bazel.build/versions/master/remote-execution-rules.html) +[Troubleshooting Remote Execution](https://docs.bazel.build/versions/master/remote-execution-sandbox.html) [WebDAV module]: http://nginx.org/en/docs/http/ngx_http_dav_module.html [docker image]: https://hub.docker.com/r/buchgr/bazel-remote-cache/ [GitHub]: https://github.com/buchgr/bazel-remote/ diff --git a/site/docs/remote-execution-rules.md b/site/docs/remote-execution-rules.md new file mode 100644 index 0000000000..a069fc370d --- /dev/null +++ b/site/docs/remote-execution-rules.md @@ -0,0 +1,172 @@ +--- +layout: documentation +title: Adapting Bazel Rules for Remote Execution +--- + +# Adapting Bazel Rules for Remote Execution + +Remote execution allows Bazel to execute actions on a separate platform, such as +a datacenter. A [gRPC protocol](https://github.com/googleapis/googleapis/blob/master/google/devtools/remoteexecution/v1test/remote_execution.proto) +is currently in development. You can try remote execution with [bazel-buildfarm](https://github.com/bazelbuild/bazel-buildfarm), +an open-source project that aims to provide a distributed remote execution +platform. This document is intended for Bazel users writing custom build and +test rules in Skylark who want to understand the requirements for Bazel rules in +the context of remote execution. + +This document uses the following terminology when referring to different +environment types or *platforms*: + +* **Host platform** - where Bazel runs. +* **Execution platform** - where Bazel actions run. +* **Target platform** - where the build outputs (and some actions) run. + +## Overview + +When configuring a Bazel build for remote execution, you must follow the +guidelines described in this document to ensure the build executes remotely +error-free. This is due to the nature of remote execution, namely: + +* **Isolated build actions.** Build tools do not retain state and dependencies + cannot leak between them. + +* **Diverse execution environments.** Local build configuration is not always + suitable for remote execution environments. + +This document describes the issues that can arise when implementing custom Bazel +build and test rules for remote execution and how to avoid them. It covers the +following topics: + +* [Invoking build tools through toolchain rules](#invoking-build-tools-through-toolchain-rules) +* [Managing dependencies](#managing-dependencies) +* [Managing platform-dependent binaries](#managing-platform-dependent-binaries) +* [Managing configure-style WORKSPACE rules](#managing-configure-style-workspace-rules) + +## Invoking build tools through toolchain rules + +A Bazel toolchain rule is a configuration provider that tells a build rule what +build tools, such as compilers and linkers, to use and how to configure them +using parameters defined by the rule's creator. A toolchain rule allows build +and test rules to invoke build tools in a predictable, preconfigured manner +that's compatible with remote execution. For example, use a toolchain rule +instead of invoking build tools via the `PATH`, `JAVA_HOME`, or other local +variables that may not be set to equivalent values (or at all) in the remote +execution environment. + +Toolchain rules currently exist for Bazel build and test rules for +[Scala](https://github.com/bazelbuild/rules_scala/blob/master/scala/scala_toolch +ain.bzl), +[Rust](https://github.com/bazelbuild/rules_rust/blob/master/rust/toolchain.bzl), +and [Go](https://github.com/bazelbuild/rules_go/blob/master/go/toolchains.rst), +and new toolchain rules are under way for other languages and tools such as +[bash](https://docs.google.com/document/d/e/2PACX-1vRCSB_n3vctL6bKiPkIa_RN_ybzoAccSe0ic8mxdFNZGNBJ3QGhcKjsL7YKf-ngVyjRZwCmhi_5KhcX/pub). +If a toolchain rule does not exist for the tool your rule uses, consider [/toolchains.html#creating-a-toolchain-rule). + +## Managing implicit dependencies + +If a build tool can access dependencies across build actions, those actions will +fail when remotely executed because each remote build action is executed +separately from others. Some build tools retain state across build actions and +access dependencies that have not been explicitly included in the tool +invocation, which will cause remotely executed build actions to fail. + +For example, when Bazel instructs a stateful compiler to locally build _foo_, +the compiler retains references to foo's build outputs. When Bazel then +instructs the compiler to build _bar_, which depends on _foo_, without +explicitly stating that dependency in the BUILD file for inclusion in the +compiler invocation, the action executes successfully as long as the same +compiler instance executes for both actions (as is typical for local execution). +However, since in a remote execution scenario each build action executes a +separate compiler instance, compiler state and _bar_'s implicit dependency on +_foo_ will be lost and the build will fail. + +To help detect and eliminate these dependency problems, Bazel 0.14.1 offers the +local Docker sandbox, which has the same restrictions for dependencies as remote +execution. Use the sandbox to prepare your build for remote execution by +identifying and resolving dependency-related build errors. See [Troubleshooting Bazel Remote Execution with Docker Sandbox](/remote-execution-sandbox.html) +for more information. + +## Managing platform-dependent binaries + +Typically, a binary built on the host platform cannot safely execute on an +arbitrary remote execution platform due to potentially mismatched dependencies. +For example, the SingleJar binary supplied with Bazel targets the host platform. +However, for remote execution, SingleJar must be compiled as part of the process +of building your code so that it targets the remote execution platform. (See the +[target selection logic](https://github.com/bazelbuild/bazel/blob/130aeadfd660336572c3da397f1f107f0c89aa8d/tools/jdk/BUILD#L115).) + +Do not ship binaries of build tools required by your build with your source code +unless you are sure they will safely run in your execution platform. Instead, do +one of the following: + +* Ship or externally reference the source code for the tool so that it can be + built for the remote execution platform. + +* Pre-install the tool into the remote execution environment (for example, a + toolchain container) if it's stable enough and use toolchain rules to run it + in your build. + +## Managing `configure`-style WORKSPACE rules + +Bazel's `WORKSPACE` rules can be used for probing the host platform for tools +and libraries required by the build, which, for local builds, is also Bazel's +execution platform. If the build explicitly depends on local build tools and +artifacts, it will fail during remote execution if the remote execution platform +is not identical to the host platform. + +The following actions performed by `WORKSPACE` rules are not compatible with +remote execution: + +* **Building binaries.** Executing compilation actions in `WORKSPACE` rules + results in binaries that are incompatible with the remote execution platform + if different from the host platform. + +* **Installing `pip` packages.** `pip` packages installed via `WORKSPACE` + rules require that their dependencies be pre-installed on the host platform. + Such packages, built specifically for the host platform, will be + incompatible with the remote execution platform if different from the host + platform. + +* **Symlinking to local tools or artifacts.** Symlinks to tools or libraries + installed on the host platform created via `WORKSPACE` rules will cause the + build to fail on the remote execution platform as Bazel will not be able to + locate them. Instead, create symlinks using standard build actions so that + the symlinked tools and libraries are accessible from Bazel's `runfiles` + tree. Do not use [`repository_ctx.symlink`](https://docs.bazel.build/versions/master/skylark/lib/repository_ctx.html#symlink) + to symlink target files outside of the external repo directory. + +* **Mutating the host platform.** Avoid creating files outside of the Bazel + `runfiles` tree, creating environment variables, and similar actions, as + they may behave unexpectedly on the remote execution platform. + +If an external dependency executes specific operations dependent on the host +platform, we recommend splitting those operations between `WORKSPACE` and build +rules as follows: + +* **Platform inspection and dependency enumeration.** These operations are + safe to execute locally via `WORKSPACE` rules, which can check which + libraries are installed, download packages that must be built, and prepare + required artifacts for compilation. For remote execution, these rules must + also support using pre-checked artifacts to provide the information that + would normally be obtained during host platform inspection. Pre-checked + artifacts allow Bazel to describe dependencies as if they were local. Use + conditional statements or the `--override_repository` flag for this. + +* **Generating or compiling target-specific artifacts and platform mutation**. + These operations must be executed via regular build rules. Actions that + produce target-specific artifacts for external dependencies must execute + during the build. + +To more easily generate pre-checked artifacts for remote execution, you can use +`WORKSPACE` rules to emit generated files. You can run those rules on each new +execution environment, such as inside each toolchain container, and check the +outputs of your remote execution build in to your source repo to reference. + +For example, for Tensorflow's rules for [`cuda`](https://github.com/tensorflow/tensorflow/blob/master/third_party/gpus/cuda_configure.bzl) +and [`python`](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl), +the `WORKSPACE` rules produce the following [`BUILD files`](https://github.com/tensorflow/tensorflow/tree/master/third_party/toolchains/cpus/py). +For local execution, files produced by checking the host environment are used. +For remote execution, a [conditional statement](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl#L304) +on an environment variable allows the rule to use files that are checked into +the repo. The `BUILD` files declare [`genrules`](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl#L84\) +that can run both locally and remotely, and perform the necessary processing +that was previously done via `repository_ctx.symlink` as shown [here](https://github.com/tensorflow/tensorflow/blob/d1ba01f81d8fa1d0171ba9ce871599063d5c7eb9/third_party/gpus/cuda_configure.bzl#L730). diff --git a/site/docs/remote-execution-sandbox.md b/site/docs/remote-execution-sandbox.md new file mode 100644 index 0000000000..a7adbbf92b --- /dev/null +++ b/site/docs/remote-execution-sandbox.md @@ -0,0 +1,272 @@ +--- +layout: documentation +title: Troubleshooting Bazel Remote Execution with Docker Sandbox + +--- + +# Troubleshooting Bazel Remote Execution with Docker Sandbox + +Contents + +* [Overview](#overview) +* [Prerequisites](#prerequisites) +* [Troubleshooting on the local machine](#troubleshooting-on-the-local-machine) + * [Step 1: Run the build](#step-1-run-the-build) + * [Step 2: Resolve detected issues](#step-2-resolve-detected-issues) +* [Troubleshooting in a Docker container](#troubleshooting-in-a-docker-container) + * [Step 1: Build the container](#step-1-build-the-container) + * [Step 2: Start the container](#step-2-start-the-container) + * [Step 3: Test the container](#step-3-test-the-container) + * [Step 4: Run the build](#step-4-run-the-build) + * [Step 5: Resolve detected issues](#step-5-resolve-detected-issues) + +## Overview + +Bazel builds that succeed locally may fail when executed remotely due to +restrictions and requirements that do not affect local builds. The most common +causes of such failures are described in [Adapting Bazel Rules for Remote Execution](https://docs.bazel.build/versions/master/remote-execution-rules.html). + +This document describes how to identify and resolve the most common issues that +arise with remote execution using the Docker sandbox feature, which imposes +restrictions upon the build equal to those of remote execution. This allows you +to troubleshoot your build without the need for a remote execution service. + +The Docker sandbox feature mimics the restrictions of remote execution as +follows: + +* **Build actions execute in toolchain containers.** You can use the same + toolchain containers to run your build locally and remotely via a service + supporting containerized remote execution. + +* **No extraneous data crosses the container boundary.** Only explicitly + declared inputs and outputs enter and leave the container, and only after + the associated build action successfully completes. + +* **Each action executes in a fresh container.** A new, unique container is + created for each spawned build action. + +**Note:** Builds take noticeably more time to complete when the Docker sandbox +feature is enabled. This is normal. + +You can troubleshoot these issues using one of the following methods: + +* **[Troubleshooting natively.](#troubleshooting-natively)** With this method, + Bazel and its build actions run natively on your local machine. The Docker + sandbox feature imposes restrictions upon the build equal to those of remote + execution. However, this method will not detect local tools, states, and + data leaking into your build, which will cause problems with remote execution. + +* **[Troubleshooting in a Docker container.](#troubleshooting-in-a-docker-container)** + With this method, Bazel and its build actions run inside a Docker container, + which allows you to detect tools, states, and data leaking from the local + machine into the build in addition to imposing restrictions + equal to those of remote execution. This method provides insight into your + build even if portions of the build are failing. This method is experimental + and not officially supported. + +## Prerequisites + +Before you begin troubleshooting, do the following if you have not already done so: + +* Install Docker and configure the permissions required to run it. +* Install Bazel 0.14.1 or later. Earlier versions do not support the Docker + sandbox feature. +* Add the [bazel-toolchains](https://releases.bazel.build/bazel-toolchains.html) + repo, pinned to the latest release version, to your build's `WORKSPACE` file + as described [here](https://releases.bazel.build/bazel-toolchains.html). +* Add the following lines to your `.bazelrc` file. Create the file in the root + directory of your Bazel project if it does not exist. + +``` +# Docker Sandbox Mode +build:docker-sandbox --host_javabase=@bazel_toolchains//configs/ubuntu16_04_clang/1.0:jdk8 +build:docker-sandbox --javabase=@bazel_toolchains//configs/ubuntu16_04_clang/1.0:jdk8 +build:docker-sandbox --crosstool_top=@bazel_toolchains//configs/ubuntu16_04_clang/1.0/bazel_0.13.0/default:toolchain +build:docker-sandbox --experimental_docker_image=gcr.io/cloud-marketplace/google/rbe-ubuntu16-04@sha256:59bf0e191a6b5cc1ab62c2224c810681d1326bad5a27b1d36c9f40113e79da7f +build:docker-sandbox --spawn_strategy=docker --strategy=Javac=docker --genrule_strategy=docker +build:docker-sandbox --define=EXECUTOR=remote +build:docker-sandbox --experimental_docker_verbose +build:docker-sandbox --experimental_enable_docker_sandbox +``` + +**Note:** The flags referenced in the `.bazelrc` file shown above are configured +to run within the [rbe-ubuntu16-04](https://console.cloud.google.com/launcher/details/google/rbe-ubuntu16-04?_ga=2.231599325.-396882850.1508873814) +container. + +If your rules require additional tools, do the following: + +1. Create a custom Docker container by installing tools using a [Dockerfile](https://docs.docker.com/engine/reference/builder/) + and [building](https://docs.docker.com/engine/reference/commandline/build/) + the image locally. + +2. Replace the value of the `--experimental_docker_image` flag above with the + name of your custom container image. + + +## Troubleshooting natively + +This method executes Bazel and all of its build actions directly on the local +machine and is a reliable way to confirm whether your build will succeed when +executed remotely. + +However, with this method, locally installed tools, binaries,and data may leak +into into your build, especially if it uses [configure-style WORKSPACE rules](https://docs.bazel.build/versions/master/remote-execution-rules.html#managing-configure-style-workspace-rules). +Such leaks will cause problems with remote execution; to detect them, [troubleshoot in a Docker container](#troubleshooting-in-a-docker-container) +in addition to troubleshooting natively. + +### Step 1: Run the build + +1. Add the `--config=docker-sandbox` flag to the Bazel command that executes + your build. For example: + + ``` + bazel --bazelrc=.bazelrc build --config=docker-sandbox + + ``` + +2. Run the build and wait for it to complete. The build will run up to four + times slower than normal due to the Docker sandbox feature. + +You may encounter the following error: + +``` +ERROR: 'docker' is an invalid value for docker spawn strategy. +``` + +If you do, run the build again with the `--experimental_docker_verbose` flag. +This flag enables verbose error messages. This error is typically caused by a +faulty Docker installation or lack of permissions to execute it under the +current user account. See the [Docker documentation](https://docs.docker.com/install/linux/linux-postinstall/) +for more information. If problems persist, skip ahead to [Troubleshooting in a Docker container](#troubleshooting-in-a-docker-container). + +### Step 2: Resolve detected issues + +The following are the most commonly encountered issues and their workarounds. + +* **A file, tool, binary, or resource referenced by the Bazel runfiles tree is + missing.**. Confirm that all dependencies of the affected targets have been [explicitly declared](/build-ref.html#dependencies). + See [Managing implicit dependencies](/remote-execution-rules.html#managing-implicit-dependencies) + for more information. + +* **A file, tool, binary, or resource referenced by an absolute path or the `PATH` + variable is missing.** Confirm that all required tools are installed within + the toolchain container and use [toolchain rules](/toolchains.html) to properly + declare dependencies pointing to the missing resource. See [Invoking build tools through toolchain rules](/remote-execution-rules.html#invoking-build-tools-through-toolchain-rules) + for more information. + +* **A binary execution fails.** One of the build rules is referencing a binary + incompatible with the execution environment (the Docker container). See [Managing platform-dependent binaries](/remote-execution-rules.html#managing-platform-dependent-binaries) + for more information. If you cannot resolve the issue, contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) + for help. + +* **A file from `@local-jdk` is missing or causing errors.** The Java binaries + on your local machine are leaking into the build while being incompatible with + it. Use [`java_toolchain`](https://docs.bazel.build/versions/master/be/java.html#java_toolchain) + in your rules and targets instead of `@local_jdk`. Contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) if you need further help. + +* **Other errors.** Contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) for help. + +## Troubleshooting in a Docker container + +With this method, Bazel runs inside a host Docker container, and Bazel's build +actions execute inside individual toolchain containers spawned by the Docker +sandbox feature. The sandbox spawns a brand new toolchain container for each +build action and only one action executes in each toolchain container. + +This method provides more granular control of tools installed in the host +environment. By separating the execution of the build from the execution of its +build actions and keeping the installed tooling to a minimum, you can verify +whether your build has any dependencies on the local execution environment. + +### Step 1: Build the container + +**Note:** The commands below are tailored specifically for a `debian:stretch` base. +For other bases, modify them as necessary. + +1. Create a `Dockerfile` that creates the Docker container and installs Bazel + with a minimal set of build tools: + + ``` + FROM debian:stretch + + RUN apt-get update && apt-get install -y apt-transport-https curl software-properties-common git gcc gnupg2 g++ openjdk-8-jdk-headless python-dev zip wget vim + + RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add - + + RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" + + RUN apt-get update && apt-get install -y docker-ce + + RUN wget https://releases.bazel.build/0.14.1/release/bazel-0.14.1-installer-linux-x86_64.sh -O ./bazel-installer.sh && chmod 755 ./bazel-installer.sh + + RUN ./bazel-installer.sh + ``` + +2. Build the container as `bazel_container`: + + ``` + docker build -t bazel_container - < Dockerfile + ``` + +### Step 2: Start the container + +Start the Docker container using the command shown below. In the command, +substitute the path to the source code on your host that you want to build. + +``` +docker run -it \ + -v /var/run/docker.sock:/var/run/docker.sock \ + -v /tmp:/tmp \ + -v :/src \ + -w /src \ + bazel_container \ + /bin/bash +``` + +This command runs the container as root, mapping the docker socket, and mounting +the `/tmp` directory. This allows Bazel to spawn other Docker containers and to +use directories under `/tmp` to share files with those containers. Your source +code is available at `/src` inside the container. + +The command intentionally starts from a `debian:stretch` base container that +includes binaries incompatible with the `rbe-ubuntu16-04` container used as a +toolchain container. If binaries from the local environment are leaking into the +toolchain container, they will cause build errors. + +### Step 3: Test the container + +Run the following commands from inside the Docker container to test it: + +``` +docker ps + +bazel version +``` + +### Step 4: Run the build + +Run the build as shown below. The output user is root so that it corresponds to +a directory that is accessible with the same absolute path from inside the host +container in which Bazel runs, from the toolchain containers spawned by the Docker +sandbox feature in which Bazel's build actions are running, and from the local +machine on which the host and action containers run. + +``` +bazel --output_user_root=/tmp/bazel_docker_root --bazelrc=.bazelrc \ build --config=docker-sandbox +``` + +### Step 5: Resolve detected issues + +You can resolve build failures as follows: + +* If the build fails with an "out of disk space" error, you can increase this + limit by starting the host container with the flag `--memory=XX` where `XX` + is the allocated disk space in gigabytes. This is experimental and may + result in unpredictable behavior. + +* If the build fails during the analysis or loading phases, one or more of + your build rules declared in the WORKSPACE file are not compatible with + remote execution. See [Adapting Bazel Rules for Remote Execution](https://docs.bazel.build/versions/master/remote-execution-rules.html) + for possible causes and workarounds. + +* If the build fails for any other reason, see the troubleshooting steps in [Step 2: Resolve detected issues](#step-2-resolve-detected-issues). -- cgit v1.2.3