Document troubleshooting Bazel remote execution using the Docker sandbox feature.

PiperOrigin-RevId: 200451804

4 files changed, 284 insertions, 6 deletions

diff --git a/site/_layouts/documentation.html b/site/_layouts/documentation.html
index f41ae43811..b55ad03236 100644
--- a/site/_layouts/documentation.html
+++ b/site/_layouts/documentation.html
@@ -230,7 +230,8 @@ nav: docs
                   <li><a href="/versions/{{ site.version }}/skylark/skylint.html">Linter</a></li>
                   <li><a href="/versions/{{ site.version }}/skylark/performance.html">Optimizing Performance</a></li>
                   <li><a href="/versions/{{ site.version }}/skylark/deploying.html">Deploying Rules</a></li>
-                  <li><a href="/versions/{{ site.version }}/remex-rules-guidelines.html">Guidelines for Remote Execution</a></li>
+                  <li><a href="/versions/{{ site.version }}/remote-execution-rules.html">Guidelines for Remote Execution</a></li>
+                  <li><a href="/versions/{{ site.version }}/remote-execution-sandbox.html">Troubleshooting Remote Execution</a></li>
                 </ul>
               </li>
 
diff --git a/site/docs/remote-caching.md b/site/docs/remote-caching.md
index 7900f573d4..416550f63c 100644
--- a/site/docs/remote-caching.md
+++ b/site/docs/remote-caching.md
@@ -374,7 +374,8 @@ separate platform, such as a datacenter.  You can try remote execution with
 [Buildfarm], an open source project that aims to provide a distributed remote
 execution platform.
 
-[Adapting Bazel Rules for Remote Execution](/remex-rules-guidelines.html)
+[Adapting Rules for Remote Execution](https://docs.bazel.build/versions/master/remote-execution-rules.html)
+[Troubleshooting Remote Execution](https://docs.bazel.build/versions/master/remote-execution-sandbox.html)
 [WebDAV module]: http://nginx.org/en/docs/http/ngx_http_dav_module.html
 [docker image]: https://hub.docker.com/r/buchgr/bazel-remote-cache/
 [GitHub]: https://github.com/buchgr/bazel-remote/
diff --git a/site/docs/remex-rules-guidelines.md b/site/docs/remote-execution-rules.md
index eac255dddf..a069fc370d 100644
--- a/site/docs/remex-rules-guidelines.md
+++ b/site/docs/remote-execution-rules.md
@@ -59,7 +59,7 @@ ain.bzl),
 and [Go](https://github.com/bazelbuild/rules_go/blob/master/go/toolchains.rst),
 and new toolchain rules are under way for other languages and tools such as
 [bash](https://docs.google.com/document/d/e/2PACX-1vRCSB_n3vctL6bKiPkIa_RN_ybzoAccSe0ic8mxdFNZGNBJ3QGhcKjsL7YKf-ngVyjRZwCmhi_5KhcX/pub).
-If a toolchain rule does not exist for the tool your rule uses, consider [creating a custom toolchain rule](https://docs.bazel.build/versions/master/toolchains.html#creating-a-toolchain-rule).
+If a toolchain rule does not exist for the tool your rule uses, consider [/toolchains.html#creating-a-toolchain-rule).
 
 ## Managing implicit dependencies
 
@@ -79,6 +79,12 @@ However, since in a remote execution scenario each build action executes a
 separate compiler instance, compiler state and _bar_'s implicit dependency on
 _foo_ will be lost and the build will fail.
 
+To help detect and eliminate these dependency problems, Bazel 0.14.1 offers the
+local Docker sandbox, which has the same restrictions for dependencies as remote
+execution. Use the sandbox to prepare your build for remote execution by
+identifying and resolving dependency-related build errors. See [Troubleshooting Bazel Remote Execution with Docker Sandbox](/remote-execution-sandbox.html)
+for more information.
+
 ## Managing platform-dependent binaries
 
 Typically, a binary built on the host platform cannot safely execute on an
@@ -86,9 +92,7 @@ arbitrary remote execution platform due to potentially mismatched dependencies.
 For example, the SingleJar binary supplied with Bazel targets the host platform.
 However, for remote execution, SingleJar must be compiled as part of the process
 of building your code so that it targets the remote execution platform. (See the
-[target selection
-logic](https://github.com/bazelbuild/bazel/blob/130aeadfd660336572c3da397f1f107f
-0c89aa8d/tools/jdk/BUILD#L115).)
+[target selection logic](https://github.com/bazelbuild/bazel/blob/130aeadfd660336572c3da397f1f107f0c89aa8d/tools/jdk/BUILD#L115).)
 
 Do not ship binaries of build tools required by your build with your source code
 unless you are sure they will safely run in your execution platform. Instead, do
diff --git a/site/docs/remote-execution-sandbox.md b/site/docs/remote-execution-sandbox.md
new file mode 100644
index 0000000000..a7adbbf92b
--- /dev/null
+++ b/site/docs/remote-execution-sandbox.md
@@ -0,0 +1,272 @@
+---
+layout: documentation
+title: Troubleshooting Bazel Remote Execution with Docker Sandbox
+
+---
+
+# Troubleshooting Bazel Remote Execution with Docker Sandbox
+
+Contents
+
+*  [Overview](#overview)
+*  [Prerequisites](#prerequisites)
+*  [Troubleshooting on the local machine](#troubleshooting-on-the-local-machine)
+   *  [Step 1: Run the build](#step-1-run-the-build)
+   *  [Step 2: Resolve detected issues](#step-2-resolve-detected-issues)
+*  [Troubleshooting in a Docker container](#troubleshooting-in-a-docker-container)
+   *  [Step 1: Build the container](#step-1-build-the-container)
+   *  [Step 2: Start the container](#step-2-start-the-container)
+   *  [Step 3: Test the container](#step-3-test-the-container)
+   *  [Step 4: Run the build](#step-4-run-the-build)
+   *  [Step 5: Resolve detected issues](#step-5-resolve-detected-issues)
+
+## Overview
+
+Bazel builds that succeed locally may fail when executed remotely due to
+restrictions and requirements that do not affect local builds. The most common
+causes of such failures are described in [Adapting Bazel Rules for Remote Execution](https://docs.bazel.build/versions/master/remote-execution-rules.html).
+
+This document describes how to identify and resolve the most common issues that
+arise with remote execution using the Docker sandbox feature, which imposes
+restrictions upon the build equal to those of remote execution. This allows you
+to troubleshoot your build without the need for a remote execution service.
+
+The Docker sandbox feature mimics the restrictions of remote execution as
+follows:
+
+*   **Build actions execute in toolchain containers.** You can use the same
+    toolchain containers to run your build locally and remotely via a service
+    supporting containerized remote execution.
+
+*   **No extraneous data crosses the container boundary.** Only explicitly
+    declared inputs and outputs enter and leave the container, and only after
+    the associated build action successfully completes.
+
+*   **Each action executes in a fresh container.** A new, unique container is
+    created for each spawned build action.
+
+**Note:** Builds take noticeably more time to complete when the Docker sandbox
+feature is enabled. This is normal.
+
+You can troubleshoot these issues using one of the following methods:
+
+*   **[Troubleshooting natively.](#troubleshooting-natively)** With this method,
+    Bazel and its build actions run natively on your local machine. The Docker
+    sandbox feature imposes restrictions upon the build equal to those of remote
+    execution. However, this method will not detect local tools, states, and
+    data leaking into your build, which will cause problems with remote execution.
+
+*   **[Troubleshooting in a Docker container.](#troubleshooting-in-a-docker-container)**
+    With this method, Bazel and its build actions run inside a Docker container,
+    which allows you to detect tools, states, and data leaking from the local
+    machine into the build in addition to imposing restrictions
+    equal to those of remote execution. This method provides insight into your
+    build even if portions of the build are failing. This method is experimental
+    and not officially supported.
+
+## Prerequisites
+
+Before you begin troubleshooting, do the following if you have not already done so:
+
+*   Install Docker and configure the permissions required to run it.
+*   Install Bazel 0.14.1 or later. Earlier versions do not support the Docker
+    sandbox feature.
+*   Add the [bazel-toolchains](https://releases.bazel.build/bazel-toolchains.html)
+    repo, pinned to the latest release version, to your build's `WORKSPACE` file
+    as described [here](https://releases.bazel.build/bazel-toolchains.html).
+*   Add the following lines to your `.bazelrc` file. Create the file in the root
+    directory of your Bazel project if it does not exist.
+
+```
+# Docker Sandbox Mode
+build:docker-sandbox --host_javabase=@bazel_toolchains//configs/ubuntu16_04_clang/1.0:jdk8
+build:docker-sandbox --javabase=@bazel_toolchains//configs/ubuntu16_04_clang/1.0:jdk8
+build:docker-sandbox --crosstool_top=@bazel_toolchains//configs/ubuntu16_04_clang/1.0/bazel_0.13.0/default:toolchain
+build:docker-sandbox --experimental_docker_image=gcr.io/cloud-marketplace/google/rbe-ubuntu16-04@sha256:59bf0e191a6b5cc1ab62c2224c810681d1326bad5a27b1d36c9f40113e79da7f
+build:docker-sandbox --spawn_strategy=docker --strategy=Javac=docker --genrule_strategy=docker
+build:docker-sandbox --define=EXECUTOR=remote
+build:docker-sandbox --experimental_docker_verbose
+build:docker-sandbox --experimental_enable_docker_sandbox
+```
+
+**Note:** The flags referenced in the `.bazelrc` file shown above are configured
+to run within the [rbe-ubuntu16-04](https://console.cloud.google.com/launcher/details/google/rbe-ubuntu16-04?_ga=2.231599325.-396882850.1508873814)
+container.
+
+If your rules require additional tools, do the following:
+
+1.  Create a custom Docker container by installing tools using a [Dockerfile](https://docs.docker.com/engine/reference/builder/)
+    and [building](https://docs.docker.com/engine/reference/commandline/build/)
+    the image locally.
+
+2.  Replace the value of the `--experimental_docker_image` flag above with the
+    name of your custom container image.
+
+
+## Troubleshooting natively
+
+This method executes Bazel and all of its build actions directly on the local
+machine and is a reliable way to confirm whether your build will succeed when
+executed remotely.
+
+However, with this method, locally installed tools, binaries,and data may leak
+into into your build, especially if it uses [configure-style WORKSPACE rules](https://docs.bazel.build/versions/master/remote-execution-rules.html#managing-configure-style-workspace-rules).
+Such leaks will cause problems with remote execution; to detect them, [troubleshoot in a Docker container](#troubleshooting-in-a-docker-container)
+in addition to troubleshooting natively.
+
+### Step 1: Run the build
+
+1.  Add the `--config=docker-sandbox` flag to the Bazel command that executes
+    your build. For example:
+
+    ```
+    bazel --bazelrc=.bazelrc build --config=docker-sandbox <target>
+
+    ```
+
+2.  Run the build and wait for it to complete. The build will run up to four
+    times slower than normal due to the Docker sandbox feature.
+
+You may encounter the following error:
+
+```
+ERROR: 'docker' is an invalid value for docker spawn strategy.
+```
+
+If you do, run the build again with the `--experimental_docker_verbose`  flag.
+This flag enables verbose error messages. This error is typically caused by a
+faulty Docker installation or lack of permissions to execute it under the
+current user account. See the [Docker documentation](https://docs.docker.com/install/linux/linux-postinstall/)
+for more information. If problems persist, skip ahead to [Troubleshooting in a Docker container](#troubleshooting-in-a-docker-container).
+
+### Step 2: Resolve detected issues
+
+The following are the most commonly encountered issues and their workarounds.
+
+*  **A file, tool, binary, or resource referenced by the Bazel runfiles tree is
+   missing.**. Confirm that all dependencies of the affected targets have been [explicitly declared](/build-ref.html#dependencies).
+   See [Managing implicit dependencies](/remote-execution-rules.html#managing-implicit-dependencies)
+   for more information.
+
+*  **A file, tool, binary, or resource referenced by an absolute path or the `PATH`
+   variable is missing.** Confirm that all required tools are installed within
+   the toolchain container and use [toolchain rules](/toolchains.html) to properly
+   declare dependencies pointing to the missing resource. See [Invoking build tools through toolchain rules](/remote-execution-rules.html#invoking-build-tools-through-toolchain-rules)
+   for more information.
+
+*  **A binary execution fails.** One of the build rules is referencing a binary
+   incompatible with the execution environment (the Docker container). See [Managing platform-dependent binaries](/remote-execution-rules.html#managing-platform-dependent-binaries)
+   for more information. If you cannot resolve the issue, contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com)
+   for help.
+
+*  **A file from `@local-jdk` is missing or causing errors.** The Java binaries
+   on your local machine are leaking into the build while being incompatible with
+   it. Use [`java_toolchain`](https://docs.bazel.build/versions/master/be/java.html#java_toolchain)
+   in your rules and targets instead of `@local_jdk`. Contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) if you need further help.
+
+*  **Other errors.** Contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) for help.
+
+## Troubleshooting in a Docker container
+
+With this method, Bazel runs inside a host Docker container, and Bazel's build
+actions execute inside individual toolchain containers spawned by the Docker
+sandbox feature. The sandbox spawns a brand new toolchain container for each
+build action and only one action executes in each toolchain container.
+
+This method provides more granular control of tools installed in the host
+environment. By separating the execution of the build from the execution of its
+build actions and keeping the installed tooling to a minimum, you can verify
+whether your build has any dependencies on the local execution environment.
+
+### Step 1: Build the container
+
+**Note:** The commands below are tailored specifically for a  `debian:stretch` base.
+For other bases, modify them as necessary.
+
+1.  Create a `Dockerfile` that creates the Docker container and installs Bazel
+    with a minimal set of build tools:
+
+    ```
+    FROM debian:stretch
+
+    RUN apt-get update && apt-get install -y apt-transport-https curl software-properties-common git gcc gnupg2 g++ openjdk-8-jdk-headless python-dev zip wget vim
+
+    RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
+
+    RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"
+
+    RUN apt-get update && apt-get install -y docker-ce
+
+    RUN wget https://releases.bazel.build/0.14.1/release/bazel-0.14.1-installer-linux-x86_64.sh -O ./bazel-installer.sh && chmod 755 ./bazel-installer.sh
+
+    RUN ./bazel-installer.sh
+    ```
+
+2.  Build the container as `bazel_container`:
+
+    ```
+    docker build -t bazel_container - < Dockerfile
+    ```
+
+### Step 2: Start the container
+
+Start the Docker container using the command shown below. In the command,
+substitute the path to the source code on your host that you want to build.
+
+```
+docker run -it \
+  -v /var/run/docker.sock:/var/run/docker.sock \
+  -v /tmp:/tmp \
+  -v <your source code directory>:/src \
+  -w /src \
+  bazel_container \
+  /bin/bash
+```
+
+This command runs the container as root, mapping the docker socket, and mounting
+the `/tmp` directory. This allows Bazel to spawn other Docker containers and to
+use directories under `/tmp` to share files with those containers. Your source
+code is available at `/src` inside the container.
+
+The command intentionally starts from a `debian:stretch` base container that
+includes binaries incompatible with the `rbe-ubuntu16-04` container used as a
+toolchain container. If binaries from the local environment are leaking into the
+toolchain container, they will cause build errors.
+
+### Step 3: Test the container
+
+Run the following commands from inside the Docker container to test it:
+
+```
+docker ps
+
+bazel version
+```
+
+### Step 4: Run the build
+
+Run the build as shown below. The output user is root so that it corresponds to
+a directory that is accessible with the same absolute path from inside the host
+container in which Bazel runs, from the toolchain containers spawned by the Docker
+sandbox feature in which Bazel's build actions are running, and from the local
+machine on which the host and action containers run.
+
+```
+bazel --output_user_root=/tmp/bazel_docker_root --bazelrc=.bazelrc \ build --config=docker-sandbox <target>
+```
+
+### Step 5: Resolve detected issues
+
+You can resolve build failures as follows:
+
+*   If the build fails with an "out of disk space" error, you  can increase this
+    limit by starting the host container with the flag `--memory=XX` where `XX`
+    is the allocated disk space in gigabytes. This is experimental and may
+    result in unpredictable behavior.
+
+*   If the build fails during the analysis or loading phases, one or more of
+    your build rules declared in the WORKSPACE file are not compatible with
+    remote execution. See [Adapting Bazel Rules for Remote Execution](https://docs.bazel.build/versions/master/remote-execution-rules.html)
+    for possible causes and workarounds.
+
+*   If the build fails for any other reason, see the troubleshooting steps in [Step 2: Resolve detected issues](#step-2-resolve-detected-issues).