Document troubleshooting Bazel remote execution using the Docker sandbox feature.

PiperOrigin-RevId: 200451804

1 files changed, 272 insertions, 0 deletions

diff --git a/site/docs/remote-execution-sandbox.md b/site/docs/remote-execution-sandbox.md
new file mode 100644
index 0000000000..a7adbbf92b
--- /dev/null
+++ b/site/docs/remote-execution-sandbox.md
@@ -0,0 +1,272 @@
+---
+layout: documentation
+title: Troubleshooting Bazel Remote Execution with Docker Sandbox
+
+---
+
+# Troubleshooting Bazel Remote Execution with Docker Sandbox
+
+Contents
+
+*  [Overview](#overview)
+*  [Prerequisites](#prerequisites)
+*  [Troubleshooting on the local machine](#troubleshooting-on-the-local-machine)
+   *  [Step 1: Run the build](#step-1-run-the-build)
+   *  [Step 2: Resolve detected issues](#step-2-resolve-detected-issues)
+*  [Troubleshooting in a Docker container](#troubleshooting-in-a-docker-container)
+   *  [Step 1: Build the container](#step-1-build-the-container)
+   *  [Step 2: Start the container](#step-2-start-the-container)
+   *  [Step 3: Test the container](#step-3-test-the-container)
+   *  [Step 4: Run the build](#step-4-run-the-build)
+   *  [Step 5: Resolve detected issues](#step-5-resolve-detected-issues)
+
+## Overview
+
+Bazel builds that succeed locally may fail when executed remotely due to
+restrictions and requirements that do not affect local builds. The most common
+causes of such failures are described in [Adapting Bazel Rules for Remote Execution](https://docs.bazel.build/versions/master/remote-execution-rules.html).
+
+This document describes how to identify and resolve the most common issues that
+arise with remote execution using the Docker sandbox feature, which imposes
+restrictions upon the build equal to those of remote execution. This allows you
+to troubleshoot your build without the need for a remote execution service.
+
+The Docker sandbox feature mimics the restrictions of remote execution as
+follows:
+
+*   **Build actions execute in toolchain containers.** You can use the same
+    toolchain containers to run your build locally and remotely via a service
+    supporting containerized remote execution.
+
+*   **No extraneous data crosses the container boundary.** Only explicitly
+    declared inputs and outputs enter and leave the container, and only after
+    the associated build action successfully completes.
+
+*   **Each action executes in a fresh container.** A new, unique container is
+    created for each spawned build action.
+
+**Note:** Builds take noticeably more time to complete when the Docker sandbox
+feature is enabled. This is normal.
+
+You can troubleshoot these issues using one of the following methods:
+
+*   **[Troubleshooting natively.](#troubleshooting-natively)** With this method,
+    Bazel and its build actions run natively on your local machine. The Docker
+    sandbox feature imposes restrictions upon the build equal to those of remote
+    execution. However, this method will not detect local tools, states, and
+    data leaking into your build, which will cause problems with remote execution.
+
+*   **[Troubleshooting in a Docker container.](#troubleshooting-in-a-docker-container)**
+    With this method, Bazel and its build actions run inside a Docker container,
+    which allows you to detect tools, states, and data leaking from the local
+    machine into the build in addition to imposing restrictions
+    equal to those of remote execution. This method provides insight into your
+    build even if portions of the build are failing. This method is experimental
+    and not officially supported.
+
+## Prerequisites
+
+Before you begin troubleshooting, do the following if you have not already done so:
+
+*   Install Docker and configure the permissions required to run it.
+*   Install Bazel 0.14.1 or later. Earlier versions do not support the Docker
+    sandbox feature.
+*   Add the [bazel-toolchains](https://releases.bazel.build/bazel-toolchains.html)
+    repo, pinned to the latest release version, to your build's `WORKSPACE` file
+    as described [here](https://releases.bazel.build/bazel-toolchains.html).
+*   Add the following lines to your `.bazelrc` file. Create the file in the root
+    directory of your Bazel project if it does not exist.
+
+```
+# Docker Sandbox Mode
+build:docker-sandbox --host_javabase=@bazel_toolchains//configs/ubuntu16_04_clang/1.0:jdk8
+build:docker-sandbox --javabase=@bazel_toolchains//configs/ubuntu16_04_clang/1.0:jdk8
+build:docker-sandbox --crosstool_top=@bazel_toolchains//configs/ubuntu16_04_clang/1.0/bazel_0.13.0/default:toolchain
+build:docker-sandbox --experimental_docker_image=gcr.io/cloud-marketplace/google/rbe-ubuntu16-04@sha256:59bf0e191a6b5cc1ab62c2224c810681d1326bad5a27b1d36c9f40113e79da7f
+build:docker-sandbox --spawn_strategy=docker --strategy=Javac=docker --genrule_strategy=docker
+build:docker-sandbox --define=EXECUTOR=remote
+build:docker-sandbox --experimental_docker_verbose
+build:docker-sandbox --experimental_enable_docker_sandbox
+```
+
+**Note:** The flags referenced in the `.bazelrc` file shown above are configured
+to run within the [rbe-ubuntu16-04](https://console.cloud.google.com/launcher/details/google/rbe-ubuntu16-04?_ga=2.231599325.-396882850.1508873814)
+container.
+
+If your rules require additional tools, do the following:
+
+1.  Create a custom Docker container by installing tools using a [Dockerfile](https://docs.docker.com/engine/reference/builder/)
+    and [building](https://docs.docker.com/engine/reference/commandline/build/)
+    the image locally.
+
+2.  Replace the value of the `--experimental_docker_image` flag above with the
+    name of your custom container image.
+
+
+## Troubleshooting natively
+
+This method executes Bazel and all of its build actions directly on the local
+machine and is a reliable way to confirm whether your build will succeed when
+executed remotely.
+
+However, with this method, locally installed tools, binaries,and data may leak
+into into your build, especially if it uses [configure-style WORKSPACE rules](https://docs.bazel.build/versions/master/remote-execution-rules.html#managing-configure-style-workspace-rules).
+Such leaks will cause problems with remote execution; to detect them, [troubleshoot in a Docker container](#troubleshooting-in-a-docker-container)
+in addition to troubleshooting natively.
+
+### Step 1: Run the build
+
+1.  Add the `--config=docker-sandbox` flag to the Bazel command that executes
+    your build. For example:
+
+    ```
+    bazel --bazelrc=.bazelrc build --config=docker-sandbox <target>
+
+    ```
+
+2.  Run the build and wait for it to complete. The build will run up to four
+    times slower than normal due to the Docker sandbox feature.
+
+You may encounter the following error:
+
+```
+ERROR: 'docker' is an invalid value for docker spawn strategy.
+```
+
+If you do, run the build again with the `--experimental_docker_verbose`  flag.
+This flag enables verbose error messages. This error is typically caused by a
+faulty Docker installation or lack of permissions to execute it under the
+current user account. See the [Docker documentation](https://docs.docker.com/install/linux/linux-postinstall/)
+for more information. If problems persist, skip ahead to [Troubleshooting in a Docker container](#troubleshooting-in-a-docker-container).
+
+### Step 2: Resolve detected issues
+
+The following are the most commonly encountered issues and their workarounds.
+
+*  **A file, tool, binary, or resource referenced by the Bazel runfiles tree is
+   missing.**. Confirm that all dependencies of the affected targets have been [explicitly declared](/build-ref.html#dependencies).
+   See [Managing implicit dependencies](/remote-execution-rules.html#managing-implicit-dependencies)
+   for more information.
+
+*  **A file, tool, binary, or resource referenced by an absolute path or the `PATH`
+   variable is missing.** Confirm that all required tools are installed within
+   the toolchain container and use [toolchain rules](/toolchains.html) to properly
+   declare dependencies pointing to the missing resource. See [Invoking build tools through toolchain rules](/remote-execution-rules.html#invoking-build-tools-through-toolchain-rules)
+   for more information.
+
+*  **A binary execution fails.** One of the build rules is referencing a binary
+   incompatible with the execution environment (the Docker container). See [Managing platform-dependent binaries](/remote-execution-rules.html#managing-platform-dependent-binaries)
+   for more information. If you cannot resolve the issue, contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com)
+   for help.
+
+*  **A file from `@local-jdk` is missing or causing errors.** The Java binaries
+   on your local machine are leaking into the build while being incompatible with
+   it. Use [`java_toolchain`](https://docs.bazel.build/versions/master/be/java.html#java_toolchain)
+   in your rules and targets instead of `@local_jdk`. Contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) if you need further help.
+
+*  **Other errors.** Contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) for help.
+
+## Troubleshooting in a Docker container
+
+With this method, Bazel runs inside a host Docker container, and Bazel's build
+actions execute inside individual toolchain containers spawned by the Docker
+sandbox feature. The sandbox spawns a brand new toolchain container for each
+build action and only one action executes in each toolchain container.
+
+This method provides more granular control of tools installed in the host
+environment. By separating the execution of the build from the execution of its
+build actions and keeping the installed tooling to a minimum, you can verify
+whether your build has any dependencies on the local execution environment.
+
+### Step 1: Build the container
+
+**Note:** The commands below are tailored specifically for a  `debian:stretch` base.
+For other bases, modify them as necessary.
+
+1.  Create a `Dockerfile` that creates the Docker container and installs Bazel
+    with a minimal set of build tools:
+
+    ```
+    FROM debian:stretch
+
+    RUN apt-get update && apt-get install -y apt-transport-https curl software-properties-common git gcc gnupg2 g++ openjdk-8-jdk-headless python-dev zip wget vim
+
+    RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
+
+    RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"
+
+    RUN apt-get update && apt-get install -y docker-ce
+
+    RUN wget https://releases.bazel.build/0.14.1/release/bazel-0.14.1-installer-linux-x86_64.sh -O ./bazel-installer.sh && chmod 755 ./bazel-installer.sh
+
+    RUN ./bazel-installer.sh
+    ```
+
+2.  Build the container as `bazel_container`:
+
+    ```
+    docker build -t bazel_container - < Dockerfile
+    ```
+
+### Step 2: Start the container
+
+Start the Docker container using the command shown below. In the command,
+substitute the path to the source code on your host that you want to build.
+
+```
+docker run -it \
+  -v /var/run/docker.sock:/var/run/docker.sock \
+  -v /tmp:/tmp \
+  -v <your source code directory>:/src \
+  -w /src \
+  bazel_container \
+  /bin/bash
+```
+
+This command runs the container as root, mapping the docker socket, and mounting
+the `/tmp` directory. This allows Bazel to spawn other Docker containers and to
+use directories under `/tmp` to share files with those containers. Your source
+code is available at `/src` inside the container.
+
+The command intentionally starts from a `debian:stretch` base container that
+includes binaries incompatible with the `rbe-ubuntu16-04` container used as a
+toolchain container. If binaries from the local environment are leaking into the
+toolchain container, they will cause build errors.
+
+### Step 3: Test the container
+
+Run the following commands from inside the Docker container to test it:
+
+```
+docker ps
+
+bazel version
+```
+
+### Step 4: Run the build
+
+Run the build as shown below. The output user is root so that it corresponds to
+a directory that is accessible with the same absolute path from inside the host
+container in which Bazel runs, from the toolchain containers spawned by the Docker
+sandbox feature in which Bazel's build actions are running, and from the local
+machine on which the host and action containers run.
+
+```
+bazel --output_user_root=/tmp/bazel_docker_root --bazelrc=.bazelrc \ build --config=docker-sandbox <target>
+```
+
+### Step 5: Resolve detected issues
+
+You can resolve build failures as follows:
+
+*   If the build fails with an "out of disk space" error, you  can increase this
+    limit by starting the host container with the flag `--memory=XX` where `XX`
+    is the allocated disk space in gigabytes. This is experimental and may
+    result in unpredictable behavior.
+
+*   If the build fails during the analysis or loading phases, one or more of
+    your build rules declared in the WORKSPACE file are not compatible with
+    remote execution. See [Adapting Bazel Rules for Remote Execution](https://docs.bazel.build/versions/master/remote-execution-rules.html)
+    for possible causes and workarounds.
+
+*   If the build fails for any other reason, see the troubleshooting steps in [Step 2: Resolve detected issues](#step-2-resolve-detected-issues).