site: add documentation about the BEP and BES

Provide an overview of the Build Event Protocol and Build Event Service.

Change-Id: If11a060cea7ce0fef80486a473b0f7c1b523f50c
PiperOrigin-RevId: 164435311

1 files changed, 243 insertions, 0 deletions

diff --git a/site/docs/build-event-protocol.md b/site/docs/build-event-protocol.md
new file mode 100644
index 0000000000..88d81fab2e
--- /dev/null
+++ b/site/docs/build-event-protocol.md
@@ -0,0 +1,243 @@
+---
+layout: documentation
+title: Build Event Protocol
+---
+
+# Build Event Protocol
+
+The [Build Event Protocol] allows third party programs to gain insight into a
+Bazel invocation. For example, you could use the Build Event Protocol to gather
+information for an IDE plugin or a dashboard that displays build results.
+
+The protocol is a set of [protocol buffer] messages with some semantics defined
+on top of it. It includes information about build and test results, build
+progress, the build configuration and much more. The Build Event Protocol is
+intended to be consumed programmatically and makes parsing Bazel’s command line
+output a thing of the past.
+
+## Contents
+
+*  [Build Event Protocol Overview](#build-event-protocol-overview)
+   *  [Build Event Graph](#build-event-graph)
+   *  [The Build Event Protocol by Example](#the-build-event-protocol-by-example)
+*  [Consuming the Build Event Protocol](#consuming-the-build-event-protocol)
+   *  [Consume in a Binary Format](#consume-in-a-binary-format)
+   *  [Consume in Text Formats](#consume-in-text-formats)
+*  [The Build Event Service](#the-build-event-service)
+   *  [Build Event Service Flags](#build-event-service-flags)
+   *  [Authentication and Security](#authentication-and-security)
+
+## Build Event Protocol Overview
+
+The [Build Event Protocol] represents information about a build as events. A
+build event is a protocol buffer message consisting of a build event identifier,
+a set of child event identifiers, and a payload.
+
+*  __Build Event Identifier:__ Depending on the kind of build event, it might be
+an [opaque string] or [structured information] revealing more about the build
+event. A build event identifier is unique within a build.
+
+*  __Children:__ A build event may announce other build events, by including
+their build event identifiers in its [children field]. For example, the
+`PatternExpanded` build event announces the targets it expands to as children.
+The protocol guarantees that all events, except for the first event, are
+announced by a previous event.
+
+* __Payload:__ The payload contains structured information about a build event,
+encoded as a protocol buffer message specific to that event. Note, that the
+payload might not be the expected type, but could be an `Aborted` message e.g.
+if the build aborted prematurely.
+
+### Build Event Graph
+All build events form a directed acyclic graph through their parent and child
+relationship. Every build event except for the initial build event has one or
+more parent events. Please note that not all parent events of a child event must
+necessarily be posted before it. When a build is complete (succeeded or failed)
+all announced events will have been posted. In case of a Bazel crash or a failed
+network transport, some announced build events may never be posted.
+
+## The Build Event Protocol by Example
+The full specification of the [Build Event Protocol] can be found in its
+protocol buffer definition and describing it here is beyond the scope of this
+document. However, it might be helpful to build up some intuition before looking
+at the specification.
+
+Consider a simple Bazel workspace that consists of two empty shell scripts
+`foo.sh` and `foo_test.sh` and the following BUILD file:
+
+```bash
+sh_binary(
+    name = "foo",
+    srcs = ["foo.sh"],
+)
+
+sh_library(
+    name = "foo_lib",
+    data = [":foo"],
+)
+
+sh_test(
+    name = "foo_test",
+    srcs = ["foo_test.sh"],
+    deps = [":foo_lib"],
+)
+```
+
+When running `bazel test ...` on this project the build graph of the generated
+build events will resemble the graph below. The arrows indicate the
+aforementioned parent and child relationship. Note that some build events and
+most fields have been omitted for brevity.
+
+![bep-graph]
+
+Initially, a `BuildStarted` event is published. The event informs us that the
+build was invoked through the `bazel test` command and it also announces five
+child events: `OptionsParsed`, `WorkspaceStatus`, `CommandLine`,
+`PatternExpanded` and `Progress`. The first three events provide information
+about how Bazel was invoked. The `PatternExpanded` build event provides insight
+into which specific targets the `...` pattern expanded to: `//:foo`, `//:foo_lib`
+and `//:foo_test`. It does so by declaring three `TargetConfigured` events as
+children.
+
+Note that the `TargetConfigured` event declares the `Configuration` event as a
+child event, even though `Configuration` has been posted before the
+`TargetConfigured` event.
+
+Besides the parent and child relationship, events may also refer to each other
+using their build event identifiers. For example, in the above graph the
+`TargetComplete` event refers to the `NamedSetOfFiles` event in its `fileSets`
+field.
+
+Build events that refer to files (i.e. outputs) usually don’t embed the file
+names and paths in the event. Instead, they contain the build event identifier
+of a `NamedSetOfFiles` event, which will then contain the actual file names and
+paths. The `NamedSetOfFiles` event allows a set of files to be reported once and
+referred to by many targets. This structure is necessary because otherwise in
+some cases the Build Event Protocol output size would grow quadratically with
+the number of files. A `NamedSetOfFiles` event may also not have all its files
+embedded, but instead refer to other `NamedSetOfFiles` events through their
+build event identifiers.
+
+Below is an instance of the `TargetComplete` event for the `//:foo_lib` target
+from the above graph, printed in protocol buffer’s JSON representation. The
+build event identifier contains the target as an opaque string and refers to the
+`Configuration` event using its build event identifier. The event does not
+announce any child events. The payload contains information about whether the
+target was built successfully, the set of output files, and the kind of target
+built.
+
+```json
+{
+  "id": {
+    "targetCompleted": {
+      "label": "//:foo_lib",
+      "configuration": {
+        "id": "544e39a7f0abdb3efdd29d675a48bc6a"
+      }
+    }
+  },
+  "completed": {
+    "success": true,
+    "outputGroup": [{
+      "name": "default",
+      "fileSets": [{
+        "id": "0"
+      }]
+    }],
+    "targetKind": "sh_library rule"
+  }
+}
+```
+
+## Consuming the Build Event Protocol
+
+### Consume in a Binary Format
+
+To consume the Build Event Protocol in a binary format:
+
+1. Have Bazel serialize the protocol buffer messages to a file by specifying the
+option `--build_event_binary_file=/path/to/file`. The file will contain
+serialized protocol buffer messages with each message being length delimited.
+Each message is prefixed with its length encoded as a variable length integer.
+This format can be read using the protocol buffer library’s
+[parseDelimitedFrom(InputStream)] method.
+
+2. Then, write a program that extracts the relevant information from the
+serialized protocol buffer message.
+
+### Consume in Text Formats
+
+The following Bazel command line flags will output the Build Event Protocol in a
+human-readable formats:
+
+```
+--build_event_text_file
+--build_event_json_file
+```
+
+## The Build Event Service
+
+The [Build Event Service] Protocol is a generic [gRPC] service for transmitting
+build events. The Build Event Service protocol is independent of the Build Event
+Protocol and treats the Build Event Protocol events as opaque bytes. Bazel ships
+with a gRPC client implementation of the Build Event Service protocol that
+transmits Build Event Protocol events. One can specify the endpoint to send the
+events to using the `--bes_backend=HOST:PORT flag`. Bazel’s implementation also
+supports TLS which can be enabled by specifying the `--tls_enabled flag`.
+
+There is currently no open source server-side Build Event Service implementation
+that we know of.
+
+### Build Event Service Flags
+
+Bazel has several flags related to the [Build Event Service] protocol:
+
+*  `--bes_backend`
+*  `--[no]bes_best_effort`
+*  `--[no]bes_lifecycle_events`
+*  `--bes_timeout`
+*  `--project_id`
+
+For a description of each of these flags, see the
+[Command-Line Reference](command-line-reference.html).
+
+### Authentication and Security
+
+Bazel’s [Build Event Service] implementation also supports authentication and
+TLS. These settings can be controlled using the below flags. Please note that
+these flags are also used for Bazel’s Remote Execution. This implies that the
+Build Event Service and Remote Execution Endpoints need to share the same
+authentication and TLS infrastructure.
+
+*  `--auth_credentials`
+*  `--[no]auth_enabled`
+*  `--auth_scope`
+*  `--tls_certificate`
+*  `--[no]tls_enabled`
+
+For a description of each of these flags, see the
+[Command-Line Reference](command-line-reference.html).
+
+[Build Event Protocol]:
+https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto
+
+[Build Event Service]:
+https://github.com/googleapis/googleapis/blob/master/google/devtools/build/v1/publish_build_event.proto
+
+[gRPC]: https://www.grpc.io
+
+[protocol buffer]: https://developers.google.com/protocol-buffers/
+
+[bep-graph]: /assets/bep-graph.svg
+
+[parseDelimitedFrom(InputStream)]:
+https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/AbstractParser#parseDelimitedFrom-java.io.InputStream-
+
+[opaque string]:
+https://github.com/bazelbuild/bazel/blob/16a107d/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L91
+
+[structured information]:
+https://github.com/bazelbuild/bazel/blob/16a107d/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L123
+
+[children field]:
+https://github.com/bazelbuild/bazel/blob/16a107d/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L469