Restructure site/ directory into docs/ which only contains Bazel documentation.

The new docs/ directory in the bazel source tree will only contain the Bazel
docs site, which is hosted at docs.bazel.build. This change deletes the
marketing site and blog, which have been migrated to the bazel-website and
bazel-blog GitHub repositories respectively.

This change also updates the serve-docs.sh and ci/build.sh under scripts/ in
preparation for publishing the docs site.

Note that to help make reviews more manageable, this change is limited to
moving files to their new locations. Here are the follow-up changes:

* Update all links in docs to remove versions/master in paths and to add
  correct bazel.build subdomain when linking to pages on the marketing site
  or the blog.
* Set up versioned directories on GCS bucket and add tooling for versioning
  docs

This change is also coordinated with
https://bazel-review.googlesource.com/c/11568/ to have the PublishSite job
publish to docs.bazel.build rather than www.bazel.build.

Issue #2397

RELNOTES: None
PiperOrigin-RevId: 157612651

1 files changed, 490 insertions, 0 deletions

diff --git a/site/docs/test-encyclopedia.html b/site/docs/test-encyclopedia.html
new file mode 100644
index 0000000000..76e326283b
--- /dev/null
+++ b/site/docs/test-encyclopedia.html
@@ -0,0 +1,490 @@
+---
+layout: documentation
+title: Test Encyclopedia
+---
+<h1>Writing tests</h1>
+
+<p class="lead">An Exhaustive Specification of the Test Execution Environment</p>
+
+<h2>Background</h2>
+
+<p>The Bazel BUILD language includes rules which can be used to define
+automated test programs in many languages.</p>
+
+<p>Tests are run using <code><a href="bazel-user-manual.html#test">bazel test</a></code>.
+
+Users may also execute test binaries directly. This is allowed but not endorsed, as such
+an invocation will not adhere to the mandates described below.</p>
+
+<p>Tests should be <i>hermetic</i>: that is, they ought to access only those
+resources on which they have a declared dependency. If tests are not properly
+hermetic then they do not give historically reproducible results. This could be a
+significant problem for culprit finding (determining which change broke a test),
+release engineering auditability, and resource isolation of tests (automated
+testing frameworks ought not DDOS a server because some tests happen to
+talk to it).<p>
+
+<h2>Objective</h2>
+
+<p>The goal of this document is to formally establish the runtime environment
+for and expected behavior of Bazel tests.  It will also impose requirements on
+the test runner and the build system.
+
+Our intent is to help test authors avoid relying on unspecified
+behavior, and thus give the testing infrastructure more freedom to make
+implementation changes.  We will also take the opportunity to tighten up some
+holes which currently allow many tests to pass despite not being
+properly hermetic, deterministic, and reentrant.</p>
+
+<p>This document is intended to be both normative and authoritative.  If
+this specification and the implemented behavior of test runner disagree, the
+specification takes precedence.</p>
+
+
+<h3>Purpose of Tests</h3>
+
+<p>The purpose of Bazel tests is to confirm some property of the source files
+checked into the repository.  (In this document, "source files" includes test data,
+golden outputs, and anything else kept under version control.)  One
+user writes a test to assert an invariant which they expect to be maintained.
+Other users execute the test later to check whether the invariant has been
+broken.  If the test depends on any variables other than source files
+(non-hermetic), its value is diminished, because the later users cannot be sure
+their changes are at fault when the test stops passing.</p>
+
+<p>Therefore the outcome of a test must depend only on:</p>
+<ul>
+  <li>source files on which the test has a declared dependency</li>
+  <li>products of the build system on which the test has a declared dependency</li>
+  <li>resources whose behavior is guaranteed by the test runner to remain constant</li>
+</ul>
+
+<p>Currently, such behavior is not enforced. However, test runners reserve the
+right to add such enforcement in the future.</p>
+
+<h3>Role of the Build System</h3>
+
+<p>Test rules are analogous to binary rules in that each must yield an
+executable program.  For some languages, this is a stub program which combines
+a language-specific harness with the test code.  Test rules must produce other
+outputs as well.  In addition to the primary test executable, the test runner
+will need a manifest of <b>runfiles</b>, input files which should be made
+available to the test at runtime, and it may need information about the type,
+size, and tags of a test.</p>
+
+<p>The build system may use the runfiles to deliver code as well as data.  (This
+might be used as an optimization to make each test binary smaller by sharing
+files across tests, e.g. through the use of dynamic linking.)  The build system
+should ensure that the generated executable loads these files via the runfiles
+image provided by the test runner, rather than hardcoded references to absolute
+locations in the source or output tree.</p>
+
+<h3>Role of the Test Runner</h3>
+
+<p>From the point of view of the test runner, each test is a program which can
+be invoked with <code>execve()</code>.  There may be other ways to execute
+tests; for example, an IDE might allow the execution of Java tests in-process.
+However, the result of running the test as a standalone process must be
+considered authoritative.  If a test process runs to completion and terminates
+normally with an exit code of zero, the test has passed.  Any other result is
+considered a test failure.  In particular, writing any of the strings
+<code>PASS</code> or <code>FAIL</code> to stdout has no significance to the test
+runner.</p>
+
+<p>If a test takes too long to execute, exceeds some resource limit, or the test
+runner otherwise detects prohibited behavior, it may choose to kill the test
+and treat the run as a failure.  The runner must not report the test as passing
+after sending a signal to the test process or any children thereof.</p>
+
+<p id="timeout">The whole test target (not individual methods or tests) is given a
+limited amount of time to run to completion. The time limit for a test is based
+on its timeout attribute according to the following table:</p>
+
+<table class="table table-bordered table-striped table-condensed">
+  <thead>
+    <tr><th>timeout</th><th>Time Limit (sec.)</th></tr>
+  </thead>
+  <tbody>
+    <tr><td><code>short</code></td><td>60</td></tr>
+    <tr><td><code>moderate</code></td><td>300</td></tr>
+    <tr><td><code>long</code></td><td>900</td></tr>
+    <tr><td><code>eternal</code></td><td>3600</td></tr>
+  </tbody>
+</table>
+
+<p id="size">Tests which do not explicitly specify a timeout have one implied based on the
+test's <code>size</code> as follows:</p>
+
+<table class="table table-bordered table-striped table-condensed">
+  <thead>
+    <tr><th>size</th><th>Implied timeout label</th></tr>
+  </thead>
+  <tbody>
+    <tr><td><code>small</code></td><td>short</td></tr>
+    <tr><td><code>medium</code></td><td>moderate</td></tr>
+    <tr><td><code>large</code></td><td>long</td></tr>
+    <tr><td><code>enormous</code></td><td>eternal</td></tr>
+  </tbody>
+</table>
+<p>For example a "large" test with no explicit timeout setting will be allotted
+900 seconds to run. A "medium" test with a timeout of "short" will be allotted
+60 seconds.</p>
+
+<p>All combinations of <code>size</code> and <code>timeout</code> labels are
+legal, so an "enormous" test may be declared to have a timeout of "short".
+Presumably it would do some really horrible things very quickly.</p>
+<p>Tests may return arbitrarily fast regardless of timeout.  A test is not
+penalized for an overgenerous timeout, although a warning may be issued: you
+should generally set your timeout as tight as you can without incurring any
+flakiness.</p>
+
+<p>There is also a recommended lower bound for test timeouts as follows: </p>
+
+<table class="table table-bordered table-striped table-condensed">
+  <thead>
+    <tr><th>size</th><th>Time minimum (sec.)</th></tr>
+  </thead>
+  <tbody>
+    <tr><td><code>short</code></td><td>0</td></tr>
+    <tr><td><code>moderate</code></td><td>30</td></tr>
+    <tr><td><code>long</code></td><td>300</td></tr>
+    <tr><td><code>eternal</code></td><td>900</td></tr>
+  </tbody>
+</table>
+
+<p>For example, if a "moderate" test completes in 5.5s, consider setting
+<code>timeout</code>="short" or <code>size</code>="small". Using the bazel
+<code>--test_verbose_timeout_warnings</code> command line option will show the
+tests whose specified size is too big.</p>
+
+<p>Test sizes and timeouts are specified in the BUILD file according to the specification
+<a href="be/common-definitions.html#common-attributes-tests">here</a>.
+Any test that does not specify a recognized size will default to being a medium
+test.</p>
+
+<p>If the main process of a test exits, but some of its children are still
+running, the test runner should consider the run complete and count it as a
+success or failure based on the exit code observed from the main process.  The
+test runner may kill any stray processes.  Tests should not leak processes in
+this fashion.</p>
+
+<p>When executing a test, the test runner must establish certain initial
+conditions.</p>
+
+<h3>Initial Conditions</h3>
+
+<p>The test runner must invoke each test with the path to the test
+executable in <code>argv[0]</code>.  This path must be relative and
+beneath the test's current directory (which is in the runfiles tree,
+see below).
+The test runner should not pass any other arguments to a
+test unless the user explicitly requests it.</p>
+
+<p>The initial environment block shall be composed as follows:</p>
+
+<table class="table table-bordered table-striped table-condensed">
+  <thead>
+    <tr><th>Variable</th><th>Value</th><th>Status</th></tr>
+  </thead>
+  <tbody>
+
+    <tr><td><code>HOME</code></td><td>value of <code>$TEST_TMPDIR</code></td><td>recommended</td></tr>
+    <tr><td><code>LANG</code></td><td><i>unset</i></td><td>required</td></tr>
+    <tr><td><code>LANGUAGE</code></td><td><i>unset</i></td><td>required</td></tr>
+    <tr><td><code>LC_ALL</code></td><td><i>unset</i></td><td>required</td></tr>
+    <tr><td><code>LC_COLLATE</code></td><td><i>unset</i></td><td>required</td></tr>
+    <tr><td><code>LC_CTYPE</code></td><td><i>unset</i></td><td>required</td></tr>
+    <tr><td><code>LC_MESSAGES</code></td><td><i>unset</i></td><td>required</td></tr>
+    <tr><td><code>LC_MONETARY</code></td><td><i>unset</i></td><td>required</td></tr>
+    <tr><td><code>LC_NUMERIC</code></td><td><i>unset</i></td><td>required</td></tr>
+    <tr><td><code>LC_TIME</code></td><td><i>unset</i></td><td>required</td></tr>
+    <tr><td><code>LD_LIBRARY_PATH</code></td><td>colon-separated list of directories containing shared libraries</td><td>optional</td></tr>
+    <tr><td><code>JAVA_RUNFILES</code></td><td>value of <code>$TEST_SRCDIR</code></td><td>deprecated</td></tr>
+
+    <tr><td><code>LOGNAME</code></td><td>value of <code>$USER</code></td><td>required</td></tr>
+
+    <tr><td><code>PATH</code></td><td><code>/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.</code></td><td>recommended</td></tr>
+    <tr><td><code>PWD</code></td><td><code>$TEST_SRCDIR/<i>workspace-name</i></code></td><td>recommended</td></tr>
+    <tr><td><code>SHLVL</code></td><td><code>2</code></td><td>recommended</td></tr>
+    <tr><td><code>TEST_PREMATURE_EXIT_FILE</code></td><td>absolute path to a private file in a writable directory (used for catching calls to exit())</td><td>optional</td></tr>
+    <tr>
+      <td><code>TEST_RANDOM_SEED</code></td>
+      <td colspan="2">If the <code class='flag'>--runs_per_test</code> option is used, TEST_RANDOM_SEED
+      is set to the <var>run number</var> (starting with 1) for each individual test run.</td>
+      <td>optional</td>
+    </tr>
+    <tr><td><code>TEST_SIZE</code></td><td>The test <a href="#size"><code>size</code></a></td><td>optional</td></tr>
+    <tr><td><code>TEST_TIMEOUT</code></td><td>The test <a href="#timeout"><code>timeout</code></a></td><td>optional</td></tr>
+    <tr><td><code>TEST_SRCDIR</code></td><td>absolute path to the base of the runfiles tree</td><td>required</td></tr>
+    <tr><td><code>TEST_TMPDIR</code></td><td>absolute path to a private writable directory</td><td>required</td></tr>
+    <tr><td><code>TEST_UNDECLARED_OUTPUTS_DIR</code></td><td>absolute path to a private writable directory (used to write undeclared test outputs)</td><td>optional</td></tr>
+    <tr><td><code>TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR</code></td><td>absolute path to a private writable directory (used to write undeclared test output annotation .part files).
+
+    </td><td>optional</td></tr>
+    <tr><td><code>TEST_WARNINGS_OUTPUT_FILE</code></td><td>absolute path to a private file in a writable directory (used to write test target warnings)</td><td>optional</td></tr>
+
+    <tr><td><code>TZ</code></td><td><code>UTC</code></td><td>required</td></tr>
+    <tr><td><code>USER</code></td><td>value of <code>getpwuid(getuid())-&gt;pw_name</code></td><td>required</td></tr>
+
+    <tr><td><code>XML_OUTPUT_FILE</code></td><td>Location of the <code>ANT</code>-like XML output file</td><td>optional</td></tr>
+    <tr><td><code>TEST_WORKSPACE</code></td><td>the local repository's workspace name</td><td>optional</td></tr>
+  </tbody>
+</table>
+<br>
+<p>The environment may contain additional entries.  Tests should not depend on the
+presence, absence, or value of any environment variable not listed above.</p>
+
+<p>The initial working directory shall be <code>$TEST_SRCDIR/$TEST_WORKSPACE</code>.</p>
+<p> The current process id, process group id, session id, and parent process
+id are unspecified.  The process may or may not be a process group leader or a
+session leader.  The process may or may not have a controlling terminal.  The
+process may have zero or more running or unreaped child processes.  The process
+should not have multiple threads when the test code gains control.</p>
+
+<p>File descriptor 0 (stdin) shall be open for reading, but what it is attached
+to is unspecified.  Tests must not read from it.  File descriptors 1 (stdout)
+and 2 (stderr) shall be open for writing, but what they are attached to is
+unspecified.  It could be a terminal, a pipe, a regular file, or anything else
+to which characters can be written.  They may share an entry in the open file
+table (meaning that they cannot seek independently).  Tests should not inherit
+any other open file descriptors.</p>
+
+<p>The initial umask shall be 022 or 027.</p>
+
+<p>No alarm or interval timer shall be pending.</p>
+
+<p>The initial mask of blocked signals shall be empty.  All signals shall be set
+to their default action.</p>
+
+<p>The initial resource limits, both soft and hard, should be set as follows:</p>
+
+<table class="table table-bordered table-striped table-condensed">
+  <thead>
+    <tr><th>Resource</th><th>Limit</th></tr>
+  </thead>
+  <tbody>
+    <tr><td>RLIMIT_AS</td><td>unlimited</td></tr>
+    <tr><td>RLIMIT_CORE</td><td>unspecified</td></tr>
+    <tr><td>RLIMIT_CPU</td><td>unlimited</td></tr>
+    <tr><td>RLIMIT_DATA</td><td>unlimited</td></tr>
+    <tr><td>RLIMIT_FSIZE</td><td>unlimited</td></tr>
+    <tr><td>RLIMIT_LOCKS</td><td>unlimited</td></tr>
+    <tr><td>RLIMIT_MEMLOCK</td><td>unlimited</td></tr>
+    <tr><td>RLIMIT_MSGQUEUE</td><td>unspecified</td></tr>
+    <tr><td>RLIMIT_NICE</td><td>unspecified</td></tr>
+    <tr><td>RLIMIT_NOFILE</td><td>at least 1024</td></tr>
+    <tr><td>RLIMIT_NPROC</td><td>unspecified</td></tr>
+    <tr><td>RLIMIT_RSS</td><td>unlimited</td></tr>
+    <tr><td>RLIMIT_RTPRIO</td><td>unspecified</td></tr>
+    <tr><td>RLIMIT_SIGPENDING</td><td>unspecified</td></tr>
+    <tr><td>RLIMIT_STACK</td><td>unlimited, or 2044KB &lt;= rlim &lt;= 8192KB</td></tr>
+  </tbody>
+</table>
+
+<p>The initial process times (as returned by <code>times()</code>) and resource
+utilization (as returned by <code>getrusage()</code>) are unspecified.</p>
+
+<p>The initial scheduling policy and priority are unspecified.</p>
+
+<h3>Role of the Host System</h3>
+
+<p>In addition to the aspects of user context under direct control of the
+test runner, the operating system on which tests execute must satisfy certain
+properties for a test run to be valid.</p>
+
+<h4>Filesystem</h4>
+
+<p>
+The root directory observed by a test may or may not be the real root directory.<br>
+<code>/proc</code> shall be mounted.<br>
+All build tools shall be present at the absolute paths under <code>/usr</code> used by a local installation.<br>
+Paths starting with <code>/home</code> may not be available. Tests should not access any such paths.<br>
+<code>/tmp</code> and <code>/export/hda3/tmp</code> shall be writable, but tests should avoid using these paths.<br>
+
+Tests must not assume that any constant path is available for their exclusive use.<br>
+Tests must not assume that atimes are enabled for any mounted filesystem.<br>
+</p>
+
+<h4>Users and groups</h4>
+
+<p>The users root, nobody, and unittest must exist. The groups root, nobody,
+and eng must exist.</p>
+
+<p>Tests must be executed as a non-root user.  The real and effective user ids
+must be equal; likewise for group ids.  Beyond this, the current user id, group
+id, user name, and group name are unspecified.  The set of supplementary group
+ids is unspecified.</p>
+
+<p>The current user id and group id must have corresponding names which can be
+retrieved with <code>getpwuid()</code> and <code>getgrgid()</code>.  The same
+may not be true for supplementary group ids.</p>
+
+<p>The current user must have a home directory. It may not be writable.  Tests
+must not attempt to write to it.</p>
+
+<h4>Networking</h4>
+
+<p>The hostname is unspecified. It may or may not contain a dot. Resolving
+the hostname must give an IP address of the current host. Resolving the
+hostname cut after the first dot must also work. The hostname localhost must
+resolve.</p>
+
+<h4>Other resources</h4>
+
+<p>Tests are granted at least one CPU core. Others may be available but this
+is not guaranteed. Other performance aspects of this core are not specified.
+You can increase the reservation to a higher number of CPU cores by adding the
+tag "cpu:n" (where n is a positive number) to a test rule. If a machine has
+less total CPU cores than requested, Bazel will still run the test. If a test
+uses sharding, each individual shard will reserve the number of CPU cores
+specified here.</p>
+
+<p>Tests may create subprocesses, but not process groups or sessions.</p>
+
+<p>There is a limit on both the number of input files a test may consume, and
+their aggregate size. Such limits are subject to change, but are currently in
+the range of tens of thousands of inputs and several GB of aggregate size.</p>
+
+<h4>Time and date</h4>
+
+<p>The current time and date are unspecified. The system timezone is unspecified.
+
+</p>
+
+<p>X Windows may or may not be available. Tests that need an X server should
+start Xvfb.</p>
+
+<h3>Test interaction with the filesystem</h3>
+<p>All file paths specified in test environment variables point to
+somewhere on the local filesystem, unless otherwise specified.</p>
+
+<p>
+Tests should create files only within the directories specified by
+<code>$TEST_TMPDIR</code> and <code>$TEST_UNDECLARED_OUTPUTS_DIR</code>
+(if set).<br>
+These directories will be initially empty.<br>
+Tests must not attempt to remove, chmod, or otherwise alter these directories.<br>
+These directories may be a symbolic links.<br>
+The filesystem type of <code>$TEST_TMPDIR/.</code> remains unspecified.<br>
+Tests may also write .part files to the <code>$TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR</code>
+to annotate undeclared output files.</p>
+
+<p>Tests must access inputs through the <b>runfiles</b> mechanism, or other
+parts of the execution environment which are specifically intended to make
+input files available.
+
+Tests must not access other outputs of the
+build system at paths inferred from the location of their own executable.</p>
+
+<p>It is unspecified whether the runfiles tree contains regular files, symbolic
+links, or a mixture. The runfiles tree may contain symlinks to directories.
+Tests should avoid using paths containing <code>..</code> components within the
+runfiles tree.</p>
+
+<p>No directory, file, or symlink within the runfiles tree (including paths
+which traverse symlinks) should be writable. (It follows that the initial
+working directory should not be writable.) Tests must not assume that any part
+of the runfiles is writable, or owned by the current user (i.e. chmod and chgrp
+may fail).</p>
+
+<p>The runfiles tree (including paths which traverse symlinks) must not change
+during test execution.  Parent directories and filesystem mounts must not change
+in any way which affects the result of resolving a path within the runfiles
+tree.</p>
+
+<p>In order to catch early exit, a test may create a file at the path specified by
+<code>TEST_PREMATURE_EXIT_FILE</code> upon start and remove it upon exit.  If
+Bazel sees the file when the test finishes, it will assume that the test exited
+prematurely and mark it as having failed.</p>
+
+<h3>Tag conventions</h3>
+
+<p>
+  Some tags in the test rules have a special
+  meaning.
+</p>
+
+<table class="table table-bordered table-striped table-condensed">
+  <thead>
+    <tr><th>Tag</th><th>Meaning</th></tr>
+  </thead>
+  <tbody>
+    <tr>
+      <th><code>exclusive</code></th>
+
+      <td>run no other test at the same time</td>
+    </tr>
+    <tr>
+      <th><code>external</code></th>
+      <td>test has an external dependency; disable test caching</td>
+    </tr>
+    <tr>
+      <th><code>large</code></th>
+      <td><code>test_suite</code> convention; suite of large tests<br/>
+
+      </td>
+    </tr>
+
+    <tr>
+      <th><code>manual</code></th>
+
+      <td>don't include test target in wildcard target patterns like <code>:...</code>, <code>:*</code>, or <code>:all</code>)</td>
+    </tr>
+    <tr>
+      <th><code>medium</code></th>
+
+      <td><code>test_suite</code> convention; suite of medium tests
+    </tr>
+
+    <tr>
+      <th><code>small</code></th>
+
+      <td><code>test_suite</code> convention; suite of small tests</td>
+    </tr>
+
+    <tr>
+      <th><code>smoke</code></th>
+
+      <td>
+        <code>test_suite</code> convention; means it should be run before committing code changes
+        into the version control system
+      </td>
+    </tr>
+
+  </tbody>
+</table>
+
+<h3>Runfiles</h3>
+
+<p>In the following, assume there is a *_binary() rule labeled <code>//foo/bar:unittest</code>,
+with a run-time dependency on the rule labeled <code>//deps/server:server</code>.</p>
+
+<h4>Location</h4>
+<p>The runfiles directory for a target <code>//foo/bar:unittest</code> is the directory
+<code>$(WORKSPACE)/$(BINDIR)/foo/bar/unittest.runfiles</code>. This path is referred to as the
+<code>runfiles_dir</code>.</p>
+
+<h4>Dependencies</h4>
+<p>The runfiles directory is declared as a compile-time dependency of the *_binary() rule.
+The runfiles directory itself depends on the set of BUILD files that affect the *_binary() rule
+or any of its compile-time or run-time dependencies. Modifying source files does not affect the
+structure of the runfiles directory, and thus does not trigger any rebuilding.</p>
+
+<h4>Contents</h4>
+<p>The runfiles directory contains the following:</p>
+<ul>
+  <li><b>Symlinks to run-time dependencies</b>: each OutputFile and CommandRule that is a run-time
+dependency of the *_binary() rule is represented by one symlink in the runfiles directory.
+The name of the symlink is <code>$(WORKSPACE)/package_name/rule_name</code>. For example, the
+symlink for server would be named <code>$(WORKSPACE)/deps/server/server</code>, and the full path
+would be <code>$(WORKSPACE)/foo/bar/unittest.runfiles/$(WORKSPACE)/deps/server/server</code>.
+The destination of the symlink is the OutputFileName() of the OutputFile or CommandRule,
+expressed as an absolute path. Thus, the destination of the symlink might be
+<code>$(WORKSPACE)/linux-dbg/deps/server/42/server</code>.</li>
+  <li><b>Symlinks to sub-runfiles</b>: for every *_binary() Z that is a run-time depdendency of
+*_binary() C, there is a second link in the runfiles directory of C to the runfiles of Z.
+The name of the symlink is <code>$(WORKSPACE)/package_name/rule_name.runfiles</code>.
+The target of the symlink is the runfiles directory. I.e. all subprograms share a common
+runfiles directory.</li>
+</ul>
+