Change the mover paths for docs to be under site/

-- MOS_MIGRATED_REVID=90252678
author: Kristina Chodorow <kchodorow@google.com> 2015-04-03 15:31:53 +0000
committer: Kristina Chodorow <kchodorow@google.com> 2015-04-03 20:37:00 +0000
commit: 5608a7aa9b3125738b605fee4836b7c6c9551d99 (patch)
tree: 06b5f4b3f8984d523e59b101a51bf6d538738cb9 /site/docs/test-encyclopedia.html
parent: 9d51432b6b51dd5d4abe5896470e6ca6a0d58d95 (diff)
1 files changed, 410 insertions, 0 deletions
diff --git a/site/docs/test-encyclopedia.html b/site/docs/test-encyclopedia.html
new file mode 100644
index 0000000000..2931933dd8
--- /dev/null
+++ b/site/docs/test-encyclopedia.html
@@ -0,0 +1,410 @@
+<!DOCTYPE html>
+
+<html>
+<head>
+  <meta charset="utf-8" />
+  <title>Bazel Test Encyclopedia</title>
+
+<body>
+<h1>Bazel Test Encyclopedia<br/>
+<small>An Exhaustive Specification of the Test Execution Environment</small></h1>
+
+<h2>Background</h2>
+
+<p>The Bazel BUILD language includes rules which can be used to define
+automated test programs in many languages.</p>
+
+<p>Tests are run using <code><a href="bazel-user-manual.html#test">bazel test</a></code>.
+
+Users may also execute test binaries directly. This is allowed but not endorsed, as such
+an invocation will not adhere to the mandates described below.</p>
+
+<p>Tests should be <i>hermetic</i>: that is, they ought to access only those
+resources on which they have a declared dependency. If tests are not properly
+hermetic then they do not give historically reproducible results. This could be a
+significant problem for culprit finding (determining which change broke a test),
+release engineering auditability, and resource isolation of tests (automated
+testing frameworks ought not DDOS a server because some tests happen to
+talk to it).<p>
+
+<h2>Objective</h2>
+
+<p>The goal of this document is to formally establish the runtime environment
+for and expected behavior of Bazel tests.  It will also impose requirements on
+the test runner and the build system.
+
+Our intent is to help test authors avoid relying on unspecified
+behavior, and thus give the testing infrastructure more freedom to make
+implementation changes.  We will also take the opportunity to tighten up some
+holes which currently allow many tests to pass despite not being
+properly hermetic, deterministic, and reentrant.</p>
+
+<p>This document is intended to be both normative and authoritative.  If
+this specification and the implemented behavior of test runner disagree, the
+specification takes precedence.</p>
+
+
+<h3>Purpose of Tests</h3>
+
+<p>The purpose of Bazel tests is to confirm some property of the source files
+checked into the repository.  (In this document, "source files" includes test data,
+golden outputs, and anything else kept under version control.)  One
+user writes a test to assert an invariant which they expect to be maintained.
+Other users execute the test later to check whether the invariant has been
+broken.  If the test depends on any variables other than source files
+(non-hermetic), its value is diminished, because the later users cannot be sure
+their changes are at fault when the test stops passing.</p>
+
+<p>Therefore the outcome of a test must depend only on:</p>
+<ul>
+  <li>source files on which the test has a declared dependency</li>
+  <li>products of the build system on which the test has a declared dependency</li>
+  <li>resources whose behavior is guaranteed by the test runner to remain constant</li>
+</ul>
+
+<p>Currently, such behavior is not enforced. However, test runners reserve the
+right to add such enforcement in the future.</p>
+
+<h3>Role of the Build System</h3>
+
+<p>Test rules are analogous to binary rules in that each must yield an
+executable program.  For some languages, this is a stub program which combines
+a language-specific harness with the test code.  Test rules must produce other
+outputs as well.  In addition to the primary test executable, the test runner
+will need a manifest of <b>runfiles</b>, input files which should be made
+available to the test at runtime, and it may need information about the type,
+size, and tags of a test.</p>
+
+<p>The build system may use the runfiles to deliver code as well as data.  (This
+might be used as an optimization to make each test binary smaller by sharing
+files across tests, e.g. through the use of dynamic linking.)  The build system
+should ensure that the generated executable loads these files via the runfiles
+image provided by the test runner, rather than hardcoded references to absolute
+locations in the source or output tree.</p>
+
+<h3>Role of the Test Runner</h3>
+
+<p>From the point of view of the test runner, each test is a program which can
+be invoked with <code>execve()</code>.  There may be other ways to execute
+tests; for example, an IDE might allow the execution of Java tests in-process.
+However, the result of running the test as a standalone process must be
+considered authoritative.  If a test process runs to completion and terminates
+normally with an exit code of zero, the test has passed.  Any other result is
+considered a test failure.  In particular, writing any of the strings
+<code>PASS</code> or <code>FAIL</code> to stdout has no significance to the test
+runner.</p>
+
+<p>If a test takes too long to execute, exceeds some resource limit, or the test
+runner otherwise detects prohibited behavior, it may choose to kill the test
+and treat the run as a failure.  The runner must not report the test as passing
+after sending a signal to the test process or any children thereof.</p>
+
+<p id="size">The whole test target (not individual methods or tests) is given a
+limited amount of time to run to completion. The time limit for a test is based
+on its timeout attribute according to the following table:</p>
+<table>
+  <tr><th>timeout</th><th>Time Limit (sec.)</th></tr>
+  <tr><td><code>short</code></td><td>60</td></tr>
+  <tr><td><code>moderate</code></td><td>300</td></tr>
+  <tr><td><code>long</code></td><td>900</td></tr>
+  <tr><td><code>eternal</code></td><td>3600</td></tr>
+</table>
+
+<p>Tests which do not explicitly specify a timeout have one implied based on the
+test's <code>size</code> as follows:</p>
+<table>
+  <tr><th>size</th><th>Implied timeout label</th></tr>
+  <tr><td><code>small</code></td><td>short</td></tr>
+  <tr><td><code>medium</code></td><td>moderate</td></tr>
+  <tr><td><code>large</code></td><td>long</td></tr>
+  <tr><td><code>enormous</code></td><td>eternal</td></tr>
+</table>
+<p>For example a "large" test with no explicit timeout setting will be allotted
+900 seconds to run. A "medium" test with a timeout of "short" will be allotted
+60 seconds.</p>
+
+<p>All combinations of <code>size</code> and <code>timeout</code> labels are
+legal, so an "enormous" test may be declared to have a timeout of "short".
+Presumably it would do some really horrible things very quickly.</p>
+<p>Tests may return arbitrarily fast regardless of timeout.  A test is not
+penalized for an overgenerous timeout, although a warning may be issued: you
+should generally set your timeout as tight as you can without incurring any
+flakiness.</p>
+
+<p>Test sizes and timeouts are specified in the BUILD file according to the specification
+<a href="build-encyclopedia.html#test">here</a>.
+Any test that does not specify a recognized size will default to being a medium
+test.</p>
+
+<p>If the main process of a test exits, but some of its children are still
+running, the test runner should consider the run complete and count it as a
+success or failure based on the exit code observed from the main process.  The
+test runner may kill any stray processes.  Tests should not leak processes in
+this fashion.</p>
+
+<p>When executing a test, the test runner must establish certain initial
+conditions.</p>
+
+<h3>Initial Conditions</h3>
+
+<p>The test runner must invoke each test with the path to the test
+executable in <code>argv[0]</code>.  This path must be relative and
+beneath the test's current directory (which is in the runfiles tree,
+see below).
+The test runner should not pass any other arguments to a
+test unless the user explicitly requests it.</p>
+
+<p>The initial environment block shall be composed as follows:</p>
+<table>
+  <tr><th>Variable</th><th>Value</th><th>Status</th></tr>
+
+  <tr><td><code>HOME</code></td><td>value of <code>$TEST_TMPDIR</code></td><td>recommended</td></tr>
+  <tr><td><code>JAVA_RUNFILES</code></td><td>value of <code>$TEST_SRCDIR</code></td><td>required</td></tr>
+  <tr><td><code>LANG</code></td><td><i>unset</i></td><td>required</td></tr>
+  <tr><td><code>LANGUAGE</code></td><td><i>unset</i></td><td>required</td></tr>
+  <tr><td><code>LC_ALL</code></td><td><i>unset</i></td><td>required</td></tr>
+  <tr><td><code>LC_COLLATE</code></td><td><i>unset</i></td><td>required</td></tr>
+  <tr><td><code>LC_CTYPE</code></td><td><i>unset</i></td><td>required</td></tr>
+  <tr><td><code>LC_MESSAGES</code></td><td><i>unset</i></td><td>required</td></tr>
+  <tr><td><code>LC_MONETARY</code></td><td><i>unset</i></td><td>required</td></tr>
+  <tr><td><code>LC_NUMERIC</code></td><td><i>unset</i></td><td>required</td></tr>
+  <tr><td><code>LC_TIME</code></td><td><i>unset</i></td><td>required</td></tr>
+  <tr><td><code>LD_LIBRARY_PATH</code></td><td>colon-separated list of directories containing shared libraries</td><td>optional</td></tr>
+
+  <tr><td><code>LOGNAME</code></td><td>value of <code>$USER</code></td><td>required</td></tr>
+
+  <tr><td><code>PATH</code></td><td><code>/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.</code></td><td>recommended</td></tr>
+  <tr><td><code>PWD</code></td><td><code>$TEST_SRCDIR</code></td><td>recommended</td></tr>
+  <tr><td><code>SHLVL</code></td><td><code>2</code></td><td>recommended</td></tr>
+  <tr><td><code>TEST_PREMATURE_EXIT_FILE</code></td><td>absolute path to a private file in a writable directory (used for catching calls to exit())</td><td>optional</td></tr>
+  <tr><td><code>TEST_RANDOM_SEED</code></td><td>See <a href="bazel-user-manual.html#other_options_for_blaze_test">this table</a>.</td><td>optional</td></tr>
+  <tr><td><code>TEST_SIZE</code></td><td>The test <a href="#size"><code>size</code></a></td><td>optional</td></tr>
+  <tr><td><code>TEST_SRCDIR</code></td><td>absolute path to the base of the runfiles tree</td><td>required</td></tr>
+  <tr><td><code>TEST_TMPDIR</code></td><td>absolute path to a private writable directory</td><td>required</td></tr>
+  <tr><td><code>TEST_TIMEOUT</code></td><td><code>300</code></td><td>optional</td></tr>
+  <tr><td><code>TEST_UNDECLARED_OUTPUTS_DIR</code></td><td>absolute path to a private writable directory (used to write undeclared test outputs)</td><td>optional</td></tr>
+  <tr><td><code>TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR</code></td><td>absolute path to a private writable directory (used to write undeclared test output annotation .part files).
+
+  </td><td>optional</td></tr>
+  <tr><td><code>TEST_WARNINGS_OUTPUT_FILE</code></td><td>absolute path to a private file in a writable directory (used to write test target warnings)</td><td>optional</td></tr>
+  <tr><td><code>TZ</code></td><td><code>US/Pacific</code></td><td>required</td></tr>
+
+  <tr><td><code>USER</code></td><td>value of <code>getpwuid(getuid())-&gt;pw_name</code></td><td>required</td></tr>
+
+</table>
+<br>
+<p>The environment may contain additional entries.  Tests should not depend on the
+presence, absence, or value of any environment variable not listed above.</p>
+
+<p>The initial working directory shall be <code>$TEST_SRCDIR</code>.</p>
+<p> The current process id, process group id, session id, and parent process
+id are unspecified.  The process may or may not be a process group leader or a
+session leader.  The process may or may not have a controlling terminal.  The
+process may have zero or more running or unreaped child processes.  The process
+should not have multiple threads when the test code gains control.</p>
+
+<p>File descriptor 0 (stdin) shall be open for reading, but what it is attached
+to is unspecified.  Tests must not read from it.  File descriptors 1 (stdout)
+and 2 (stderr) shall be open for writing, but what they are attached to is
+unspecified.  It could be a terminal, a pipe, a regular file, or anything else
+to which characters can be written.  They may share an entry in the open file
+table (meaning that they cannot seek independently).  Tests should not inherit
+any other open file descriptors.</p>
+
+<p>The initial umask shall be 022 or 027.</p>
+
+<p>No alarm or interval timer shall be pending.</p>
+
+<p>The initial mask of blocked signals shall be empty.  All signals shall be set
+to their default action.</p>
+
+<p>The initial resource limits, both soft and hard, should be set as follows:</p>
+<table>
+  <tr><th>Resource</th><th>Limit</th></tr>
+  <tr><td>RLIMIT_AS</td><td>unlimited</td></tr>
+  <tr><td>RLIMIT_CORE</td><td>unspecified</td></tr>
+  <tr><td>RLIMIT_CPU</td><td>unlimited</td></tr>
+  <tr><td>RLIMIT_DATA</td><td>unlimited</td></tr>
+  <tr><td>RLIMIT_FSIZE</td><td>unlimited</td></tr>
+  <tr><td>RLIMIT_LOCKS</td><td>unlimited</td></tr>
+  <tr><td>RLIMIT_MEMLOCK</td><td>unlimited</td></tr>
+  <tr><td>RLIMIT_MSGQUEUE</td><td>unspecified</td></tr>
+  <tr><td>RLIMIT_NICE</td><td>unspecified</td></tr>
+  <tr><td>RLIMIT_NOFILE</td><td>at least 1024</td></tr>
+  <tr><td>RLIMIT_NPROC</td><td>unspecified</td></tr>
+  <tr><td>RLIMIT_RSS</td><td>unlimited</td></tr>
+  <tr><td>RLIMIT_RTPRIO</td><td>unspecified</td></tr>
+  <tr><td>RLIMIT_SIGPENDING</td><td>unspecified</td></tr>
+  <tr><td>RLIMIT_STACK</td><td>unlimited, or 2044KB &lt;= rlim &lt;= 8192KB</td></tr>
+</table>
+
+<p>The initial process times (as returned by <code>times()</code>) and resource
+utilization (as returned by <code>getrusage()</code>) are unspecified.</p>
+
+<p>The initial scheduling policy and priority are unspecified.</p>
+
+<h3>Role of the Host System</h3>
+
+<p>In addition to the aspects of user context under direct control of the
+test runner, the operating system on which tests execute must satisfy certain
+properties for a test run to be valid.</p>
+
+<h4>Filesystem</h4>
+
+<p>
+The root directory observed by a test may or may not be the real root directory.<br>
+<code>/proc</code> shall be mounted.<br>
+All build tools shall be present at the absolute paths under <code>/usr</code> used by a local installation.<br>
+Paths starting with <code>/home</code> may not be available. Tests should not access any such paths.<br>
+<code>/tmp</code> and <code>/export/hda3/tmp</code> shall be writable, but tests should avoid using these paths.<br>
+
+Tests must not assume that any constant path is available for their exclusive use.<br>
+Tests must not assume that atimes are enabled for any mounted filesystem.<br>
+</p>
+
+<h4>Users and groups</h4>
+
+<p>The users root, nobody, and unittest must exist. The groups root, nobody,
+and eng must exist.</p>
+
+<p>Tests must be executed as a non-root user.  The real and effective user ids
+must be equal; likewise for group ids.  Beyond this, the current user id, group
+id, user name, and group name are unspecified.  The set of supplementary group
+ids is unspecified.</p>
+
+<p>The current user id and group id must have corresponding names which can be
+retrieved with <code>getpwuid()</code> and <code>getgrgid()</code>.  The same
+may not be true for supplementary group ids.</p>
+
+<p>The current user must have a home directory. It may not be writable.  Tests
+must not attempt to write to it.</p>
+
+<h4>Networking</h4>
+
+<p>The hostname is unspecified. It may or may not contain a dot. Resolving
+the hostname must give an IP address of the current host. Resolving the
+hostname cut after the first dot must also work. The hostname localhost must
+resolve.</p>
+
+<h4>Other resources</h4>
+
+<p>Tests are granted at least one CPU core. Others may be available but this
+is not guaranteed. Other performance aspects of this core are not specified.</p>
+
+<p>Tests may create subprocesses, but not process groups or sessions.</p>
+
+<p>There is a limit on both the number of input files a test may consume, and
+their aggregate size. Such limits are subject to change, but are currently in
+the range of tens of thousands of inputs and several GB of aggregate size.</p>
+
+<h4>Time and date</h4>
+
+<p>The current time and date are unspecified. The system timezone is unspecified.
+
+</p>
+
+<p>X Windows may or may not be available. Tests that need an X server should
+start Xvfb.</p>
+
+<h3>Test interaction with the filesystem</h3>
+<p>All file paths specified in test environment variables point to
+somewhere on the local filesystem, unless otherwise specified.</p>
+
+<p>
+Tests should create files only within the directories specified by
+<code>$TEST_TMPDIR</code> and <code>$TEST_UNDECLARED_OUTPUTS_DIR</code>
+(if set).<br>
+These directories will be initially empty.<br>
+Tests must not attempt to remove, chmod, or otherwise alter these directories.<br>
+These directories may be a symbolic links.<br>
+The filesystem type of <code>$TEST_TMPDIR/.</code> remains unspecified.<br>
+Tests may also write .part files to the <code>$TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR</code>
+to annotate undeclared output files.</p>
+
+<p>Tests must access inputs through the <b>runfiles</b> mechanism, or other
+parts of the execution environment which are specifically intended to make
+input files available.
+
+Tests must not access other outputs of the
+build system at paths inferred from the location of their own executable.</p>
+
+<p>It is unspecified whether the runfiles tree contains regular files, symbolic
+links, or a mixture. The runfiles tree may contain symlinks to directories.
+Tests should avoid using paths containing <code>..</code> components within the
+runfiles tree.</p>
+
+<p>No directory, file, or symlink within the runfiles tree (including paths
+which traverse symlinks) should be writable. (It follow that the initial
+working directory should not be writable.) Tests must not assume that any part
+of the runfiles is writable, or owned by the current user (i.e. chmod and chgrp
+may fail).</p>
+
+<p>The runfiles tree (including paths which traverse symlinks) must not change
+during test execution.  Parent directories and filesystem mounts must not change
+in any way which affects the result of resolving a path within the runfiles
+tree.</p>
+
+<p>In order to catch early exit, a test may create a file at the path specified by
+<code>TEST_PREMATURE_EXIT_FILE</code> upon start and remove it upon exit.  If
+Bazel sees the file when the test finishes, it will assume that the test exited
+prematurely and mark it as having failed.</p>
+
+<h3>Tag conventions</h3>
+
+<p>
+  Some <a href="bazel-user-manual.html#tags_keywords">tags</a> in the test rules have a special
+  meaning.
+</p>
+
+<table>
+<tr>
+  <th>Tag</th>
+  <th>Meaning</th>
+</tr>
+<tr>
+  <th><code>exclusive</code></th>
+
+  <td>run no other test at the same time</td>
+</tr>
+<tr>
+  <th><code>external</code></th>
+  <td>test has an external dependency; disable test caching</td>
+</tr>
+<tr>
+  <th><code>large</code></th>
+  <td><code>test_suite</code> convention; suite of large tests<br/>
+
+  </td>
+</tr>
+
+<tr>
+  <th><code>manual</code></th>
+
+  <td>run test only if mentioned explicitly (not in a wildcard <code>:...</code>, <code>:*</code>, or <code>:all</code>)</td>
+</tr>
+<tr>
+  <th><code>medium</code></th>
+
+  <td><code>test_suite</code> convention; suite of medium tests
+</tr>
+
+<tr>
+  <th><code>small</code></th>
+
+  <td><code>test_suite</code> convention; suite of small tests</td>
+</tr>
+
+<tr>
+  <th><code>smoke</code></th>
+
+  <td>
+    <code>test_suite</code> convention; means it should be run before committing code changes
+    into the version control system
+  </td>
+</tr>
+
+</table>
+
+</body>
+
+</html>
author	Kristina Chodorow <kchodorow@google.com>	2015-04-03 15:31:53 +0000
committer	Kristina Chodorow <kchodorow@google.com>	2015-04-03 20:37:00 +0000
commit	5608a7aa9b3125738b605fee4836b7c6c9551d99 (patch)
tree	06b5f4b3f8984d523e59b101a51bf6d538738cb9 /site/docs/test-encyclopedia.html
parent	9d51432b6b51dd5d4abe5896470e6ca6a0d58d95 (diff)