diff options
author | David Chen <dzc@google.com> | 2016-07-26 20:54:03 +0000 |
---|---|---|
committer | Damien Martin-Guillerez <dmarting@google.com> | 2016-07-27 11:15:14 +0000 |
commit | 3e8bcae69a0718cf6972be086706b1841e0ed6b7 (patch) | |
tree | ce6b37e16350f164d9ef937a69ba51558c99e53d /site/docs/test-encyclopedia.html | |
parent | 3b47b1fdc6b24bb2c947d02316c1cf4e6a02cf09 (diff) |
Move Bazel docs into versioned directory.
* Move all Bazel docs (excluding main page, search page, and blog)
into versions/master directory.
* Replace all original pages with redirects.
* Add Jekyll config with default_version setting to specify the default
version to redirect docs to.
* Add Jekyll config with version_prefix setting specific to pages under
each version directory.
* Update layouts to generate links to pages for the same version with the
version_prefix.
* Update Blaze release script to copy docs from
third_party/bazel/site/versions/master
Changes to follow this CL:
* Separate navigation from layouts so that navigation can be versioned
as well.
* Add tool for cutting a release of Bazel docs and copies them into a new
version directory.
Bug: #579
--
MOS_MIGRATED_REVID=128510319
Diffstat (limited to 'site/docs/test-encyclopedia.html')
-rw-r--r-- | site/docs/test-encyclopedia.html | 483 |
1 files changed, 0 insertions, 483 deletions
diff --git a/site/docs/test-encyclopedia.html b/site/docs/test-encyclopedia.html deleted file mode 100644 index bdb7e787f6..0000000000 --- a/site/docs/test-encyclopedia.html +++ /dev/null @@ -1,483 +0,0 @@ ---- -layout: documentation -title: Test Encyclopedia ---- -<h1>Writing tests</h1> - -<p class="lead">An Exhaustive Specification of the Test Execution Environment</p> - -<h2>Background</h2> - -<p>The Bazel BUILD language includes rules which can be used to define -automated test programs in many languages.</p> - -<p>Tests are run using <code><a href="bazel-user-manual.html#test">bazel test</a></code>. - -Users may also execute test binaries directly. This is allowed but not endorsed, as such -an invocation will not adhere to the mandates described below.</p> - -<p>Tests should be <i>hermetic</i>: that is, they ought to access only those -resources on which they have a declared dependency. If tests are not properly -hermetic then they do not give historically reproducible results. This could be a -significant problem for culprit finding (determining which change broke a test), -release engineering auditability, and resource isolation of tests (automated -testing frameworks ought not DDOS a server because some tests happen to -talk to it).<p> - -<h2>Objective</h2> - -<p>The goal of this document is to formally establish the runtime environment -for and expected behavior of Bazel tests. It will also impose requirements on -the test runner and the build system. - -Our intent is to help test authors avoid relying on unspecified -behavior, and thus give the testing infrastructure more freedom to make -implementation changes. We will also take the opportunity to tighten up some -holes which currently allow many tests to pass despite not being -properly hermetic, deterministic, and reentrant.</p> - -<p>This document is intended to be both normative and authoritative. If -this specification and the implemented behavior of test runner disagree, the -specification takes precedence.</p> - - -<h3>Purpose of Tests</h3> - -<p>The purpose of Bazel tests is to confirm some property of the source files -checked into the repository. (In this document, "source files" includes test data, -golden outputs, and anything else kept under version control.) One -user writes a test to assert an invariant which they expect to be maintained. -Other users execute the test later to check whether the invariant has been -broken. If the test depends on any variables other than source files -(non-hermetic), its value is diminished, because the later users cannot be sure -their changes are at fault when the test stops passing.</p> - -<p>Therefore the outcome of a test must depend only on:</p> -<ul> - <li>source files on which the test has a declared dependency</li> - <li>products of the build system on which the test has a declared dependency</li> - <li>resources whose behavior is guaranteed by the test runner to remain constant</li> -</ul> - -<p>Currently, such behavior is not enforced. However, test runners reserve the -right to add such enforcement in the future.</p> - -<h3>Role of the Build System</h3> - -<p>Test rules are analogous to binary rules in that each must yield an -executable program. For some languages, this is a stub program which combines -a language-specific harness with the test code. Test rules must produce other -outputs as well. In addition to the primary test executable, the test runner -will need a manifest of <b>runfiles</b>, input files which should be made -available to the test at runtime, and it may need information about the type, -size, and tags of a test.</p> - -<p>The build system may use the runfiles to deliver code as well as data. (This -might be used as an optimization to make each test binary smaller by sharing -files across tests, e.g. through the use of dynamic linking.) The build system -should ensure that the generated executable loads these files via the runfiles -image provided by the test runner, rather than hardcoded references to absolute -locations in the source or output tree.</p> - -<h3>Role of the Test Runner</h3> - -<p>From the point of view of the test runner, each test is a program which can -be invoked with <code>execve()</code>. There may be other ways to execute -tests; for example, an IDE might allow the execution of Java tests in-process. -However, the result of running the test as a standalone process must be -considered authoritative. If a test process runs to completion and terminates -normally with an exit code of zero, the test has passed. Any other result is -considered a test failure. In particular, writing any of the strings -<code>PASS</code> or <code>FAIL</code> to stdout has no significance to the test -runner.</p> - -<p>If a test takes too long to execute, exceeds some resource limit, or the test -runner otherwise detects prohibited behavior, it may choose to kill the test -and treat the run as a failure. The runner must not report the test as passing -after sending a signal to the test process or any children thereof.</p> - -<p id="timeout">The whole test target (not individual methods or tests) is given a -limited amount of time to run to completion. The time limit for a test is based -on its timeout attribute according to the following table:</p> - -<table class="table table-bordered table-striped table-condensed"> - <thead> - <tr><th>timeout</th><th>Time Limit (sec.)</th></tr> - </thead> - <tbody> - <tr><td><code>short</code></td><td>60</td></tr> - <tr><td><code>moderate</code></td><td>300</td></tr> - <tr><td><code>long</code></td><td>900</td></tr> - <tr><td><code>eternal</code></td><td>3600</td></tr> - </tbody> -</table> - -<p id="size">Tests which do not explicitly specify a timeout have one implied based on the -test's <code>size</code> as follows:</p> - -<table class="table table-bordered table-striped table-condensed"> - <thead> - <tr><th>size</th><th>Implied timeout label</th></tr> - </thead> - <tbody> - <tr><td><code>small</code></td><td>short</td></tr> - <tr><td><code>medium</code></td><td>moderate</td></tr> - <tr><td><code>large</code></td><td>long</td></tr> - <tr><td><code>enormous</code></td><td>eternal</td></tr> - </tbody> -</table> -<p>For example a "large" test with no explicit timeout setting will be allotted -900 seconds to run. A "medium" test with a timeout of "short" will be allotted -60 seconds.</p> - -<p>All combinations of <code>size</code> and <code>timeout</code> labels are -legal, so an "enormous" test may be declared to have a timeout of "short". -Presumably it would do some really horrible things very quickly.</p> -<p>Tests may return arbitrarily fast regardless of timeout. A test is not -penalized for an overgenerous timeout, although a warning may be issued: you -should generally set your timeout as tight as you can without incurring any -flakiness.</p> - -<p>There is also a recommended lower bound for test timeouts as follows: </p> - -<table class="table table-bordered table-striped table-condensed"> - <thead> - <tr><th>size</th><th>Time minimum (sec.)</th></tr> - </thead> - <tbody> - <tr><td><code>short</code></td><td>0</td></tr> - <tr><td><code>moderate</code></td><td>30</td></tr> - <tr><td><code>long</code></td><td>300</td></tr> - <tr><td><code>eternal</code></td><td>900</td></tr> - </tbody> -</table> - -<p>For example, if a "moderate" test completes in 5.5s, consider setting -<code>timeout</code>="short" or <code>size</code>="small". Using the bazel -<code>--test_verbose_timeout_warnings</code> command line option will show the -tests whose specified size is too big.</p> - -<p>Test sizes and timeouts are specified in the BUILD file according to the specification -<a href="be/common-definitions.html#common-attributes-tests">here</a>. -Any test that does not specify a recognized size will default to being a medium -test.</p> - -<p>If the main process of a test exits, but some of its children are still -running, the test runner should consider the run complete and count it as a -success or failure based on the exit code observed from the main process. The -test runner may kill any stray processes. Tests should not leak processes in -this fashion.</p> - -<p>When executing a test, the test runner must establish certain initial -conditions.</p> - -<h3>Initial Conditions</h3> - -<p>The test runner must invoke each test with the path to the test -executable in <code>argv[0]</code>. This path must be relative and -beneath the test's current directory (which is in the runfiles tree, -see below). -The test runner should not pass any other arguments to a -test unless the user explicitly requests it.</p> - -<p>The initial environment block shall be composed as follows:</p> - -<table class="table table-bordered table-striped table-condensed"> - <thead> - <tr><th>Variable</th><th>Value</th><th>Status</th></tr> - </thead> - <tbody> - - <tr><td><code>HOME</code></td><td>value of <code>$TEST_TMPDIR</code></td><td>recommended</td></tr> - <tr><td><code>LANG</code></td><td><i>unset</i></td><td>required</td></tr> - <tr><td><code>LANGUAGE</code></td><td><i>unset</i></td><td>required</td></tr> - <tr><td><code>LC_ALL</code></td><td><i>unset</i></td><td>required</td></tr> - <tr><td><code>LC_COLLATE</code></td><td><i>unset</i></td><td>required</td></tr> - <tr><td><code>LC_CTYPE</code></td><td><i>unset</i></td><td>required</td></tr> - <tr><td><code>LC_MESSAGES</code></td><td><i>unset</i></td><td>required</td></tr> - <tr><td><code>LC_MONETARY</code></td><td><i>unset</i></td><td>required</td></tr> - <tr><td><code>LC_NUMERIC</code></td><td><i>unset</i></td><td>required</td></tr> - <tr><td><code>LC_TIME</code></td><td><i>unset</i></td><td>required</td></tr> - <tr><td><code>LD_LIBRARY_PATH</code></td><td>colon-separated list of directories containing shared libraries</td><td>optional</td></tr> - <tr><td><code>JAVA_RUNFILES</code></td><td>value of <code>$TEST_SRCDIR</code></td><td>deprecated</td></tr> - - <tr><td><code>LOGNAME</code></td><td>value of <code>$USER</code></td><td>required</td></tr> - - <tr><td><code>PATH</code></td><td><code>/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.</code></td><td>recommended</td></tr> - <tr><td><code>PWD</code></td><td><code>$TEST_SRCDIR/<i>workspace-name</i></code></td><td>recommended</td></tr> - <tr><td><code>SHLVL</code></td><td><code>2</code></td><td>recommended</td></tr> - <tr><td><code>TEST_PREMATURE_EXIT_FILE</code></td><td>absolute path to a private file in a writable directory (used for catching calls to exit())</td><td>optional</td></tr> - <tr><td><code>TEST_RANDOM_SEED</code></td><td>Inherited from the client environment. If <code class='flag'>--runs_per_test</code> option is used and client environment does not contain this variable, it is set to the <var>run number</var> (starting with 1) for each individual test run.</td><td>optional</td></tr> - <tr><td><code>TEST_SIZE</code></td><td>The test <a href="#size"><code>size</code></a></td><td>optional</td></tr> - <tr><td><code>TEST_TIMEOUT</code></td><td>The test <a href="#timeout"><code>timeout</code></a></td><td>optional</td></tr> - <tr><td><code>TEST_SRCDIR</code></td><td>absolute path to the base of the runfiles tree</td><td>required</td></tr> - <tr><td><code>TEST_TMPDIR</code></td><td>absolute path to a private writable directory</td><td>required</td></tr> - <tr><td><code>TEST_TIMEOUT</code></td><td><code>300</code></td><td>optional</td></tr> - <tr><td><code>TEST_UNDECLARED_OUTPUTS_DIR</code></td><td>absolute path to a private writable directory (used to write undeclared test outputs)</td><td>optional</td></tr> - <tr><td><code>TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR</code></td><td>absolute path to a private writable directory (used to write undeclared test output annotation .part files). - - </td><td>optional</td></tr> - <tr><td><code>TEST_WARNINGS_OUTPUT_FILE</code></td><td>absolute path to a private file in a writable directory (used to write test target warnings)</td><td>optional</td></tr> - - <tr><td><code>TZ</code></td><td><code>UTC</code></td><td>required</td></tr> - - <tr><td><code>USER</code></td><td>value of <code>getpwuid(getuid())->pw_name</code></td><td>required</td></tr> - - <tr><td><code>XML_OUTPUT_FILE</code></td><td>Location of the <code>ANT</code>-like XML output file</td><td>optional</td></tr> - <tr><td><code>TEST_WORKSPACE</code></td><td>the local repository's workspace name</td><td>optional</td></tr> - - </tbody> -</table> -<br> -<p>The environment may contain additional entries. Tests should not depend on the -presence, absence, or value of any environment variable not listed above.</p> - -<p>The initial working directory shall be <code>$TEST_SRCDIR/$TEST_WORKSPACE</code>.</p> -<p> The current process id, process group id, session id, and parent process -id are unspecified. The process may or may not be a process group leader or a -session leader. The process may or may not have a controlling terminal. The -process may have zero or more running or unreaped child processes. The process -should not have multiple threads when the test code gains control.</p> - -<p>File descriptor 0 (stdin) shall be open for reading, but what it is attached -to is unspecified. Tests must not read from it. File descriptors 1 (stdout) -and 2 (stderr) shall be open for writing, but what they are attached to is -unspecified. It could be a terminal, a pipe, a regular file, or anything else -to which characters can be written. They may share an entry in the open file -table (meaning that they cannot seek independently). Tests should not inherit -any other open file descriptors.</p> - -<p>The initial umask shall be 022 or 027.</p> - -<p>No alarm or interval timer shall be pending.</p> - -<p>The initial mask of blocked signals shall be empty. All signals shall be set -to their default action.</p> - -<p>The initial resource limits, both soft and hard, should be set as follows:</p> - -<table class="table table-bordered table-striped table-condensed"> - <thead> - <tr><th>Resource</th><th>Limit</th></tr> - </thead> - <tbody> - <tr><td>RLIMIT_AS</td><td>unlimited</td></tr> - <tr><td>RLIMIT_CORE</td><td>unspecified</td></tr> - <tr><td>RLIMIT_CPU</td><td>unlimited</td></tr> - <tr><td>RLIMIT_DATA</td><td>unlimited</td></tr> - <tr><td>RLIMIT_FSIZE</td><td>unlimited</td></tr> - <tr><td>RLIMIT_LOCKS</td><td>unlimited</td></tr> - <tr><td>RLIMIT_MEMLOCK</td><td>unlimited</td></tr> - <tr><td>RLIMIT_MSGQUEUE</td><td>unspecified</td></tr> - <tr><td>RLIMIT_NICE</td><td>unspecified</td></tr> - <tr><td>RLIMIT_NOFILE</td><td>at least 1024</td></tr> - <tr><td>RLIMIT_NPROC</td><td>unspecified</td></tr> - <tr><td>RLIMIT_RSS</td><td>unlimited</td></tr> - <tr><td>RLIMIT_RTPRIO</td><td>unspecified</td></tr> - <tr><td>RLIMIT_SIGPENDING</td><td>unspecified</td></tr> - <tr><td>RLIMIT_STACK</td><td>unlimited, or 2044KB <= rlim <= 8192KB</td></tr> - </tbody> -</table> - -<p>The initial process times (as returned by <code>times()</code>) and resource -utilization (as returned by <code>getrusage()</code>) are unspecified.</p> - -<p>The initial scheduling policy and priority are unspecified.</p> - -<h3>Role of the Host System</h3> - -<p>In addition to the aspects of user context under direct control of the -test runner, the operating system on which tests execute must satisfy certain -properties for a test run to be valid.</p> - -<h4>Filesystem</h4> - -<p> -The root directory observed by a test may or may not be the real root directory.<br> -<code>/proc</code> shall be mounted.<br> -All build tools shall be present at the absolute paths under <code>/usr</code> used by a local installation.<br> -Paths starting with <code>/home</code> may not be available. Tests should not access any such paths.<br> -<code>/tmp</code> and <code>/export/hda3/tmp</code> shall be writable, but tests should avoid using these paths.<br> - -Tests must not assume that any constant path is available for their exclusive use.<br> -Tests must not assume that atimes are enabled for any mounted filesystem.<br> -</p> - -<h4>Users and groups</h4> - -<p>The users root, nobody, and unittest must exist. The groups root, nobody, -and eng must exist.</p> - -<p>Tests must be executed as a non-root user. The real and effective user ids -must be equal; likewise for group ids. Beyond this, the current user id, group -id, user name, and group name are unspecified. The set of supplementary group -ids is unspecified.</p> - -<p>The current user id and group id must have corresponding names which can be -retrieved with <code>getpwuid()</code> and <code>getgrgid()</code>. The same -may not be true for supplementary group ids.</p> - -<p>The current user must have a home directory. It may not be writable. Tests -must not attempt to write to it.</p> - -<h4>Networking</h4> - -<p>The hostname is unspecified. It may or may not contain a dot. Resolving -the hostname must give an IP address of the current host. Resolving the -hostname cut after the first dot must also work. The hostname localhost must -resolve.</p> - -<h4>Other resources</h4> - -<p>Tests are granted at least one CPU core. Others may be available but this -is not guaranteed. Other performance aspects of this core are not specified.</p> - -<p>Tests may create subprocesses, but not process groups or sessions.</p> - -<p>There is a limit on both the number of input files a test may consume, and -their aggregate size. Such limits are subject to change, but are currently in -the range of tens of thousands of inputs and several GB of aggregate size.</p> - -<h4>Time and date</h4> - -<p>The current time and date are unspecified. The system timezone is unspecified. - -</p> - -<p>X Windows may or may not be available. Tests that need an X server should -start Xvfb.</p> - -<h3>Test interaction with the filesystem</h3> -<p>All file paths specified in test environment variables point to -somewhere on the local filesystem, unless otherwise specified.</p> - -<p> -Tests should create files only within the directories specified by -<code>$TEST_TMPDIR</code> and <code>$TEST_UNDECLARED_OUTPUTS_DIR</code> -(if set).<br> -These directories will be initially empty.<br> -Tests must not attempt to remove, chmod, or otherwise alter these directories.<br> -These directories may be a symbolic links.<br> -The filesystem type of <code>$TEST_TMPDIR/.</code> remains unspecified.<br> -Tests may also write .part files to the <code>$TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR</code> -to annotate undeclared output files.</p> - -<p>Tests must access inputs through the <b>runfiles</b> mechanism, or other -parts of the execution environment which are specifically intended to make -input files available. - -Tests must not access other outputs of the -build system at paths inferred from the location of their own executable.</p> - -<p>It is unspecified whether the runfiles tree contains regular files, symbolic -links, or a mixture. The runfiles tree may contain symlinks to directories. -Tests should avoid using paths containing <code>..</code> components within the -runfiles tree.</p> - -<p>No directory, file, or symlink within the runfiles tree (including paths -which traverse symlinks) should be writable. (It follows that the initial -working directory should not be writable.) Tests must not assume that any part -of the runfiles is writable, or owned by the current user (i.e. chmod and chgrp -may fail).</p> - -<p>The runfiles tree (including paths which traverse symlinks) must not change -during test execution. Parent directories and filesystem mounts must not change -in any way which affects the result of resolving a path within the runfiles -tree.</p> - -<p>In order to catch early exit, a test may create a file at the path specified by -<code>TEST_PREMATURE_EXIT_FILE</code> upon start and remove it upon exit. If -Bazel sees the file when the test finishes, it will assume that the test exited -prematurely and mark it as having failed.</p> - -<h3>Tag conventions</h3> - -<p> - Some tags in the test rules have a special - meaning. -</p> - -<table class="table table-bordered table-striped table-condensed"> - <thead> - <tr><th>Tag</th><th>Meaning</th></tr> - </thead> - <tbody> - <tr> - <th><code>exclusive</code></th> - - <td>run no other test at the same time</td> - </tr> - <tr> - <th><code>external</code></th> - <td>test has an external dependency; disable test caching</td> - </tr> - <tr> - <th><code>large</code></th> - <td><code>test_suite</code> convention; suite of large tests<br/> - - </td> - </tr> - - <tr> - <th><code>manual</code></th> - - <td>don't include test target in wildcard target patterns like <code>:...</code>, <code>:*</code>, or <code>:all</code>)</td> - </tr> - <tr> - <th><code>medium</code></th> - - <td><code>test_suite</code> convention; suite of medium tests - </tr> - - <tr> - <th><code>small</code></th> - - <td><code>test_suite</code> convention; suite of small tests</td> - </tr> - - <tr> - <th><code>smoke</code></th> - - <td> - <code>test_suite</code> convention; means it should be run before committing code changes - into the version control system - </td> - </tr> - - </tbody> -</table> - -<h3>Runfiles</h3> - -<p>In the following, assume there is a *_binary() rule labeled <code>//foo/bar:unittest</code>, -with a run-time dependency on the rule labeled <code>//deps/server:server</code>.</p> - -<h4>Location</h4> -<p>The runfiles directory for a target <code>//foo/bar:unittest</code> is the directory -<code>$(WORKSPACE)/$(BINDIR)/foo/bar/unittest.runfiles</code>. This path is referred to as the -<code>runfiles_dir</code>.</p> - -<h4>Dependencies</h4> -<p>The runfiles directory is declared as a compile-time dependency of the *_binary() rule. -The runfiles directory itself depends on the set of BUILD files that affect the *_binary() rule -or any of its compile-time or run-time dependencies. Modifying source files does not affect the -structure of the runfiles directory, and thus does not trigger any rebuilding.</p> - -<h4>Contents</h4> -<p>The runfiles directory contains the following:</p> -<ul> - <li><b>Symlinks to run-time dependencies</b>: each OutputFile and CommandRule that is a run-time -dependency of the *_binary() rule is represented by one symlink in the runfiles directory. -The name of the symlink is <code>$(WORKSPACE)/package_name/rule_name</code>. For example, the -symlink for server would be named <code>$(WORKSPACE)/deps/server/server</code>, and the full path -would be <code>$(WORKSPACE)/foo/bar/unittest.runfiles/$(WORKSPACE)/deps/server/server</code>. -The destination of the symlink is the OutputFileName() of the OutputFile or CommandRule, -expressed as an absolute path. Thus, the destination of the symlink might be -<code>$(WORKSPACE)/linux-dbg/deps/server/42/server</code>.</li> - <li><b>Symlinks to sub-runfiles</b>: for every *_binary() Z that is a run-time depdendency of -*_binary() C, there is a second link in the runfiles directory of C to the runfiles of Z. -The name of the symlink is <code>$(WORKSPACE)/package_name/rule_name.runfiles</code>. -The target of the symlink is the runfiles directory. I.e. all subprograms share a common -runfiles directory.</li> -</ul> - |