From 95a54b98368a1c680c14e13d841b14a27aba01ca Mon Sep 17 00:00:00 2001 From: Damien Martin-Guillerez Date: Thu, 28 Jul 2016 12:47:11 +0000 Subject: Rollback of commit 3e8bcae69a0718cf6972be086706b1841e0ed6b7. *** Reason for rollback *** Breaks design docs links *** Original change description *** Move Bazel docs into versioned directory. * Move all Bazel docs (excluding main page, search page, and blog) into versions/master directory. * Replace all original pages with redirects. * Add Jekyll config with default_version setting to specify the default version to redirect docs to. * Add Jekyll config with version_prefix setting specific to pages under each version directory. * Update layouts to generate links to pages for the same version with the version_prefix. * Update Blaze rel... *** -- MOS_MIGRATED_REVID=128690580 --- site/docs/test-encyclopedia.html | 483 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 483 insertions(+) create mode 100644 site/docs/test-encyclopedia.html (limited to 'site/docs/test-encyclopedia.html') diff --git a/site/docs/test-encyclopedia.html b/site/docs/test-encyclopedia.html new file mode 100644 index 0000000000..bdb7e787f6 --- /dev/null +++ b/site/docs/test-encyclopedia.html @@ -0,0 +1,483 @@ +--- +layout: documentation +title: Test Encyclopedia +--- +

Writing tests

+ +

An Exhaustive Specification of the Test Execution Environment

+ +

Background

+ +

The Bazel BUILD language includes rules which can be used to define +automated test programs in many languages.

+ +

Tests are run using bazel test. + +Users may also execute test binaries directly. This is allowed but not endorsed, as such +an invocation will not adhere to the mandates described below.

+ +

Tests should be hermetic: that is, they ought to access only those +resources on which they have a declared dependency. If tests are not properly +hermetic then they do not give historically reproducible results. This could be a +significant problem for culprit finding (determining which change broke a test), +release engineering auditability, and resource isolation of tests (automated +testing frameworks ought not DDOS a server because some tests happen to +talk to it).

+ +

Objective

+ +

The goal of this document is to formally establish the runtime environment +for and expected behavior of Bazel tests. It will also impose requirements on +the test runner and the build system. + +Our intent is to help test authors avoid relying on unspecified +behavior, and thus give the testing infrastructure more freedom to make +implementation changes. We will also take the opportunity to tighten up some +holes which currently allow many tests to pass despite not being +properly hermetic, deterministic, and reentrant.

+ +

This document is intended to be both normative and authoritative. If +this specification and the implemented behavior of test runner disagree, the +specification takes precedence.

+ + +

Purpose of Tests

+ +

The purpose of Bazel tests is to confirm some property of the source files +checked into the repository. (In this document, "source files" includes test data, +golden outputs, and anything else kept under version control.) One +user writes a test to assert an invariant which they expect to be maintained. +Other users execute the test later to check whether the invariant has been +broken. If the test depends on any variables other than source files +(non-hermetic), its value is diminished, because the later users cannot be sure +their changes are at fault when the test stops passing.

+ +

Therefore the outcome of a test must depend only on:

+ + +

Currently, such behavior is not enforced. However, test runners reserve the +right to add such enforcement in the future.

+ +

Role of the Build System

+ +

Test rules are analogous to binary rules in that each must yield an +executable program. For some languages, this is a stub program which combines +a language-specific harness with the test code. Test rules must produce other +outputs as well. In addition to the primary test executable, the test runner +will need a manifest of runfiles, input files which should be made +available to the test at runtime, and it may need information about the type, +size, and tags of a test.

+ +

The build system may use the runfiles to deliver code as well as data. (This +might be used as an optimization to make each test binary smaller by sharing +files across tests, e.g. through the use of dynamic linking.) The build system +should ensure that the generated executable loads these files via the runfiles +image provided by the test runner, rather than hardcoded references to absolute +locations in the source or output tree.

+ +

Role of the Test Runner

+ +

From the point of view of the test runner, each test is a program which can +be invoked with execve(). There may be other ways to execute +tests; for example, an IDE might allow the execution of Java tests in-process. +However, the result of running the test as a standalone process must be +considered authoritative. If a test process runs to completion and terminates +normally with an exit code of zero, the test has passed. Any other result is +considered a test failure. In particular, writing any of the strings +PASS or FAIL to stdout has no significance to the test +runner.

+ +

If a test takes too long to execute, exceeds some resource limit, or the test +runner otherwise detects prohibited behavior, it may choose to kill the test +and treat the run as a failure. The runner must not report the test as passing +after sending a signal to the test process or any children thereof.

+ +

The whole test target (not individual methods or tests) is given a +limited amount of time to run to completion. The time limit for a test is based +on its timeout attribute according to the following table:

+ + + + + + + + + + + +
timeoutTime Limit (sec.)
short60
moderate300
long900
eternal3600
+ +

Tests which do not explicitly specify a timeout have one implied based on the +test's size as follows:

+ + + + + + + + + + + +
sizeImplied timeout label
smallshort
mediummoderate
largelong
enormouseternal
+

For example a "large" test with no explicit timeout setting will be allotted +900 seconds to run. A "medium" test with a timeout of "short" will be allotted +60 seconds.

+ +

All combinations of size and timeout labels are +legal, so an "enormous" test may be declared to have a timeout of "short". +Presumably it would do some really horrible things very quickly.

+

Tests may return arbitrarily fast regardless of timeout. A test is not +penalized for an overgenerous timeout, although a warning may be issued: you +should generally set your timeout as tight as you can without incurring any +flakiness.

+ +

There is also a recommended lower bound for test timeouts as follows:

+ + + + + + + + + + + +
sizeTime minimum (sec.)
short0
moderate30
long300
eternal900
+ +

For example, if a "moderate" test completes in 5.5s, consider setting +timeout="short" or size="small". Using the bazel +--test_verbose_timeout_warnings command line option will show the +tests whose specified size is too big.

+ +

Test sizes and timeouts are specified in the BUILD file according to the specification +here. +Any test that does not specify a recognized size will default to being a medium +test.

+ +

If the main process of a test exits, but some of its children are still +running, the test runner should consider the run complete and count it as a +success or failure based on the exit code observed from the main process. The +test runner may kill any stray processes. Tests should not leak processes in +this fashion.

+ +

When executing a test, the test runner must establish certain initial +conditions.

+ +

Initial Conditions

+ +

The test runner must invoke each test with the path to the test +executable in argv[0]. This path must be relative and +beneath the test's current directory (which is in the runfiles tree, +see below). +The test runner should not pass any other arguments to a +test unless the user explicitly requests it.

+ +

The initial environment block shall be composed as follows:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
VariableValueStatus
HOMEvalue of $TEST_TMPDIRrecommended
LANGunsetrequired
LANGUAGEunsetrequired
LC_ALLunsetrequired
LC_COLLATEunsetrequired
LC_CTYPEunsetrequired
LC_MESSAGESunsetrequired
LC_MONETARYunsetrequired
LC_NUMERICunsetrequired
LC_TIMEunsetrequired
LD_LIBRARY_PATHcolon-separated list of directories containing shared librariesoptional
JAVA_RUNFILESvalue of $TEST_SRCDIRdeprecated
LOGNAMEvalue of $USERrequired
PATH/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.recommended
PWD$TEST_SRCDIR/workspace-namerecommended
SHLVL2recommended
TEST_PREMATURE_EXIT_FILEabsolute path to a private file in a writable directory (used for catching calls to exit())optional
TEST_RANDOM_SEEDInherited from the client environment. If --runs_per_test option is used and client environment does not contain this variable, it is set to the run number (starting with 1) for each individual test run.optional
TEST_SIZEThe test sizeoptional
TEST_TIMEOUTThe test timeoutoptional
TEST_SRCDIRabsolute path to the base of the runfiles treerequired
TEST_TMPDIRabsolute path to a private writable directoryrequired
TEST_TIMEOUT300optional
TEST_UNDECLARED_OUTPUTS_DIRabsolute path to a private writable directory (used to write undeclared test outputs)optional
TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIRabsolute path to a private writable directory (used to write undeclared test output annotation .part files). + + optional
TEST_WARNINGS_OUTPUT_FILEabsolute path to a private file in a writable directory (used to write test target warnings)optional
TZUTCrequired
USERvalue of getpwuid(getuid())->pw_namerequired
XML_OUTPUT_FILELocation of the ANT-like XML output fileoptional
TEST_WORKSPACEthe local repository's workspace nameoptional
+
+

The environment may contain additional entries. Tests should not depend on the +presence, absence, or value of any environment variable not listed above.

+ +

The initial working directory shall be $TEST_SRCDIR/$TEST_WORKSPACE.

+

The current process id, process group id, session id, and parent process +id are unspecified. The process may or may not be a process group leader or a +session leader. The process may or may not have a controlling terminal. The +process may have zero or more running or unreaped child processes. The process +should not have multiple threads when the test code gains control.

+ +

File descriptor 0 (stdin) shall be open for reading, but what it is attached +to is unspecified. Tests must not read from it. File descriptors 1 (stdout) +and 2 (stderr) shall be open for writing, but what they are attached to is +unspecified. It could be a terminal, a pipe, a regular file, or anything else +to which characters can be written. They may share an entry in the open file +table (meaning that they cannot seek independently). Tests should not inherit +any other open file descriptors.

+ +

The initial umask shall be 022 or 027.

+ +

No alarm or interval timer shall be pending.

+ +

The initial mask of blocked signals shall be empty. All signals shall be set +to their default action.

+ +

The initial resource limits, both soft and hard, should be set as follows:

+ + + + + + + + + + + + + + + + + + + + + + +
ResourceLimit
RLIMIT_ASunlimited
RLIMIT_COREunspecified
RLIMIT_CPUunlimited
RLIMIT_DATAunlimited
RLIMIT_FSIZEunlimited
RLIMIT_LOCKSunlimited
RLIMIT_MEMLOCKunlimited
RLIMIT_MSGQUEUEunspecified
RLIMIT_NICEunspecified
RLIMIT_NOFILEat least 1024
RLIMIT_NPROCunspecified
RLIMIT_RSSunlimited
RLIMIT_RTPRIOunspecified
RLIMIT_SIGPENDINGunspecified
RLIMIT_STACKunlimited, or 2044KB <= rlim <= 8192KB
+ +

The initial process times (as returned by times()) and resource +utilization (as returned by getrusage()) are unspecified.

+ +

The initial scheduling policy and priority are unspecified.

+ +

Role of the Host System

+ +

In addition to the aspects of user context under direct control of the +test runner, the operating system on which tests execute must satisfy certain +properties for a test run to be valid.

+ +

Filesystem

+ +

+The root directory observed by a test may or may not be the real root directory.
+/proc shall be mounted.
+All build tools shall be present at the absolute paths under /usr used by a local installation.
+Paths starting with /home may not be available. Tests should not access any such paths.
+/tmp and /export/hda3/tmp shall be writable, but tests should avoid using these paths.
+ +Tests must not assume that any constant path is available for their exclusive use.
+Tests must not assume that atimes are enabled for any mounted filesystem.
+

+ +

Users and groups

+ +

The users root, nobody, and unittest must exist. The groups root, nobody, +and eng must exist.

+ +

Tests must be executed as a non-root user. The real and effective user ids +must be equal; likewise for group ids. Beyond this, the current user id, group +id, user name, and group name are unspecified. The set of supplementary group +ids is unspecified.

+ +

The current user id and group id must have corresponding names which can be +retrieved with getpwuid() and getgrgid(). The same +may not be true for supplementary group ids.

+ +

The current user must have a home directory. It may not be writable. Tests +must not attempt to write to it.

+ +

Networking

+ +

The hostname is unspecified. It may or may not contain a dot. Resolving +the hostname must give an IP address of the current host. Resolving the +hostname cut after the first dot must also work. The hostname localhost must +resolve.

+ +

Other resources

+ +

Tests are granted at least one CPU core. Others may be available but this +is not guaranteed. Other performance aspects of this core are not specified.

+ +

Tests may create subprocesses, but not process groups or sessions.

+ +

There is a limit on both the number of input files a test may consume, and +their aggregate size. Such limits are subject to change, but are currently in +the range of tens of thousands of inputs and several GB of aggregate size.

+ +

Time and date

+ +

The current time and date are unspecified. The system timezone is unspecified. + +

+ +

X Windows may or may not be available. Tests that need an X server should +start Xvfb.

+ +

Test interaction with the filesystem

+

All file paths specified in test environment variables point to +somewhere on the local filesystem, unless otherwise specified.

+ +

+Tests should create files only within the directories specified by +$TEST_TMPDIR and $TEST_UNDECLARED_OUTPUTS_DIR +(if set).
+These directories will be initially empty.
+Tests must not attempt to remove, chmod, or otherwise alter these directories.
+These directories may be a symbolic links.
+The filesystem type of $TEST_TMPDIR/. remains unspecified.
+Tests may also write .part files to the $TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR +to annotate undeclared output files.

+ +

Tests must access inputs through the runfiles mechanism, or other +parts of the execution environment which are specifically intended to make +input files available. + +Tests must not access other outputs of the +build system at paths inferred from the location of their own executable.

+ +

It is unspecified whether the runfiles tree contains regular files, symbolic +links, or a mixture. The runfiles tree may contain symlinks to directories. +Tests should avoid using paths containing .. components within the +runfiles tree.

+ +

No directory, file, or symlink within the runfiles tree (including paths +which traverse symlinks) should be writable. (It follows that the initial +working directory should not be writable.) Tests must not assume that any part +of the runfiles is writable, or owned by the current user (i.e. chmod and chgrp +may fail).

+ +

The runfiles tree (including paths which traverse symlinks) must not change +during test execution. Parent directories and filesystem mounts must not change +in any way which affects the result of resolving a path within the runfiles +tree.

+ +

In order to catch early exit, a test may create a file at the path specified by +TEST_PREMATURE_EXIT_FILE upon start and remove it upon exit. If +Bazel sees the file when the test finishes, it will assume that the test exited +prematurely and mark it as having failed.

+ +

Tag conventions

+ +

+ Some tags in the test rules have a special + meaning. +

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TagMeaning
exclusiverun no other test at the same time
externaltest has an external dependency; disable test caching
largetest_suite convention; suite of large tests
+ +
manualdon't include test target in wildcard target patterns like :..., :*, or :all)
mediumtest_suite convention; suite of medium tests +
smalltest_suite convention; suite of small tests
smoke + test_suite convention; means it should be run before committing code changes + into the version control system +
+ +

Runfiles

+ +

In the following, assume there is a *_binary() rule labeled //foo/bar:unittest, +with a run-time dependency on the rule labeled //deps/server:server.

+ +

Location

+

The runfiles directory for a target //foo/bar:unittest is the directory +$(WORKSPACE)/$(BINDIR)/foo/bar/unittest.runfiles. This path is referred to as the +runfiles_dir.

+ +

Dependencies

+

The runfiles directory is declared as a compile-time dependency of the *_binary() rule. +The runfiles directory itself depends on the set of BUILD files that affect the *_binary() rule +or any of its compile-time or run-time dependencies. Modifying source files does not affect the +structure of the runfiles directory, and thus does not trigger any rebuilding.

+ +

Contents

+

The runfiles directory contains the following:

+ + -- cgit v1.2.3