diff options
Diffstat (limited to 'test/README')
| -rw-r--r-- | test/README | 232 |
1 files changed, 49 insertions, 183 deletions
diff --git a/test/README b/test/README index af27d603..5861cc49 100644 --- a/test/README +++ b/test/README @@ -1,46 +1,21 @@ -Core GIT Tests -============== - -This directory holds many test scripts for core GIT tools. The -first part of this short document describes how to run the tests -and read their output. - -When fixing the tools or adding enhancements, you are strongly -encouraged to add tests in this directory to cover what you are -trying to fix or enhance. The later part of this short document -describes how your test scripts should be organized. +Notmuch test suite +================== +This directory contains the test suite for notmuch. +When fixing bugs or enhancing notmuch, you are strongly encouraged to +add tests in this directory to cover what you are trying to fix or +enhance. Running Tests ------------- +The easiest way to run tests is to say "make test", (or simply run the +notmuch-test script). Either command will run all available tests. + +Alternately, you can run a specific subset of tests by simply invoking +one of the executable scripts in this directory, (such as ./search, +./reply, etc.) -The easiest way to run tests is to say "make". This runs all -the tests. - - *** t0000-basic.sh *** - * ok 1: .git/objects should be empty after git-init in an empty repo. - * ok 2: .git/objects should have 256 subdirectories. - * ok 3: git-update-index without --add should fail adding. - ... - * ok 23: no diff after checkout and git-update-index --refresh. - * passed all 23 test(s) - *** t0100-environment-names.sh *** - * ok 1: using old names should issue warnings. - * ok 2: using old names but having new names should not issue warnings. - ... - -Or you can run each test individually from command line, like -this: - - $ sh ./t3001-ls-files-killed.sh - * ok 1: git-update-index --add to add various paths. - * ok 2: git-ls-files -k to show killed files. - * ok 3: validate git-ls-files -k output. - * passed all 3 test(s) - -You can pass --verbose (or -v), --debug (or -d), and --immediate -(or -i) command line argument to the test, or by setting GIT_TEST_OPTS -appropriately before running "make". +The following command-line options are available when running tests: --verbose:: This makes the test more verbose. Specifically, the @@ -55,12 +30,8 @@ appropriately before running "make". This causes the test to immediately exit upon the first failed test. ---long-tests:: - This causes additional long-running tests to be run (where - available), for more exhaustive testing. - --valgrind:: - Execute all Git binaries with valgrind and exit with status + Execute notmuch with valgrind and exit with status 126 on errors (just like regular tests, this will only stop the test script when running under -i). Valgrind errors go to stderr, so you might want to pass the -v option, too. @@ -75,95 +46,33 @@ appropriately before running "make". As the names depend on the tests' file names, it is safe to run the tests with this option in parallel. ---with-dashes:: - By default tests are run without dashed forms of - commands (like git-commit) in the PATH (it only uses - wrappers from ../bin-wrappers). Use this option to include - the build directory (..) in the PATH, which contains all - the dashed forms of commands. This option is currently - implied by other options like --valgrind and - GIT_TEST_INSTALLED. - -You can also set the GIT_TEST_INSTALLED environment variable to -the bindir of an existing git installation to test that installation. -You still need to have built this git sandbox, from which various -test-* support programs, templates, and perl libraries are used. -If your installed git is incomplete, it will silently test parts of -your built version instead. - -When using GIT_TEST_INSTALLED, you can also set GIT_TEST_EXEC_PATH to -override the location of the dashed-form subcommands (what -GIT_EXEC_PATH would be used for during normal operation). -GIT_TEST_EXEC_PATH defaults to `$GIT_TEST_INSTALLED/git --exec-path`. +When invoking the test suite via "make test" any of the above options +can be specified as follows: + make test OPTIONS="--verbose" Skipping Tests -------------- +If, for any reason, you need to skip one or more tests, you can do so +by setting the NOTMUCH_SKIP_TESTS variable to the name of one or more +sections of tests. -In some environments, certain tests have no way of succeeding -due to platform limitation, such as lack of 'unzip' program, or -filesystem that do not allow arbitrary sequence of non-NUL bytes -as pathnames. - -You should be able to say something like - - $ GIT_SKIP_TESTS=t9200.8 sh ./t9200-git-cvsexport-commit.sh - -and even: - - $ GIT_SKIP_TESTS='t[0-4]??? t91?? t9200.8' make - -to omit such tests. The value of the environment variable is a -SP separated list of patterns that tells which tests to skip, -and either can match the "t[0-9]{4}" part to skip the whole -test, or t[0-9]{4} followed by ".$number" to say which -particular test to skip. - -Note that some tests in the existing test suite rely on previous -test item, so you cannot arbitrarily disable one and expect the -remainder of test to check what the test originally was intended -to check. - - -Naming Tests ------------- - -The test files are named as: - - tNNNN-commandname-details.sh - -where N is a decimal digit. +For example: -First digit tells the family: + $ NOTMUCH_SKIP_TESTS="search reply" make test - 0 - the absolute basics and global stuff - 1 - the basic commands concerning database - 2 - the basic commands concerning the working tree - 3 - the other basic commands (e.g. ls-files) - 4 - the diff commands - 5 - the pull and exporting commands - 6 - the revision tree commands (even e.g. merge-base) - 7 - the porcelainish commands concerning the working tree - 8 - the porcelainish commands concerning forensics - 9 - the git tools +Even more fine-grained skipping is possible by appending a test number +(or glob pattern) after the section name. For example, the first +search test and the second reply test could be skipped with: -Second digit tells the particular command we are testing. - -Third digit (optionally) tells the particular switch or group of switches -we are testing. - -If you create files under t/ directory (i.e. here) that is not -the top-level test script, never name the file to match the above -pattern. The Makefile here considers all such files as the -top-level test script and tries to run all of them. A care is -especially needed if you are creating a common test library -file, similar to test-lib.sh, because such a library file may -not be suitable for standalone execution. + $ NOTMUCH_SKIP_TESTS="search.1 reply.2" make test +Note that some tests in the existing test suite rely on previous test +items, so you cannot arbitrarily skip any test and expect the +remaining tests to be unaffected. Writing Tests ------------- - The test script is written as a shell script. It should start with the standard "#!/bin/bash" with copyright notices, and an assignment to variable 'test_description', like this: @@ -175,13 +84,11 @@ assignment to variable 'test_description', like this: test_description='xxx test (option --frotz) - This test registers the following structure in the cache - and tries to run git-ls-files with option --frotz.' - + This test exercises the "notmuch xxx" command when + given the option --frotz.' Source 'test-lib.sh' -------------------- - After assigning test_description, the test script should source test-lib.sh like this: @@ -192,9 +99,8 @@ This test harness library does the following things: - If the script is invoked with command line argument --help (or -h), it shows the test_description and exits. - - Creates a test directory with default notmuch-config and empty mail - store. This directory is 't/trash directory.<test-basename>' (note - the space) if you must know, but I do not think you care. The path + - Creates a temporary directory with default notmuch-config and empty + mail store. This directory is 'test/tmp.<test-basename>'. The path to notmuch-config is exported in NOTMUCH_CONFIG environment variable and mail store path is stored in MAIL_DIR variable. @@ -203,18 +109,14 @@ This test harness library does the following things: consistently when command line arguments --verbose (or -v), --debug (or -d), and --immediate (or -i) is given. - End with test_done ------------------ - Your script will be a sequence of tests, using helper functions from the test harness library. At the end of the script, call 'test_done'. - Test harness library -------------------- - There are a handful helper functions defined in the test harness library for your script to use. @@ -224,12 +126,6 @@ library for your script to use. <script>. If it yields success, test is considered successful. <message> should state what it is testing. - Example: - - test_expect_success \ - 'git-write-tree should be able to write an empty tree.' \ - 'tree=$(git-write-tree)' - - test_expect_failure <message> <script> This is NOT the opposite of test_expect_success, but is used @@ -239,6 +135,21 @@ library for your script to use. success and "still broken" on failure. Failures from these tests won't cause -i (immediate) to stop. + - test_begin_subtest <message> + + Set the test description message for a subsequent test_expect_equal + invocation (see below). + + - test_expect_equal <output> <expected> + + This is an often-used convenience function built on top of + test_expect_success. It uses the message from the last + test_begin_subtest call, so call before calling + test_expect_equal. This function generates a successful test if + both the <output> and <expected> strings are identical. If not, it + will generate a failure and print the difference of the two + strings. + - test_debug <script> This takes a single argument, <script>, and evaluates it only @@ -251,49 +162,4 @@ library for your script to use. Your test script must have test_done at the end. Its purpose is to summarize successes and failures in the test script and exit with an appropriate error code. - - - test_tick - - Make commit and tag names consistent by setting the author and - committer times to defined stated. Subsequent calls will - advance the times by a fixed amount. - - - test_commit <message> [<filename> [<contents>]] - - Creates a commit with the given message, committing the given - file with the given contents (default for both is to reuse the - message string), and adds a tag (again reusing the message - string as name). Calls test_tick to make the SHA-1s - reproducible. - - - test_merge <message> <commit-or-tag> - - Merges the given rev using the given message. Like test_commit, - creates a tag and calls test_tick before committing. - -Tips for Writing Tests ----------------------- - -As with any programming projects, existing programs are the best -source of the information. However, do _not_ emulate -t0000-basic.sh when writing your tests. The test is special in -that it tries to validate the very core of GIT. For example, it -knows that there will be 256 subdirectories under .git/objects/, -and it knows that the object ID of an empty tree is a certain -40-byte string. This is deliberately done so in t0000-basic.sh -because the things the very basic core test tries to achieve is -to serve as a basis for people who are changing the GIT internal -drastically. For these people, after making certain changes, -not seeing failures from the basic test _is_ a failure. And -such drastic changes to the core GIT that even changes these -otherwise supposedly stable object IDs should be accompanied by -an update to t0000-basic.sh. - -However, other tests that simply rely on basic parts of the core -GIT working properly should not have that level of intimate -knowledge of the core GIT internals. If all the test scripts -hardcoded the object IDs like t0000-basic.sh does, that defeats -the purpose of t0000-basic.sh, which is to isolate that level of -validation in one place. Your test also ends up needing -updating when such a change to the internal happens, so do _not_ -do it and leave the low level of validation to t0000-basic.sh. +k
\ No newline at end of file |