aboutsummaryrefslogtreecommitdiffhomepage
path: root/test/README
Commit message (Collapse)AuthorAge
* test: implement and document NOTMUCH_TEST_QUIET variable usageGravatar Tomi Ollila2013-12-09
| | | | | | When NOTMUCH_TEST_QUIET environment variable is set to non-null value messages when new test script starts and when test PASSes are disabled. This eases picking the cases when tests FAIL (as those are still printed).
* test/README: mention the test_expect_equal_json and *sanitize* functionsGravatar Aaron Ecay2013-03-07
| | | | also fix one typo
* test: Make the emacsclient binary user-configurableGravatar Austin Clements2012-11-29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | And require that if TEST_EMACS is specified, so is TEST_EMACSCLIENT. Previously, the test framework always used "emacsclient", even if the Emacs in use was overridden by TEST_EMACS. This causes problems if both Emacs 23 and Emacs 24 are installed, the Emacs 23 emacsclient is the system default, but TEST_EMACS is set to emacs24. Specifically, with an Emacs 24 server and an Emacs 23 client, emacs tests that run very quickly may produce no output from emacsclient, causing the test to fail. The Emacs server uses a very simple line-oriented protocol in which the client sends a request to evaluate an expression and the server sends a request to print the result of evaluation. Prior to Emacs bzr commit 107565 on March 11th, 2012 (released in Emacs 24.1), if multiple commands were sent to the emacsclient between when it sent the evaluation command and when it entered its receive loop, it would only process the first response command, ignoring the rest of the received buffer. This wasn't a problem with the Emacs 23 server because it sent only the command to print the evaluation result. However, the Emacs 24 server first sends an unprompted command specifying the PID of the Emacs server, then processes the evaluation request, then sends the command to print the result. If the evaluation is fast enough, it can send both of these commands before emacsclient enters the receive loop. Hence, if an Emacs 24 server is used with an Emacs 23 emacsclient, it may miss the response printing command, ultimately causing intermittent notmuch test failures.
* Fixed 2 misspellings of word 'separate'Gravatar Tomi Ollila2012-11-16
| | | | | | | "Seperate: The second most common misspelling on the Internet..." (from http://www.re-vision.com/spelling/separate.html). s/seperate/separate/ in debian/NEWS.Debian & test/README done.
* test: make test_expect_equal_file() arguments flexibleGravatar Dmitry Kurochkin2012-09-01
| | | | | | | | | | | | | | | | | | | | | | | Before the change, test_expect_equal_file() function treated the first argument as "actual output file" and the second argument as "expected output file". When the test fails, the files are copied for later inspection. The first files was copied to "$testname.output" and the second file to "$testname.expected". The argument order for test_expect_equal_file() is often wrong which results in confusing diff output and incorrectly named files. The patch solves the issue by changing test_expect_equal_file() to treat arguments just as two files, without any special properties (like "actual" and "expected"). The file names for copying is now based on the given file name: "$testname.$file1" and "$testname.$file2". E.g. if test_expect_equal_file() is called with "OUTPUT" and "EXPECTED", the copied files can be named "emacs.1.OUTPUT" and "emacs.1.EXPECTED". The down side of this approach is that diff argument order depends on test_expect_equal_file() argument order. So sometimes we get diff from expected to actual results, and sometimes the other way around. But the files are always named correctly.
* test: Add `test_emacs_expect_t'.Gravatar David Edmondson2012-01-25
| | | | | | | | | Add a new test function to allow simpler testing of emacs functionality. `test_emacs_expect_t' takes one argument - a lisp expression to evaluate. The test passes if the expression returns `t', otherwise it fails and the output is reported to the tester.
* Document external dependencies in the test suiteGravatar Ethan Glasser-Camp2012-01-21
| | | | | | | | Add an explicit note to the README explaining what programs are necessary and the perhaps-surprising behavior of skipping tests if they aren't present. Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
* test: allow user to choose which emacs to run tests with.Gravatar David Bremner2011-12-28
| | | | | | | | | | | | As we start to pay more attention to emacs24, it helps to be able to select a different version of emacs to run the tests with to verify version specific bugs. A separate variable TEST_EMACS is needed to avoid being overwritten by the make variable EMACS in Makefile.config For what it's worth, the value of emacs is chosen at the time tmp.emacs/run_emacs is created, so is fixed for all subtests.
* test: add functions to count how much times notmuch was calledGravatar Dmitry Kurochkin2011-12-07
| | | | | | | | | | | | | | | | The patch adds two auxiliary functions and a variable: notmuch_counter_reset $notmuch_counter_command notmuch_counter_value They allow to count how many times notmuch binary is called. notmuch_counter_reset() function generates a script that counts how many times it is called and resets the counter to zero. The function sets $notmuch_counter_command variable to the path to the generated script that should be called instead of notmuch to do the counting. The notmuch_counter_value() function returns the current counter value.
* test: add test-binaries targetGravatar David Bremner2011-12-06
| | | | | The goal here is to have a simple way of making sure all of the binaries needed to run tests are available.
* test: remove documentation for test_expect_equal_failure.Gravatar David Bremner2011-09-10
| | | | The actual function was deleted in commit 4cc67276
* test: improve known broken tests supportGravatar Dmitry Kurochkin2011-09-10
| | | | | | | | | | | | | | | | | There is existing support for broken tests. But it is not convenient to use. The primary issue is that we have to maintain a set of test_expect_*_failure functions which are equivalent to the normal test_expect_* counterparts except for what functions are called for result reporting. The patch adds test_subtest_known_broken function which marks a subset as broken, making the normal test_expect_* functions behave as test_expect_*_failure. All test_expect_*_failure functions are removed. Test_known_broken_failure_ is changed to format details the same way as test_failure_ does. Another benefit of this change is that the diff when a broken test is fixed would be small and nice. Documentation is updated accordingly.
* test: update documentation for test_emacs in test/READMEGravatar Dmitry Kurochkin2011-09-10
| | | | | | Update test_emacs documentation in test/README according to the latest changes in emacs tests. Move the note regarding setting variables from test/emacs to test/README.
* test: document test_expect_equal_fileGravatar Jameson Graef Rollins2011-06-29
| | | | | This test was not properly documented when it was originally added (my bad).
* test:Improve test behaviors when --root is usedGravatar Mark Anderson2011-06-28
| | | | | | | | | | | Change add_email_corpus, emacs_deliver_message and tests to use $TEST_DIRECTORY instead of '..'. This improves the behavior of the usage of --root=<dir>, as the assumption of what '..' means will usually be incorrect. Document -root option in README and update valgrind to work with -root.
* fix sum moar typos [text files]Gravatar Pieter Praet2011-06-23
| | | | | | | | | Various typo fixes in auxiliary text files included with the source, (README, TODO, etc.). Signed-off-by: Pieter Praet <pieter@praet.org> Edited-by: Carl Worth <cworth@cworth.org> Restricted to just text files.
* test: change "#!/bin/bash" to "#!/usr/bin/env bash" enhances portabilityGravatar Joel Borggrén-Franck2011-05-27
| | | | | | Change #!/bin/bash at start of tests to "#!/usr/bin/env bash". That way systems running on bash < 4 can prepend bash >= 4 to path before running the tests.
* test: Add a new test_expect_equal_failureGravatar Carl Worth2010-10-22
| | | | | Which allows us to have a known-broken test that would otherwise use test_expect_equal.
* test: Add a new test_emacs function to test-lib.shGravatar Carl Worth2010-10-22
| | | | | | | | This should be quite handy for doing automated testing of the emacs-based functionality in notmuch. This function invokes emacs with the necessary command-line arguments, (to run in batch mode with no local initialization, to load the notmuch code from the source directory, and to ensure an 80-column width).
* test/README: Document add_email_corpus (and add_message/generate_message)Gravatar Carl Worth2010-09-20
| | | | | | While adding the documentation here for add_email_corpus I noticed that the other email-adding functions in test-lib.sh were not yet documented here, so add all of that documentation.
* test: Make the --valgrind option useful, (and drop --verbose).Gravatar Carl Worth2010-09-20
| | | | | | | | | In order for --valgrind to be useful, we drop noisy additional output of all of the commands being executed in verbose mode. This makes --verbose alone quite useless, so we don't document it any more. Also, add a zlib valgrind suppression that was showing up frequently in the test suite.
* test/README: Update to become notmuch-specific rather than git-specificGravatar Carl Worth2010-09-20
| | | | | | This file was obviously describing the git test suite previously, and would have been very hard to understand in the context of the notmuch test suite. HOpefully it's easier to follow now.
* Update test framework for use with notmuchGravatar Michal Sojka2010-09-16
| | | | | | | | This removes Git specific things from the test-lib.sh and adds helper functions for notmuch taken from Carl's notmuch-test script. README is also slightly modified to reflect the current state. Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
* Copy test framework from GitGravatar Michal Sojka2010-09-16
Git uses a simple and yet powerful test framework, written in shell. The framework is easy to use for both users and developers so I think it would help if it is used in notmuch as well. This is a copy of Git's test framework from commit b6b0afdc30e066788592ca07c9a6c6936c68cc11 in git repository. Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>