diff options
author | David Bremner <bremner@debian.org> | 2012-12-16 08:33:17 -0400 |
---|---|---|
committer | David Bremner <bremner@debian.org> | 2012-12-25 08:49:24 -0400 |
commit | 098ef4af4d0a52a6b4daed5324a7c77f6c9108da (patch) | |
tree | 64bda0a95d08df65c86f2448d6d988e53bb9e3d6 /performance-test/README | |
parent | 047cd7a595f760a23415b9e38d7136325f28b114 (diff) |
perf-test: initial version of memory test infrastructure.
The idea is run some code under valgrind --leak-check=full and report
a summary, leaving the user to peruse the log file if they want.
We go to some lengths to preserve the log files from accidental
overwriting; the full corpus takes about 3 hours to run under valgrind
on my machine.
The naming of the log directories may be slightly controversial; in
the unlikely event of two runs in less than a second, the log will be
overwritten. A previous version with mktemp+timestamp was dismissed as
overkill; just mktemp alone does not sort nicely.
One new test is included, to check notmuch new for memory leaks.
Diffstat (limited to 'performance-test/README')
-rw-r--r-- | performance-test/README | 57 |
1 files changed, 39 insertions, 18 deletions
diff --git a/performance-test/README b/performance-test/README index d1fb6de4..996724cd 100644 --- a/performance-test/README +++ b/performance-test/README @@ -1,3 +1,10 @@ +Performance Tests +----------------- + +This directory contains two kinds of performance tests: time tests, +and memory tests. The former use gnu time, and the latter use +valgrind. + Pre-requisites -------------- @@ -5,9 +12,10 @@ In addition to having notmuch, you need: - gpg - gnu tar -- gnu time +- gnu time (for the time tests) - xz. Some speedup can be gotten by installing "pixz", but this is probably only worthwhile if you are debugging the tests. +- valgrind (for the memory tests) Getting set up to run tests: ---------------------------- @@ -36,34 +44,47 @@ for a list of mirrors. Running tests ------------- -The easiest way to run performance tests is to say "make time-test", (or -simply run the notmuch-time-test script). Either command will run all -available performance tests. - -Alternately, you can run a specific subset of tests by simply invoking -one of the executable scripts in this directory, (such as ./basic). -Each test script supports the following arguments +The easiest way to run performance tests is to say "make perf-test". +This will run all time and memory tests. Be aware that the memory +tests are quite time consuming when run on the full corpus, and that +depending on your interests it may be more sensible to run "make +time-test" or "make memory-test". You can also invoke one of the +scripts notmuch-time-test or notmuch-memory-test or run a more +specific subset of tests by simply invoking one of the executable +scripts in this directory, (such as ./T00-new). Each test script +supports the following arguments --small / --medium / --large Choose corpus size. --debug Enable debugging. In particular don't delete temporary directories. +When using the make targets, you can pass arguments to all test +scripts by defining the make variable OPTIONS. + Writing tests ------------- -Have a look at "T01-dump-restore" for an example. Sourcing -"perf-test-lib.sh" is mandatory. Utility functions include +Have a look at "T01-dump-restore" for an example time test and +"M00-new" for an example memory test. In both cases sourcing +"perf-test-lib.sh" is mandatory. -- 'add_email_corpus' unpacks a set of messages and adds them to the database. -- 'cache_database': makes a snapshot of the current database -- 'uncache_database': forces the next 'add_email_corpus' to rebuild the - database. -- 'time_start' unpacks the mail corpus and calls notmuch new if it +Basics: + +- '(time|memory)_start' unpacks the mail corpus and calls notmuch new if it cannot find a cache of the appropriate corpus. -- 'time_done' does the cleanup; comment it out or pass --debug to the +- '(time|memory)_run' runs the command under time or valgrind. Currently + "memory_run" does not support i/o redirection in the command. +- '(time|memory)_done' does the cleanup; comment it out or pass --debug to the script to leave the temporary files around. +Utility functions include + +- 'add_email_corpus' unpacks a set of messages and tags +- 'cache_database': makes a snapshot of the current database +- 'uncache_database': forces the next '(time|memory)_start' to rebuild the + database. + Scripts are run in the order specified in notmuch-perf-test. In the future this order might be chosen automatically so please follow the -convention of starting the name with 'T' followed by two digits to -specify the order. +convention of starting the name with 'T' or 'M' followed by two digits +to specify the order. |