Merge branch 'release'

merge 0.13.1 bugfix patches back to master, fixes for emacs reply and
spurious directory document creation.

10 files changed, 120 insertions, 103 deletions

diff --git a/NEWS b/NEWS
index 69a72030..668281c5 100644
--- a/NEWS
+++ b/NEWS
@@ -1,24 +1,17 @@
-Notmuch 0.13 (2012-xx-xx)
+Notmuch 0.13 (2012-05-15)
 =========================
 
 Command-Line Interface
 ----------------------
 
-Reply to sender
-
-  "notmuch reply" has gained the ability to create a reply template
-  for replying just to the sender of the message, in addition to reply
-  to all. The feature is available through the new command line option
-  --reply-to=(all|sender).
-
 JSON reply format
 
-  "notmuch reply" can now produce JSON output that contains the headers
+  `notmuch reply` can now produce JSON output that contains the headers
   for a reply message and full information about the original message
   begin replied to. This allows MUAs to create replies intelligently.
   For example, an MUA that can parse HTML might quote HTML parts.
 
-  Calling notmuch reply with --format=json imposes the restriction that
+  Calling notmuch reply with `--format=json` imposes the restriction that
   only a single message is returned by the search, as replying to
   multiple messages does not have a well-defined behavior. The default
   retains its current behavior for multiple message replies.
@@ -26,24 +19,24 @@ JSON reply format
 Tag exclusion
 
   Tags can be automatically excluded from search results by adding them
-  to the new 'search.exclude_tags' option in the Notmuch config file.
+  to the new `search.exclude_tags` option in the Notmuch config file.
 
   This behaviour can be overridden by explicitly including an excluded
   tag in your query, for example:
 
-    notmuch search $your_query and tag:$excluded_tag
+        notmuch search $your_query and tag:$excluded_tag
 
-  Existing users will probably want to run "notmuch setup" again to add
+  Existing users will probably want to run `notmuch setup` again to add
   the new well-commented [search] section to the configuration file.
 
   For new configurations, accepting the default setting will cause the
   tags "deleted" and "spam" to be excluded, equivalent to running:
 
-    notmuch config set search.exclude_tags deleted spam
+        notmuch config set search.exclude_tags deleted spam
 
 Raw show format changes
 
-  The output of show --format=raw has changed for multipart and
+  The output of show `--format=raw` has changed for multipart and
   message parts.  Previously, the output was a mash of somewhat-parsed
   headers and transfer-decoded bodies.  Now, such parts are reproduced
   faithfully from the original source.  Message parts (which includes
@@ -54,7 +47,7 @@ Raw show format changes
 
 Listing configuration items
 
-  The new "config list" command prints out all configuration items and
+  The new `config list` command prints out all configuration items and
   their values.
 
 Emacs Interface
@@ -64,7 +57,7 @@ Changes to tagging interface
 
   The user-facing tagging functions in the Emacs interface have been
   normalized across all notmuch modes.  The tagging functions are now
-  'notmuch-search-tag' in search-mode, and 'notmuch-show-tag' in
+  notmuch-search-tag in search-mode, and notmuch-show-tag in
   show-mode.  They accept a string representing a single tag change,
   or a list of tag changes.  See 'M-x describe-function notmuch-tag'
   for more information.
@@ -83,11 +76,11 @@ Reply improvement using the JSON format
 New add-on tool: notmuch-mutt
 -----------------------------
 
-The new contrib/ tool "notmuch-mutt" provides Notmuch integration for
+The new contrib/ tool `notmuch-mutt` provides Notmuch integration for
 the Mutt mail user agent. Using it, Mutt users can perform mail
 search, thread reconstruction, and mail tagging/untagging without
 leaving Mutt.  notmuch-mutt, formerly distributed under the name
-"mutt-notmuch" by Stefano Zacchiroli, will be maintained as a notmuch
+`mutt-notmuch` by Stefano Zacchiroli, will be maintained as a notmuch
 contrib/ from now on.
 
 Library changes
@@ -96,26 +89,26 @@ Library changes
 The API changes detailed below break binary and source compatibility,
 so libnotmuch has been bumped to version 3.0.0.
 
-The function notmuch_database_close has been split into
-notmuch_database_close and notmuch_database_destroy
+The function `notmuch_database_close` has been split into
+`notmuch_database_close` and `notmuch_database_destroy`
 
   This makes it possible for long running programs to close the xapian
   database and thus release the lock associated with it without
   destroying the data structures obtained from it.
 
-notmuch_database_open, notmuch_database_create, and
-notmuch_database_get_directory now return errors
+`notmuch_database_open`, `notmuch_database_create`, and
+`notmuch_database_get_directory` now return errors
 
   The type signatures of these functions have changed so that the
-  functions now return a notmuch_status_t and take an out-argument for
+  functions now return a `notmuch_status_t` and take an out-argument for
   returning the new database object or directory object.
 
-go bindings changes
+Go bindings changes
 -------------------
 
 Go 1 compatibility
 
-  The go bindings and the notmuch-addrlookup utility are now
+  The go bindings and the `notmuch-addrlookup` utility are now
   compatible with go 1.
 
 Notmuch 0.12 (2012-03-20)
diff --git a/bindings/python/notmuch/database.py b/bindings/python/notmuch/database.py
index ee0366cd..5931f41b 100644
--- a/bindings/python/notmuch/database.py
+++ b/bindings/python/notmuch/database.py
@@ -347,7 +347,6 @@ class Database(object):
 
     def get_directory(self, path):
         """Returns a :class:`Directory` of path,
-        (creating it if it does not exist(?))
 
         :param path: An unicode string containing the path relative to the path
               of database (see :meth:`get_path`), or else should be an absolute
@@ -355,8 +354,6 @@ class Database(object):
         :returns: :class:`Directory` or raises an exception.
         :raises: :exc:`FileError` if path is not relative database or absolute
                  with initial components same as database.
-        :raises: :exc:`ReadOnlyDatabaseError` if the database has not been
-                 opened in read-write mode
         """
         self._assert_db_is_initialized()
 
@@ -530,19 +527,10 @@ class Database(object):
                  retry.
         :raises: :exc:`NotInitializedError` if the database was not
                  intitialized.
-        :raises: :exc:`ReadOnlyDatabaseError` if the database has not been
-                 opened in read-write mode
 
         *Added in notmuch 0.9*"""
         self._assert_db_is_initialized()
 
-        # work around libnotmuch calling exit(3), see
-        # id:20120221002921.8534.57091@thinkbox.jade-hamburg.de
-        # TODO: remove once this issue is resolved
-        if self.mode != Database.MODE.READ_WRITE:
-            raise ReadOnlyDatabaseError('The database has to be opened in '
-                                        'read-write mode for get_directory')
-
         msg_p = NotmuchMessageP()
         status = Database._find_message_by_filename(self._db, _str(filename),
                                                     byref(msg_p))
diff --git a/emacs/notmuch-lib.el b/emacs/notmuch-lib.el
index 7fa441af..e99b48d1 100644
--- a/emacs/notmuch-lib.el
+++ b/emacs/notmuch-lib.el
@@ -244,7 +244,12 @@ the given type."
 current buffer, if possible."
   (let ((display-buffer (current-buffer)))
     (with-temp-buffer
-      (let* ((charset (plist-get part :content-charset))
+      ;; In case there is :content, the content string is already converted
+      ;; into emacs internal format. `gnus-decoded' is a fake charset,
+      ;; which means no further decoding (to be done by mm- functions).
+      (let* ((charset (if (plist-member part :content)
+			  'gnus-decoded
+			(plist-get part :content-charset)))
 	     (handle (mm-make-handle (current-buffer) `(,content-type (charset . ,charset)))))
 	;; If the user wants the part inlined, insert the content and
 	;; test whether we are able to inline it (which includes both
diff --git a/lib/database.cc b/lib/database.cc
index f8c4a7d1..761dc1a2 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -956,7 +956,7 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
 		    document.get_value (NOTMUCH_VALUE_TIMESTAMP));
 
 		directory = _notmuch_directory_create (notmuch, term.c_str() + 10,
-						       &status);
+						       NOTMUCH_FIND_CREATE, &status);
 		notmuch_directory_set_mtime (directory, mtime);
 		notmuch_directory_destroy (directory);
 	    }
@@ -1197,9 +1197,17 @@ _notmuch_database_split_path (void *ctx,
     return NOTMUCH_STATUS_SUCCESS;
 }
 
+/* Find the document ID of the specified directory.
+ *
+ * If (flags & NOTMUCH_FIND_CREATE), a new directory document will be
+ * created if one does not exist for 'path'.  Otherwise, if the
+ * directory document does not exist, this sets *directory_id to
+ * ((unsigned int)-1) and returns NOTMUCH_STATUS_SUCCESS.
+ */
 notmuch_status_t
 _notmuch_database_find_directory_id (notmuch_database_t *notmuch,
 				     const char *path,
+				     notmuch_find_flags_t flags,
 				     unsigned int *directory_id)
 {
     notmuch_directory_t *directory;
@@ -1210,8 +1218,8 @@ _notmuch_database_find_directory_id (notmuch_database_t *notmuch,
 	return NOTMUCH_STATUS_SUCCESS;
     }
 
-    directory = _notmuch_directory_create (notmuch, path, &status);
-    if (status) {
+    directory = _notmuch_directory_create (notmuch, path, flags, &status);
+    if (status || !directory) {
 	*directory_id = -1;
 	return status;
     }
@@ -1240,13 +1248,16 @@ _notmuch_database_get_directory_path (void *ctx,
  * database path), return a new string (with 'ctx' as the talloc
  * owner) suitable for use as a direntry term value.
  *
- * The necessary directory documents will be created in the database
- * as needed.
+ * If (flags & NOTMUCH_FIND_CREATE), the necessary directory documents
+ * will be created in the database as needed.  Otherwise, if the
+ * necessary directory documents do not exist, this sets
+ * *direntry to NULL and returns NOTMUCH_STATUS_SUCCESS.
  */
 notmuch_status_t
 _notmuch_database_filename_to_direntry (void *ctx,
 					notmuch_database_t *notmuch,
 					const char *filename,
+					notmuch_find_flags_t flags,
 					char **direntry)
 {
     const char *relative, *directory, *basename;
@@ -1260,10 +1271,12 @@ _notmuch_database_filename_to_direntry (void *ctx,
     if (status)
 	return status;
 
-    status = _notmuch_database_find_directory_id (notmuch, directory,
+    status = _notmuch_database_find_directory_id (notmuch, directory, flags,
 						  &directory_id);
-    if (status)
+    if (status || directory_id == (unsigned int)-1) {
+	*direntry = NULL;
 	return status;
+    }
 
     *direntry = talloc_asprintf (ctx, "%u:%s", directory_id, basename);
 
@@ -1315,12 +1328,9 @@ notmuch_database_get_directory (notmuch_database_t *notmuch,
 	return NOTMUCH_STATUS_NULL_POINTER;
     *directory = NULL;
 
-    /* XXX Handle read-only databases properly */
-    if (notmuch->mode == NOTMUCH_DATABASE_MODE_READ_ONLY)
-	return NOTMUCH_STATUS_READ_ONLY_DATABASE;
-
     try {
-	*directory = _notmuch_directory_create (notmuch, path, &status);
+	*directory = _notmuch_directory_create (notmuch, path,
+						NOTMUCH_FIND_LOOKUP, &status);
     } catch (const Xapian::Error &error) {
 	fprintf (stderr, "A Xapian exception occurred getting directory: %s.\n",
 		 error.get_msg().c_str());
@@ -1884,9 +1894,9 @@ notmuch_database_find_message_by_filename (notmuch_database_t *notmuch,
     local = talloc_new (notmuch);
 
     try {
-	status = _notmuch_database_filename_to_direntry (local, notmuch,
-							 filename, &direntry);
-	if (status)
+	status = _notmuch_database_filename_to_direntry (
+	    local, notmuch, filename, NOTMUCH_FIND_LOOKUP, &direntry);
+	if (status || !direntry)
 	    goto DONE;
 
 	term = talloc_asprintf (local, "%s%s", prefix, direntry);
diff --git a/lib/directory.cc b/lib/directory.cc
index 70e1693e..6a3ffed7 100644
--- a/lib/directory.cc
+++ b/lib/directory.cc
@@ -82,28 +82,41 @@ find_directory_document (notmuch_database_t *notmuch,
     return NOTMUCH_PRIVATE_STATUS_SUCCESS;
 }
 
+/* Find or create a directory document.
+ *
+ * 'path' should be a path relative to the path of 'database', or else
+ * should be an absolute path with initial components that match the
+ * path of 'database'.
+ *
+ * If (flags & NOTMUCH_FIND_CREATE), then the directory document will
+ * be created if it does not exist.  Otherwise, if the directory
+ * document does not exist, *status_ret is set to
+ * NOTMUCH_STATUS_SUCCESS and this returns NULL.
+ */
 notmuch_directory_t *
 _notmuch_directory_create (notmuch_database_t *notmuch,
 			   const char *path,
+			   notmuch_find_flags_t flags,
 			   notmuch_status_t *status_ret)
 {
     Xapian::WritableDatabase *db;
     notmuch_directory_t *directory;
     notmuch_private_status_t private_status;
     const char *db_path;
+    notmuch_bool_t create = (flags & NOTMUCH_FIND_CREATE);
 
     *status_ret = NOTMUCH_STATUS_SUCCESS;
 
     path = _notmuch_database_relative_path (notmuch, path);
 
-    if (notmuch->mode == NOTMUCH_DATABASE_MODE_READ_ONLY)
+    if (create && notmuch->mode == NOTMUCH_DATABASE_MODE_READ_ONLY)
 	INTERNAL_ERROR ("Failure to ensure database is writable");
 
-    db = static_cast <Xapian::WritableDatabase *> (notmuch->xapian_db);
-
     directory = talloc (notmuch, notmuch_directory_t);
-    if (unlikely (directory == NULL))
+    if (unlikely (directory == NULL)) {
+	*status_ret = NOTMUCH_STATUS_OUT_OF_MEMORY;
 	return NULL;
+    }
 
     directory->notmuch = notmuch;
 
@@ -122,6 +135,13 @@ _notmuch_directory_create (notmuch_database_t *notmuch,
 	directory->document_id = directory->doc.get_docid ();
 
 	if (private_status == NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND) {
+	    if (!create) {
+		notmuch_directory_destroy (directory);
+		directory = NULL;
+		*status_ret = NOTMUCH_STATUS_SUCCESS;
+		goto DONE;
+	    }
+
 	    void *local = talloc_new (directory);
 	    const char *parent, *basename;
 	    Xapian::docid parent_id;
@@ -133,7 +153,13 @@ _notmuch_directory_create (notmuch_database_t *notmuch,
 
 	    _notmuch_database_split_path (local, path, &parent, &basename);
 
-	    _notmuch_database_find_directory_id (notmuch, parent, &parent_id);
+	    *status_ret = _notmuch_database_find_directory_id (
+		notmuch, parent, NOTMUCH_FIND_CREATE, &parent_id);
+	    if (*status_ret) {
+		notmuch_directory_destroy (directory);
+		directory = NULL;
+		goto DONE;
+	    }
 
 	    if (basename) {
 		term = talloc_asprintf (local, "%s%u:%s",
@@ -145,6 +171,8 @@ _notmuch_directory_create (notmuch_database_t *notmuch,
 	    directory->doc.add_value (NOTMUCH_VALUE_TIMESTAMP,
 				      Xapian::sortable_serialise (0));
 
+	    db = static_cast <Xapian::WritableDatabase *> (notmuch->xapian_db);
+
 	    directory->document_id = _notmuch_database_generate_doc_id (notmuch);
 	    db->replace_document (directory->document_id, directory->doc);
 	    talloc_free (local);
@@ -158,10 +186,11 @@ _notmuch_directory_create (notmuch_database_t *notmuch,
 		 error.get_msg().c_str());
 	notmuch->exception_reported = TRUE;
 	notmuch_directory_destroy (directory);
+	directory = NULL;
 	*status_ret = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
-	return NULL;
     }
 
+  DONE:
     if (db_path != path)
 	free ((char *) db_path);
 
diff --git a/lib/message.cc b/lib/message.cc
index 00754254..67875065 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -495,9 +495,8 @@ _notmuch_message_add_filename (notmuch_message_t *message,
     if (status)
 	return status;
 
-    status = _notmuch_database_filename_to_direntry (local,
-						     message->notmuch,
-						     filename, &direntry);
+    status = _notmuch_database_filename_to_direntry (
+	local, message->notmuch, filename, NOTMUCH_FIND_CREATE, &direntry);
     if (status)
 	return status;
 
@@ -541,9 +540,9 @@ _notmuch_message_remove_filename (notmuch_message_t *message,
     notmuch_status_t status;
     Xapian::TermIterator i, last;
 
-    status = _notmuch_database_filename_to_direntry (local, message->notmuch,
-						     filename, &direntry);
-    if (status)
+    status = _notmuch_database_filename_to_direntry (
+	local, message->notmuch, filename, NOTMUCH_FIND_LOOKUP, &direntry);
+    if (status || !direntry)
 	return status;
 
     /* Unlink this file from its parent directory. */
diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index 3886e0ca..bfb41116 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -146,6 +146,16 @@ typedef enum _notmuch_private_status {
      :									\
      (notmuch_status_t) private_status)
 
+/* Flags shared by various lookup functions. */
+typedef enum _notmuch_find_flags {
+    /* Lookup without creating any documents.  This is the default
+     * behavior. */
+    NOTMUCH_FIND_LOOKUP = 0,
+    /* If set, create the necessary document (or documents) if they
+     * are missing.  Requires a read/write database. */
+    NOTMUCH_FIND_CREATE = 1<<0,
+} notmuch_find_flags_t;
+
 typedef struct _notmuch_doc_id_set notmuch_doc_id_set_t;
 
 typedef struct _notmuch_string_list notmuch_string_list_t;
@@ -188,6 +198,7 @@ _notmuch_database_find_unique_doc_id (notmuch_database_t *notmuch,
 notmuch_status_t
 _notmuch_database_find_directory_id (notmuch_database_t *database,
 				     const char *path,
+				     notmuch_find_flags_t flags,
 				     unsigned int *directory_id);
 
 const char *
@@ -199,6 +210,7 @@ notmuch_status_t
 _notmuch_database_filename_to_direntry (void *ctx,
 					notmuch_database_t *notmuch,
 					const char *filename,
+					notmuch_find_flags_t flags,
 					char **direntry);
 
 /* directory.cc */
@@ -206,6 +218,7 @@ _notmuch_database_filename_to_direntry (void *ctx,
 notmuch_directory_t *
 _notmuch_directory_create (notmuch_database_t *notmuch,
 			   const char *path,
+			   notmuch_find_flags_t flags,
 			   notmuch_status_t *status_ret);
 
 unsigned int
diff --git a/lib/notmuch.h b/lib/notmuch.h
index bbb17e4a..3633bedd 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -300,10 +300,8 @@ notmuch_database_end_atomic (notmuch_database_t *notmuch);
  * (see notmuch_database_get_path), or else should be an absolute path
  * with initial components that match the path of 'database'.
  *
- * Note: Currently this will create the directory object if it doesn't
- * exist.  In the future, when a directory object does not exist this
- * will return NOTMUCH_STATUS_SUCCESS and set *directory to NULL.
- * Callers should be prepared for this.
+ * If this directory object does not exist in the database, this
+ * returns NOTMUCH_STATUS_SUCCESS and sets *directory to NULL.
  *
  * Return value:
  *
@@ -313,10 +311,6 @@ notmuch_database_end_atomic (notmuch_database_t *notmuch);
  *
  * NOTMUCH_STATUS_XAPIAN_EXCEPTION: A Xapian exception occurred;
  *	directory not retrieved.
- *
- * NOTMUCH_STATUS_READ_ONLY_DATABASE: Database was opened in read-only
- *	mode so the directory cannot be created (this case will be
- *	removed in the future).
  */
 notmuch_status_t
 notmuch_database_get_directory (notmuch_database_t *database,
diff --git a/notmuch-new.c b/notmuch-new.c
index a3a8bece..72dd558d 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -256,7 +256,7 @@ add_files_recursive (notmuch_database_t *notmuch,
     notmuch_filenames_t *db_subdirs = NULL;
     time_t stat_time;
     struct stat st;
-    notmuch_bool_t is_maildir, new_directory;
+    notmuch_bool_t is_maildir;
     const char **tag;
 
     if (stat (path, &st)) {
@@ -281,33 +281,12 @@ add_files_recursive (notmuch_database_t *notmuch,
     }
     db_mtime = directory ? notmuch_directory_get_mtime (directory) : 0;
 
-    new_directory = db_mtime ? FALSE : TRUE;
-
-    /* XXX This is a temporary workaround.  If we don't update the
-     * database mtime until after processing messages in this
-     * directory, then a 0 mtime is *not* sufficient to indicate that
-     * this directory has no messages or subdirs in the database (for
-     * example, if an earlier run skipped the mtime update because
-     * fs_mtime == stat_time, or was interrupted before updating the
-     * mtime at the end).  To address this, we record a (bogus)
-     * non-zero value before processing any child messages so that a
-     * later run won't mistake this for a new directory (and, for
-     * example, fail to detect removed files and subdirs).
-     *
-     * A better solution would be for notmuch_database_get_directory
-     * to indicate if it really created a new directory or not, either
-     * by a new out-argument, or by recording this information and
-     * providing an accessor.
-     */
-    if (new_directory && directory)
-	notmuch_directory_set_mtime (directory, -1);
-
     /* If the database knows about this directory, then we sort based
      * on strcmp to match the database sorting. Otherwise, we can do
      * inode-based sorting for faster filesystem operation. */
     num_fs_entries = scandir (path, &fs_entries, 0,
-			      new_directory ?
-			      dirent_sort_inode : dirent_sort_strcmp_name);
+			      directory ?
+			      dirent_sort_strcmp_name : dirent_sort_inode);
 
     if (num_fs_entries == -1) {
 	fprintf (stderr, "Error opening directory %s: %s\n",
@@ -376,13 +355,12 @@ add_files_recursive (notmuch_database_t *notmuch,
      * being discovered until the clock catches up and the directory
      * is modified again).
      */
-    if (fs_mtime == db_mtime)
+    if (directory && fs_mtime == db_mtime)
 	goto DONE;
 
-    /* new_directory means a directory that the database has never
-     * seen before. In that case, we can simply leave db_files and
-     * db_subdirs NULL. */
-    if (!new_directory) {
+    /* If the database has never seen this directory before, we can
+     * simply leave db_files and db_subdirs NULL. */
+    if (directory) {
 	db_files = notmuch_directory_get_child_files (directory);
 	db_subdirs = notmuch_directory_get_child_directories (directory);
     }
diff --git a/test/python b/test/python
index 6018c2d0..3f03a2e3 100755
--- a/test/python
+++ b/test/python
@@ -28,4 +28,12 @@ EOF
 notmuch search --sort=oldest-first --output=messages tag:inbox | sed s/^id:// > EXPECTED
 test_expect_equal_file OUTPUT EXPECTED
 
+test_begin_subtest "get non-existent file"
+test_python <<EOF
+import notmuch
+db = notmuch.Database(mode=notmuch.Database.MODE.READ_ONLY)
+print db.find_message_by_filename("i-dont-exist")
+EOF
+test_expect_equal "$(cat OUTPUT)" "None"
+
 test_done