Avoid database corruption by not adding partially-constructed mail documents.

Previously we were using Xapian's add_document to allocate document ID
values for notmuch_message_t objects.  This had the drawback of adding
a partially constructed mail document to the database. If notmuch was
subsequently interrupted before fully populating this document, then
later runs would be quite confused when seeing the partial documents.

There are reports from the wild of people hitting internal errors of
the form "Message ... has no thread ID" for example, (which is
currently an unrecoverable error).

We fix this by manually allocating document IDs without adding
documents. With this change, we never call Xapian's add_document
method, but only replace_document with either the current document ID
of a message or a new one that we have allocated.

1 files changed, 26 insertions, 0 deletions

diff --git a/lib/database.cc b/lib/database.cc
index 6afc8d93..dd1fc637 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -622,6 +622,7 @@ notmuch_database_open (const char *path,
 	    }
 	}
 
+	notmuch->last_doc_id = notmuch->xapian_db->get_lastdocid ();
 	last_thread_id = notmuch->xapian_db->get_metadata ("last_thread_id");
 	if (last_thread_id.empty ()) {
 	    notmuch->last_thread_id = 0;
@@ -1169,6 +1170,31 @@ notmuch_database_get_directory (notmuch_database_t *notmuch,
     }
 }
 
+/* Allocate a document ID that satisfies the following criteria:
+ *
+ * 1. The ID does not exist for any document in the Xapian database
+ *
+ * 2. The ID was not previously returned from this function
+ *
+ * 3. The ID is the smallest integer satisfying (1) and (2)
+ *
+ * This function will trigger an internal error if these constraints
+ * cannot all be satisfied, (that is, the pool of available document
+ * IDs has been exhausted).
+ */
+unsigned int
+_notmuch_database_generate_doc_id (notmuch_database_t *notmuch)
+{
+    assert (notmuch->last_doc_id >= notmuch->xapian_db->get_lastdocid ());
+
+    notmuch->last_doc_id++;
+
+    if (notmuch->last_doc_id == 0)
+	INTERNAL_ERROR ("Xapian document IDs are exhausted.\n");	
+
+    return notmuch->last_doc_id;
+}
+
 static const char *
 _notmuch_database_generate_thread_id (notmuch_database_t *notmuch)
 {