lib: replace the header parser with gmime

The notmuch library includes a full blown message header parser. Yet
the same message headers are parsed by gmime during indexing. Switch
to gmime parsing completely.

These are the main changes:

* Gmime stops header parsing at the first invalid header, and presumes
  the message body starts from there. The current parser is quite
  liberal in accepting broken headers. The change means we will be
  much pickier about accepting invalid messages.

* The current parser converts tabs used in header folding to
  spaces. Gmime preserve the tabs. Due to a broken python library used
  in mailman, there are plenty of mailing lists that produce headers
  with tabs in header folding, and we'll see plenty of tabs. (This
  change has been mitigated in preparatory patches.)

* For pure header parsing, the current parser is likely faster than
  gmime, which parses the whole message rather than just the
  headers. Since we parse the message and its headers using gmime for
  indexing anyway, this avoids and extra header parsing round when
  adding new messages. In case of duplicate messages, we'll end up
  parsing the full message although just headers would be
  sufficient. All in all this should still speed up 'notmuch new'.

* Calls to notmuch_message_get_header() may be slightly slower than
  previously for headers that are not indexed in the database, due to
  parsing of the whole message. Within the notmuch code base, notmuch
  reply is the only such user.

1 files changed, 28 insertions, 27 deletions

diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index 59eb2bc2..703ae7bb 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -46,6 +46,8 @@ NOTMUCH_BEGIN_DECLS
 
 #include <talloc.h>
 
+#include <gmime/gmime.h>
+
 #include "xutil.h"
 #include "error_util.h"
 
@@ -320,13 +322,6 @@ notmuch_message_set_author (notmuch_message_t *message, const char *author);
 const char *
 notmuch_message_get_author (notmuch_message_t *message);
 
-
-/* index.cc */
-
-notmuch_status_t
-_notmuch_message_index_file (notmuch_message_t *message,
-			     const char *filename);
-
 /* message-file.c */
 
 /* XXX: I haven't decided yet whether these will actually get exported
@@ -352,32 +347,32 @@ _notmuch_message_file_open_ctx (void *ctx, const char *filename);
 void
 notmuch_message_file_close (notmuch_message_file_t *message);
 
-/* Restrict 'message' to only save the named headers.
+/* Parse the message.
  *
- * When the caller is only interested in a short list of headers,
- * known in advance, calling this function can avoid wasted time and
- * memory parsing/saving header values that will never be needed.
+ * This will be done automatically as necessary on other calls
+ * depending on it, but an explicit call allows for better error
+ * status reporting.
+ */
+notmuch_status_t
+_notmuch_message_file_parse (notmuch_message_file_t *message);
+
+/* Get the gmime message of a message file.
  *
- * The variable arguments should be a list of const char * with a
- * final '(const char *) NULL' to terminate the list.
+ * The message file is parsed as necessary.
  *
- * If this function is called, it must be called before any calls to
- * notmuch_message_get_header for this message.
+ * The GMimeMessage* is set to *mime_message on success (which the
+ * caller must not unref).
  *
- * After calling this function, if notmuch_message_get_header is
- * called with a header name not in this list, then NULL will be
- * returned even if that header exists in the actual message.
+ * XXX: Would be nice to not have to expose GMimeMessage here.
  */
-void
-notmuch_message_file_restrict_headers (notmuch_message_file_t *message, ...);
-
-/* Identical to notmuch_message_restrict_headers but accepting a va_list. */
-void
-notmuch_message_file_restrict_headersv (notmuch_message_file_t *message,
-					va_list va_headers);
+notmuch_status_t
+_notmuch_message_file_get_mime_message (notmuch_message_file_t *message,
+					GMimeMessage **mime_message);
 
 /* Get the value of the specified header from the message as a UTF-8 string.
  *
+ * The message file is parsed as necessary.
+ *
  * The header name is case insensitive.
  *
  * The Received: header is special - for it all Received: headers in
@@ -387,13 +382,19 @@ notmuch_message_file_restrict_headersv (notmuch_message_file_t *message,
  * only until the message is closed. The caller should copy it if
  * needing to modify the value or to hold onto it for longer.
  *
- * Returns NULL if the message does not contain a header line matching
- * 'header'.
+ * Returns NULL on errors, empty string if the message does not
+ * contain a header line matching 'header'.
  */
 const char *
 notmuch_message_file_get_header (notmuch_message_file_t *message,
 				 const char *header);
 
+/* index.cc */
+
+notmuch_status_t
+_notmuch_message_index_file (notmuch_message_t *message,
+			     notmuch_message_file_t *message_file);
+
 /* messages.c */
 
 typedef struct _notmuch_message_node {