gmime 3.0 related preliminaries

gmime 3.0 related preliminaries

[David Bremner]

I've been working away on porting to gmime 3.0 [1]; currently more
than 50 tests fail, at least some of which due to [2].

I've also found a few bugs in notmuch:

We currently use GmimeFileStream to output to stdout, but this assumes
the underlying file descriptor is seekable; the seek errors are masked
in gmime 2.6 but not in gmime 3.0

[PATCH 1/5] util: convenience function to create gmime stream for
[PATCH 2/5] cli/reply: direct all output for text format to gmime
[PATCH 3/5] cli/show: use single stream for printf / gmime object

There is also memory leak in notmuch reply; because notmuch-reply
doesn't run for a long time, the practical impact is low. The first
patch is just so we have some way of verifying the fix. It seems a bit
heavy to start using valgrind in the main test suite.


[1]: http://pivot.cs.unb.ca/git?p=notmuch.git;a=shortlog;h=refs/heads/gmime-3.0
[2]: https://bugzilla.gnome.org/show_bug.cgi?id=782915

[PATCH 1/5] util: convenience function to create gmime stream for stdout

[David Bremner]

It turns out that our use of GMimeStreamPipe has only succeeded
because gmime has been ignoring some seek failures; this will no
longer be the case in gmime 3.0, so we use a GMimeStreamPipe, which
does not assume seekability, wrapped in a buffering stream.
---
 util/Makefile.local |  2 +-
 util/gmime-extra.c  | 15 +++++++++++++++
 util/gmime-extra.h  |  6 ++++++
 3 files changed, 22 insertions(+), 1 deletion(-)
 create mode 100644 util/gmime-extra.c
 create mode 100644 util/gmime-extra.h

diff --git a/util/Makefile.local b/util/Makefile.local
index a6962d49..3027880b 100644
--- a/util/Makefile.local
+++ b/util/Makefile.local
@@ -5,7 +5,7 @@ extra_cflags += -I$(srcdir)/$(dir)
 
 libnotmuch_util_c_srcs := $(dir)/xutil.c $(dir)/error_util.c $(dir)/hex-escape.c \
 		  $(dir)/string-util.c $(dir)/talloc-extra.c $(dir)/zlib-extra.c \
-		$(dir)/util.c
+		$(dir)/util.c $(dir)/gmime-extra.c
 
 libnotmuch_util_modules := $(libnotmuch_util_c_srcs:.c=.o)
 
diff --git a/util/gmime-extra.c b/util/gmime-extra.c
new file mode 100644
index 00000000..d0366812
--- /dev/null
+++ b/util/gmime-extra.c
@@ -0,0 +1,15 @@
+#include "gmime-extra.h"
+
+GMimeStream *
+g_mime_stream_stdout_new()
+{
+    GMimeStream *stream_stdout = NULL;
+
+    stream_stdout = g_mime_stream_pipe_new (STDOUT_FILENO);
+    if (!stream_stdout)
+	return NULL;
+
+    g_mime_stream_pipe_set_owner (GMIME_STREAM_PIPE (stream_stdout), FALSE);
+
+    return g_mime_stream_buffer_new (stream_stdout, GMIME_STREAM_BUFFER_BLOCK_WRITE);
+}
diff --git a/util/gmime-extra.h b/util/gmime-extra.h
new file mode 100644
index 00000000..e0432a94
--- /dev/null
+++ b/util/gmime-extra.h
@@ -0,0 +1,6 @@
+#ifndef _GMIME_EXTRA_H
+#define _GMIME_EXTRA_H
+#include <gmime/gmime.h>
+
+GMimeStream *g_mime_stream_stdout_new(void);
+#endif
-- 
2.11.0

Re: [PATCH 1/5] util: convenience function to create gmime stream for stdout

[David Bremner]

David Bremner <david@tethera.net> writes:
> +GMimeStream *
> +g_mime_stream_stdout_new()
> +{
> +    GMimeStream *stream_stdout = NULL;
> +
> +    stream_stdout = g_mime_stream_pipe_new (STDOUT_FILENO);
> +    if (!stream_stdout)
> +	return NULL;
> +
> +    g_mime_stream_pipe_set_owner (GMIME_STREAM_PIPE (stream_stdout), FALSE);
> +
> +    return g_mime_stream_buffer_new (stream_stdout, GMIME_STREAM_BUFFER_BLOCK_WRITE);
> +}

speaking of memory leaks, I _think_ stream_stdout is leaking here,
although I can't find it with valgrind.

Re: [PATCH 1/5] util: convenience function to create gmime stream for stdout

[David Bremner]

David Bremner <david@tethera.net> writes:

> David Bremner <david@tethera.net> writes:
>> +GMimeStream *
>> +g_mime_stream_stdout_new()
>> +{
>> +    GMimeStream *stream_stdout = NULL;
>> +
>> +    stream_stdout = g_mime_stream_pipe_new (STDOUT_FILENO);
>> +    if (!stream_stdout)
>> +	return NULL;
>> +
>> +    g_mime_stream_pipe_set_owner (GMIME_STREAM_PIPE (stream_stdout), FALSE);
>> +
>> +    return g_mime_stream_buffer_new (stream_stdout, GMIME_STREAM_BUFFER_BLOCK_WRITE);
>> +}
>
> speaking of memory leaks, I _think_ stream_stdout is leaking here,
> although I can't find it with valgrind.

Err. Now I see it, so that's another thing to clean up in this series.

d

[PATCH 2/5] cli/reply: direct all output for text format to gmime stream

[David Bremner]

Interleaving printfs with writes to the gmime stream worked when the
gmime stream was backed by the FILE *stdout, but that is no longer the
case.  Create one stream and pass it into the two functions where
needed, as well well as replacing printfs with g_mime_stream_printf.
---
 notmuch-client.h |  2 +-
 notmuch-reply.c  | 63 +++++++++++++++++++++++++++-----------------------------
 2 files changed, 31 insertions(+), 34 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index a6f70eae..5692caf3 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -29,7 +29,7 @@
 
 #include "compat.h"
 
-#include <gmime/gmime.h>
+#include "gmime-extra.h"
 
 typedef GMimeCryptoContext notmuch_crypto_context_t;
 /* This is automatically included only since gmime 2.6.10 */
diff --git a/notmuch-reply.c b/notmuch-reply.c
index 2fa6e5a3..9239aac2 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -25,47 +25,42 @@
 #include "sprinter.h"
 
 static void
-show_reply_headers (GMimeMessage *message)
+show_reply_headers (GMimeStream *stream, GMimeMessage *message)
 {
-    GMimeStream *stream_stdout = NULL;
-
-    stream_stdout = g_mime_stream_file_new (stdout);
-    if (stream_stdout) {
-	g_mime_stream_file_set_owner (GMIME_STREAM_FILE (stream_stdout), FALSE);
-	/* Output RFC 2822 formatted (and RFC 2047 encoded) headers. */
-	g_mime_object_write_to_stream (GMIME_OBJECT(message), stream_stdout);
-	g_object_unref(stream_stdout);
+    /* Output RFC 2822 formatted (and RFC 2047 encoded) headers. */
+    if (g_mime_object_write_to_stream (GMIME_OBJECT(message), stream) < 0) {
+	INTERNAL_ERROR("failed to write headers to stdout\n");
     }
 }
 
 static void
-format_part_reply (mime_node_t *node)
+format_part_reply (GMimeStream *stream, mime_node_t *node)
 {
     int i;
 
     if (node->envelope_file) {
-	printf ("On %s, %s wrote:\n",
-		notmuch_message_get_header (node->envelope_file, "date"),
-		notmuch_message_get_header (node->envelope_file, "from"));
+	g_mime_stream_printf (stream, "On %s, %s wrote:\n",
+			      notmuch_message_get_header (node->envelope_file, "date"),
+			      notmuch_message_get_header (node->envelope_file, "from"));
     } else if (GMIME_IS_MESSAGE (node->part)) {
 	GMimeMessage *message = GMIME_MESSAGE (node->part);
 	InternetAddressList *recipients;
 	const char *recipients_string;
 
-	printf ("> From: %s\n", g_mime_message_get_sender (message));
+	g_mime_stream_printf (stream, "> From: %s\n", g_mime_message_get_sender (message));
 	recipients = g_mime_message_get_recipients (message, GMIME_RECIPIENT_TYPE_TO);
 	recipients_string = internet_address_list_to_string (recipients, 0);
 	if (recipients_string)
-	    printf ("> To: %s\n",
-		    recipients_string);
+	    g_mime_stream_printf (stream, "> To: %s\n",
+				  recipients_string);
 	recipients = g_mime_message_get_recipients (message, GMIME_RECIPIENT_TYPE_CC);
 	recipients_string = internet_address_list_to_string (recipients, 0);
 	if (recipients_string)
-	    printf ("> Cc: %s\n",
-		    recipients_string);
-	printf ("> Subject: %s\n", g_mime_message_get_subject (message));
-	printf ("> Date: %s\n", g_mime_message_get_date_as_string (message));
-	printf (">\n");
+	    g_mime_stream_printf (stream, "> Cc: %s\n",
+				  recipients_string);
+	g_mime_stream_printf (stream, "> Subject: %s\n", g_mime_message_get_subject (message));
+	g_mime_stream_printf (stream, "> Date: %s\n", g_mime_message_get_date_as_string (message));
+	g_mime_stream_printf (stream, ">\n");
     } else if (GMIME_IS_PART (node->part)) {
 	GMimeContentType *content_type = g_mime_object_get_content_type (node->part);
 	GMimeContentDisposition *disposition = g_mime_object_get_content_disposition (node->part);
@@ -75,24 +70,21 @@ format_part_reply (mime_node_t *node)
 	    /* Ignore PGP/MIME cruft parts */
 	} else if (g_mime_content_type_is_type (content_type, "text", "*") &&
 		   !g_mime_content_type_is_type (content_type, "text", "html")) {
-	    GMimeStream *stream_stdout = g_mime_stream_file_new (stdout);
-	    g_mime_stream_file_set_owner (GMIME_STREAM_FILE (stream_stdout), FALSE);
-	    show_text_part_content (node->part, stream_stdout, NOTMUCH_SHOW_TEXT_PART_REPLY);
-	    g_object_unref(stream_stdout);
+	    show_text_part_content (node->part, stream, NOTMUCH_SHOW_TEXT_PART_REPLY);
 	} else if (disposition &&
 		   strcasecmp (g_mime_content_disposition_get_disposition (disposition),
 			       GMIME_DISPOSITION_ATTACHMENT) == 0) {
 	    const char *filename = g_mime_part_get_filename (GMIME_PART (node->part));
-	    printf ("Attachment: %s (%s)\n", filename,
-		    g_mime_content_type_to_string (content_type));
+	    g_mime_stream_printf (stream, "Attachment: %s (%s)\n", filename,
+				  g_mime_content_type_to_string (content_type));
 	} else {
-	    printf ("Non-text part: %s\n",
-		    g_mime_content_type_to_string (content_type));
+	    g_mime_stream_printf (stream, "Non-text part: %s\n",
+				  g_mime_content_type_to_string (content_type));
 	}
     }
 
     for (i = 0; i < node->nchildren; i++)
-	format_part_reply (mime_node_child (node, i));
+	format_part_reply (stream, mime_node_child (node, i));
 }
 
 typedef enum {
@@ -678,9 +670,14 @@ static int do_reply(notmuch_config_t *config,
 	    /* End */
 	    sp->end (sp);
 	} else {
-	    show_reply_headers (reply);
-	    if (format == FORMAT_DEFAULT)
-		format_part_reply (node);
+	    GMimeStream *stream_stdout = stream_stdout = g_mime_stream_stdout_new ();
+	    if (stream_stdout) {
+		show_reply_headers (stream_stdout, reply);
+		if (format == FORMAT_DEFAULT)
+		    format_part_reply (stream_stdout, node);
+	    }
+	    g_mime_stream_flush (stream_stdout);
+	    g_object_unref(stream_stdout);
 	}
 
 	g_object_unref (G_OBJECT (reply));
-- 
2.11.0

[PATCH 3/5] cli/show: use single stream for printf / gmime object output

[David Bremner]

This is again motivated by the need to transition away from
GMimeStreamFile for output to stdout.

format_part_mbox is left alone for now, as this cannot be mixed in
with output using gmime object output.
---
 notmuch-client.h |  1 +
 notmuch-show.c   | 70 ++++++++++++++++++++++++++------------------------------
 2 files changed, 33 insertions(+), 38 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index 5692caf3..62d4bcec 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -86,6 +86,7 @@ typedef struct notmuch_show_params {
     int part;
     notmuch_crypto_t crypto;
     notmuch_bool_t include_html;
+    GMimeStream *out_stream;
 } notmuch_show_params_t;
 
 /* There's no point in continuing when we've detected that we've done
diff --git a/notmuch-show.c b/notmuch-show.c
index 7021008e..accea48a 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -426,6 +426,7 @@ format_part_text (const void *ctx, sprinter_t *sp, mime_node_t *node,
 	GMIME_OBJECT (node->envelope_part) : node->part;
     GMimeContentType *content_type = g_mime_object_get_content_type (meta);
     const notmuch_bool_t leaf = GMIME_IS_PART (node->part);
+    GMimeStream *stream = params->out_stream;
     const char *part_type;
     int i;
 
@@ -433,13 +434,13 @@ format_part_text (const void *ctx, sprinter_t *sp, mime_node_t *node,
 	notmuch_message_t *message = node->envelope_file;
 
 	part_type = "message";
-	printf ("\f%s{ id:%s depth:%d match:%d excluded:%d filename:%s\n",
-		part_type,
-		notmuch_message_get_message_id (message),
-		indent,
-		notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_MATCH) ? 1 : 0,
-		notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED) ? 1 : 0,
-		notmuch_message_get_filename (message));
+	g_mime_stream_printf (stream, "\f%s{ id:%s depth:%d match:%d excluded:%d filename:%s\n",
+			      part_type,
+			      notmuch_message_get_message_id (message),
+			      indent,
+			      notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_MATCH) ? 1 : 0,
+			      notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED) ? 1 : 0,
+			      notmuch_message_get_filename (message));
     } else {
 	char *content_string;
 	const char *disposition = _get_disposition (meta);
@@ -453,14 +454,14 @@ format_part_text (const void *ctx, sprinter_t *sp, mime_node_t *node,
 	else
 	    part_type = "part";
 
-	printf ("\f%s{ ID: %d", part_type, node->part_num);
+	g_mime_stream_printf (stream, "\f%s{ ID: %d", part_type, node->part_num);
 	if (filename)
-	    printf (", Filename: %s", filename);
+	    g_mime_stream_printf (stream, ", Filename: %s", filename);
 	if (cid)
-	    printf (", Content-id: %s", cid);
+	    g_mime_stream_printf (stream, ", Content-id: %s", cid);
 
 	content_string = g_mime_content_type_to_string (content_type);
-	printf (", Content-type: %s\n", content_string);
+	g_mime_stream_printf (stream, ", Content-type: %s\n", content_string);
 	g_free (content_string);
     }
 
@@ -470,40 +471,37 @@ format_part_text (const void *ctx, sprinter_t *sp, mime_node_t *node,
 	char *recipients_string;
 	char *date_string;
 
-	printf ("\fheader{\n");
+	g_mime_stream_printf (stream, "\fheader{\n");
 	if (node->envelope_file)
-	    printf ("%s\n", _get_one_line_summary (ctx, node->envelope_file));
-	printf ("Subject: %s\n", g_mime_message_get_subject (message));
-	printf ("From: %s\n", g_mime_message_get_sender (message));
+	    g_mime_stream_printf (stream, "%s\n", _get_one_line_summary (ctx, node->envelope_file));
+	g_mime_stream_printf (stream, "Subject: %s\n", g_mime_message_get_subject (message));
+	g_mime_stream_printf (stream, "From: %s\n", g_mime_message_get_sender (message));
 	recipients = g_mime_message_get_recipients (message, GMIME_RECIPIENT_TYPE_TO);
 	recipients_string = internet_address_list_to_string (recipients, 0);
 	if (recipients_string)
-	    printf ("To: %s\n", recipients_string);
+	    g_mime_stream_printf (stream, "To: %s\n", recipients_string);
 	g_free (recipients_string);
 	recipients = g_mime_message_get_recipients (message, GMIME_RECIPIENT_TYPE_CC);
 	recipients_string = internet_address_list_to_string (recipients, 0);
 	if (recipients_string)
-	    printf ("Cc: %s\n", recipients_string);
+	    g_mime_stream_printf (stream, "Cc: %s\n", recipients_string);
 	g_free (recipients_string);
 	date_string = g_mime_message_get_date_as_string (message);
-	printf ("Date: %s\n", date_string);
+	g_mime_stream_printf (stream, "Date: %s\n", date_string);
 	g_free (date_string);
-	printf ("\fheader}\n");
+	g_mime_stream_printf (stream, "\fheader}\n");
 
-	printf ("\fbody{\n");
+	g_mime_stream_printf (stream, "\fbody{\n");
     }
 
     if (leaf) {
 	if (g_mime_content_type_is_type (content_type, "text", "*") &&
 	    !g_mime_content_type_is_type (content_type, "text", "html"))
 	{
-	    GMimeStream *stream_stdout = g_mime_stream_file_new (stdout);
-	    g_mime_stream_file_set_owner (GMIME_STREAM_FILE (stream_stdout), FALSE);
-	    show_text_part_content (node->part, stream_stdout, 0);
-	    g_object_unref(stream_stdout);
+	    show_text_part_content (node->part, stream, 0);
 	} else {
 	    char *content_string = g_mime_content_type_to_string (content_type);
-	    printf ("Non-text part: %s\n", content_string);
+	    g_mime_stream_printf (stream, "Non-text part: %s\n", content_string);
 	    g_free (content_string);
 	}
     }
@@ -512,9 +510,9 @@ format_part_text (const void *ctx, sprinter_t *sp, mime_node_t *node,
 	format_part_text (ctx, sp, mime_node_child (node, i), indent, params);
 
     if (GMIME_IS_MESSAGE (node->part))
-	printf ("\fbody}\n");
+	g_mime_stream_printf (stream, "\fbody}\n");
 
-    printf ("\f%s}\n", part_type);
+    g_mime_stream_printf (stream, "\f%s}\n", part_type);
 
     return NOTMUCH_STATUS_SUCCESS;
 }
@@ -741,7 +739,7 @@ format_part_mbox (const void *ctx, unused (sprinter_t *sp), mime_node_t *node,
 static notmuch_status_t
 format_part_raw (unused (const void *ctx), unused (sprinter_t *sp),
 		 mime_node_t *node, unused (int indent),
-		 unused (const notmuch_show_params_t *params))
+		 const notmuch_show_params_t *params)
 {
     if (node->envelope_file) {
 	/* Special case the entire message to avoid MIME parsing. */
@@ -781,13 +779,7 @@ format_part_raw (unused (const void *ctx), unused (sprinter_t *sp),
 	return NOTMUCH_STATUS_SUCCESS;
     }
 
-    GMimeStream *stream_stdout;
-    GMimeStream *stream_filter = NULL;
-
-    stream_stdout = g_mime_stream_file_new (stdout);
-    g_mime_stream_file_set_owner (GMIME_STREAM_FILE (stream_stdout), FALSE);
-
-    stream_filter = g_mime_stream_filter_new (stream_stdout);
+    GMimeStream *stream_filter = g_mime_stream_filter_new (params->out_stream);
 
     if (GMIME_IS_PART (node->part)) {
 	/* For leaf parts, we emit only the transfer-decoded
@@ -810,9 +802,6 @@ format_part_raw (unused (const void *ctx), unused (sprinter_t *sp),
     if (stream_filter)
 	g_object_unref (stream_filter);
 
-    if (stream_stdout)
-	g_object_unref(stream_stdout);
-
     return NOTMUCH_STATUS_SUCCESS;
 }
 
@@ -1167,6 +1156,8 @@ notmuch_show_command (notmuch_config_t *config, int argc, char *argv[])
     formatter = formatters[format];
     sprinter = formatter->new_sprinter(config, stdout);
 
+    params.out_stream = g_mime_stream_stdout_new ();
+
     /* If a single message is requested we do not use search_excludes. */
     if (single_message) {
 	ret = do_show_single (config, query, formatter, sprinter, &params);
@@ -1200,6 +1191,9 @@ notmuch_show_command (notmuch_config_t *config, int argc, char *argv[])
     }
 
  DONE:
+    g_mime_stream_flush (params.out_stream);
+    g_object_unref (params.out_stream);
+
     notmuch_crypto_cleanup (&params.crypto);
     notmuch_query_destroy (query);
     notmuch_database_destroy (notmuch);
-- 
2.11.0

[PATCH 4/5] perf-test: add memory test for reply

[David Bremner]

Looking at the code for notmuch-reply, there seems to be several gmime
related memory leaks. This test is supposed to help eliminate those.
---
 performance-test/M04-reply.sh | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
 create mode 100755 performance-test/M04-reply.sh

diff --git a/performance-test/M04-reply.sh b/performance-test/M04-reply.sh
new file mode 100755
index 00000000..ca8a899b
--- /dev/null
+++ b/performance-test/M04-reply.sh
@@ -0,0 +1,15 @@
+#!/bin/bash
+
+test_description='search'
+
+. ./perf-test-lib.sh || exit 1
+
+memory_start
+
+for id in $(notmuch search --output=messages '*' | shuf -n 5); do
+    memory_run "reply $id" "notmuch reply $id 1>/dev/null"
+    memory_run "reply --format=json $id" "notmuch reply --format=json $id 1>/dev/null"
+    memory_run "reply --format=sexp $id" "notmuch reply --format=sexp $id 1>/dev/null"
+done
+
+memory_done
-- 
2.11.0

[PATCH 5/5] cli/reply: fix memory leak

[David Bremner]

internet_address_list_to_string returns an allocated string, which
needs to be freed with g_free.
---
 notmuch-reply.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/notmuch-reply.c b/notmuch-reply.c
index 9239aac2..6cab75bf 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -45,7 +45,7 @@ format_part_reply (GMimeStream *stream, mime_node_t *node)
     } else if (GMIME_IS_MESSAGE (node->part)) {
 	GMimeMessage *message = GMIME_MESSAGE (node->part);
 	InternetAddressList *recipients;
-	const char *recipients_string;
+	char *recipients_string;
 
 	g_mime_stream_printf (stream, "> From: %s\n", g_mime_message_get_sender (message));
 	recipients = g_mime_message_get_recipients (message, GMIME_RECIPIENT_TYPE_TO);
-- 
2.11.0

Re: [PATCH 5/5] cli/reply: fix memory leak

[David Bremner]

David Bremner <david@tethera.net> writes:

> internet_address_list_to_string returns an allocated string, which
> needs to be freed with g_free.
> ---
>  notmuch-reply.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/notmuch-reply.c b/notmuch-reply.c
> index 9239aac2..6cab75bf 100644
> --- a/notmuch-reply.c
> +++ b/notmuch-reply.c
> @@ -45,7 +45,7 @@ format_part_reply (GMimeStream *stream, mime_node_t *node)
>      } else if (GMIME_IS_MESSAGE (node->part)) {
>  	GMimeMessage *message = GMIME_MESSAGE (node->part);
>  	InternetAddressList *recipients;
> -	const char *recipients_string;
> +	char *recipients_string;
>  
>  	g_mime_stream_printf (stream, "> From: %s\n", g_mime_message_get_sender (message));
>  	recipients = g_mime_message_get_recipients (message, GMIME_RECIPIENT_TYPE_TO);
> -- 
> 2.11.0

Oops. Apparently I rebased away the actual fix. It needs added calls to
g_free (recipients_string) like those in notmuch_show.c. There is also a
trickier leak involving internet_address_list_parse_string call
indirectly by add_recipients_from_message.