Non-ASCII characters in headers (éščřžýáíé)

Non-ASCII characters in headers (éščřžýáíé)

[Michal Sojka z česka]

Hi,

if you try to reply to this message in notmuch (Emacs UI), you will see
something like:

Subject: Re: Non-ASCII characters in headers
 =?iso-8859-2?b?KOm56Pi+/eHt6Sk=?=
To: Michal Sojka z =?iso-8859-2?b?6GVza2E=?= <sojkam1@fel.cvut.cz>,

I've tried to write a fix for this, which I send as a reply to this
mail.

Michal

[PATCH] Decode headers in reply

[Michal Sojka]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 13877 bytes --]

When headers contain non-ASCII characters, they are encoded according
to rfc2047. Nomtuch reply command emits the headers in the encoded
form, which makes them hard to read by humans who compose the reply.

For example instead of "Subject: Re: Rozlučka" one currently sees
"Subject: Re: =?iso-8859-2?q?Rozlu=E8ka?=".

This patch adds a new GMime filter which is used to decode headers to
UTF-8 and uses this filter when notmuch reply outputs headers.

Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
---
I tested this patch with valgrind to debug some invalid writes in an
earlier version of the patch. There are no such problems in this
version, but valgrind reports many memory leaks (even on master
branch). I'm not sure whether these are false positives or not.

 Makefile.local         |    1 +
 gmime-filter-headers.c |  263 ++++++++++++++++++++++++++++++++++++++++++++++++
 gmime-filter-headers.h |   69 +++++++++++++
 notmuch-reply.c        |   26 ++++-
 4 files changed, 354 insertions(+), 5 deletions(-)

diff --git a/Makefile.local b/Makefile.local
index 04bac83..0cacb5f 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -4,6 +4,7 @@ notmuch_client_srcs =		\
 	$(notmuch_compat_srcs)	\
 	debugger.c		\
 	gmime-filter-reply.c	\
+	gmime-filter-headers.c	\
 	notmuch.c		\
 	notmuch-config.c	\
 	notmuch-count.c		\
diff --git a/gmime-filter-headers.c b/gmime-filter-headers.c
new file mode 100644
index 0000000..2f3df80
--- /dev/null
+++ b/gmime-filter-headers.c
@@ -0,0 +1,263 @@
+/*
+ * Copyright © 2009 Keith Packard <keithp@keithp.com>
+ * Copyright © 2010 Michal Sojka <sojkam1@fel.cvut.cz>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA.
+ */
+
+#include "gmime-filter-headers.h"
+#include <string.h>
+#include <gmime/gmime-utils.h>
+#include <glib/gprintf.h>
+#include <stdlib.h>
+#include <xutil.h>
+
+/**
+ * SECTION: gmime-filter-headers
+ * @title: GMimeFilterHeaders
+ * @short_description: Add/remove headers markers
+ *
+ * A #GMimeFilter for decoding rfc2047 encoded headers to UTF-8
+ **/
+
+
+static void g_mime_filter_headers_class_init (GMimeFilterHeadersClass *klass);
+static void g_mime_filter_headers_init (GMimeFilterHeaders *filter, GMimeFilterHeadersClass *klass);
+static void g_mime_filter_headers_finalize (GObject *object);
+
+static GMimeFilter *filter_copy (GMimeFilter *filter);
+static void filter_filter (GMimeFilter *filter, char *in, size_t len, size_t prespace,
+			   char **out, size_t *outlen, size_t *outprespace);
+static void filter_complete (GMimeFilter *filter, char *in, size_t len, size_t prespace,
+			     char **out, size_t *outlen, size_t *outprespace);
+static void filter_reset (GMimeFilter *filter);
+
+
+static GMimeFilterClass *parent_class = NULL;
+
+GType
+g_mime_filter_headers_get_type (void)
+{
+	static GType type = 0;
+
+	if (!type) {
+		static const GTypeInfo info = {
+			sizeof (GMimeFilterHeadersClass),
+			NULL, /* base_class_init */
+			NULL, /* base_class_finalize */
+			(GClassInitFunc) g_mime_filter_headers_class_init,
+			NULL, /* class_finalize */
+			NULL, /* class_data */
+			sizeof (GMimeFilterHeaders),
+			0,    /* n_preallocs */
+			(GInstanceInitFunc) g_mime_filter_headers_init,
+			NULL	/* value_table */
+		};
+
+		type = g_type_register_static (GMIME_TYPE_FILTER, "GMimeFilterHeaders", &info, (GTypeFlags) 0);
+	}
+
+	return type;
+}
+
+
+static void
+g_mime_filter_headers_class_init (GMimeFilterHeadersClass *klass)
+{
+	GObjectClass *object_class = G_OBJECT_CLASS (klass);
+	GMimeFilterClass *filter_class = GMIME_FILTER_CLASS (klass);
+
+	parent_class = (GMimeFilterClass *) g_type_class_ref (GMIME_TYPE_FILTER);
+
+	object_class->finalize = g_mime_filter_headers_finalize;
+
+	filter_class->copy = filter_copy;
+	filter_class->filter = filter_filter;
+	filter_class->complete = filter_complete;
+	filter_class->reset = filter_reset;
+}
+
+static void
+g_mime_filter_headers_init (GMimeFilterHeaders *filter, GMimeFilterHeadersClass *klass)
+{
+	(void) klass;
+	filter->saw_nl = TRUE;
+	filter->line = NULL;
+	filter->line_size = 0;
+	filter->lineptr = NULL;
+}
+
+static void
+g_mime_filter_headers_finalize (GObject *object)
+{
+	free (GMIME_FILTER_HEADERS (object)->line);
+	G_OBJECT_CLASS (parent_class)->finalize (object);
+}
+
+
+static GMimeFilter *
+filter_copy (GMimeFilter *filter)
+{
+	(void) filter;
+	return g_mime_filter_headers_new ();
+}
+
+static void
+output_decoded_header (GMimeFilterHeaders *headers, char **outptr)
+{
+	char *colon, *name, *s, *decoded_value;
+	size_t offset;
+	gint ret;
+
+	colon = strchr (headers->line, ':');
+	if (colon == NULL)
+		return;
+
+	name = headers->line;
+	*colon = '\0';
+	s = colon + 1;
+	while (*s == ' ' || *s == '\t')
+		s++;
+	decoded_value = g_mime_utils_header_decode_text(s);
+	if (decoded_value == NULL)
+		return;
+	offset = *outptr - GMIME_FILTER (headers)->outbuf;
+	g_mime_filter_set_size (GMIME_FILTER (headers), strlen(name) + 2 +
+			       strlen(decoded_value) + 2, TRUE);
+	*outptr = GMIME_FILTER (headers)->outbuf + offset;
+	ret = g_sprintf (*outptr, "%s: %s\n", name, decoded_value);
+	if (ret > 0)
+		*outptr += ret;
+	free (decoded_value);
+}
+
+static void
+output_final_newline (GMimeFilterHeaders *headers, char **outptr)
+{
+	size_t offset;
+
+	offset = *outptr - GMIME_FILTER (headers)->outbuf;
+	g_mime_filter_set_size (GMIME_FILTER (headers), 1, TRUE);
+	*outptr = GMIME_FILTER (headers)->outbuf + offset;
+	*(*outptr)++ = '\n';
+}
+
+static void
+filter_filter (GMimeFilter *filter, char *inbuf, size_t inlen, size_t prespace,
+	       char **outbuf, size_t *outlen, size_t *outprespace)
+{
+	GMimeFilterHeaders *headers = (GMimeFilterHeaders *) filter;
+	register const char *inptr = inbuf;
+	const char *inend = inbuf + inlen;
+	char *lineptr, *lineend, *outptr;
+
+	(void) prespace;
+	if (headers->line == NULL) {
+		headers->line_size = 200;
+		headers->lineptr = headers->line = malloc (headers->line_size);
+	}
+	lineptr = headers->lineptr;
+	lineend = headers->line + headers->line_size;
+	if (lineptr == NULL)
+		return;
+	outptr = filter->outbuf;
+	while (inptr < inend) {
+		if (*inptr == '\n') {
+			if (headers->saw_nl)
+				output_final_newline(headers, &outptr);
+			headers->saw_nl = TRUE;
+			inptr++;
+			continue;
+		}
+
+		if (lineptr == lineend) {
+			headers->line_size *= 2;
+			headers->line = xrealloc (headers->line, headers->line_size);
+			lineptr = headers->line + headers->line_size / 2;
+			lineend = headers->line + headers->line_size;
+		}
+
+		if (headers->saw_nl && *inptr != ' ' && *inptr != '\t') {
+			*lineptr = '\0';
+			output_decoded_header (headers, &outptr);
+			lineptr = headers->line;
+		}
+		if (headers->saw_nl && (*inptr == ' ' || *inptr == '\t')) {
+			*lineptr = ' ';
+			lineptr++;
+			while (inptr < inend && (*inptr == ' ' || *inptr == '\t'))
+				inptr++;
+			headers->saw_nl = FALSE;
+			continue;
+		}
+		headers->saw_nl = FALSE;
+
+		if (*inptr != '\r')
+			*lineptr++ = *inptr;
+		inptr++;
+	}
+	if (headers->saw_nl) {
+		*lineptr = '\0';
+		output_decoded_header (headers, &outptr);
+		lineptr = headers->line;
+	}
+	headers->lineptr = lineptr;
+	*outlen = outptr - filter->outbuf;
+	*outprespace = filter->outpre;
+	*outbuf = filter->outbuf;
+}
+
+static void
+filter_complete (GMimeFilter *filter, char *inbuf, size_t inlen, size_t prespace,
+		 char **outbuf, size_t *outlen, size_t *outprespace)
+{
+	if (inbuf && inlen)
+		filter_filter (filter, inbuf, inlen, prespace, outbuf, outlen, outprespace);
+}
+
+static void
+filter_reset (GMimeFilter *filter)
+{
+	GMimeFilterHeaders *headers = (GMimeFilterHeaders *) filter;
+
+	headers->saw_nl = TRUE;
+	free(headers->line);
+	headers->line = NULL;
+	headers->line_size = 0;
+}
+
+
+/**
+ * g_mime_filter_headers_new:
+ * @encode: %TRUE if the filter should encode or %FALSE otherwise
+ * @dots: encode/decode dots (as for SMTP)
+ *
+ * Creates a new #GMimeFilterHeaders filter.
+ *
+ * If @encode is %TRUE, then all lines will be prefixed by "> ",
+ * otherwise any lines starting with "> " will have that removed
+ *
+ * Returns: a new #GMimeFilterHeaders filter.
+ **/
+GMimeFilter *
+g_mime_filter_headers_new (void)
+{
+	GMimeFilterHeaders *new_headers;
+
+	new_headers = (GMimeFilterHeaders *) g_object_newv (GMIME_TYPE_FILTER_HEADERS, 0, NULL);
+
+	return (GMimeFilter *) new_headers;
+}
+
diff --git a/gmime-filter-headers.h b/gmime-filter-headers.h
new file mode 100644
index 0000000..47d1d45
--- /dev/null
+++ b/gmime-filter-headers.h
@@ -0,0 +1,69 @@
+/*
+ * Copyright © 2009 Keith Packard <keithp@keithp.com>
+ * Copyright © 2010 Michal Sojka <sojkam1@fel.cvut.cz>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA.
+ */
+
+#ifndef _GMIME_FILTER_HEADERS_H_
+#define _GMIME_FILTER_HEADERS_H_
+
+#include <gmime/gmime-filter.h>
+
+G_BEGIN_DECLS
+
+#define GMIME_TYPE_FILTER_HEADERS            (g_mime_filter_headers_get_type ())
+#define GMIME_FILTER_HEADERS(obj)            (G_TYPE_CHECK_INSTANCE_CAST ((obj), GMIME_TYPE_FILTER_HEADERS, GMimeFilterHeaders))
+#define GMIME_FILTER_HEADERS_CLASS(klass)    (G_TYPE_CHECK_CLASS_CAST ((klass), GMIME_TYPE_FILTER_HEADERS, GMimeFilterHeadersClass))
+#define GMIME_IS_FILTER_HEADERS(obj)         (G_TYPE_CHECK_INSTANCE_TYPE ((obj), GMIME_TYPE_FILTER_HEADERS))
+#define GMIME_IS_FILTER_HEADERS_CLASS(klass) (G_TYPE_CHECK_CLASS_TYPE ((klass), GMIME_TYPE_FILTER_HEADERS))
+#define GMIME_FILTER_HEADERS_GET_CLASS(obj)  (G_TYPE_INSTANCE_GET_CLASS ((obj), GMIME_TYPE_FILTER_HEADERS, GMimeFilterHeadersClass))
+
+typedef struct _GMimeFilterHeaders GMimeFilterHeaders;
+typedef struct _GMimeFilterHeadersClass GMimeFilterHeadersClass;
+
+/**
+ * GMimeFilterHeaders:
+ * @parent_object: parent #GMimeFilter
+ * @saw_nl: previous char was a \n
+ * @line: temporary buffer for line unfolding
+ * @line_size: size of currently allocated nemory for @line
+ * @lineptr: pointer to the first unused character in @line
+ *
+ * A filter to decode rfc2047 encoded headers
+ **/
+struct _GMimeFilterHeaders {
+	GMimeFilter parent_object;
+
+	gboolean saw_nl;
+	char *line;
+	size_t line_size;
+	char *lineptr;
+};
+
+struct _GMimeFilterHeadersClass {
+	GMimeFilterClass parent_class;
+
+};
+
+
+GType g_mime_filter_headers_get_type (void);
+
+GMimeFilter *g_mime_filter_headers_new (void);
+
+G_END_DECLS
+
+
+#endif /* _GMIME_FILTER_HEADERS_H_ */
diff --git a/notmuch-reply.c b/notmuch-reply.c
index 98f6442..a1a536c 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -22,6 +22,7 @@
 
 #include "notmuch-client.h"
 #include "gmime-filter-reply.h"
+#include "gmime-filter-headers.h"
 
 static void
 reply_part_content (GMimeObject *part)
@@ -52,6 +53,25 @@ reply_part_content (GMimeObject *part)
 }
 
 static void
+show_reply_headers (GMimeMessage *message)
+{
+    GMimeStream *stream_stdout = NULL, *stream_filter = NULL;
+
+    stream_stdout = g_mime_stream_file_new (stdout);
+    if (stream_stdout) {
+	g_mime_stream_file_set_owner (GMIME_STREAM_FILE (stream_stdout), FALSE);
+	stream_filter = g_mime_stream_filter_new(stream_stdout);
+	if (stream_filter) {
+		g_mime_stream_filter_add(GMIME_STREAM_FILTER(stream_filter),
+					 g_mime_filter_headers_new());
+		g_mime_object_write_to_stream(GMIME_OBJECT(message), stream_filter);
+		g_object_unref(stream_filter);
+	}
+	g_object_unref(stream_stdout);
+    }
+}
+
+static void
 reply_part (GMimeObject *part, int *part_count)
 {
     GMimeContentDisposition *disposition;
@@ -290,7 +310,6 @@ notmuch_reply_format_default(void *ctx, notmuch_config_t *config, notmuch_query_
     notmuch_message_t *message;
     const char *subject, *from_addr = NULL;
     const char *in_reply_to, *orig_references, *references;
-    char *reply_headers;
 
     for (messages = notmuch_query_search_messages (query);
 	 notmuch_messages_has_more (messages);
@@ -306,7 +325,6 @@ notmuch_reply_format_default(void *ctx, notmuch_config_t *config, notmuch_query_
 	}
 
 	subject = notmuch_message_get_header (message, "subject");
-
 	if (strncasecmp (subject, "Re:", 3))
 	    subject = talloc_asprintf (ctx, "Re: %s", subject);
 	g_mime_message_set_subject (reply, subject);
@@ -339,9 +357,7 @@ notmuch_reply_format_default(void *ctx, notmuch_config_t *config, notmuch_query_
 	g_mime_object_set_header (GMIME_OBJECT (reply),
 				  "References", references);
 
-	reply_headers = g_mime_object_to_string (GMIME_OBJECT (reply));
-	printf ("%s", reply_headers);
-	free (reply_headers);
+	show_reply_headers (reply);
 
 	g_object_unref (G_OBJECT (reply));
 	reply = NULL;
-- 
tg: (3910000..) t/decode-headers-in-reply (depends on: master)

Re: [PATCH] Decode headers in reply

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 2204 bytes --]

On Wed,  3 Mar 2010 08:50:56 +0100, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> When headers contain non-ASCII characters, they are encoded according
> to rfc2047. Nomtuch reply command emits the headers in the encoded
> form, which makes them hard to read by humans who compose the reply.

This feature is obviously extremely handy. So thank you, Michal! I've
now pushed this out.

I think the implementation is over-complicated, but that's almost
entirely the fault of GMime and GObject, not yours. We are creating a
GMime object to represent the message and then asking GMime to print
it. GMime then does what it should (assuming the message is going to be
sent over SMTP) and encodes the headers.

We don't happen to want this encoding, since the client wants to allow
the user to edit the message and only encode the final result. There is
a single function that GMime provides for the decoding
(g_mime_utils_header_decode_text) but arranging for GMime to call this
function on each header requires over 200 lines of code with the hideous
GObject boilerplate and a bunch of replicated parsing code.

Meanwhile, what's GMime actually doing for us here? We have the original
decoded strings in the first place. And we know how to print "Subject:",
"To:", and "From:". So what do we need GMime for the headers?

The only thing I can think of is that GMime is also inserting
RFC-compliant wrapping of long lines. But since we want to let the user
edit the subject, etc. anyway, don't we have to rely on the client to do
that *anyway*?

Incidentally, it looks like message-mode does have support to do
RCF-compliant line wrapping if I hit M-q (`fill-paragraph') on a long
Subject: line but it doesn't appear to do that automatically. Instead,
it will just send a non-RFC-compliant message if I just type a long
Subject line. That looks like yet another reason to have a
notmuch-message-mode that extends message-mode and fixes several issues
we've been identifying.

Back to the patch though, I have pushed this out, but I've also added a
TODO item for simplifying this stuff. I think a handful of calls to
printf would be much better than this big pile of GObject boilerplate
and ad-hoc parsing.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]