RFC: output json from notmuch?

RFC: output json from notmuch?

[David Bremner]

It would be nice to have more structured output from notmuch-show.  I
decided to investigate sexp (i.e. lisp) and json output.

Sexp has the obvious advantage that it is trivially parsable in emacs;
for other clients it is a little more work.  I started looking at
sfsexp (http://sexpr.sourceforge.net).  It looks ok; I got a little
irritated that it doesn't support cons cells (i.e. dotted pairs). 

Then I found that json parsing is provided by the library json.el
shipped with emacs23, so I decided to play with json a bit.

I settled on the jansson library (http://www.digip.org/jansson/)
because it seemed to have a sane api and documentation, but there are
many choices.  I'll follow up with the actual patch, but the idea is to
replace the printfs in show_message with calls to set (key,value) pairs
in a json object, and output it at the end.

This is not in any sense a production patch (e.g. it needs to actually
return a thread object rather than just dumping messages out; nothing at
all has been done on the emacs side), but it gives you some idea of
would be involved.

So, do people think this is a reasonable idea to persue?

    1) it adds a dependency, but not a heavy one. jansson is about 300k
    installed on debian/i386

    2) It might mean that people using emacs22 might have to score
    json.el from somewhere; I'm not sure.

    3) There is some increase in memory use since the whole thread has
    to be built as a json object before being output.

    4) Of course jansson is doing it's own reference counting memory
    managment, and not using talloc. But we already have glib...

Of course we won't really know if it is good idea until we try it, but
if it already looks like a no-go, I'll stop.

Attachments:  json output of a message from carl, and the equivalent
sexpr as parsed by json.el.  




-- 
David Bremner                                  Professor, UNB Computer Science
bremner@unb.ca			           
http://www.cs.unb.ca/~bremner               Cross Appointment, UNB Mathematics
http://www.mitacs.ca/			   MITACS Atlantic Scientific Director

Re: RFC: output json from notmuch?

[David Bremner]

[-- Attachment #1: Type: text/plain, Size: 216 bytes --]

On Sun, 13 Dec 2009 19:21:02 -0400, David Bremner <david@tethera.net> wrote:
> 
> Attachments:  json output of a message from carl, and the equivalent
> sexpr as parsed by json.el.  
> 
uh, right. Second time lucky.

[-- Attachment #2: json encoded message --]
[-- Type: application/octet-stream, Size: 2283 bytes --]

{
    "match": true,
    "header": {
        "Cc": "",
        "Subject": "Re: [notmuch] Patch for xapian defect #250",
        "From": "Carl Worth <cworth@cworth.org>",
        "summary": "Carl Worth <cworth@cworth.org> (Fri. 01:13) ()",
        "To": "Kan-Ru Chen <kanru@kanru.info>, notmuch <notmuch@notmuchmail.org>",
        "Bcc": "",
        "Date": "Thu, 10 Dec 2009 21:13:07 -0800"
    },
    "id": "87638e3xgc.fsf@yoom.home.cworth.org",
    "filename": "/home/bremner/Maildir/.list.notmuch/new/1260508402.30807.pivot.cs.unb.ca",
    "depth": 1,
    "parts": [
        {
            "content-type": "text/plain",
            "id": 3,
            "text": "On Thu, 10 Dec 2009 15:00:42 +0800, Kan-Ru Chen <kanru@kanru.info> wrote:\n> The termlist is already sorted, so this is the patch trying to minimize\n> the modification of database as suggested in the comment and Carl's\n> TODO file.\n\nFantastic, Kan-Ru!\n\n> My poor profiling shows not much, but some improvement.\n\nNow you're just understating for sake of the pun. A 5-6x performance\nimprovement looks great. And I see that as well in my testing:\n\nBefore:\n\n\t$ time notmuch tag +foo tag:sent\n\treal 3m18.067s\n\t$ time notmuch tag -foo tag:sent\n\treal 3m12.940s\n\nAfter:\n\n\t$ time notmuch tag +foo tag:sent\n        real 0m31.497s\n\t$ time notmuch tag -foo tag:sent\n\treal 0m28.722s\n\nI didn't flush the OS caches between runs, but a subsequent run of the\n\"before\" code still performed similarly slow:\n\n\t$ time notmuch tag +foo tag:sent\n\treal 3m7.172s\n\nAnd if I *had* used cold caches for every run the benefit of the patch\nwould have looked even better.\n\nAnyway, we should get this upstream to the Xapian folks straight\naway. I expect they'll want to see a patch to the chert backend as well\nas the flint backend, (but fortunately the relevant code looks very\nsimilar if not identical).\n\nThanks again,\n\n-Carl\n"
        },
        {
            "content-type": "application/pgp-signature",
            "id": 4
        },
        {
            "content-type": "text/plain",
            "id": 5,
            "text": "_______________________________________________\nnotmuch mailing list\nnotmuch@notmuchmail.org\nhttp://notmuchmail.org/mailman/listinfo/notmuch\n"
        }
    ]
}

[-- Attachment #3: msg converted to sexpr by json.el --]
[-- Type: application/octet-stream, Size: 2065 bytes --]

((parts .
	[((text . "On Thu, 10 Dec 2009 15:00:42 +0800, Kan-Ru Chen <kanru@kanru.info> wrote:\n> The termlist is already sorted, so this is the patch trying to minimize\n> the modification of database as suggested in the comment and Carl's\n> TODO file.\n\nFantastic, Kan-Ru!\n\n> My poor profiling shows not much, but some improvement.\n\nNow you're just understating for sake of the pun. A 5-6x performance\nimprovement looks great. And I see that as well in my testing:\n\nBefore:\n\n	$ time notmuch tag +foo tag:sent\n	real 3m18.067s\n	$ time notmuch tag -foo tag:sent\n	real 3m12.940s\n\nAfter:\n\n	$ time notmuch tag +foo tag:sent\n        real 0m31.497s\n	$ time notmuch tag -foo tag:sent\n	real 0m28.722s\n\nI didn't flush the OS caches between runs, but a subsequent run of the\n\"before\" code still performed similarly slow:\n\n	$ time notmuch tag +foo tag:sent\n	real 3m7.172s\n\nAnd if I *had* used cold caches for every run the benefit of the patch\nwould have looked even better.\n\nAnyway, we should get this upstream to the Xapian folks straight\naway. I expect they'll want to see a patch to the chert backend as well\nas the flint backend, (but fortunately the relevant code looks very\nsimilar if not identical).\n\nThanks again,\n\n-Carl\n")
	  (id . 3)
	  (content-type . "text/plain"))
	 ((id . 4)
	  (content-type . "application/pgp-signature"))
	 ((text . "_______________________________________________\nnotmuch mailing list\nnotmuch@notmuchmail.org\nhttp://notmuchmail.org/mailman/listinfo/notmuch\n")
	  (id . 5)
	  (content-type . "text/plain"))])
 (depth . 1)
 (filename . "/home/bremner/Maildir/.list.notmuch/new/1260508402.30807.pivot.cs.unb.ca")
 (id . "87638e3xgc.fsf@yoom.home.cworth.org")
 (header
  (Date . "Thu, 10 Dec 2009 21:13:07 -0800")
  (Bcc . "")
  (To . "Kan-Ru Chen <kanru@kanru.info>, notmuch <notmuch@notmuchmail.org>")
  (summary . "Carl Worth <cworth@cworth.org> (Fri. 01:13) ()")
  (From . "Carl Worth <cworth@cworth.org>")
  (Subject . "Re: [notmuch] Patch for xapian defect #250")
  (Cc . ""))
 (match . t))

[-- Attachment #4: Type: text/plain, Size: 255 bytes --]


-- 
David Bremner                                  Professor, UNB Computer Science
bremner@unb.ca			           
http://www.cs.unb.ca/~bremner               Cross Appointment, UNB Mathematics
http://www.mitacs.ca/			   MITACS Atlantic Scientific Director

Re: RFC: output json from notmuch?

[Scott Robinson]

Excerpts from David Bremner's message of Sun Dec 13 15:26:02 -0800 2009:
> On Sun, 13 Dec 2009 19:21:02 -0400, David Bremner <david@tethera.net> wrote:
> > 
> > Attachments:  json output of a message from carl, and the equivalent
> > sexpr as parsed by json.el.  
> > 
> uh, right. Second time lucky.
> 

I have a patch for a --output=(text|json) for both notmuch-show and
notmuch-search. I mentioned it earlier on the list, and no one seemed to have
any interest.

Is it worth updating to HEAD and trying again?
-- 
Scott Robinson | http://quadhome.com/

Q: Why are my replies five sentences or less?
A: http://five.sentenc.es/

Re: RFC: output json from notmuch?

[David Bremner]

On Sun, 13 Dec 2009 16:05:07 -0800, Scott Robinson <scott@quadhome.com> wrote:

> I have a patch for a --output=(text|json) for both notmuch-show and
> notmuch-search. I mentioned it earlier on the list, and no one seemed to have
> any interest.

Ahh, I missed that. I think it was just before I subscribed to the list,
and I skipped over the subject about perl and python.

> Is it worth updating to HEAD and trying again?

Sure, it sounds more complete than what I have. You could also wait and
see what feedback this thread gathers.

On a different topic (and in response to Carl and your earlier
discussion), as a counter-weight to the desire to avoid dependencies
(which I agree with), I also think we should be careful about how many
embedded copies of code there are, from the point of keeping up with
security problems.  This is probably a bias I picked up from Debian.


-- 
David Bremner                                  Professor, UNB Computer Science
bremner@unb.ca			           
http://www.cs.unb.ca/~bremner               Cross Appointment, UNB Mathematics
http://www.mitacs.ca/			   MITACS Atlantic Scientific Director

Re: RFC: output json from notmuch?

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1271 bytes --]

On Sun, 13 Dec 2009 20:42:07 -0400, David Bremner <bremner@unb.ca> wrote:
> On a different topic (and in response to Carl and your earlier
> discussion), as a counter-weight to the desire to avoid dependencies
> (which I agree with), I also think we should be careful about how many
> embedded copies of code there are, from the point of keeping up with
> security problems.  This is probably a bias I picked up from Debian.

Embedded copies of code already packaged in Debian is something to
avoid, definitely. I'm not actively against having dependencies for
functionality that makes sense.

For example, the current libsha1 code in notmuch has been the subject of
a debate about embedded code copies here on the list already. If
somebody would like to maintain that code as a Debian package, then I
would be happy to depend on it that way rather than having an embedded
copy inside notmuch.

For something like JSON output, I really can't see how we need an
external library. The job we're talking about is changing our current
delimiters and then fixing our code to properly quote delimiters
appearing in the content. That doesn't sound like a job that needs a
library.

(Meanwhile, if we were parsing JSON, then I'd be happy to depend on a
library for that.)

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: RFC: output json from notmuch?

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 652 bytes --]

On Sun, 13 Dec 2009 16:05:07 -0800, Scott Robinson <scott@quadhome.com> wrote:
> I have a patch for a --output=(text|json) for both notmuch-show and
> notmuch-search. I mentioned it earlier on the list, and no one seemed to have
> any interest.

Hi Scott,

I remember you mentioning this earlier, but I didn't remember seeing the
actual patch? Did I miss it? I also didn't find it again when searching
and re-reading the thread. hopefully this isn't a case of my mail
client's search feature failing me... ;-)

> Is it worth updating to HEAD and trying again?

If you'd like to, please feel free. I'd be glad to evaluate different
options here.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: RFC: output json from notmuch?

[Marten Veldthuis]

Excerpts from David Bremner's message of Mon Dec 14 00:21:02 +0100 2009:
> So, do people think this is a reasonable idea to persue?

JSON was actually the first thing I thought of when I first saw the
output of notmuch-show. I think it's a much more natural fit for notmuch
than say, sexps, since most languages will have libraries for JSON,
which will be nicer to interfaces other than the emacs one.
-- 
- Marten

Re: RFC: output json from notmuch?

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 2204 bytes --]

On Sun, 13 Dec 2009 19:21:02 -0400, David Bremner <david@tethera.net> wrote:
> It would be nice to have more structured output from notmuch-show.  I
> decided to investigate sexp (i.e. lisp) and json output.

Thanks, David!

> Then I found that json parsing is provided by the library json.el
> shipped with emacs23, so I decided to play with json a bit.

That sounds very promising. I wasn't aware this existed inside emacs
already.

> I settled on the jansson library (http://www.digip.org/jansson/)
> because it seemed to have a sane api and documentation, but there are
> many choices.  I'll follow up with the actual patch, but the idea is to
> replace the printfs in show_message with calls to set (key,value) pairs
> in a json object, and output it at the end.

I'm still not sure of the need to depend on a library just to *generate*
any particular format. I would expect that job to be so constrained as
to be almost trivial. I won't necessarily block a patch based on that,
but I think we should at least look at how easy it would be to just emit
the syntax manually.

> So, do people think this is a reasonable idea to persue?

I do, yes. The current lack of full quoting is an obnoxious bug.

>     1) it adds a dependency, but not a heavy one. jansson is about 300k
>     installed on debian/i386

That might be acceptable. And we might rewrite the patch to avoid it as
well. I'd like to see what that would take.

>     2) It might mean that people using emacs22 might have to score
>     json.el from somewhere; I'm not sure.

Currently, it sounds like notmuch is entirely unusable from emacs22, so
I'm really not too concerned about one more thing that requires emacs
23.

>     3) There is some increase in memory use since the whole thread has
>     to be built as a json object before being output.

Right. That would be another reason to just stream the output
manually. The syntax doesn't require things like knowing the size of an
array before emitting the first element, right?

>     4) Of course jansson is doing it's own reference counting memory
>     managment, and not using talloc. But we already have glib...

It's not evil to not use talloc. So not an issue there.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: RFC: output json from notmuch?

[David Bremner]

On Mon, 14 Dec 2009 14:30:45 -0800, Carl Worth <cworth@cworth.org> wrote:

> I'm still not sure of the need to depend on a library just to *generate*
> any particular format. I would expect that job to be so constrained as
> to be almost trivial. I won't necessarily block a patch based on that,
> but I think we should at least look at how easy it would be to just emit
> the syntax manually.

OK, I'll wait for Scott's patch, and see how he is using cJSON. Just
stealing a few functions might be the way to go.

The cJSON code is less horrifying after running through indent. Now all
we need is "indent --carl" :)

d

[PATCH] JSON output for notmuch-search and notmuch-show.

[Scott Robinson]

Excerpts from David Bremner's message of Mon Dec 14 15:10:12 -0800 2009:
> OK, I'll wait for Scott's patch, and see how he is using cJSON. Just
> stealing a few functions might be the way to go.
> 
> The cJSON code is less horrifying after running through indent. Now all
> we need is "indent --carl" :)
> 

I took an earlier suggestion and didn't use cJSON, instead writing custom code
for emitting the new format.



Added an "--output=(json|text|)" command-line option to both
notmuch-search and notmuch-show.

In the case of notmuch-show, "--output=json" also implies
"--entire-thread" as the thread structure is implicit in the emitted
document tree.

As a coincidence to the implementation, multipart message ID numbers are
now incremented with each part printed. This changes the previous
semantics, which were unclear and not necessary related to the actual
ordering of the message parts.
---
 Makefile.local   |    3 +-
 json.c           |   73 ++++++++++++++
 notmuch-client.h |    3 +
 notmuch-search.c |  163 +++++++++++++++++++++++++++++---
 notmuch-show.c   |  275 ++++++++++++++++++++++++++++++++++++++++++++++--------
 notmuch.c        |   24 ++++--
 show-message.c   |    4 +-
 7 files changed, 481 insertions(+), 64 deletions(-)
 create mode 100644 json.c

diff --git a/Makefile.local b/Makefile.local
index 933ff4c..53b474b 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -18,7 +18,8 @@ notmuch_client_srcs =		\
 	notmuch-tag.c		\
 	notmuch-time.c		\
 	query-string.c		\
-	show-message.c
+	show-message.c		\
+	json.c
 
 notmuch_client_modules = $(notmuch_client_srcs:.c=.o)
 notmuch: $(notmuch_client_modules) lib/notmuch.a
diff --git a/json.c b/json.c
new file mode 100644
index 0000000..ee563d6
--- /dev/null
+++ b/json.c
@@ -0,0 +1,73 @@
+/* notmuch - Not much of an email program, (just index and search)
+ *
+ * Copyright © 2009 Carl Worth
+ * Copyright © 2009 Keith Packard
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see http://www.gnu.org/licenses/ .
+ *
+ * Authors: Carl Worth <cworth@cworth.org>
+ *	    Keith Packard <keithp@keithp.com>
+ */
+
+#include "notmuch-client.h"
+
+/*
+ * json_quote_str derived from cJSON's print_string_ptr,
+ * Copyright (c) 2009 Dave Gamble
+ */
+
+char *
+json_quote_str(const void *ctx, const char *str)
+{
+    const char *ptr;
+    char *ptr2;
+    char *out;
+    int len = 0;
+
+    if (!str)
+	return NULL;
+
+    for (ptr = str; *ptr; len++, ptr++) {
+	if (*ptr < 32 || *ptr == '\"' || *ptr == '\\')
+	    len++;
+    }
+
+    out = talloc_array (ctx, char, len + 3);
+
+    ptr = str;
+    ptr2 = out;
+
+    *ptr2++ = '\"';
+    while (*ptr) {
+	    if (*ptr > 31 && *ptr != '\"' && *ptr != '\\') {
+		*ptr2++ = *ptr++;
+	    } else {
+		*ptr2++ = '\\';
+		switch (*ptr++) {
+		    case '\"':	*ptr2++ = '\"';	break;
+		    case '\\':	*ptr2++ = '\\';	break;
+		    case '\b':	*ptr2++ = 'b';	break;
+		    case '\f':	*ptr2++ = 'f';	break;
+		    case '\n':	*ptr2++ = 'n';	break;
+		    case '\r':	*ptr2++ = 'r';	break;
+		    case '\t':	*ptr2++ = 't';	break;
+		    default:	 ptr2--;	break;
+		}
+	    }
+    }
+    *ptr2++ = '\"';
+    *ptr2++ = '\0';
+
+    return out;
+}
diff --git a/notmuch-client.h b/notmuch-client.h
index 50a30fe..7b844b9 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -143,6 +143,9 @@ notmuch_status_t
 show_message_body (const char *filename,
 		   void (*show_part) (GMimeObject *part, int *part_count));
 
+char *
+json_quote_str (const void *ctx, const char *str);
+
 /* notmuch-config.c */
 
 typedef struct _notmuch_config notmuch_config_t;
diff --git a/notmuch-search.c b/notmuch-search.c
index dc44eb6..e243747 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -20,8 +20,120 @@
 
 #include "notmuch-client.h"
 
+typedef struct search_format {
+    const char *results_start;
+    const char *thread_start;
+    void (*thread) (const void *ctx,
+		    const char *id,
+		    const time_t date,
+		    const int matched,
+		    const int total,
+		    const char *authors,
+		    const char *subject);
+    const char *tag_start;
+    const char *tag;
+    const char *tag_sep;
+    const char *tag_end;
+    const char *thread_sep;
+    const char *thread_end;
+    const char *results_end;
+} search_format_t;
+
+static void
+format_thread_text (const void *ctx,
+		    const char *id,
+		    const time_t date,
+		    const int matched,
+		    const int total,
+		    const char *authors,
+		    const char *subject);
+static const search_format_t format_text = {
+    "",
+	"",
+	    format_thread_text,
+	    " (",
+		"%s", " ",
+	    ")", "",
+	"\n",
+    "",
+};
+
+static void
+format_thread_json (const void *ctx,
+		    const char *id,
+		    const time_t date,
+		    const int matched,
+		    const int total,
+		    const char *authors,
+		    const char *subject);
+static const search_format_t format_json = {
+    "[",
+	"{",
+	    format_thread_json,
+	    "\"tags\": [",
+		"\"%s\"", ", ",
+	    "]", ",\n",
+	"}",
+    "]\n",
+};
+
+static void
+format_thread_text (const void *ctx,
+		    const char *id,
+		    const time_t date,
+		    const int matched,
+		    const int total,
+		    const char *authors,
+		    const char *subject)
+{
+    printf ("thread:%s %12s [%d/%d] %s; %s",
+	    id,
+	    notmuch_time_relative_date (ctx, date),
+	    matched,
+	    total,
+	    authors,
+	    subject);
+}
+
+static void
+format_thread_json (const void *ctx,
+		    const char *id,
+		    const time_t date,
+		    const int matched,
+		    const int total,
+		    const char *authors,
+		    const char *subject)
+{
+    struct tm *tm;
+    char timestamp[40];
+    void *ctx_quote = talloc_new (ctx);
+
+    tm = gmtime (&date);
+    if (tm == NULL)
+	INTERNAL_ERROR ("gmtime failed on thread %s.", id);
+
+    if (strftime (timestamp, sizeof (timestamp), "%s", tm) == 0)
+	INTERNAL_ERROR ("strftime failed on thread %s.", id);
+
+    printf ("\"id\": %s,\n"
+	    "\"timestamp\": %s,\n"
+	    "\"matched\": %d,\n"
+	    "\"total\": %d,\n"
+	    "\"authors\": %s,\n"
+	    "\"subject\": %s,\n",
+	    json_quote_str (ctx_quote, id),
+	    timestamp,
+	    matched,
+	    total,
+	    json_quote_str (ctx_quote, authors),
+	    json_quote_str (ctx_quote, subject));
+
+    talloc_free (ctx_quote);
+}
+
 static void
 do_search_threads (const void *ctx,
+		   const search_format_t *format,
 		   notmuch_query_t *query,
 		   notmuch_sort_t sort)
 {
@@ -29,7 +141,9 @@ do_search_threads (const void *ctx,
     notmuch_threads_t *threads;
     notmuch_tags_t *tags;
     time_t date;
-    const char *relative_date;
+    int first_thread = 1;
+
+    fputs (format->results_start, stdout);
 
     for (threads = notmuch_query_search_threads (query);
 	 notmuch_threads_has_more (threads);
@@ -37,6 +151,9 @@ do_search_threads (const void *ctx,
     {
 	int first_tag = 1;
 
+	if (! first_thread)
+	    fputs (format->thread_sep, stdout);
+
 	thread = notmuch_threads_get (threads);
 
 	if (sort == NOTMUCH_SORT_OLDEST_FIRST)
@@ -44,30 +161,37 @@ do_search_threads (const void *ctx,
 	else
 	    date = notmuch_thread_get_newest_date (thread);
 
-	relative_date = notmuch_time_relative_date (ctx, date);
+	fputs (format->thread_start, stdout);
+
+	format->thread (ctx,
+			notmuch_thread_get_thread_id (thread),
+			date,
+			notmuch_thread_get_matched_messages (thread),
+			notmuch_thread_get_total_messages (thread),
+			notmuch_thread_get_authors (thread),
+			notmuch_thread_get_subject (thread));
 
-	printf ("thread:%s %12s [%d/%d] %s; %s",
-		notmuch_thread_get_thread_id (thread),
-		relative_date,
-		notmuch_thread_get_matched_messages (thread),
-		notmuch_thread_get_total_messages (thread),
-		notmuch_thread_get_authors (thread),
-		notmuch_thread_get_subject (thread));
+	fputs (format->tag_start, stdout);
 
-	printf (" (");
 	for (tags = notmuch_thread_get_tags (thread);
 	     notmuch_tags_has_more (tags);
 	     notmuch_tags_advance (tags))
 	{
 	    if (! first_tag)
-		printf (" ");
-	    printf ("%s", notmuch_tags_get (tags));
+		fputs (format->tag_sep, stdout);
+	    printf (format->tag, notmuch_tags_get (tags));
 	    first_tag = 0;
 	}
-	printf (")\n");
+
+	fputs (format->tag_end, stdout);
+	fputs (format->thread_end, stdout);
+
+	first_thread = 0;
 
 	notmuch_thread_destroy (thread);
     }
+
+    fputs (format->results_end, stdout);
 }
 
 int
@@ -79,6 +203,7 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
     char *query_str;
     char *opt;
     notmuch_sort_t sort = NOTMUCH_SORT_NEWEST_FIRST;
+    const search_format_t *format = &format_text;
     int i;
 
     for (i = 0; i < argc && argv[i][0] == '-'; i++) {
@@ -96,6 +221,16 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
 		fprintf (stderr, "Invalid value for --sort: %s\n", opt);
 		return 1;
 	    }
+	} else if (STRNCMP_LITERAL (argv[i], "--output=") == 0) {
+	    opt = argv[i] + sizeof ("--output=") - 1;
+	    if (strcmp (opt, "text") == 0) {
+		format = &format_text;
+	    } else if (strcmp (opt, "json") == 0) {
+		format = &format_json;
+	    } else {
+		fprintf (stderr, "Invalid value for --output: %s\n", opt);
+		return 1;
+	    }
 	} else {
 	    fprintf (stderr, "Unrecognized option: %s\n", argv[i]);
 	    return 1;
@@ -132,7 +267,7 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
 
     notmuch_query_set_sort (query, sort);
 
-    do_search_threads (ctx, query, sort);
+    do_search_threads (ctx, format, query, sort);
 
     notmuch_query_destroy (query);
     notmuch_database_close (notmuch);
diff --git a/notmuch-show.c b/notmuch-show.c
index 376aacd..70d2605 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -20,8 +20,65 @@
 
 #include "notmuch-client.h"
 
+typedef struct show_format {
+    const char *message_set_start;
+    const char *message_start;
+    void (*message) (const void *ctx,
+		     notmuch_message_t *message,
+		     int indent);
+    const char *header_start;
+    void (*header) (const void *ctx,
+		    notmuch_message_t *message);
+    const char *header_end;
+    const char *body_start;
+    void (*part) (GMimeObject *part,
+		  int *part_count);
+    const char *body_end;
+    const char *message_end;
+    const char *message_set_sep;
+    const char *message_set_end;
+} show_format_t;
+
+static void
+format_message_text (unused (const void *ctx),
+		     notmuch_message_t *message,
+		     int indent);
+static void
+format_headers_text (const void *ctx,
+		     notmuch_message_t *message);
+static void
+format_part_text (GMimeObject *part,
+		  int *part_count);
+static const show_format_t format_text = {
+    "",
+	"\fmessage{ ", format_message_text,
+	    "\fheader{\n", format_headers_text, "\fheader}\n",
+	    "\fbody{\n", format_part_text, "\fbody}\n",
+	"\fmessage}\n", "",
+    ""
+};
+
+static void
+format_message_json (const void *ctx,
+		     notmuch_message_t *message,
+		     unused (int indent));
+static void
+format_headers_json (const void *ctx,
+		     notmuch_message_t *message);
+static void
+format_part_json (GMimeObject *part,
+		  int *part_count);
+static const show_format_t format_json = {
+    "[",
+	"{", format_message_json,
+	    ", \"headers\": {", format_headers_json, "}",
+	    ", \"body\": [", format_part_json, "]",
+	"}", ", ",
+    "]"
+};
+
 static const char *
-_get_tags_as_string (void *ctx, notmuch_message_t *message)
+_get_tags_as_string (const void *ctx, notmuch_message_t *message)
 {
     notmuch_tags_t *tags;
     int first = 1;
@@ -48,7 +105,7 @@ _get_tags_as_string (void *ctx, notmuch_message_t *message)
 
 /* Get a nice, single-line summary of message. */
 static const char *
-_get_one_line_summary (void *ctx, notmuch_message_t *message)
+_get_one_line_summary (const void *ctx, notmuch_message_t *message)
 {
     const char *from;
     time_t date;
@@ -67,18 +124,87 @@ _get_one_line_summary (void *ctx, notmuch_message_t *message)
 }
 
 static void
-show_part_content (GMimeObject *part)
+format_message_text (unused (const void *ctx), notmuch_message_t *message, int indent)
+{
+    printf ("id:%s depth:%d match:%d filename:%s\n",
+	    notmuch_message_get_message_id (message),
+	    indent,
+	    notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_MATCH),
+	    notmuch_message_get_filename (message));
+}
+
+static void
+format_message_json (const void *ctx, notmuch_message_t *message, unused (int indent))
+{
+    void *ctx_quote = talloc_new (ctx);
+    
+    printf ("\"id\": %s, \"match\": %s, \"filename\": %s",
+	    json_quote_str (ctx_quote, notmuch_message_get_message_id (message)),
+	    notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_MATCH) ? "true" : "false",
+	    json_quote_str (ctx_quote, notmuch_message_get_filename (message)));
+
+    talloc_free (ctx_quote);
+}
+
+static void
+format_headers_text (const void *ctx, notmuch_message_t *message)
+{
+    const char *headers[] = {
+	"Subject", "From", "To", "Cc", "Bcc", "Date"
+    };
+    const char *name, *value;
+    unsigned int i;
+
+    printf ("%s\n", _get_one_line_summary (ctx, message));
+
+    for (i = 0; i < ARRAY_SIZE (headers); i++) {
+	name = headers[i];
+	value = notmuch_message_get_header (message, name);
+	if (value)
+	    printf ("%s: %s\n", name, value);
+    }
+}
+
+static void
+format_headers_json (const void *ctx, notmuch_message_t *message)
+{
+    const char *headers[] = {
+	"Subject", "From", "To", "Cc", "Bcc", "Date"
+    };
+    const char *name, *value;
+    unsigned int i;
+    int first_header = 1;
+    void *ctx_quote = talloc_new (ctx);
+
+    for (i = 0; i < ARRAY_SIZE (headers); i++) {
+	name = headers[i];
+	value = notmuch_message_get_header (message, name);
+	if (value)
+	{
+	    if (!first_header)
+		fputs (", ", stdout);
+	    first_header = 0;
+
+	    printf ("%s: %s",
+		    json_quote_str (ctx_quote, name),
+		    json_quote_str (ctx_quote, value));
+	}
+    }
+
+    talloc_free (ctx_quote);
+}
+
+static void
+show_part_content (GMimeObject *part, GMimeStream *stream_out)
 {
-    GMimeStream *stream_stdout = g_mime_stream_file_new (stdout);
     GMimeStream *stream_filter = NULL;
     GMimeDataWrapper *wrapper;
     const char *charset;
 
     charset = g_mime_object_get_content_type_parameter (part, "charset");
 
-    if (stream_stdout) {
-	g_mime_stream_file_set_owner (GMIME_STREAM_FILE (stream_stdout), FALSE);
-	stream_filter = g_mime_stream_filter_new(stream_stdout);
+    if (stream_out) {
+	stream_filter = g_mime_stream_filter_new(stream_out);
 	g_mime_stream_filter_add(GMIME_STREAM_FILTER(stream_filter),
 				 g_mime_filter_crlf_new(FALSE, FALSE));
         if (charset) {
@@ -92,15 +218,16 @@ show_part_content (GMimeObject *part)
 	g_mime_data_wrapper_write_to_stream (wrapper, stream_filter);
     if (stream_filter)
 	g_object_unref(stream_filter);
-    if (stream_stdout)
-	g_object_unref(stream_stdout);
 }
 
 static void
-show_part (GMimeObject *part, int *part_count)
+format_part_text (GMimeObject *part, int *part_count)
 {
     GMimeContentDisposition *disposition;
     GMimeContentType *content_type;
+    GMimeStream *stream_stdout = g_mime_stream_file_new (stdout);
+
+    g_mime_stream_file_set_owner (GMIME_STREAM_FILE (stream_stdout), FALSE);
 
     disposition = g_mime_object_get_content_disposition (part);
     if (disposition &&
@@ -118,11 +245,14 @@ show_part (GMimeObject *part, int *part_count)
 	if (g_mime_content_type_is_type (content_type, "text", "*") &&
 	    !g_mime_content_type_is_type (content_type, "text", "html"))
 	{
-	    show_part_content (part);
+	    show_part_content (part, stream_stdout);
 	}
 
 	printf ("\fattachment}\n");
 
+	if (stream_stdout)
+	    g_object_unref(stream_stdout);
+
 	return;
     }
 
@@ -135,7 +265,7 @@ show_part (GMimeObject *part, int *part_count)
     if (g_mime_content_type_is_type (content_type, "text", "*") &&
 	!g_mime_content_type_is_type (content_type, "text", "html"))
     {
-	show_part_content (part);
+	show_part_content (part, stream_stdout);
     }
     else
     {
@@ -144,57 +274,93 @@ show_part (GMimeObject *part, int *part_count)
     }
 
     printf ("\fpart}\n");
+
+    if (stream_stdout)
+	g_object_unref(stream_stdout);
 }
 
 static void
-show_message (void *ctx, notmuch_message_t *message, int indent)
+format_part_json (GMimeObject *part, int *part_count)
 {
-    const char *headers[] = {
-	"Subject", "From", "To", "Cc", "Bcc", "Date"
-    };
-    const char *name, *value;
-    unsigned int i;
+    GMimeContentType *content_type;
+    GMimeContentDisposition *disposition;
+    void *ctx = talloc_new (NULL);
+    GMimeStream *stream_memory = g_mime_stream_mem_new ();
+    GByteArray *part_content;
 
-    printf ("\fmessage{ id:%s depth:%d match:%d filename:%s\n",
-	    notmuch_message_get_message_id (message),
-	    indent,
-	    notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_MATCH),
-	    notmuch_message_get_filename (message));
+    content_type = g_mime_object_get_content_type (GMIME_OBJECT (part));
 
-    printf ("\fheader{\n");
+    if (*part_count > 1)
+	fputs (", ", stdout);
 
-    printf ("%s\n", _get_one_line_summary (ctx, message));
+    printf ("{\"id\": %d, \"content-type\": %s",
+	    *part_count,
+	    json_quote_str (ctx, g_mime_content_type_to_string (content_type)));
 
-    for (i = 0; i < ARRAY_SIZE (headers); i++) {
-	name = headers[i];
-	value = notmuch_message_get_header (message, name);
-	if (value)
-	    printf ("%s: %s\n", name, value);
+    disposition = g_mime_object_get_content_disposition (part);
+    if (disposition &&
+	strcmp (disposition->disposition, GMIME_DISPOSITION_ATTACHMENT) == 0)
+    {
+	const char *filename = g_mime_part_get_filename (GMIME_PART (part));
+
+	printf (", \"filename\": %s", json_quote_str (ctx, filename));
     }
 
-    printf ("\fheader}\n");
-    printf ("\fbody{\n");
+    if (g_mime_content_type_is_type (content_type, "text", "*") &&
+	!g_mime_content_type_is_type (content_type, "text", "html"))
+    {
+	show_part_content (part, stream_memory);
+	part_content = g_mime_stream_mem_get_byte_array (GMIME_STREAM_MEM (stream_memory));
+
+	printf (", \"content\": %s", json_quote_str (ctx, (char *) part_content->data));
+    }
+
+    fputs ("}", stdout);
+
+    talloc_free (ctx);
+    if (stream_memory)
+	g_object_unref (stream_memory);
+}
+
+static void
+show_message (void *ctx, const show_format_t *format, notmuch_message_t *message, int indent)
+{
+    fputs (format->message_start, stdout);
+    format->message(ctx, message, indent);
 
-    show_message_body (notmuch_message_get_filename (message), show_part);
+    fputs (format->header_start, stdout);
+    format->header(ctx, message);
+    fputs (format->header_end, stdout);
 
-    printf ("\fbody}\n");
+    fputs (format->body_start, stdout);
+    show_message_body (notmuch_message_get_filename (message), format->part);
+    fputs (format->body_end, stdout);
 
-    printf ("\fmessage}\n");
+    fputs (format->message_end, stdout);
 }
 
 
 static void
-show_messages (void *ctx, notmuch_messages_t *messages, int indent,
+show_messages (void *ctx, const show_format_t *format, notmuch_messages_t *messages, int indent,
 	       notmuch_bool_t entire_thread)
 {
     notmuch_message_t *message;
     notmuch_bool_t match;
+    int first_set = 1;
     int next_indent;
 
+    fputs (format->message_set_start, stdout);
+
     for (;
 	 notmuch_messages_has_more (messages);
 	 notmuch_messages_advance (messages))
     {
+	if (!first_set)
+	    fputs (format->message_set_sep, stdout);
+	first_set = 0;
+
+	fputs (format->message_set_start, stdout);
+
 	message = notmuch_messages_get (messages);
 
 	match = notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_MATCH);
@@ -202,15 +368,21 @@ show_messages (void *ctx, notmuch_messages_t *messages, int indent,
 	next_indent = indent;
 
 	if (match || entire_thread) {
-	    show_message (ctx, message, indent);
+	    show_message (ctx, format, message, indent);
 	    next_indent = indent + 1;
+
+	    fputs (format->message_set_sep, stdout);
 	}
 
-	show_messages (ctx, notmuch_message_get_replies (message),
+	show_messages (ctx, format, notmuch_message_get_replies (message),
 		       next_indent, entire_thread);
 
 	notmuch_message_destroy (message);
+
+	fputs (format->message_set_end, stdout);
     }
+
+    fputs (format->message_set_end, stdout);
 }
 
 int
@@ -223,15 +395,29 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
     notmuch_thread_t *thread;
     notmuch_messages_t *messages;
     char *query_string;
+    char *opt;
+    const show_format_t *format = &format_text;
     int entire_thread = 0;
     int i;
+    int first_toplevel = 1;
 
     for (i = 0; i < argc && argv[i][0] == '-'; i++) {
 	if (strcmp (argv[i], "--") == 0) {
 	    i++;
 	    break;
 	}
-        if (strcmp(argv[i], "--entire-thread") == 0) {
+	if (STRNCMP_LITERAL (argv[i], "--output=") == 0) {
+	    opt = argv[i] + sizeof ("--output=") - 1;
+	    if (strcmp (opt, "text") == 0) {
+		format = &format_text;
+	    } else if (strcmp (opt, "json") == 0) {
+		format = &format_json;
+		entire_thread = 1;
+	    } else {
+		fprintf (stderr, "Invalid value for --output: %s\n", opt);
+		return 1;
+	    }
+	} else if (STRNCMP_LITERAL (argv[i], "--entire-thread") == 0) {
 	    entire_thread = 1;
 	} else {
 	    fprintf (stderr, "Unrecognized option: %s\n", argv[i]);
@@ -268,6 +454,8 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
 	return 1;
     }
 
+    fputs (format->message_set_start, stdout);
+
     for (threads = notmuch_query_search_threads (query);
 	 notmuch_threads_has_more (threads);
 	 notmuch_threads_advance (threads))
@@ -280,11 +468,18 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
 	    INTERNAL_ERROR ("Thread %s has no toplevel messages.\n",
 			    notmuch_thread_get_thread_id (thread));
 
-	show_messages (ctx, messages, 0, entire_thread);
+	if (!first_toplevel)
+	    fputs (format->message_set_sep, stdout);
+	first_toplevel = 0;
+
+	show_messages (ctx, format, messages, 0, entire_thread);
 
 	notmuch_thread_destroy (thread);
+
     }
 
+    fputs (format->message_set_end, stdout);
+
     notmuch_query_destroy (query);
     notmuch_database_close (notmuch);
 
diff --git a/notmuch.c b/notmuch.c
index 2ac8a59..aa2fc12 100644
--- a/notmuch.c
+++ b/notmuch.c
@@ -162,6 +162,11 @@ command_t commands[] = {
       "\n"
       "\t\tSupported options for search include:\n"
       "\n"
+      "\t\t--output=(json|text)\n"
+      "\n"
+      "\t\t\tPresents the results in either JSON or plain-text\n"
+      "\t\t\tformat, which is the default.\n"
+      "\n"
       "\t\t--sort=(newest-first|oldest-first)\n"
       "\n"
       "\t\t\tPresent results in either chronological order\n"
@@ -186,13 +191,18 @@ command_t commands[] = {
       "\t\t\tall messages in the same thread as any matched\n"
       "\t\t\tmessage will be displayed.\n"
       "\n"
-      "\t\tThe output format is plain-text, with all text-content\n"
-      "\t\tMIME parts decoded. Various components in the output,\n"
-      "\t\t('message', 'header', 'body', 'attachment', and MIME 'part')\n"
-      "\t\tare delimited by easily-parsed markers. Each marker consists\n"
-      "\t\tof a Control-L character (ASCII decimal 12), the name of\n"
-      "\t\tthe marker, and then either an opening or closing brace,\n"
-      "\t\t'{' or '}' to either open or close the component.\n"
+      "\t\t--output=(json|text)\n"
+      "\n"
+      "\t\t\tPresents the results in either JSON or plain-text\n"
+      "\t\t\tformat, which is the default.\n"
+      "\n"
+      "\t\tThe plain-text has all text-content MIME parts decoded.\n"
+      "\t\tVarious components in the output, ('message', 'header',\n"
+      "\t\t'body', 'attachment', and MIME 'part') are delimited by\n"
+      "\t\teasily-parsed markers. Each marker consists of a Control-L\n"
+      "\t\tcharacter (ASCII decimal 12), the name of the marker, and\n"
+      "\t\tthen either an opening or closing brace, '{' or '}' to\n"
+      "\t\teither open or close the component.\n"
       "\n"
       "\t\tA common use of \"notmuch show\" is to display a single\n"
       "\t\tthread of email messages. For this, use a search term of\n"
diff --git a/show-message.c b/show-message.c
index 784981b..05ced9c 100644
--- a/show-message.c
+++ b/show-message.c
@@ -26,8 +26,6 @@ static void
 show_message_part (GMimeObject *part, int *part_count,
 		   void (*show_part) (GMimeObject *part, int *part_count))
 {
-    *part_count = *part_count + 1;
-
     if (GMIME_IS_MULTIPART (part)) {
 	GMimeMultipart *multipart = GMIME_MULTIPART (part);
 	int i;
@@ -56,6 +54,8 @@ show_message_part (GMimeObject *part, int *part_count,
 	return;
     }
 
+    *part_count = *part_count + 1;
+
     (*show_part) (part, part_count);
 }
-- 
Scott Robinson | http://quadhome.com/

Q: Why are my replies five sentences or less?
A: http://five.sentenc.es/

[PATCH] Add an "--output=(json|text|)" command-line option to both notmuch-search and notmuch-show.

[david]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 25645 bytes --]

From: Scott Robinson <scott@quadhome.com>

In the case of notmuch-show, "--output=json" also implies
"--entire-thread" as the thread structure is implicit in the emitted
document tree.

As a coincidence to the implementation, multipart message ID numbers are
now incremented with each part printed. This changes the previous
semantics, which were unclear and not necessary related to the actual
ordering of the message parts.

Edited-By: David Bremner <david@tethera.net>
Reviewed-By: David Bremner <david@tethera.net>
---

It took me a little work to apply Scott's patch, so rather than asking
him to resend it from git-send-email, I am just sending. I hope no-one
is offended (much).

Other than manually extracting the patch from the output of notmuch
show (for me the message arrived base64 encoded), I deleted trailing
whitespace on line 465. 

It compiles, it doesn't seem to screw up the original output, and at
least in a few tests, it generates parseable json. Yay!.

I'm thinking that the patch I sent out last night to only dump message
ids could be reworked to use the framework of this patch.  I also
think it would be reasonably simple to add an --output=mbox option,
for archiving and so on.

 Makefile.local   |    3 +-
 json.c           |   73 ++++++++++++++
 notmuch-client.h |    3 +
 notmuch-search.c |  163 +++++++++++++++++++++++++++++---
 notmuch-show.c   |  275 ++++++++++++++++++++++++++++++++++++++++++++++--------
 notmuch.c        |   24 ++++--
 show-message.c   |    4 +-
 7 files changed, 481 insertions(+), 64 deletions(-)
 create mode 100644 json.c

diff --git a/Makefile.local b/Makefile.local
index 933ff4c..53b474b 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -18,7 +18,8 @@ notmuch_client_srcs =		\
 	notmuch-tag.c		\
 	notmuch-time.c		\
 	query-string.c		\
-	show-message.c
+	show-message.c		\
+	json.c
 
 notmuch_client_modules = $(notmuch_client_srcs:.c=.o)
 notmuch: $(notmuch_client_modules) lib/notmuch.a
diff --git a/json.c b/json.c
new file mode 100644
index 0000000..ee563d6
--- /dev/null
+++ b/json.c
@@ -0,0 +1,73 @@
+/* notmuch - Not much of an email program, (just index and search)
+ *
+ * Copyright © 2009 Carl Worth
+ * Copyright © 2009 Keith Packard
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see http://www.gnu.org/licenses/ .
+ *
+ * Authors: Carl Worth <cworth@cworth.org>
+ *	    Keith Packard <keithp@keithp.com>
+ */
+
+#include "notmuch-client.h"
+
+/*
+ * json_quote_str derived from cJSON's print_string_ptr,
+ * Copyright (c) 2009 Dave Gamble
+ */
+
+char *
+json_quote_str(const void *ctx, const char *str)
+{
+    const char *ptr;
+    char *ptr2;
+    char *out;
+    int len = 0;
+
+    if (!str)
+	return NULL;
+
+    for (ptr = str; *ptr; len++, ptr++) {
+	if (*ptr < 32 || *ptr == '\"' || *ptr == '\\')
+	    len++;
+    }
+
+    out = talloc_array (ctx, char, len + 3);
+
+    ptr = str;
+    ptr2 = out;
+
+    *ptr2++ = '\"';
+    while (*ptr) {
+	    if (*ptr > 31 && *ptr != '\"' && *ptr != '\\') {
+		*ptr2++ = *ptr++;
+	    } else {
+		*ptr2++ = '\\';
+		switch (*ptr++) {
+		    case '\"':	*ptr2++ = '\"';	break;
+		    case '\\':	*ptr2++ = '\\';	break;
+		    case '\b':	*ptr2++ = 'b';	break;
+		    case '\f':	*ptr2++ = 'f';	break;
+		    case '\n':	*ptr2++ = 'n';	break;
+		    case '\r':	*ptr2++ = 'r';	break;
+		    case '\t':	*ptr2++ = 't';	break;
+		    default:	 ptr2--;	break;
+		}
+	    }
+    }
+    *ptr2++ = '\"';
+    *ptr2++ = '\0';
+
+    return out;
+}
diff --git a/notmuch-client.h b/notmuch-client.h
index 50a30fe..7b844b9 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -143,6 +143,9 @@ notmuch_status_t
 show_message_body (const char *filename,
 		   void (*show_part) (GMimeObject *part, int *part_count));
 
+char *
+json_quote_str (const void *ctx, const char *str);
+
 /* notmuch-config.c */
 
 typedef struct _notmuch_config notmuch_config_t;
diff --git a/notmuch-search.c b/notmuch-search.c
index dc44eb6..e243747 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -20,8 +20,120 @@
 
 #include "notmuch-client.h"
 
+typedef struct search_format {
+    const char *results_start;
+    const char *thread_start;
+    void (*thread) (const void *ctx,
+		    const char *id,
+		    const time_t date,
+		    const int matched,
+		    const int total,
+		    const char *authors,
+		    const char *subject);
+    const char *tag_start;
+    const char *tag;
+    const char *tag_sep;
+    const char *tag_end;
+    const char *thread_sep;
+    const char *thread_end;
+    const char *results_end;
+} search_format_t;
+
+static void
+format_thread_text (const void *ctx,
+		    const char *id,
+		    const time_t date,
+		    const int matched,
+		    const int total,
+		    const char *authors,
+		    const char *subject);
+static const search_format_t format_text = {
+    "",
+	"",
+	    format_thread_text,
+	    " (",
+		"%s", " ",
+	    ")", "",
+	"\n",
+    "",
+};
+
+static void
+format_thread_json (const void *ctx,
+		    const char *id,
+		    const time_t date,
+		    const int matched,
+		    const int total,
+		    const char *authors,
+		    const char *subject);
+static const search_format_t format_json = {
+    "[",
+	"{",
+	    format_thread_json,
+	    "\"tags\": [",
+		"\"%s\"", ", ",
+	    "]", ",\n",
+	"}",
+    "]\n",
+};
+
+static void
+format_thread_text (const void *ctx,
+		    const char *id,
+		    const time_t date,
+		    const int matched,
+		    const int total,
+		    const char *authors,
+		    const char *subject)
+{
+    printf ("thread:%s %12s [%d/%d] %s; %s",
+	    id,
+	    notmuch_time_relative_date (ctx, date),
+	    matched,
+	    total,
+	    authors,
+	    subject);
+}
+
+static void
+format_thread_json (const void *ctx,
+		    const char *id,
+		    const time_t date,
+		    const int matched,
+		    const int total,
+		    const char *authors,
+		    const char *subject)
+{
+    struct tm *tm;
+    char timestamp[40];
+    void *ctx_quote = talloc_new (ctx);
+
+    tm = gmtime (&date);
+    if (tm == NULL)
+	INTERNAL_ERROR ("gmtime failed on thread %s.", id);
+
+    if (strftime (timestamp, sizeof (timestamp), "%s", tm) == 0)
+	INTERNAL_ERROR ("strftime failed on thread %s.", id);
+
+    printf ("\"id\": %s,\n"
+	    "\"timestamp\": %s,\n"
+	    "\"matched\": %d,\n"
+	    "\"total\": %d,\n"
+	    "\"authors\": %s,\n"
+	    "\"subject\": %s,\n",
+	    json_quote_str (ctx_quote, id),
+	    timestamp,
+	    matched,
+	    total,
+	    json_quote_str (ctx_quote, authors),
+	    json_quote_str (ctx_quote, subject));
+
+    talloc_free (ctx_quote);
+}
+
 static void
 do_search_threads (const void *ctx,
+		   const search_format_t *format,
 		   notmuch_query_t *query,
 		   notmuch_sort_t sort)
 {
@@ -29,7 +141,9 @@ do_search_threads (const void *ctx,
     notmuch_threads_t *threads;
     notmuch_tags_t *tags;
     time_t date;
-    const char *relative_date;
+    int first_thread = 1;
+
+    fputs (format->results_start, stdout);
 
     for (threads = notmuch_query_search_threads (query);
 	 notmuch_threads_has_more (threads);
@@ -37,6 +151,9 @@ do_search_threads (const void *ctx,
     {
 	int first_tag = 1;
 
+	if (! first_thread)
+	    fputs (format->thread_sep, stdout);
+
 	thread = notmuch_threads_get (threads);
 
 	if (sort == NOTMUCH_SORT_OLDEST_FIRST)
@@ -44,30 +161,37 @@ do_search_threads (const void *ctx,
 	else
 	    date = notmuch_thread_get_newest_date (thread);
 
-	relative_date = notmuch_time_relative_date (ctx, date);
+	fputs (format->thread_start, stdout);
+
+	format->thread (ctx,
+			notmuch_thread_get_thread_id (thread),
+			date,
+			notmuch_thread_get_matched_messages (thread),
+			notmuch_thread_get_total_messages (thread),
+			notmuch_thread_get_authors (thread),
+			notmuch_thread_get_subject (thread));
 
-	printf ("thread:%s %12s [%d/%d] %s; %s",
-		notmuch_thread_get_thread_id (thread),
-		relative_date,
-		notmuch_thread_get_matched_messages (thread),
-		notmuch_thread_get_total_messages (thread),
-		notmuch_thread_get_authors (thread),
-		notmuch_thread_get_subject (thread));
+	fputs (format->tag_start, stdout);
 
-	printf (" (");
 	for (tags = notmuch_thread_get_tags (thread);
 	     notmuch_tags_has_more (tags);
 	     notmuch_tags_advance (tags))
 	{
 	    if (! first_tag)
-		printf (" ");
-	    printf ("%s", notmuch_tags_get (tags));
+		fputs (format->tag_sep, stdout);
+	    printf (format->tag, notmuch_tags_get (tags));
 	    first_tag = 0;
 	}
-	printf (")\n");
+
+	fputs (format->tag_end, stdout);
+	fputs (format->thread_end, stdout);
+
+	first_thread = 0;
 
 	notmuch_thread_destroy (thread);
     }
+
+    fputs (format->results_end, stdout);
 }
 
 int
@@ -79,6 +203,7 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
     char *query_str;
     char *opt;
     notmuch_sort_t sort = NOTMUCH_SORT_NEWEST_FIRST;
+    const search_format_t *format = &format_text;
     int i;
 
     for (i = 0; i < argc && argv[i][0] == '-'; i++) {
@@ -96,6 +221,16 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
 		fprintf (stderr, "Invalid value for --sort: %s\n", opt);
 		return 1;
 	    }
+	} else if (STRNCMP_LITERAL (argv[i], "--output=") == 0) {
+	    opt = argv[i] + sizeof ("--output=") - 1;
+	    if (strcmp (opt, "text") == 0) {
+		format = &format_text;
+	    } else if (strcmp (opt, "json") == 0) {
+		format = &format_json;
+	    } else {
+		fprintf (stderr, "Invalid value for --output: %s\n", opt);
+		return 1;
+	    }
 	} else {
 	    fprintf (stderr, "Unrecognized option: %s\n", argv[i]);
 	    return 1;
@@ -132,7 +267,7 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
 
     notmuch_query_set_sort (query, sort);
 
-    do_search_threads (ctx, query, sort);
+    do_search_threads (ctx, format, query, sort);
 
     notmuch_query_destroy (query);
     notmuch_database_close (notmuch);
diff --git a/notmuch-show.c b/notmuch-show.c
index 376aacd..b5b3eba 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -20,8 +20,65 @@
 
 #include "notmuch-client.h"
 
+typedef struct show_format {
+    const char *message_set_start;
+    const char *message_start;
+    void (*message) (const void *ctx,
+		     notmuch_message_t *message,
+		     int indent);
+    const char *header_start;
+    void (*header) (const void *ctx,
+		    notmuch_message_t *message);
+    const char *header_end;
+    const char *body_start;
+    void (*part) (GMimeObject *part,
+		  int *part_count);
+    const char *body_end;
+    const char *message_end;
+    const char *message_set_sep;
+    const char *message_set_end;
+} show_format_t;
+
+static void
+format_message_text (unused (const void *ctx),
+		     notmuch_message_t *message,
+		     int indent);
+static void
+format_headers_text (const void *ctx,
+		     notmuch_message_t *message);
+static void
+format_part_text (GMimeObject *part,
+		  int *part_count);
+static const show_format_t format_text = {
+    "",
+	"\fmessage{ ", format_message_text,
+	    "\fheader{\n", format_headers_text, "\fheader}\n",
+	    "\fbody{\n", format_part_text, "\fbody}\n",
+	"\fmessage}\n", "",
+    ""
+};
+
+static void
+format_message_json (const void *ctx,
+		     notmuch_message_t *message,
+		     unused (int indent));
+static void
+format_headers_json (const void *ctx,
+		     notmuch_message_t *message);
+static void
+format_part_json (GMimeObject *part,
+		  int *part_count);
+static const show_format_t format_json = {
+    "[",
+	"{", format_message_json,
+	    ", \"headers\": {", format_headers_json, "}",
+	    ", \"body\": [", format_part_json, "]",
+	"}", ", ",
+    "]"
+};
+
 static const char *
-_get_tags_as_string (void *ctx, notmuch_message_t *message)
+_get_tags_as_string (const void *ctx, notmuch_message_t *message)
 {
     notmuch_tags_t *tags;
     int first = 1;
@@ -48,7 +105,7 @@ _get_tags_as_string (void *ctx, notmuch_message_t *message)
 
 /* Get a nice, single-line summary of message. */
 static const char *
-_get_one_line_summary (void *ctx, notmuch_message_t *message)
+_get_one_line_summary (const void *ctx, notmuch_message_t *message)
 {
     const char *from;
     time_t date;
@@ -67,18 +124,87 @@ _get_one_line_summary (void *ctx, notmuch_message_t *message)
 }
 
 static void
-show_part_content (GMimeObject *part)
+format_message_text (unused (const void *ctx), notmuch_message_t *message, int indent)
+{
+    printf ("id:%s depth:%d match:%d filename:%s\n",
+	    notmuch_message_get_message_id (message),
+	    indent,
+	    notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_MATCH),
+	    notmuch_message_get_filename (message));
+}
+
+static void
+format_message_json (const void *ctx, notmuch_message_t *message, unused (int indent))
+{
+    void *ctx_quote = talloc_new (ctx);
+
+    printf ("\"id\": %s, \"match\": %s, \"filename\": %s",
+	    json_quote_str (ctx_quote, notmuch_message_get_message_id (message)),
+	    notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_MATCH) ? "true" : "false",
+	    json_quote_str (ctx_quote, notmuch_message_get_filename (message)));
+
+    talloc_free (ctx_quote);
+}
+
+static void
+format_headers_text (const void *ctx, notmuch_message_t *message)
+{
+    const char *headers[] = {
+	"Subject", "From", "To", "Cc", "Bcc", "Date"
+    };
+    const char *name, *value;
+    unsigned int i;
+
+    printf ("%s\n", _get_one_line_summary (ctx, message));
+
+    for (i = 0; i < ARRAY_SIZE (headers); i++) {
+	name = headers[i];
+	value = notmuch_message_get_header (message, name);
+	if (value)
+	    printf ("%s: %s\n", name, value);
+    }
+}
+
+static void
+format_headers_json (const void *ctx, notmuch_message_t *message)
+{
+    const char *headers[] = {
+	"Subject", "From", "To", "Cc", "Bcc", "Date"
+    };
+    const char *name, *value;
+    unsigned int i;
+    int first_header = 1;
+    void *ctx_quote = talloc_new (ctx);
+
+    for (i = 0; i < ARRAY_SIZE (headers); i++) {
+	name = headers[i];
+	value = notmuch_message_get_header (message, name);
+	if (value)
+	{
+	    if (!first_header)
+		fputs (", ", stdout);
+	    first_header = 0;
+
+	    printf ("%s: %s",
+		    json_quote_str (ctx_quote, name),
+		    json_quote_str (ctx_quote, value));
+	}
+    }
+
+    talloc_free (ctx_quote);
+}
+
+static void
+show_part_content (GMimeObject *part, GMimeStream *stream_out)
 {
-    GMimeStream *stream_stdout = g_mime_stream_file_new (stdout);
     GMimeStream *stream_filter = NULL;
     GMimeDataWrapper *wrapper;
     const char *charset;
 
     charset = g_mime_object_get_content_type_parameter (part, "charset");
 
-    if (stream_stdout) {
-	g_mime_stream_file_set_owner (GMIME_STREAM_FILE (stream_stdout), FALSE);
-	stream_filter = g_mime_stream_filter_new(stream_stdout);
+    if (stream_out) {
+	stream_filter = g_mime_stream_filter_new(stream_out);
 	g_mime_stream_filter_add(GMIME_STREAM_FILTER(stream_filter),
 				 g_mime_filter_crlf_new(FALSE, FALSE));
         if (charset) {
@@ -92,15 +218,16 @@ show_part_content (GMimeObject *part)
 	g_mime_data_wrapper_write_to_stream (wrapper, stream_filter);
     if (stream_filter)
 	g_object_unref(stream_filter);
-    if (stream_stdout)
-	g_object_unref(stream_stdout);
 }
 
 static void
-show_part (GMimeObject *part, int *part_count)
+format_part_text (GMimeObject *part, int *part_count)
 {
     GMimeContentDisposition *disposition;
     GMimeContentType *content_type;
+    GMimeStream *stream_stdout = g_mime_stream_file_new (stdout);
+
+    g_mime_stream_file_set_owner (GMIME_STREAM_FILE (stream_stdout), FALSE);
 
     disposition = g_mime_object_get_content_disposition (part);
     if (disposition &&
@@ -118,11 +245,14 @@ show_part (GMimeObject *part, int *part_count)
 	if (g_mime_content_type_is_type (content_type, "text", "*") &&
 	    !g_mime_content_type_is_type (content_type, "text", "html"))
 	{
-	    show_part_content (part);
+	    show_part_content (part, stream_stdout);
 	}
 
 	printf ("\fattachment}\n");
 
+	if (stream_stdout)
+	    g_object_unref(stream_stdout);
+
 	return;
     }
 
@@ -135,7 +265,7 @@ show_part (GMimeObject *part, int *part_count)
     if (g_mime_content_type_is_type (content_type, "text", "*") &&
 	!g_mime_content_type_is_type (content_type, "text", "html"))
     {
-	show_part_content (part);
+	show_part_content (part, stream_stdout);
     }
     else
     {
@@ -144,57 +274,93 @@ show_part (GMimeObject *part, int *part_count)
     }
 
     printf ("\fpart}\n");
+
+    if (stream_stdout)
+	g_object_unref(stream_stdout);
 }
 
 static void
-show_message (void *ctx, notmuch_message_t *message, int indent)
+format_part_json (GMimeObject *part, int *part_count)
 {
-    const char *headers[] = {
-	"Subject", "From", "To", "Cc", "Bcc", "Date"
-    };
-    const char *name, *value;
-    unsigned int i;
+    GMimeContentType *content_type;
+    GMimeContentDisposition *disposition;
+    void *ctx = talloc_new (NULL);
+    GMimeStream *stream_memory = g_mime_stream_mem_new ();
+    GByteArray *part_content;
 
-    printf ("\fmessage{ id:%s depth:%d match:%d filename:%s\n",
-	    notmuch_message_get_message_id (message),
-	    indent,
-	    notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_MATCH),
-	    notmuch_message_get_filename (message));
+    content_type = g_mime_object_get_content_type (GMIME_OBJECT (part));
 
-    printf ("\fheader{\n");
+    if (*part_count > 1)
+	fputs (", ", stdout);
 
-    printf ("%s\n", _get_one_line_summary (ctx, message));
+    printf ("{\"id\": %d, \"content-type\": %s",
+	    *part_count,
+	    json_quote_str (ctx, g_mime_content_type_to_string (content_type)));
 
-    for (i = 0; i < ARRAY_SIZE (headers); i++) {
-	name = headers[i];
-	value = notmuch_message_get_header (message, name);
-	if (value)
-	    printf ("%s: %s\n", name, value);
+    disposition = g_mime_object_get_content_disposition (part);
+    if (disposition &&
+	strcmp (disposition->disposition, GMIME_DISPOSITION_ATTACHMENT) == 0)
+    {
+	const char *filename = g_mime_part_get_filename (GMIME_PART (part));
+
+	printf (", \"filename\": %s", json_quote_str (ctx, filename));
     }
 
-    printf ("\fheader}\n");
-    printf ("\fbody{\n");
+    if (g_mime_content_type_is_type (content_type, "text", "*") &&
+	!g_mime_content_type_is_type (content_type, "text", "html"))
+    {
+	show_part_content (part, stream_memory);
+	part_content = g_mime_stream_mem_get_byte_array (GMIME_STREAM_MEM (stream_memory));
+
+	printf (", \"content\": %s", json_quote_str (ctx, (char *) part_content->data));
+    }
+
+    fputs ("}", stdout);
+
+    talloc_free (ctx);
+    if (stream_memory)
+	g_object_unref (stream_memory);
+}
+
+static void
+show_message (void *ctx, const show_format_t *format, notmuch_message_t *message, int indent)
+{
+    fputs (format->message_start, stdout);
+    format->message(ctx, message, indent);
 
-    show_message_body (notmuch_message_get_filename (message), show_part);
+    fputs (format->header_start, stdout);
+    format->header(ctx, message);
+    fputs (format->header_end, stdout);
 
-    printf ("\fbody}\n");
+    fputs (format->body_start, stdout);
+    show_message_body (notmuch_message_get_filename (message), format->part);
+    fputs (format->body_end, stdout);
 
-    printf ("\fmessage}\n");
+    fputs (format->message_end, stdout);
 }
 
 
 static void
-show_messages (void *ctx, notmuch_messages_t *messages, int indent,
+show_messages (void *ctx, const show_format_t *format, notmuch_messages_t *messages, int indent,
 	       notmuch_bool_t entire_thread)
 {
     notmuch_message_t *message;
     notmuch_bool_t match;
+    int first_set = 1;
     int next_indent;
 
+    fputs (format->message_set_start, stdout);
+
     for (;
 	 notmuch_messages_has_more (messages);
 	 notmuch_messages_advance (messages))
     {
+	if (!first_set)
+	    fputs (format->message_set_sep, stdout);
+	first_set = 0;
+
+	fputs (format->message_set_start, stdout);
+
 	message = notmuch_messages_get (messages);
 
 	match = notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_MATCH);
@@ -202,15 +368,21 @@ show_messages (void *ctx, notmuch_messages_t *messages, int indent,
 	next_indent = indent;
 
 	if (match || entire_thread) {
-	    show_message (ctx, message, indent);
+	    show_message (ctx, format, message, indent);
 	    next_indent = indent + 1;
+
+	    fputs (format->message_set_sep, stdout);
 	}
 
-	show_messages (ctx, notmuch_message_get_replies (message),
+	show_messages (ctx, format, notmuch_message_get_replies (message),
 		       next_indent, entire_thread);
 
 	notmuch_message_destroy (message);
+
+	fputs (format->message_set_end, stdout);
     }
+
+    fputs (format->message_set_end, stdout);
 }
 
 int
@@ -223,15 +395,29 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
     notmuch_thread_t *thread;
     notmuch_messages_t *messages;
     char *query_string;
+    char *opt;
+    const show_format_t *format = &format_text;
     int entire_thread = 0;
     int i;
+    int first_toplevel = 1;
 
     for (i = 0; i < argc && argv[i][0] == '-'; i++) {
 	if (strcmp (argv[i], "--") == 0) {
 	    i++;
 	    break;
 	}
-        if (strcmp(argv[i], "--entire-thread") == 0) {
+	if (STRNCMP_LITERAL (argv[i], "--output=") == 0) {
+	    opt = argv[i] + sizeof ("--output=") - 1;
+	    if (strcmp (opt, "text") == 0) {
+		format = &format_text;
+	    } else if (strcmp (opt, "json") == 0) {
+		format = &format_json;
+		entire_thread = 1;
+	    } else {
+		fprintf (stderr, "Invalid value for --output: %s\n", opt);
+		return 1;
+	    }
+	} else if (STRNCMP_LITERAL (argv[i], "--entire-thread") == 0) {
 	    entire_thread = 1;
 	} else {
 	    fprintf (stderr, "Unrecognized option: %s\n", argv[i]);
@@ -268,6 +454,8 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
 	return 1;
     }
 
+    fputs (format->message_set_start, stdout);
+
     for (threads = notmuch_query_search_threads (query);
 	 notmuch_threads_has_more (threads);
 	 notmuch_threads_advance (threads))
@@ -280,11 +468,18 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
 	    INTERNAL_ERROR ("Thread %s has no toplevel messages.\n",
 			    notmuch_thread_get_thread_id (thread));
 
-	show_messages (ctx, messages, 0, entire_thread);
+	if (!first_toplevel)
+	    fputs (format->message_set_sep, stdout);
+	first_toplevel = 0;
+
+	show_messages (ctx, format, messages, 0, entire_thread);
 
 	notmuch_thread_destroy (thread);
+
     }
 
+    fputs (format->message_set_end, stdout);
+
     notmuch_query_destroy (query);
     notmuch_database_close (notmuch);
 
diff --git a/notmuch.c b/notmuch.c
index 2ac8a59..aa2fc12 100644
--- a/notmuch.c
+++ b/notmuch.c
@@ -162,6 +162,11 @@ command_t commands[] = {
       "\n"
       "\t\tSupported options for search include:\n"
       "\n"
+      "\t\t--output=(json|text)\n"
+      "\n"
+      "\t\t\tPresents the results in either JSON or plain-text\n"
+      "\t\t\tformat, which is the default.\n"
+      "\n"
       "\t\t--sort=(newest-first|oldest-first)\n"
       "\n"
       "\t\t\tPresent results in either chronological order\n"
@@ -186,13 +191,18 @@ command_t commands[] = {
       "\t\t\tall messages in the same thread as any matched\n"
       "\t\t\tmessage will be displayed.\n"
       "\n"
-      "\t\tThe output format is plain-text, with all text-content\n"
-      "\t\tMIME parts decoded. Various components in the output,\n"
-      "\t\t('message', 'header', 'body', 'attachment', and MIME 'part')\n"
-      "\t\tare delimited by easily-parsed markers. Each marker consists\n"
-      "\t\tof a Control-L character (ASCII decimal 12), the name of\n"
-      "\t\tthe marker, and then either an opening or closing brace,\n"
-      "\t\t'{' or '}' to either open or close the component.\n"
+      "\t\t--output=(json|text)\n"
+      "\n"
+      "\t\t\tPresents the results in either JSON or plain-text\n"
+      "\t\t\tformat, which is the default.\n"
+      "\n"
+      "\t\tThe plain-text has all text-content MIME parts decoded.\n"
+      "\t\tVarious components in the output, ('message', 'header',\n"
+      "\t\t'body', 'attachment', and MIME 'part') are delimited by\n"
+      "\t\teasily-parsed markers. Each marker consists of a Control-L\n"
+      "\t\tcharacter (ASCII decimal 12), the name of the marker, and\n"
+      "\t\tthen either an opening or closing brace, '{' or '}' to\n"
+      "\t\teither open or close the component.\n"
       "\n"
       "\t\tA common use of \"notmuch show\" is to display a single\n"
       "\t\tthread of email messages. For this, use a search term of\n"
diff --git a/show-message.c b/show-message.c
index 784981b..05ced9c 100644
--- a/show-message.c
+++ b/show-message.c
@@ -26,8 +26,6 @@ static void
 show_message_part (GMimeObject *part, int *part_count,
 		   void (*show_part) (GMimeObject *part, int *part_count))
 {
-    *part_count = *part_count + 1;
-
     if (GMIME_IS_MULTIPART (part)) {
 	GMimeMultipart *multipart = GMIME_MULTIPART (part);
 	int i;
@@ -56,6 +54,8 @@ show_message_part (GMimeObject *part, int *part_count,
 	return;
     }
 
+    *part_count = *part_count + 1;
+
     (*show_part) (part, part_count);
 }
 
-- 
1.6.5.3

Re: [PATCH] Add an "--output=(json|text|)" command-line option to both notmuch-search and notmuch-show.

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 795 bytes --]

On Fri, 18 Dec 2009 08:59:55 -0400, david@tethera.net wrote:
> It took me a little work to apply Scott's patch, so rather than asking
> him to resend it from git-send-email, I am just sending. I hope no-one
> is offended (much).

I think that's great! Collaboration is what this is all about.

> I'm thinking that the patch I sent out last night to only dump message
> ids could be reworked to use the framework of this patch.  I also
> think it would be reasonably simple to add an --output=mbox option,
> for archiving and so on.

I think that selecting *what* to emit is orthogonal from selecting *how*
to format that output. See some ideas in the TODO file, (where I
proposed --for and --format options for these). Having a way to do mbox
output for export would indeed be very nice.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH] Add an "--output=(json|text|)" command-line option to both notmuch-search and notmuch-show.

[Scott Robinson]

Excerpts from Carl Worth's message of Fri Dec 18 09:33:43 -0800 2009:
> On Fri, 18 Dec 2009 08:59:55 -0400, david@tethera.net wrote:
> > It took me a little work to apply Scott's patch, so rather than asking
> > him to resend it from git-send-email, I am just sending. I hope no-one
> > is offended (much).
> 
> I think that's great! Collaboration is what this is all about.

Me too!

I've never used git-send-email. I'll give it a whirl on my next patch.

> > I'm thinking that the patch I sent out last night to only dump message
> > ids could be reworked to use the framework of this patch.  I also
> > think it would be reasonably simple to add an --output=mbox option,
> > for archiving and so on.
> 
> I think that selecting *what* to emit is orthogonal from selecting *how*
> to format that output. See some ideas in the TODO file, (where I
> proposed --for and --format options for these). Having a way to do mbox
> output for export would indeed be very nice.

Haha! I originally used "--format" and changed for some reason that escapes me
now.

Implementing an "mbox" formatted output in the current logic wouldn't be
archive perfect. The message body is emitted on a per-part basis.

What I would do is change the semantics of format->body to be called from
show_message. Then the text and json parts would point at the original
implementation passing off their per-part function pointers. And, a new mbox
implementation would just dump the full message body.
-- 
Scott Robinson | http://quadhome.com/

Q: Why are my replies five sentences or less?
A: http://five.sentenc.es/

Re: [PATCH] Add an "--output=(json|text|)" command-line option to both notmuch-search and notmuch-show.

[David Bremner]

On Fri, 18 Dec 2009 09:33:43 -0800, Carl Worth <cworth@cworth.org> wrote:

> I think that selecting *what* to emit is orthogonal from selecting *how*
> to format that output. 

I can see that point of view.

> See some ideas in the TODO file, (where I proposed --for and --format
> options for these).

It's a detail, but could you choose two names that are not substrings of
each other?  Eventually we do want tab completion on the command line to
work :).  Also, "search --for tags foo" suggests to me that
searching for tags matching foo.  What about using --output for that?
One thing that is not completely clear to me at this point is what the
difference is between 

    notmuch search --for messages  search-terms

and 

    notmuch show search-terms

David

Re: [PATCH] Add an "--output=(json|text|)" command-line option to both notmuch-search and notmuch-show.

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1791 bytes --]

On Fri, 18 Dec 2009 20:36:34 -0400, David Bremner <david@tethera.net> wrote:
> It's a detail, but could you choose two names that are not substrings of
> each other?  Eventually we do want tab completion on the command line to
> work :).

Yes, that's a good point.

> Also, "search --for tags foo" suggests to me that
> searching for tags matching foo.  What about using --output for that?

OK. "--output" sounds good to me here.

> One thing that is not completely clear to me at this point is what the
> difference is between 
> 
>     notmuch search --for messages  search-terms
> 
> and 
> 
>     notmuch show search-terms

So, "notmuch show <search-terms>" is clear enough---it works as it does
today.

The new command, ("notmuch search --output=messages"), would be quite
different. It would have single-line output for each message, (as
"notmuch search" has single line-output already, but for threads by
default). You can see behavior like this in the "notmuch
search-messages" command for which I sent a patch a while ago, (but have
never merged).

The idea is that "notmuch search" would always give single-line output
suitable for various kinds of processing.

For example. How much mail have I sent?

	notmuch search --output=messages tag:sent | wc -l

That's something you can't do with a thread-based search, (and it's not
convenient to get a robust result from "notmuch show").

Once this is combined with a new --format to select what gets printed, I
can imagine a lot of useful things, like collecting email addresses:

	notmuch search --output=messages --format="${FROM}" to:cworth@cworth.org

or whatever. I can imagine a lot of different queries I'd like to be
able to make of my mail store with things like this.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Prototype of --show option to control what is shown.

[david]

Here is a set of 3 patches to implement (some) output control.  

[PATCH 1/3] rename option to select output format to --format from --output.
[PATCH 2/3] notmuch-show.c: make calls to format functions conditional

The first two are essentially suggestions for Scott.  In particular the first 
is not needed at all for this little experiment.

[PATCH 3/3] notmuch-show.c: prototype tabular output format, with output control

The third implements a new "tabular" output format, and shows how it
might be conditionally controlled.  At the moment the only
possibilities to output are message-id, match flag, and filename.
This is mainly a matter of laziness.  I didn't want to go too crazy
before Scott's reached some final(ish) version.

[PATCH 1/3] rename option to select output format to --format from --output.

[david]

From: David Bremner <bremner@unb.ca>

---
 notmuch-search.c |    6 +++---
 notmuch-show.c   |    6 +++---
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/notmuch-search.c b/notmuch-search.c
index e243747..482c6e8 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -221,14 +221,14 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
 		fprintf (stderr, "Invalid value for --sort: %s\n", opt);
 		return 1;
 	    }
-	} else if (STRNCMP_LITERAL (argv[i], "--output=") == 0) {
-	    opt = argv[i] + sizeof ("--output=") - 1;
+	} else if (STRNCMP_LITERAL (argv[i], "--format=") == 0) {
+	    opt = argv[i] + sizeof ("--format=") - 1;
 	    if (strcmp (opt, "text") == 0) {
 		format = &format_text;
 	    } else if (strcmp (opt, "json") == 0) {
 		format = &format_json;
 	    } else {
-		fprintf (stderr, "Invalid value for --output: %s\n", opt);
+		fprintf (stderr, "Invalid value for --format: %s\n", opt);
 		return 1;
 	    }
 	} else {
diff --git a/notmuch-show.c b/notmuch-show.c
index b5b3eba..b6e3f44 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -406,15 +406,15 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
 	    i++;
 	    break;
 	}
-	if (STRNCMP_LITERAL (argv[i], "--output=") == 0) {
-	    opt = argv[i] + sizeof ("--output=") - 1;
+	if (STRNCMP_LITERAL (argv[i], "--format=") == 0) {
+	    opt = argv[i] + sizeof ("--format=") - 1;
 	    if (strcmp (opt, "text") == 0) {
 		format = &format_text;
 	    } else if (strcmp (opt, "json") == 0) {
 		format = &format_json;
 		entire_thread = 1;
 	    } else {
-		fprintf (stderr, "Invalid value for --output: %s\n", opt);
+		fprintf (stderr, "Invalid value for --format: %s\n", opt);
 		return 1;
 	    }
 	} else if (STRNCMP_LITERAL (argv[i], "--entire-thread") == 0) {
-- 
1.6.5.3

[PATCH 2/3] notmuch-show.c: make calls to format functions conditional

[david]

From: David Bremner <bremner@unb.ca>

This makes it easier to define minimal formats without defining several
dummy functions that do nothing.
---
 notmuch-show.c |   13 +++++++++----
 1 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/notmuch-show.c b/notmuch-show.c
index b6e3f44..51aa87d 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -326,14 +326,17 @@ static void
 show_message (void *ctx, const show_format_t *format, notmuch_message_t *message, int indent)
 {
     fputs (format->message_start, stdout);
-    format->message(ctx, message, indent);
+    if (format->message)
+	format->message(ctx, message, indent);
 
     fputs (format->header_start, stdout);
-    format->header(ctx, message);
+    if (format->header) 
+	format->header(ctx, message);
     fputs (format->header_end, stdout);
 
     fputs (format->body_start, stdout);
-    show_message_body (notmuch_message_get_filename (message), format->part);
+    if (format->part) 
+	show_message_body (notmuch_message_get_filename (message), format->part);
     fputs (format->body_end, stdout);
 
     fputs (format->message_end, stdout);
@@ -408,7 +411,9 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
 	}
 	if (STRNCMP_LITERAL (argv[i], "--format=") == 0) {
 	    opt = argv[i] + sizeof ("--format=") - 1;
-	    if (strcmp (opt, "text") == 0) {
+	    if (strcmp (opt, "tabular") == 0) {
+		format = &format_tabular;
+	    } else if (strcmp (opt, "text") == 0) {
 		format = &format_text;
 	    } else if (strcmp (opt, "json") == 0) {
 		format = &format_json;
-- 
1.6.5.3

[PATCH 3/3] notmuch-show.c: prototype tabular output format, with output control

[david]

From: David Bremner <bremner@unb.ca>

Currently this only outputs the information from the "message header";
i.e. the part before the rfc2822 header or body.

Adding this required adding an extra parameter, currently unused, to
format_message_text and format_message_json. Also the struct
definition is changed to match the new function prototypes.

An int (enum) is used as a mask, each bit corresponds to some part to
be shown or not shown.  This enum will need to be extended as more
things are controllable via --show.
---
 notmuch-show.c |   89 ++++++++++++++++++++++++++++++++++++++++++++++++-------
 1 files changed, 77 insertions(+), 12 deletions(-)

diff --git a/notmuch-show.c b/notmuch-show.c
index 51aa87d..99a4d3c 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -20,10 +20,18 @@
 
 #include "notmuch-client.h"
 
+/* These should be powers of 2 */
+typedef enum {SHOW_MESSAGE_ID = 1, 
+	      SHOW_MATCH = 2, 
+	      SHOW_FILENAME = 4
+} show_mask_t;
+
+static const show_mask_t SHOW_ALL = ~0;
+
 typedef struct show_format {
     const char *message_set_start;
     const char *message_start;
-    void (*message) (const void *ctx,
+    void (*message) (const void *ctx, show_mask_t show_mask,
 		     notmuch_message_t *message,
 		     int indent);
     const char *header_start;
@@ -40,7 +48,7 @@ typedef struct show_format {
 } show_format_t;
 
 static void
-format_message_text (unused (const void *ctx),
+format_message_text (unused (const void *ctx), show_mask_t show_mask,
 		     notmuch_message_t *message,
 		     int indent);
 static void
@@ -59,7 +67,21 @@ static const show_format_t format_text = {
 };
 
 static void
-format_message_json (const void *ctx,
+format_message_tabular (unused (const void *ctx), show_mask_t show_mask,
+		     notmuch_message_t *message,
+		     int indent);
+
+static const show_format_t format_tabular = {
+    "",	 
+    "", format_message_tabular, 
+    "", NULL, "", 
+    "", NULL, "",    
+    "\n", "", 
+    ""
+};
+
+static void
+format_message_json (const void *ctx, show_mask_t show_mask,
 		     notmuch_message_t *message,
 		     unused (int indent));
 static void
@@ -124,7 +146,30 @@ _get_one_line_summary (const void *ctx, notmuch_message_t *message)
 }
 
 static void
-format_message_text (unused (const void *ctx), notmuch_message_t *message, int indent)
+format_message_tabular (unused (const void *ctx), show_mask_t show_mask,
+			notmuch_message_t *message, 
+			unused(int indent))
+{
+    int count=0;
+
+    if (show_mask & SHOW_MESSAGE_ID) {
+	fputs ( notmuch_message_get_message_id (message), stdout);
+	count++;
+    }
+
+    if (show_mask & SHOW_MATCH) 
+	printf ("%s%d",	count++ ? "\t" : "", 
+		notmuch_message_get_flag (message, 
+					  NOTMUCH_MESSAGE_FLAG_MATCH));
+
+    if (show_mask & SHOW_FILENAME) 
+	printf("%s%s", count++ ? "\t" : "", 
+	       notmuch_message_get_filename (message));
+}
+
+static void
+format_message_text (unused (const void *ctx), unused(show_mask_t show_mask),
+		     notmuch_message_t *message, int indent)
 {
     printf ("id:%s depth:%d match:%d filename:%s\n",
 	    notmuch_message_get_message_id (message),
@@ -134,7 +179,8 @@ format_message_text (unused (const void *ctx), notmuch_message_t *message, int i
 }
 
 static void
-format_message_json (const void *ctx, notmuch_message_t *message, unused (int indent))
+format_message_json (const void *ctx, unused(show_mask_t show_mask),
+		     notmuch_message_t *message, unused (int indent))
 {
     void *ctx_quote = talloc_new (ctx);
 
@@ -323,11 +369,12 @@ format_part_json (GMimeObject *part, int *part_count)
 }
 
 static void
-show_message (void *ctx, const show_format_t *format, notmuch_message_t *message, int indent)
+show_message (void *ctx, const show_format_t *format, show_mask_t show_mask,
+	      notmuch_message_t *message, int indent)
 {
     fputs (format->message_start, stdout);
     if (format->message)
-	format->message(ctx, message, indent);
+	format->message(ctx, show_mask, message, indent);
 
     fputs (format->header_start, stdout);
     if (format->header) 
@@ -344,8 +391,8 @@ show_message (void *ctx, const show_format_t *format, notmuch_message_t *message
 
 
 static void
-show_messages (void *ctx, const show_format_t *format, notmuch_messages_t *messages, int indent,
-	       notmuch_bool_t entire_thread)
+show_messages (void *ctx, const show_format_t *format,  show_mask_t show_mask, 
+	       notmuch_messages_t *messages, int indent, notmuch_bool_t entire_thread)
 {
     notmuch_message_t *message;
     notmuch_bool_t match;
@@ -371,13 +418,14 @@ show_messages (void *ctx, const show_format_t *format, notmuch_messages_t *messa
 	next_indent = indent;
 
 	if (match || entire_thread) {
-	    show_message (ctx, format, message, indent);
+	    show_message (ctx, format, show_mask, message, indent);
 	    next_indent = indent + 1;
 
 	    fputs (format->message_set_sep, stdout);
 	}
 
-	show_messages (ctx, format, notmuch_message_get_replies (message),
+	show_messages (ctx, format, show_mask, 
+		       notmuch_message_get_replies (message),
 		       next_indent, entire_thread);
 
 	notmuch_message_destroy (message);
@@ -403,6 +451,7 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
     int entire_thread = 0;
     int i;
     int first_toplevel = 1;
+    show_mask_t show_mask=SHOW_ALL;
 
     for (i = 0; i < argc && argv[i][0] == '-'; i++) {
 	if (strcmp (argv[i], "--") == 0) {
@@ -422,6 +471,22 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
 		fprintf (stderr, "Invalid value for --format: %s\n", opt);
 		return 1;
 	    }
+	} else if (STRNCMP_LITERAL (argv[i], "--show=") == 0) {
+	    char *keyword=NULL;
+	    show_mask=0;
+	    opt = argv[i] + sizeof ("--show=") - 1;
+	    while ((keyword=strsep (&opt,", "))) {
+		if (strcmp (keyword, "message-id") == 0) {
+		    show_mask |= SHOW_MESSAGE_ID;
+		} else if (strcmp (keyword, "match") == 0) {
+		    show_mask |= SHOW_MATCH;
+		} else if (strcmp (keyword, "filename") == 0) {
+		    show_mask |= SHOW_FILENAME;
+		} else {
+		    fprintf (stderr, "Invalid value for --show: %s\n", opt);
+		    return 1;
+		}
+	    }
 	} else if (STRNCMP_LITERAL (argv[i], "--entire-thread") == 0) {
 	    entire_thread = 1;
 	} else {
@@ -477,7 +542,7 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
 	    fputs (format->message_set_sep, stdout);
 	first_toplevel = 0;
 
-	show_messages (ctx, format, messages, 0, entire_thread);
+	show_messages (ctx, format, show_mask, messages, 0, entire_thread);
 
 	notmuch_thread_destroy (thread);
 
-- 
1.6.5.3

Re: [PATCH 3/3] notmuch-show.c: prototype tabular output format, with output control

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 507 bytes --]

On Sat, 19 Dec 2009 10:55:24 -0400, david@tethera.net wrote:
> From: David Bremner <bremner@unb.ca>
> 
> Currently this only outputs the information from the "message header";
> i.e. the part before the rfc2822 header or body.

Hi David, 

Are you proposing the tabular format for inclusion in notmuch?

Or does "prototype" suggest that you were just showing a demonstration
of what could be done here?

If you do want this included, what's the motivation for this format?

Thanks,

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH 3/3] notmuch-show.c: prototype tabular output format, with output control

[David Bremner]

On Tue, 09 Mar 2010 11:51:46 -0800, Carl Worth <cworth@cworth.org> wrote:

> Are you proposing the tabular format for inclusion in notmuch?
> 
> Or does "prototype" suggest that you were just showing a demonstration
> of what could be done here?
> 
> If you do want this included, what's the motivation for this format?

I think some tabular format would be handy because it is very easy to
parse with scripts, and because it is a natural natural way to get
e.g. a list of message-ids.

--format=tabular --output=message-id

On the other hand, I'm not yet using this code, so I guess the main
point is to discuss how output control should be implemented, and in
that sense the patch is a demo. I suspect once we start using json
output more, we will definitely want it there.

All the best, 

David

Re: [PATCH 3/3] notmuch-show.c: prototype tabular output format, with output control

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 859 bytes --]

On Tue, 09 Mar 2010 16:19:16 -0400, David Bremner <david@tethera.net> wrote:
> I think some tabular format would be handy because it is very easy to
> parse with scripts, and because it is a natural natural way to get
> e.g. a list of message-ids.
> 
> --format=tabular --output=message-id

Ah, so maybe your "notmuch show --format=tabular" is basically the same
thing I imagined for "notmuch search" (once we teach it to accept a list
of the desired elements to output).

> On the other hand, I'm not yet using this code, so I guess the main
> point is to discuss how output control should be implemented, and in
> that sense the patch is a demo. I suspect once we start using json
> output more, we will definitely want it there.

Yes. I think we next need to implement some --output options and then
see what we want from there.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH 3/3] notmuch-show.c: prototype tabular output format, with output control

[David Bremner]

On Wed, 10 Mar 2010 01:25:22 -0800, Carl Worth <cworth@cworth.org> wrote:
> On Tue, 09 Mar 2010 16:19:16 -0400, David Bremner <david@tethera.net> wrote:
> > 
> > --format=tabular --output=message-id
> 
> Ah, so maybe your "notmuch show --format=tabular" is basically the same
> thing I imagined for "notmuch search" (once we teach it to accept a list
> of the desired elements to output).

There is a third "control axis", namely one piece of output per thread
(current "notmuch search" style), or all of the messages ("notmuch show
style")

[PATCH] notmuch-query.el: new file to support access to the notmuch database.

[david]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 4853 bytes --]

From: David Bremner <bremner@unb.ca>

Initially this file provides one main function
notmuch-query-get-threads, which takes a set of search terms, and
returns a parsed set of matching threads as a lisp data structure.

A set of notmuch-query-map-* functions are provided to help map
functions over the data structure.

The function notmuch-query-get-message-ids uses this machinery to get
the set of message-ids matching a query.

This patch relies on the json patch of
<1261141195-5469-1-git-send-email-david@tethera.net> and the argument
parsing patch 1261234524-25522-2-git-send-email-david@tethera.net,
although the latter would be trivial to avoid, by replace --format
with --output in the file.
---

This patch is the start of a new emacs api built on top of the json
output.  It roughly mimics the notion of returning lists of messages
and traversing them as used in the C code.  If nothing else, the
docstring for notmuch-query-get-threads provides some documentation
explaining the structure of the json output by Scott's patch.

I'd be interested if people who have extremely large threads find it
annoyingly slow.  The plan (subject to change as always) is to allow
the api functions (currently only notmuch-query-get-message-ids) to
pass options to the invokation of "notmuch show" to trim down the json
output (i.e. omit message bodies). The parsing and traversal code
should all unaffected by this change. In theory.

I didn't know if you (Carl) want to do copyright assignment, or just
the GPL license headers are enough. I don't mind pretty much any
variation.

 notmuch-query.el |   89 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 89 insertions(+), 0 deletions(-)
 create mode 100644 notmuch-query.el

diff --git a/notmuch-query.el b/notmuch-query.el
new file mode 100644
index 0000000..5911fa6
--- /dev/null
+++ b/notmuch-query.el
@@ -0,0 +1,89 @@
+; notmuch.el --- run notmuch within emacs
+;
+; Copyright © David Bremner
+;
+; This file is part of Notmuch.
+;
+; Notmuch is free software: you can redistribute it and/or modify it
+; under the terms of the GNU General Public License as published by
+; the Free Software Foundation, either version 3 of the License, or
+; (at your option) any later version.
+;
+; Notmuch is distributed in the hope that it will be useful, but
+; WITHOUT ANY WARRANTY; without even the implied warranty of
+; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+; General Public License for more details.
+;
+; You should have received a copy of the GNU General Public License
+; along with Notmuch.  If not, see <http://www.gnu.org/licenses/>.
+;
+; Authors: David Bremner <david@tethera.net>
+
+(require 'json)
+
+(defun notmuch-query-get-threads (search-terms &rest options)
+  "Return a list of threads of messages matching SEARCH-TERMS. 
+
+A thread is a forest or list of trees. A tree is a two element
+list where the first element is a message, and the second element
+is a possibly empty forest of replies.
+"
+  (let  ((args (append '("show" "--format=json") search-terms))
+	 (json-object-type 'plist)
+	 (json-array-type 'list)
+	 (json-false 'nil))
+    (with-temp-buffer 
+      (progn 
+	(apply 'call-process (append (list notmuch-command nil t nil) args))
+	(goto-char (point-min))
+	(json-read)))))
+
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;; Mapping functions across collections of messages.
+
+(defun notmuch-query-map-aux  (mapper function seq)
+  "private function to do the actual mapping and flattening"
+
+  (apply 'append 
+	 (mapcar 
+	   (lambda (tree) 
+	     (funcall mapper fn tree))
+	   seq)))
+
+
+(defun notmuch-query-map-threads (fn threads)
+  "apply FN to every thread in  THREADS. Flatten results to a list.
+
+See the function notmuch-query-get-threads for more information."
+
+  (notmuch-query-map-aux 'notmuch-query-map-forest fn threads))
+
+
+(defun notmuch-query-map-forest (fn forest)
+  "apply function to every message in a forest. Flatten results to a list.
+
+See the function notmuch-query-get-threads for more information.
+"
+
+  (notmuch-query-map-aux 'notmuch-query-map-tree fn forest))
+
+
+(defun notmuch-query-map-tree (fn tree)
+  "Apply function FN to every message in TREE. Flatten results to a list
+
+See the function notmuch-query-get-threads for more information."
+
+  (cons (funcall fn (car tree)) (notmuch-query-map-forest fn (cadr tree))))
+
+
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;; Predefined queries
+
+(defun notmuch-query-get-message-ids (search-terms)
+  "Return a list of message-ids of messages that match SEARCH-TERMS"
+
+  (notmuch-query-map-threads 
+   (lambda (msg) (plist-get msg :id))
+   (notmuch-query-get-threads search-terms)))
+    
+(provide 'notmuch-query)
\ No newline at end of file
-- 
1.6.5.7

Re: [PATCH] notmuch-query.el: new file to support access to the notmuch database.

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 788 bytes --]

On Sun, 20 Dec 2009 16:31:21 -0400, david@tethera.net wrote:
> I didn't know if you (Carl) want to do copyright assignment, or just
> the GPL license headers are enough. I don't mind pretty much any
> variation.

Please feel free to retain your own copyright. I certainly don't deserve
anything being assigned to me.

I like that you're doing a new file, separate from our current
notmuch.el. That file has already become extremely large and unwieldy
and needs to be broken up.

But to do that, I really want to get the emacs lisp files out of the
top-level directory, (and down into contrib/emacs, say). And it might be
nice to get some of the pending patches merged in before doing a big
restructuring here.

Hopefully I'll start making some real progress on the backlog soon...

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH] notmuch-query.el: new file to support access to the notmuch database.

[David Bremner]

On Mon, 21 Dec 2009 09:21:40 -0800, Carl Worth <cworth@cworth.org> wrote:
> I like that you're doing a new file, separate from our current
> notmuch.el. That file has already become extremely large and unwieldy
> and needs to be broken up.
> 
> But to do that, I really want to get the emacs lisp files out of the
> top-level directory, (and down into contrib/emacs, say). And it might be
> nice to get some of the pending patches merged in before doing a big
> restructuring here.

OK, I'll no doubt have at least one more version of this patch before it
get merged in.  I already have a few minor changes, but I'm using it in 
"production" with the org-mode link stuff, so we'll see if that shakes
out any more issues.  I'll post an updated version if/when I see the
json output patches merged to master.  

David

[PATCH v2] notmuch-query.el: new file to support access to the notmuch database.

[david]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 4388 bytes --]

From: David Bremner <bremner@unb.ca>

Initially this file provides one main function
notmuch-query-get-threads, which takes a set of search terms, and
returns a parsed set of matching threads as a lisp data structure.

A set of notmuch-query-map-* functions are provided to help map
functions over the data structure.

The function notmuch-query-get-message-ids uses this machinery to get
the set of message-ids matching a query.

This patch relies on the emacs directory  patch of
<1265773528-30794-1-git-send-email-david@tethera.net>
---

 This version is rebased against the json patches that eventually made
 it into master. Oddly, no actual changes were required based on
 Carl's change of "id" to "thread".   Some trailing whitespace was
 also removed.

 emacs/Makefile.local   |    3 +-
 emacs/notmuch-query.el |   89 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 91 insertions(+), 1 deletions(-)
 create mode 100644 emacs/notmuch-query.el

diff --git a/emacs/Makefile.local b/emacs/Makefile.local
index c6ca142..6c87a60 100644
--- a/emacs/Makefile.local
+++ b/emacs/Makefile.local
@@ -1,6 +1,7 @@
 dir=emacs
 emacs_sources=                 \
-	$(dir)/notmuch.el
+	$(dir)/notmuch.el      \
+	$(dir)/notmuch-query.el
 
 emacs_bytecode=$(subst .el,.elc,$(emacs_sources))
 
diff --git a/emacs/notmuch-query.el b/emacs/notmuch-query.el
new file mode 100644
index 0000000..86230ba
--- /dev/null
+++ b/emacs/notmuch-query.el
@@ -0,0 +1,89 @@
+; notmuch.el --- run notmuch within emacs
+;
+; Copyright © David Bremner
+;
+; This file is part of Notmuch.
+;
+; Notmuch is free software: you can redistribute it and/or modify it
+; under the terms of the GNU General Public License as published by
+; the Free Software Foundation, either version 3 of the License, or
+; (at your option) any later version.
+;
+; Notmuch is distributed in the hope that it will be useful, but
+; WITHOUT ANY WARRANTY; without even the implied warranty of
+; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+; General Public License for more details.
+;
+; You should have received a copy of the GNU General Public License
+; along with Notmuch.  If not, see <http://www.gnu.org/licenses/>.
+;
+; Authors: David Bremner <david@tethera.net>
+
+(require 'json)
+
+(defun notmuch-query-get-threads (search-terms &rest options)
+  "Return a list of threads of messages matching SEARCH-TERMS.
+
+A thread is a forest or list of trees. A tree is a two element
+list where the first element is a message, and the second element
+is a possibly empty forest of replies.
+"
+  (let  ((args (append '("show" "--format=json") search-terms))
+	 (json-object-type 'plist)
+	 (json-array-type 'list)
+	 (json-false 'nil))
+    (with-temp-buffer
+      (progn
+	(apply 'call-process (append (list notmuch-command nil t nil) args))
+	(goto-char (point-min))
+	(json-read)))))
+
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;; Mapping functions across collections of messages.
+
+(defun notmuch-query-map-aux  (mapper function seq)
+  "private function to do the actual mapping and flattening"
+
+  (apply 'append
+	 (mapcar
+	   (lambda (tree)
+	     (funcall mapper fn tree))
+	   seq)))
+
+
+(defun notmuch-query-map-threads (fn threads)
+  "apply FN to every thread in  THREADS. Flatten results to a list.
+
+See the function notmuch-query-get-threads for more information."
+
+  (notmuch-query-map-aux 'notmuch-query-map-forest fn threads))
+
+
+(defun notmuch-query-map-forest (fn forest)
+  "apply function to every message in a forest. Flatten results to a list.
+
+See the function notmuch-query-get-threads for more information.
+"
+
+  (notmuch-query-map-aux 'notmuch-query-map-tree fn forest))
+
+
+(defun notmuch-query-map-tree (fn tree)
+  "Apply function FN to every message in TREE. Flatten results to a list
+
+See the function notmuch-query-get-threads for more information."
+
+  (cons (funcall fn (car tree)) (notmuch-query-map-forest fn (cadr tree))))
+
+
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;; Predefined queries
+
+(defun notmuch-query-get-message-ids (&rest search-terms)
+  "Return a list of message-ids of messages that match SEARCH-TERMS"
+
+  (notmuch-query-map-threads
+   (lambda (msg) (plist-get msg :id))
+   (notmuch-query-get-threads search-terms)))
+
+(provide 'notmuch-query)
-- 
1.6.5

[PATCH v3] notmuch-query.el: new file to support access to the notmuch database.

[david]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 4328 bytes --]

From: David Bremner <bremner@unb.ca>

Initially this file provides one main function
notmuch-query-get-threads, which takes a set of search terms, and
returns a parsed set of matching threads as a lisp data structure.

A set of notmuch-query-map-* functions are provided to help map
functions over the data structure.

The function notmuch-query-get-message-ids uses this machinery to get
the set of message-ids matching a query.
---

Commentary on this revision of the patch.
	- Rebased to current master.
	- Last time I checked, used by dme's json rewrite of the emacs ui
        - Warnings eliminated by including the new notmuch-lib.el

 emacs/Makefile.local   |    1 +
 emacs/notmuch-query.el |   90 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 91 insertions(+), 0 deletions(-)
 create mode 100644 emacs/notmuch-query.el

diff --git a/emacs/Makefile.local b/emacs/Makefile.local
index 95b6fa4..626a4e5 100644
--- a/emacs/Makefile.local
+++ b/emacs/Makefile.local
@@ -4,6 +4,7 @@ dir := emacs
 emacs_sources := \
 	$(dir)/notmuch-lib.el \
 	$(dir)/notmuch.el \
+	$(dir)/notmuch-query.el \
 	$(dir)/notmuch-show.el
 
 emacs_bytecode := $(subst .el,.elc,$(emacs_sources))
diff --git a/emacs/notmuch-query.el b/emacs/notmuch-query.el
new file mode 100644
index 0000000..3f1a7b3
--- /dev/null
+++ b/emacs/notmuch-query.el
@@ -0,0 +1,90 @@
+; notmuch-query.el --- provide an emacs api to query notmuch
+;
+; Copyright © David Bremner
+;
+; This file is part of Notmuch.
+;
+; Notmuch is free software: you can redistribute it and/or modify it
+; under the terms of the GNU General Public License as published by
+; the Free Software Foundation, either version 3 of the License, or
+; (at your option) any later version.
+;
+; Notmuch is distributed in the hope that it will be useful, but
+; WITHOUT ANY WARRANTY; without even the implied warranty of
+; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+; General Public License for more details.
+;
+; You should have received a copy of the GNU General Public License
+; along with Notmuch.  If not, see <http://www.gnu.org/licenses/>.
+;
+; Authors: David Bremner <david@tethera.net>
+
+(require 'notmuch-lib)
+(require 'json)
+
+(defun notmuch-query-get-threads (search-terms &rest options)
+  "Return a list of threads of messages matching SEARCH-TERMS.
+
+A thread is a forest or list of trees. A tree is a two element
+list where the first element is a message, and the second element
+is a possibly empty forest of replies.
+"
+  (let  ((args (append '("show" "--format=json") search-terms))
+	 (json-object-type 'plist)
+	 (json-array-type 'list)
+	 (json-false 'nil))
+    (with-temp-buffer
+      (progn
+	(apply 'call-process (append (list notmuch-command nil t nil) args))
+	(goto-char (point-min))
+	(json-read)))))
+
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;; Mapping functions across collections of messages.
+
+(defun notmuch-query-map-aux  (mapper function seq)
+  "private function to do the actual mapping and flattening"
+
+  (apply 'append
+	 (mapcar
+	   (lambda (tree)
+	     (funcall mapper fn tree))
+	   seq)))
+
+
+(defun notmuch-query-map-threads (fn threads)
+  "apply FN to every thread in  THREADS. Flatten results to a list.
+
+See the function notmuch-query-get-threads for more information."
+
+  (notmuch-query-map-aux 'notmuch-query-map-forest fn threads))
+
+
+(defun notmuch-query-map-forest (fn forest)
+  "apply function to every message in a forest. Flatten results to a list.
+
+See the function notmuch-query-get-threads for more information.
+"
+
+  (notmuch-query-map-aux 'notmuch-query-map-tree fn forest))
+
+
+(defun notmuch-query-map-tree (fn tree)
+  "Apply function FN to every message in TREE. Flatten results to a list
+
+See the function notmuch-query-get-threads for more information."
+
+  (cons (funcall fn (car tree)) (notmuch-query-map-forest fn (cadr tree))))
+
+
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;; Predefined queries
+
+(defun notmuch-query-get-message-ids (&rest search-terms)
+  "Return a list of message-ids of messages that match SEARCH-TERMS"
+
+  (notmuch-query-map-threads
+   (lambda (msg) (plist-get msg :id))
+   (notmuch-query-get-threads search-terms)))
+
+(provide 'notmuch-query)
-- 
1.7.0

Re: [PATCH v3] notmuch-query.el: new file to support access to the notmuch database.

[David Edmondson]

On Mon,  5 Apr 2010 13:46:16 -0300, david@tethera.net wrote:
> diff --git a/emacs/notmuch-query.el b/emacs/notmuch-query.el
> new file mode 100644
> index 0000000..3f1a7b3
> --- /dev/null
> +++ b/emacs/notmuch-query.el
> @@ -0,0 +1,90 @@
> +; notmuch-query.el --- provide an emacs api to query notmuch

Should this comment prefix be ';;'? A single ';' is indented way over to
the right by default.

> +(defun notmuch-query-map-aux  (mapper function seq)
> +  "private function to do the actual mapping and flattening"
> +
> +  (apply 'append
> +	 (mapcar
> +	   (lambda (tree)
> +	     (funcall mapper fn tree))
> +	   seq)))
> +
> +

There are a bunch of spurious blank lines in various places.

dme.
-- 
David Edmondson, http://dme.org

Re: [PATCH v3] notmuch-query.el: new file to support access to the notmuch database.

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 622 bytes --]

On Mon,  5 Apr 2010 13:46:16 -0300, david@tethera.net wrote:
> Commentary on this revision of the patch.
> 	- Rebased to current master.
> 	- Last time I checked, used by dme's json rewrite of the emacs ui
>         - Warnings eliminated by including the new notmuch-lib.el

Very nice stuff indeed. I incorporated the two minor style issues that
David E. mentioned in his reply. I do think that
"notmuch-query-get-threads" would be better named something much closer
to "notmuch-search". But we can wait to clear up the naming until we get
everything switched to this and the old calls to "notmuch search"
removed.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH] Add an "--output=(json|text|)" command-line option to both notmuch-search and notmuch-show.

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 3222 bytes --]

On Fri, 18 Dec 2009 08:59:55 -0400, david@tethera.net wrote:
> From: Scott Robinson <scott@quadhome.com>

Hi Scott (and David),

I'm finally getting around to looking closely at the JSON patches
(hurrah!). Here are some comments:

> In the case of notmuch-show, "--output=json" also implies
> "--entire-thread" as the thread structure is implicit in the emitted
> document tree.

I think we've all agreed to use --output for selecting what to print and
to instead use --format for how to format it. I also think David's got a
patch to change that, but if we have to change the current patch anyway,
we might change that at the same time.

> --- /dev/null
> +++ b/json.c
> @@ -0,0 +1,73 @@
> +/* notmuch - Not much of an email program, (just index and search)
> + *
> + * Copyright © 2009 Carl Worth
> + * Copyright © 2009 Keith Packard

I know I didn't contribute any code to this file, so my name shouldn't
be here. Scott, I imagine you have made some non-trivial contributions
so your name (or suitable copyright-claiming entity) should be here.

> + * Authors: Carl Worth <cworth@cworth.org>
> + *	    Keith Packard <keithp@keithp.com>

Same thing here.

> +/*
> + * json_quote_str derived from cJSON's print_string_ptr,
> + * Copyright (c) 2009 Dave Gamble
> + */

Thanks for attributing the source here, but let's please keep all the
copyright statements up at the top of the file.

It would still be reasonable to have a comment at this point that this
particular code came from Dave Gamble and cJSON. But it should also
mention the license under which the code is being integrated.

I suggest leaving the notmuch-standar GPLv3+ blurb at the top of the
file, but then quoting the license itself of the external code, (it's a
short MIT-style license, right?).

> +char *
> +json_quote_str(const void *ctx, const char *str)

It would be nice to have a little comment here describing what the
function does, (what characters get quoted and how, that the return
value is talloced with 'ctx' as a parent, and perhaps a pointer to the
appropriate JSON reference/specification).

But I definitely like this nice little function as opposed to some JSON
library. Thanks!


> +typedef struct search_format {
> +    const char *results_start;
> +    const char *thread_start;
> +    void (*thread) (const void *ctx,
> +		    const char *id,
> +		    const time_t date,
> +		    const int matched,
> +		    const int total,
> +		    const char *authors,
> +		    const char *subject);
...

Definitely missing at least a quick comment for internal documentation
here as well. But I do like the way this works.

> --- a/notmuch.c
> +++ b/notmuch.c
> @@ -162,6 +162,11 @@ command_t commands[] = {
>        "\n"
>        "\t\tSupported options for search include:\n"
>        "\n"
> +      "\t\t--output=(json|text)\n"
> +      "\n"
> +      "\t\t\tPresents the results in either JSON or plain-text\n"
> +      "\t\t\tformat, which is the default.\n"
> +      "\n"

Thanks for adding the documentation here. But please add to notmuch.1 as
well.

This all looks really great. And I can't wait to apply it and play with
it more.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

JSON output as default [was: Re: [PATCH] Add an "--output=(json|text|)" command-line option...]

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 1873 bytes --]

Hey, Carl.  I mentioned this is an email a while back
(id:87eil4ygar.fsf@servo.finestructure.net), but I want to bring it up
again now that you're looking at the JSON output stuff.  I would like to
make the case for JSON being the one and only output format.  Here's the
arguments:

* JSON is a very well spec'd API
* JSON is a *very* well supported standard (it's basically well
  supported by every language)
* the JSONformat is perfectly suited for notmuch's output
* emacs has a json.el library, so it would not be hard at all to modify
  the notmuch.el to parse json output
* using only one output format would considerably reduce the code base
  in notmuch, keeping things simpler and easier to maintain

I think these are pretty good reason to just move to JSON as *the*
output format.

While I'm on this topic, let me rehash another point I made previously:
If we move to JSON output, it would be *so* sweet to provide means to
filter the output JSON fields.  The "--output" flag could be used to
pick the desired output fields.  For instance, the current "search"
output could be something like:

search --output=thread_id,date,number,author,subject,tags

while the current "show" output could be something like:

search --output=message_id,tags,path,header,body,attachments

This would all make things *much* easier for clients and wrappers.  For
instance, here are just a couple of things that would be made much
easier for wrapper scripts:

* Proper maildir sync ("search --output=message_id,tags,path" ...)

* Purging "delete" tagged messages ("search --output=path" tags:delete)

* Moving/archiving messages based on search results ("search --output=path" ...)

I would personally love to see this.  I think it would make notmuch
cleaner, simpler, and more powerful all at the same time.

Interested in hearing what other's thoughts on this idea are.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: JSON output as default [was: Re: [PATCH] Add an "--output=(json|text|)" command-line option...]

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1908 bytes --]

On Tue, 23 Feb 2010 16:00:01 -0500, Jameson Rollins <jrollins@finestructure.net> wrote:
> Hey, Carl.  I mentioned this is an email a while back
> (id:87eil4ygar.fsf@servo.finestructure.net), but I want to bring it up
> again now that you're looking at the JSON output stuff.  I would like to
> make the case for JSON being the one and only output format.  Here's the
> arguments:

Oh, I'm definitely in favor of killing the %message{ format.

As soon as we get the emacs and vi frontends switched to the json
output, then that old thing can be eliminated.

> * using only one output format would considerably reduce the code base
>   in notmuch, keeping things simpler and easier to maintain

But I actually still want text-only output. It will be even less
"parseable" than the current output, (no delimiters at all), but also
have some niceties like thread indentation, etc.

The idea there is to support direct, human-readable output for quick
command-line use. (It also means that notmuch provides a much gentler
introduction to people just experimenting with it). And I'm willing to
keep the code base just a "little" bit complicated for that.

> If we move to JSON output, it would be *so* sweet to provide means to
> filter the output JSON fields.  The "--output" flag could be used to
> pick the desired output fields.  For instance, the current "search"
> output could be something like:

I definitely want to enable the user to select which fields are printed.

But I don't see how that depends on JSON output. In fact, for easy shell
scripts, etc. I want to be able to do search that give me filenames,
email addresses, etc. without any delimiters at all.

I definitely want to be able to pipe single-field lists coming from
notmuch to grep, xargs, shell for loops, etc. without having to decode
JSON.

So I'm still in favor of having a "plain text" output going forward,
(and by default).

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: JSON output as default [was: Re: [PATCH] Add an "--output=(json|text|)" command-line option...]

[Sebastian Spaeth]

> I definitely want to be able to pipe single-field lists coming from
> notmuch to grep, xargs, shell for loops, etc. without having to decode
> JSON.

While I would love to see JSON (even by default), I agree. If I just
want to code up a notmuch-based address book with sth like:

notmuch show to:Diana --output=to --sort=relevance --limit=20

just getting back a plain list of mail addresses is the easiest to handle.

Re: [PATCH] JSON output for notmuch-search and notmuch-show.

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1027 bytes --]

On Thu, 17 Dec 2009 21:33:54 -0800, Scott Robinson <scott@quadhome.com> wrote:
> I took an earlier suggestion and didn't use cJSON, instead writing custom code
> for emitting the new format.

Nice! I have a few comments below.

> Added an "--output=(json|text|)" command-line option to both
> notmuch-search and notmuch-show.

I don't know why, but I think I'd prefer --format for the name here.

> In the case of notmuch-show, "--output=json" also implies
> "--entire-thread" as the thread structure is implicit in the emitted
> document tree.

It looks like the new documentation is missing that point, (and the man
page in notmuch.1 is missing an update as well).

> As a coincidence to the implementation, multipart message ID numbers are
> now incremented with each part printed. This changes the previous
> semantics, which were unclear and not necessary related to the actual
> ordering of the message parts.

That's just fine. The old numbering semantics were quite bizarre and
nothing I wanted to set it stone.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH] JSON output for notmuch-search and notmuch-show.

[Scott Robinson]

Excerpts from Carl Worth's message of Fri Dec 18 09:31:39 -0800 2009:
> [...]
> I don't know why, but I think I'd prefer --format for the name here.

ACK

> [...]
> It looks like the new documentation is missing that point, (and the man
> page in notmuch.1 is missing an update as well).

ACK

> [...]
> That's just fine. The old numbering semantics were quite bizarre and
> nothing I wanted to set it stone.

Cool. :-)

Resubmit a full patch, or submit another one on top of it?
-- 
Scott Robinson | http://quadhome.com/

Q: Why are my replies five sentences or less?
A: http://five.sentenc.es/

Re: [PATCH] JSON output for notmuch-search and notmuch-show.

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 189 bytes --]

On Fri, 18 Dec 2009 10:47:33 -0800, Scott Robinson <scott@quadhome.com> wrote:
> Resubmit a full patch, or submit another one on top of it?

A new full patch would be great, thanks!

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH] JSON output for notmuch-search and notmuch-show.

[Scott Robinson]

Excerpts from Carl Worth's message of Tue Dec 22 21:48:34 -0800 2009:
> On Fri, 18 Dec 2009 10:47:33 -0800, Scott Robinson <scott@quadhome.com> wrote:
> > Resubmit a full patch, or submit another one on top of it?
> 
> A new full patch would be great, thanks!

Sadly, I won't be able to make the changes requested. I'm falling off the
Internet shortly.

Hopefully David, who already put in a couple improvements, can do the finishing
touches?

Re: [PATCH] JSON output for notmuch-search and notmuch-show.

[David Bremner]

On Thu, 31 Dec 2009 00:54:55 -0800, Scott Robinson <scott@quadhome.com> wrote:
> Excerpts from Carl Worth's message of Tue Dec 22 21:48:34 -0800 2009:
> > On Fri, 18 Dec 2009 10:47:33 -0800, Scott Robinson <scott@quadhome.com> wrote:
> > > Resubmit a full patch, or submit another one on top of it?
> > 
> > A new full patch would be great, thanks!
> 
> Sadly, I won't be able to make the changes requested. I'm falling off the
> Internet shortly.
> 
> Hopefully David, who already put in a couple improvements, can do the finishing
> touches?

Happy to. Thanks for your big effort so far!

d

Re: [PATCH] JSON output for notmuch-search and notmuch-show.

[David Bremner]

On Fri, 18 Dec 2009 10:47:33 -0800, Scott Robinson <scott@quadhome.com> wrote:

> Resubmit a full patch, or submit another one on top of it?

My own feeling is that in general if a patch hasn't been merged into
master, and the changes don't form a separate logical change (i.e. "oh,
if you take that patch, you could have this feature too"), then
resubmitting makes the most sense. This also provides a chance to rebase
against master.

Of course, if I'm way off base here, please (notmuch list readers) let
me know.

David