Notmuch command interface

Notmuch command interface

[Ben Gamari]

Hey all,

While the notmuch command-line interface is fantastic, there defintely are a few
gaps which hurt performance. Since I've been following, I've seen several
threads on how to handle message addition and initial tagging[1][2]. I recall
that some time ago Carl had the idea of applying only a 'new' tag during notmuch
new. I strongly supported this proposal yet it seems that nothing ever became of
it. Is there a reason for this? Would a patch be accepted?

Secondly, there was at one point talk of allowing notmuch new to be called with
an explicit list of messages to be added. Is there support for such an idea? It
seems like this might help mail fetching performance, if only slightly.

Thanks!

- Ben


[1] p87hbp5j9dv.fsf@hackervisions.or
[2] 20100114084713.GA22273@harikalardiyar

Re: Notmuch command interface

[David A Benjamin]

On Thu, 11 Mar 2010, Ben Gamari wrote:
> While the notmuch command-line interface is fantastic, there defintely are a few
> gaps which hurt performance. Since I've been following, I've seen several
> threads on how to handle message addition and initial tagging[1][2]. I recall
> that some time ago Carl had the idea of applying only a 'new' tag during notmuch
> new. I strongly supported this proposal yet it seems that nothing ever became of
> it. Is there a reason for this? Would a patch be accepted?
>

To avoid breaking the current behavior, what if the set of tags applied to 
new/moved/deleted(?) messages could be set on the command line or some 
other library interface, while just perhaps defaulting to unread/inbox? 
This would support both the current behavior and a hooks-like workflow 
without notmuch itself treating any tags (new, unread, inbox, etc.) as 
"special".

David

Re: Notmuch command interface

[Ben Gamari]

On Thu, 11 Mar 2010 17:52:34 -0500 (EST), David A Benjamin <davidben@MIT.EDU> wrote:
> To avoid breaking the current behavior, what if the set of tags applied to 
> new/moved/deleted(?) messages could be set on the command line or some 
> other library interface, while just perhaps defaulting to unread/inbox? 
> This would support both the current behavior and a hooks-like workflow 
> without notmuch itself treating any tags (new, unread, inbox, etc.) as 
> "special".

Seems reasonable. I might take a stab at this. Anyone else have any thoughts
about the above?

- Ben

Re: Notmuch command interface

[Michal Sojka]

On Thu, 11 Mar 2010, David A Benjamin wrote:
> On Thu, 11 Mar 2010, Ben Gamari wrote:
> > While the notmuch command-line interface is fantastic, there defintely are a few
> > gaps which hurt performance. Since I've been following, I've seen several
> > threads on how to handle message addition and initial tagging[1][2]. I recall
> > that some time ago Carl had the idea of applying only a 'new' tag during notmuch
> > new. I strongly supported this proposal yet it seems that nothing ever became of
> > it. Is there a reason for this? Would a patch be accepted?
> >
> 
> To avoid breaking the current behavior, what if the set of tags applied to 
> new/moved/deleted(?) messages could be set on the command line or some 
> other library interface, while just perhaps defaulting to unread/inbox? 
> This would support both the current behavior and a hooks-like workflow 
> without notmuch itself treating any tags (new, unread, inbox, etc.) as 
> "special".

Hi,

in id:87ljecmnbd.fsf@steelpick.localdomain, I outlined my idea about
mail store abstraction. To go on with the idea, I think that the set of
initial tags should be determined by mail store implementation.
Currently I think that new messages should be tagged as follows:

1) immutable file-based mail store: inbox + unread (backward compatibility
   with current notmuch)

2) Maildir-based mail store: inbox + tags derived from maildir flags.

3) git-based mail store: arbitrary tags stored in git repository (perhaps
   by MDA).

In 3) you could therfore use any policy you want to do initial tagging
of your messages. What do you think?

Unfortunately, I didn't make much progress in implementating this, but
it's definitely a priority for me because I want to get rid of slow
notmuchsync.

Cheers,
Michal

Re: Notmuch command interface

[Sebastian Spaeth]

> Unfortunately, I didn't make much progress in implementating this, but
> it's definitely a priority for me because I want to get rid of slow
> notmuchsync.

notmuchsync master is now using the json output, by the way. I think
that suppressing the output of msg bodies in notmuch show will help
notmuchsync performance *a lot*.

Sebastian

Re: Notmuch command interface

[Michal Sojka]

On Fri, 12 Mar 2010, Sebastian Spaeth wrote:
> > Unfortunately, I didn't make much progress in implementating this, but
> > it's definitely a priority for me because I want to get rid of slow
> > notmuchsync.
> 
> notmuchsync master is now using the json output, by the way. I think
> that suppressing the output of msg bodies in notmuch show will help
> notmuchsync performance *a lot*.

I've just tried it :-(

+ /home/wsh/src/notmuchsync/notmuchsync -s -n -d
20100312 15:59:37 - DEBUG - Execute ['notmuch', 'show', '--format=json', '1265813977..1270997977']
Traceback (most recent call last):
  File "/home/wsh/src/notmuchsync/notmuchsync", line 76, in <module>
    all_mails=all_mails)
  File "/home/wsh/src/notmuchsync/notmuch/notmuch.py", line 434, in syncTags
    msgs = self.show(searchterm, wholeThread=False)
  File "/home/wsh/src/notmuchsync/notmuch/notmuch.py", line 372, in show
    json_forest = json.loads(stdout)
  File "/usr/lib/pymodules/python2.5/simplejson/__init__.py", line 307, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/pymodules/python2.5/simplejson/decoder.py", line 335, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/pymodules/python2.5/simplejson/decoder.py", line 353, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Do you know what's wrong? I use notmuch based on commit
d30049d2b9fd5eb113cd99e88d6fa7b40c36d3c5 (current master).

Michal

Re: Notmuch command interface

[Sebastian Spaeth]

On Fri, 12 Mar 2010 16:03:34 +0100, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> On Fri, 12 Mar 2010, Sebastian Spaeth wrote:
> > > Unfortunately, I didn't make much progress in implementating this, but
> > > it's definitely a priority for me because I want to get rid of slow
> > > notmuchsync.
> > 
> > notmuchsync master is now using the json output, by the way. I think
> > that suppressing the output of msg bodies in notmuch show will help
> > notmuchsync performance *a lot*.
> 
> I've just tried it :-(
>   File "/usr/lib/pymodules/python2.5/simplejson/decoder.py", line 353, in raw_decode
>     raise ValueError("No JSON object could be decoded")
> ValueError: No JSON object could be decoded

Wow, that is bad. All it does is taking "nomuch show" output and feeding
it to the json parser. That probably means that the json was not valid
(which would be bad).

Having said that, I remember that I got that when I used a notmuch that
doesn't include the notmuch tags in the json output, ie you need notmuch
from current master less than 26h ago. Do you use that?

Also Python 2.6 is a requirement, but I see you used that already...
Sebastian 

Re: Notmuch command interface

[Michal Sojka]

On Fri, 12 Mar 2010 17:03:04 +0100, "Sebastian Spaeth" <Sebastian@SSpaeth.de> wrote:
> On Fri, 12 Mar 2010 16:03:34 +0100, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> > On Fri, 12 Mar 2010, Sebastian Spaeth wrote:
> > > > Unfortunately, I didn't make much progress in implementating this, but
> > > > it's definitely a priority for me because I want to get rid of slow
> > > > notmuchsync.
> > > 
> > > notmuchsync master is now using the json output, by the way. I think
> > > that suppressing the output of msg bodies in notmuch show will help
> > > notmuchsync performance *a lot*.
> > 
> > I've just tried it :-(
> >   File "/usr/lib/pymodules/python2.5/simplejson/decoder.py", line 353, in raw_decode
> >     raise ValueError("No JSON object could be decoded")
> > ValueError: No JSON object could be decoded
> 
> Wow, that is bad. All it does is taking "nomuch show" output and feeding
> it to the json parser. That probably means that the json was not valid
> (which would be bad).

Yes. I found that the problem is caused by emails with an empty body.
I'll post a patch to fix this in a while.

> Having said that, I remember that I got that when I used a notmuch that
> doesn't include the notmuch tags in the json output, ie you need notmuch
> from current master less than 26h ago. Do you use that?
> 
> Also Python 2.6 is a requirement, but I see you used that already...

Strange, I use default python from Debian which is 2.5.5 and current
notmuchsync master runs without problems.

-Michal

[PATCH] Produce valid JSON output even if mail body is empty

[Michal Sojka]

Mails with empty body produced the following output:
  "body": [{"id": 1, "content-type": "text/plain", "content": (null)}]
The (null) is not valid JSON syntax.

This patch changes the output to:
  "body": [{"id": 1, "content-type": "text/plain", "content": ""}]

Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
---
 notmuch-show.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/notmuch-show.c b/notmuch-show.c
index ff1fecb..cc2fb6a 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -323,7 +323,7 @@ format_part_json (GMimeObject *part, int *part_count)
 	show_part_content (part, stream_memory);
 	part_content = g_mime_stream_mem_get_byte_array (GMIME_STREAM_MEM (stream_memory));
 
-	printf (", \"content\": %s", json_quote_str (ctx, (char *) part_content->data));
+	printf (", \"content\": %s", json_quote_str (ctx, part_content->data ? (char *)part_content->data : ""));
     }
 
     fputs ("}", stdout);
-- 
1.7.0

Re: [PATCH] Produce valid JSON output even if mail body is empty

[James Westby]

On Sun, 14 Mar 2010 19:19:11 +0100, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> Mails with empty body produced the following output:
>   "body": [{"id": 1, "content-type": "text/plain", "content": (null)}]
> The (null) is not valid JSON syntax.

Is this just something that can happen with the body?

I've see (null) in the emacs interface when I've done something silly
such as opening a newer notmuch db with an old client.

Should all the attributes be guarded in a similar manner to ensure valid
JSON?

Thanks,

James

[PATCH] Fix invalid JSON output for NULL strings

[Michal Sojka]

On Sun, 14 Mar 2010 15:01:28 -0400, James Westby <jw+debian@jameswestby.net> wrote:
> On Sun, 14 Mar 2010 19:19:11 +0100, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> > Mails with empty body produced the following output:
> >   "body": [{"id": 1, "content-type": "text/plain", "content": (null)}]
> > The (null) is not valid JSON syntax.
> 
> Is this just something that can happen with the body?
> 
> I've see (null) in the emacs interface when I've done something silly
> such as opening a newer notmuch db with an old client.
> 
> Should all the attributes be guarded in a similar manner to ensure valid
> JSON?


You're right. When you import messages from mail archive such as
http://notmuchmail.org/pipermail/notmuch/, there are no To: headers.
This case is already handled by the current code, but what about this
more universal solution? It also solves my problem with notmuchsynch.

-- >8 --
Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
---
 json.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/json.c b/json.c
index 9614143..0885d3d 100644
--- a/json.c
+++ b/json.c
@@ -56,7 +56,7 @@ json_quote_str(const void *ctx, const char *str)
     int len = 0;
 
     if (!str)
-	return NULL;
+	return (char*)"null";
 
     for (ptr = str; *ptr; len++, ptr++) {
 	if (*ptr < 32 || *ptr == '\"' || *ptr == '\\')
-- 
1.7.0

Re: Notmuch command interface

[Sebastian Spaeth]

> To avoid breaking the current behavior, what if the set of tags applied to 
> new/moved/deleted(?) messages could be set on the command line or some 
> other library interface, while just perhaps defaulting to unread/inbox? 
> This would support both the current behavior and a hooks-like workflow 
> without notmuch itself treating any tags (new, unread, inbox, etc.) as 
> "special".

+1. For some, the default "inbox unread" might just be the correct
thing, while others may want "new". I know that I don't need any special
tag for new mails, as I applied the "sync maildir flags" patch. So
configurable in .notmuch-config with a default of "inbox unread" to keep
behavior consistent seems the best way to me.

Sebastian