Request for high-priority improvements to notmuch

Request for high-priority improvements to notmuch

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 4798 bytes --]

Hello Carl et. al.

I know you're trying to work through a boat load of backlog patches, but
after using notmuch for a couple weeks now, I would like to highlight
what I think are the two most important things that need to be be
implemented in the notmuch CLI and why.

Ability to apply tags based on folder paths in "notmuch new"
------------------------------------------------------------
Notmuch needs to be able to be configured to tag messages differently if
they appear in different folders (mentioned previously in
id:87tyu9dfhs.fsf@servo.finestructure.net).  For instance, for
"inbox"/"sent"/"drafts"/"archive" maildir subdirectories of your primary
mail directory "/home/user/mail", a notmuch config could look like this:

   [database]
   path=/home/user/mail

   [tags]
   inbox = +inbox,+unread
   sent = +sent
   drafts = +draft
   archive = -inbox

In other words, a user could specify that new messages to the maildir
"/home/user/mail/sent" would be tagged *only* with the "sent" tag.

This is very important for being able to handle different kinds of new
messages differently, which is very important for being able to have
notmuch handle things that it is currently unable to handle.  If this is
*not* implemented, then there is no way to distinguish different kinds
of new mail in notmuch, which severely restricts subtle handling of
different mail situations.

For instance, this feature would deal with the following problems:

* Notmuch currently does not handle sent mail.  If sent messages are to
be indexed, then they need to be delivered or fcc'd to a maildir which
is indexed by notmuch.  But this means they currently always show up as
"inbox"/"unread", which means they always appear in the inbox.  Folder
based tagging would allow sent mail to instead be delivered or fcc'd to
a sent mail or archive folder that would not get the "inbox"/"unread"
tags.

* Notmuch currently does not handle draft messages.  If this folder
tagging scheme is available, draft messages could be stored in an
indexed directory, which would allow viewing of draft messages
(currently also not supported), and therefore building simple functions
to resume saved drafts.

I really believe this needs to be implemented in "notmuch new".  If it
were implemented in a separate function it would require two calls to
notmuch for every notmuch new, which would be a waste.

JSON output for "notmuch search/show" with ability to filter output fields
--------------------------------------------------------------------------
This is very important to enable more effective, streamlined, and faster
notmuch helper functions and scripts.  There has been a lot of
discussion about what notmuch should or should not be handling.  No
matter what eventually happens, it is clear that notmuch needs to make
it as easy as possible for external applications to parse it's output.
Clearly the best way to do that is to provide JSON output, which is a
widely adopted standard that is well supported in any language one
might want to use to build a notmuch wrapper.

If notmuch could also be told to output only certain desired fields,
then parsing by clients would be made significantly more efficient.  Not
to mention the notmuch UI itself could be streamlined as the "search"
and "show" functions could be combined into a single subcommand:

"search" --> "search --output=thread_id,date,number,author,subject,tags"
 "show"  --> "search --output=message_id,tags,path,header,body,attachments"

This would all make things *much* easier for clients.  For instance,
here are just a couple of things that could handled in wrapper scripts
that would be greatly facilitated by this proposal (including proposed
notmuch search filtered output commands):

* Proper maildir sync ("search --output=message_id,tags,path" ...)

* Purging "delete" tagged messages ("search --output=path" tags:delete)

* Moving/archiving messages based on search results ("search --output=path" ...)

* New client viewers

I believe there is already a patch in the queue to move to JSON output,
so hopefully this can be implemented without much further work.

--------------------------------------------------

I really think these enhancements are very important, and notmuch should
be aiming to implement them soon.  They would greatly extend the
capabilities of notmuch, without restricting it's usage at all.  They
would blow open the doors for making notmuch helper applications, which
would essentially eliminate further discussion on what notmuch should
be handling natively or not.  A lot of the current shortcomings of
notmuch could be easily dealt with if these enhancements are
implemented.

Thanks so much for the time, and I look forward to any discussion on
these ideas, and to seeing them in near-future releases.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Request for high-priority improvements to notmuch

[sebastian]

Let me second those thoughts. I'll even put forward 2 more pet suggestions
:).

1)
> Ability to apply tags based on folder paths in "notmuch new"

This will let me sync tags with notmuchsync much more easily than going
through all mails and detecting their IMAP path from the filename. Support
for searching/tagging mails in certain folders would be really nice. I
would appreciate but do not urgently need a [tags] section in notmuch
proper, as I'll extend notmuchsync to do that for me.

2)
> JSON output for "notmuch search/show" with ability to filter output fields
> "search" --> "search --output=thread_id,date,number,author,subject,tags"
>  "show"  --> "search
> --output=message_id,tags,path,header,body,attachments"

YES PLEASE :-). notmuch seems designed to work in an ecosystem of
surrounding scripts, feeding data in and out. But we are all currently
limited to regexes for that. And heck, I hard a hard time understanding
why all hell broke out until I found that i had added a tag containing
parentheses which made my regex fail. :-). XML, JSON, any structured
output would be nice.

And as for filtering: YES, PLEASE :-). notmuchsync and many other 3rd
party apps would love that. As father of notmuchsync, I can tell you my
little script hickups very badly when slurping in 200k mails (including
text bodies) just to find out the maildir tags of those mails.

> * Proper maildir sync ("search --output=message_id,tags,path" ...)
[snip many sensible proposals]

Sebastian

P.S. as a bonus: My very ugly way of getting a distribution of existing
tags (extra points for beautified versions):

notmuch dump|sed -e 's/^.*(//'|sed -e 's/)$//'|sed 's/ /\n/g'|sort|uniq
-c|sort -r

Re: Request for high-priority improvements to notmuch

[Scott Robinson]

Excerpts from sebastian's message of Mon Feb 01 14:29:44 -0600 2010:
> 2)
> > JSON output for "notmuch search/show" with ability to filter output fields
> > "search" --> "search --output=thread_id,date,number,author,subject,tags"
> >  "show"  --> "search
> > --output=message_id,tags,path,header,body,attachments"
> 
> YES PLEASE :-). notmuch seems designed to work in an ecosystem of
> surrounding scripts, feeding data in and out. But we are all currently
> limited to regexes for that. And heck, I hard a hard time understanding
> why all hell broke out until I found that i had added a tag containing
> parentheses which made my regex fail. :-). XML, JSON, any structured
> output would be nice.
> 
> And as for filtering: YES, PLEASE :-). notmuchsync and many other 3rd
> party apps would love that. As father of notmuchsync, I can tell you my
> little script hickups very badly when slurping in 200k mails (including
> text bodies) just to find out the maildir tags of those mails.
> 

There's been a JSON patch waiting for a month now.

Re: Request for high-priority improvements to notmuch

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 761 bytes --]

also sprach sebastian@sspaeth.de <sebastian@sspaeth.de> [2010.02.02.0929 +1300]:
> YES PLEASE :-). notmuch seems designed to work in an ecosystem of
> surrounding scripts, feeding data in and out. But we are all currently
> limited to regexes for that. And heck, I hard a hard time understanding
> why all hell broke out until I found that i had added a tag containing
> parentheses which made my regex fail. :-). XML, JSON, any structured
> output would be nice.

Shoot me, but I'd say mbox output would be nice too.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"time flies like an arrow. fruit flies like a banana."
                                                       -- groucho marx
 
spamtraps: madduck.bogus@madduck.net

[-- Attachment #2: Digital signature (see http://martin-krafft.net/gpg/) --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Request for high-priority improvements to notmuch

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 1191 bytes --]

also sprach Scott Robinson <scott@quadhome.com> [2010.02.02.1043 +1300]:
> > YES PLEASE :-). notmuch seems designed to work in an ecosystem of
> > surrounding scripts, feeding data in and out. But we are all currently
> > limited to regexes for that. And heck, I hard a hard time understanding
> > why all hell broke out until I found that i had added a tag containing
> > parentheses which made my regex fail. :-). XML, JSON, any structured
> > output would be nice.
> > 
> > And as for filtering: YES, PLEASE :-). notmuchsync and many other 3rd
> > party apps would love that. As father of notmuchsync, I can tell you my
> > little script hickups very badly when slurping in 200k mails (including
> > text bodies) just to find out the maildir tags of those mails.
> > 
> 
> There's been a JSON patch waiting for a month now.

The last month has been busy for everyone, mainly due to LCA. We
should now all work together to help Carl go through the patch
queue. Maybe http://patchwork.madduck.net can help.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
beauty, brains, availability, personality; pick any two.
 
spamtraps: madduck.bogus@madduck.net

[-- Attachment #2: Digital signature (see http://martin-krafft.net/gpg/) --]
[-- Type: application/pgp-signature, Size: 198 bytes --]