Thoughts on notmuch and Lua

Thoughts on notmuch and Lua

[Ali Polatel]

[-- Attachment #1: Type: text/plain, Size: 1186 bytes --]

Before trying to implement anything I decided to send a mail to the list
to ask people's opinion.

What's the problem?
===================
Notmuch isn't very configurable.

How can Lua integration solve this?
===================================
Here are initial thoughts on how to integrate Lua with notmuch.
Any comments appreciated.

1. Configuration file:
The configuration file can be a Lua script that allows more dynamic
configuration. Here's an example:

# notmuch configuration file:
config = {}
config.dbpath = "/path/to/maildir"
config.exclude = function (maildir)
    return not string.match(maildir, ".*Trash.*")
    end
...

2. Hooks:
This is a feature I really miss having switched from sup.
There can be many hooks, a hook that formats search output,
a hook that is called before adding messages to the database which may
be used to add initial tags depending on headers etc.

Why Lua?
========
Lua has many advantages over other scripting languages when it comes to
integration with a C program. It has a very clean and easy C API, the
overhead of running Lua scripts is not noticable among other things.

-- 
Regards,
Ali Polatel

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: Thoughts on notmuch and Lua

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 3365 bytes --]

On Thu, 14 Jan 2010 10:47:13 +0200, Ali Polatel <alip@exherbo.org> wrote:
> Before trying to implement anything I decided to send a mail to the list
> to ask people's opinion.

Hi Ali, welcome to notmuch!

I appreciate you soliciting opinions, but I hope that my answer won't
discourage you. By all means, please feel free to experiment!

> What's the problem?
> ===================
> Notmuch isn't very configurable.

I'll grant that. And as can be seen in TODO and in code comments, we
definitely want to fix that.

> 1. Configuration file:
> The configuration file can be a Lua script that allows more dynamic
> configuration. Here's an example:
> 
> # notmuch configuration file:
> config = {}
> config.dbpath = "/path/to/maildir"
> config.exclude = function (maildir)
>     return not string.match(maildir, ".*Trash.*")
>     end
> ...

That doesn't look very compelling to me.

I'd much rather have:

	[database]
	path=/home/cworth/mail
	maildir_exclude=.*Trash.*

with the exact same functionality.

Granted, having a full programming language in the configuration file
makes thing much more dynamic, but it also makes it much harder for the
user to read, edit, and ensure the syntax is correct.

> 2. Hooks:
> This is a feature I really miss having switched from sup.
> There can be many hooks, a hook that formats search output,
> a hook that is called before adding messages to the database which may
> be used to add initial tags depending on headers etc.

I understand that some people really like their hooks. They let users
invent all kinds of interesting, custom functionality.

But I think hooks also have problems. Sometimes the most interesting
functionality has to be pieced together by every user going to a wiki
page and finding the "standard" hooks. I'd much rather avoid that by
getting the most useful functionality into the program in the first
place.

Hooks also impose a particular amount of maintenance burden on the
software. And they are often implemented in a way that makes them very
hard to be discovered.

I wrote a message to the sup mailing list describing some of these
issues. The context there was a patch I wrote adding a configuration
option, (and the sup maintainer preferring it be added as a patch
instead):

	id:1254417826-sup-6584@yoom.home.cworth.org
	Subject: Re: [sup-talk] [PATCH] Add new :crypto_default configuration option.

I did find out later that the sup hooks were more self-documenting than
I had understood. (There was a sup command-line option that printed
documentation for all available hooks.) Something like that is
definitely a requirement for providing hooks.

So I'm not entirely opposed to the idea of adding hooks to notmuch, but
I'll definitely need to be convinced that any particular functionality
can't be better integrated without the hook.

> Why Lua?
> ========
> Lua has many advantages over other scripting languages when it comes to
> integration with a C program. It has a very clean and easy C API, the
> overhead of running Lua scripts is not noticable among other things.

I've definitely heard lots of good things about "lua embedability". So
if we do decide to provide hooks, then lua would seem like a logical
option to look at first. I've never looked at it closely myself though.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Thoughts on notmuch and Lua

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 1149 bytes --]

also sprach Carl Worth <cworth@cworth.org> [2010.01.15.1200 +1300]:
> > Lua has many advantages over other scripting languages when it
> > comes to integration with a C program. It has a very clean and
> > easy C API, the overhead of running Lua scripts is not noticable
> > among other things.
> 
> I've definitely heard lots of good things about "lua
> embedability". So if we do decide to provide hooks, then lua would
> seem like a logical option to look at first. I've never looked at
> it closely myself though.

Lua for hooks has the advantage that the hooks can be executed in
the context of manipulateable objects. On the other hand, hooks in
the style of run-parts directories are more flexible and accessible,
and could always be invoked as filters for the manipulateable data.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
"imagine if every thursday your shoes exploded if you
 tied them the usual way. this happens to us all the time
 with computers, and nobody thinks of complaining."
                                                        -- jeff raskin
 
spamtraps: madduck.bogus@madduck.net

[-- Attachment #2: Digital signature (see http://martin-krafft.net/gpg/) --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Thoughts on notmuch and Lua

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1018 bytes --]

On Fri, 15 Jan 2010 13:16:00 +1300, martin f krafft <madduck@madduck.net> wrote:
> Lua for hooks has the advantage that the hooks can be executed in
> the context of manipulateable objects. On the other hand, hooks in
> the style of run-parts directories are more flexible and accessible,
> and could always be invoked as filters for the manipulateable data.

Good point.

One thing that notmuch has in its favor already is that there's a
command-line interface that provides all of the interesting
functionality. So it's already a fairly trivial matter to "hook" things
like "notmuch new" by writing a script in whatever language you prefer
to invoke things like "notmuch search" and "notmuch tag".

And that's what I'm doing already, (with a shell script), to tag
newly-arrived messages.

The only part that's really missing here is a directory where scripts
could be dropped in and get automatically invoked.

What do you think, Ali? Would an approach like that satisfy the things
you had in mind for hooks?

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Thoughts on notmuch and Lua

[Ali Polatel]

[-- Attachment #1: Type: text/plain, Size: 2418 bytes --]

Carl Worth yazmış:
> On Fri, 15 Jan 2010 13:16:00 +1300, martin f krafft <madduck@madduck.net> wrote:
> > Lua for hooks has the advantage that the hooks can be executed in
> > the context of manipulateable objects. On the other hand, hooks in
> > the style of run-parts directories are more flexible and accessible,
> > and could always be invoked as filters for the manipulateable data.
> 
> Good point.
> 
> One thing that notmuch has in its favor already is that there's a
> command-line interface that provides all of the interesting
> functionality. So it's already a fairly trivial matter to "hook" things
> like "notmuch new" by writing a script in whatever language you prefer
> to invoke things like "notmuch search" and "notmuch tag".
> 
> And that's what I'm doing already, (with a shell script), to tag
> newly-arrived messages.
> 
> The only part that's really missing here is a directory where scripts
> could be dropped in and get automatically invoked.
> 
> What do you think, Ali? Would an approach like that satisfy the things
> you had in mind for hooks?
> 

It might, here are some thoughts and questions to help you elaborate:

- How will these scripts manipulate data?
  e.g.: A "tagger" script may get the new mail from stdin and print out
  the new tags which notmuch will read and apply to the message.
  A "search-filter" script may get search results from stdin and filter
  them. Just my initial thoughts.

This script approach has both upsides and downsides. I have some
experience about this because I wrote a tool called mpdcron¹ which
executes hooks based on mpd events. First I thought it would be cool to
execute scripts based on events by passing data using environment
variables. However this approach didn't give hooks enough power to save
state information etc. so I decided to add module support through
GModule.

I may be going a bit off topic here but what I'm trying to explain is
scripts may not give power-users enough options and is practically
slower than having Lua, or another scripting language, support. In my
humble opinion the former is just a hack while the latter is the proper
way to do it.

P.S.: As I stated in the beginning you could have elaborated what you
meant by script support and I'm just commenting on what I've understood.

¹: http://alip.github.com/mpdcron/

> -Carl

-- 
Regards,
Ali Polatel

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: Thoughts on notmuch and Lua

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 4716 bytes --]

On Fri, 15 Jan 2010 23:09:34 +0200, Ali Polatel <alip@exherbo.org> wrote:
> Carl Worth yazmış:
> > What do you think, Ali? Would an approach like that satisfy the things
> > you had in mind for hooks?
> 
> It might, here are some thoughts and questions to help you elaborate:
> 
> - How will these scripts manipulate data?
>   e.g.: A "tagger" script may get the new mail from stdin and print out
>   the new tags which notmuch will read and apply to the message.

This can happen with current notmuch entirely with no real "hook"
support.

We've talked about switching from default tags of "inbox" and "unread"
to simply having new mail tagged with a "new" tag. So a "tagger" script
could operate simply by doing a "notmuch search" for messages with the
"new" tag and could iterate over the filenames to process actual
messages. (We don't have support now for emitting just filenames from a
"notmuch search", but we have patches for that, and I'll be applying one
soon.)

So that's a taste for the "scriptability" I see in the current notmuch
system that makes it really much more flexible than any "hooks"
system. Additional flexibility comes from:

  * User can run a script like this at any point---not merely when
    messages are added.

  * User script isn't restricted to dealing only with "new" messages,
    but can act on any set of messages based on any search constraint,
    (or even all messages in the database).

The results can then be applied by simply calling "notmuch tag" as
needed.

And if there are any performance problems there we can fix them, (such
as, perhaps we'll end up wanting this script to be able to invoke a
single process for all of its tagging rather than calling "notmuch tag"
over and over).

>   A "search-filter" script may get search results from stdin and filter
>   them. Just my initial thoughts.

And how would this search functionality and filtering be different than
the search functionality provided by notmuch itself?

I can think of at least a couple of ways it might be different:

  1. It would be nice to be able to filter based on tags that are
     present in a thread, though perhaps not present in any message
     matching the original search.

     An obvious application of this is the "thread muting" feature,
     where once a message is tagged as "muted", no messages delivered to
     that thread in the future will appear in the inbox.

     This is a feature I'd like to put into the core of notmuch such
     that one passes a query to match messages and then also a second
     query to filter based on the collected tags in threads. Something
     like:

	notmuch search tag:inbox --filter="not tag:muted"

 2. There are other details available at the thread level that are not
    available at the level at which message-based searching happens.

    A simple example of this would be the ability to search for threads
    with a single message, (perhaps checking to ensure that all requests
    had gotten at least one reply). But one can imagine more complex
    things as well, "Show me all threads where ImportantPerson sent a
    message and where I never replied in the thread."

    For this kind of thing, I think we simply want to build on the
    output of "notmuch search". The current output isn't very usable for
    this, but with things like the structured json output, etc. (which,
    again, I hope to be merging soon), it would be quite easy to write
    new tools that accept that output and provide additional searching
    and filtering, etc. And that tool could provide lua-based scripting
    or whatever else is desired.

So my feeling is that if anything can live outside of notmuch, then it
should, and should simply build on top of notmuch output. (And we should
fix notmuch output to support that well.)

And anything that must live within notmuch (or is best supported there),
we should see if we can't just make that a core part of notmuch itself,
(such as the --filter option I showed above).

I'm *still* not wanting to squelch any experimentation with embedding
scripting languages inside notmuch, or anything else. I'm just still not
seeing anything that requires this.

Look at the amount of emacs-lisp code we've written, for example, and
the various things it does, (hiding away citations, etc.). That's all
"scripting code", but that sits easily on top of the existing notmuch
command-line tool.

I think I'd prefer to keep that nice clean boundary until we find
something that really requires changing that. But, show me something
really cool that requires it, and you might convince me. :-)

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Thoughts on notmuch and Lua

[Ali Polatel]

[-- Attachment #1: Type: text/plain, Size: 5544 bytes --]

Carl Worth yazmış:
> On Fri, 15 Jan 2010 23:09:34 +0200, Ali Polatel <alip@exherbo.org> wrote:
> > Carl Worth yazmış:
> > > What do you think, Ali? Would an approach like that satisfy the things
> > > you had in mind for hooks?
> > 
> > It might, here are some thoughts and questions to help you elaborate:
> > 
> > - How will these scripts manipulate data?
> >   e.g.: A "tagger" script may get the new mail from stdin and print out
> >   the new tags which notmuch will read and apply to the message.
> 
> This can happen with current notmuch entirely with no real "hook"
> support.
> 
> We've talked about switching from default tags of "inbox" and "unread"
> to simply having new mail tagged with a "new" tag. So a "tagger" script
> could operate simply by doing a "notmuch search" for messages with the
> "new" tag and could iterate over the filenames to process actual
> messages. (We don't have support now for emitting just filenames from a
> "notmuch search", but we have patches for that, and I'll be applying one
> soon.)
> 

This is one of the reasons I wanted hook support in the beginning.
I'm looking forward to seeing this in notmuch.

> So that's a taste for the "scriptability" I see in the current notmuch
> system that makes it really much more flexible than any "hooks"
> system. Additional flexibility comes from:
> 
>   * User can run a script like this at any point---not merely when
>     messages are added.
> 
>   * User script isn't restricted to dealing only with "new" messages,
>     but can act on any set of messages based on any search constraint,
>     (or even all messages in the database).
> 
> The results can then be applied by simply calling "notmuch tag" as
> needed.
> 

+1. I totally agree.

> And if there are any performance problems there we can fix them, (such
> as, perhaps we'll end up wanting this script to be able to invoke a
> single process for all of its tagging rather than calling "notmuch tag"
> over and over).
> 
> >   A "search-filter" script may get search results from stdin and filter
> >   them. Just my initial thoughts.
> 
> And how would this search functionality and filtering be different than
> the search functionality provided by notmuch itself?
> 

I accept, this search-filter idea is kinda stupid now that I think about
it.

> I can think of at least a couple of ways it might be different:
> 
>   1. It would be nice to be able to filter based on tags that are
>      present in a thread, though perhaps not present in any message
>      matching the original search.
> 
>      An obvious application of this is the "thread muting" feature,
>      where once a message is tagged as "muted", no messages delivered to
>      that thread in the future will appear in the inbox.
> 
>      This is a feature I'd like to put into the core of notmuch such
>      that one passes a query to match messages and then also a second
>      query to filter based on the collected tags in threads. Something
>      like:
> 
> 	notmuch search tag:inbox --filter="not tag:muted"
> 

Looks like a good idea indeed.

>  2. There are other details available at the thread level that are not
>     available at the level at which message-based searching happens.
> 
>     A simple example of this would be the ability to search for threads
>     with a single message, (perhaps checking to ensure that all requests
>     had gotten at least one reply). But one can imagine more complex
>     things as well, "Show me all threads where ImportantPerson sent a
>     message and where I never replied in the thread."
> 
>     For this kind of thing, I think we simply want to build on the
>     output of "notmuch search". The current output isn't very usable for
>     this, but with things like the structured json output, etc. (which,
>     again, I hope to be merging soon), it would be quite easy to write
>     new tools that accept that output and provide additional searching
>     and filtering, etc. And that tool could provide lua-based scripting
>     or whatever else is desired.
> 

Makes sense. Structured output would make things really simpler.

> So my feeling is that if anything can live outside of notmuch, then it
> should, and should simply build on top of notmuch output. (And we should
> fix notmuch output to support that well.)
> 

Works for me, as long as it solves my problems and as I stated above I
think it will.

> And anything that must live within notmuch (or is best supported there),
> we should see if we can't just make that a core part of notmuch itself,
> (such as the --filter option I showed above).
> 
> I'm *still* not wanting to squelch any experimentation with embedding
> scripting languages inside notmuch, or anything else. I'm just still not
> seeing anything that requires this.
> 
> Look at the amount of emacs-lisp code we've written, for example, and
> the various things it does, (hiding away citations, etc.). That's all
> "scripting code", but that sits easily on top of the existing notmuch
> command-line tool.
> 
> I think I'd prefer to keep that nice clean boundary until we find
> something that really requires changing that. But, show me something
> really cool that requires it, and you might convince me. :-)
> 

I don't have anything in mind right now but when I do we can talk
further :)

Thanks for the descriptive response, I really appreciate it.

> -Carl

-- 
Regards,
Ali Polatel

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

inbox/unread tags for new messages [was: Re: Thoughts on notmuch and Lua]

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 2292 bytes --]

On Fri, Jan 15, 2010 at 03:15:13PM -0800, Carl Worth wrote:
> We've talked about switching from default tags of "inbox" and "unread"
> to simply having new mail tagged with a "new" tag. So a "tagger" script
> could operate simply by doing a "notmuch search" for messages with the
> "new" tag and could iterate over the filenames to process actual
> messages. (We don't have support now for emitting just filenames from a
> "notmuch search", but we have patches for that, and I'll be applying one
> soon.)

Hey, Carl.  I would actually like to argue against replacing the
'inbox' and 'unread' tags with a single "new" tag.  Here's why:

At first I didn't think I liked the fact that new messages were doubly
tagged with 'inbox' and 'unread'.  But then I realized that what I was
really not liking was the way that the emacs UI was handling those
tags, and the more I thought about it I realized that those two tags
are very useful and we just need to streamline how they're handled.

I think that the 'unread' tag should be tightly synced to the maildir
'S' flags, which indicate the read status on the maildir message files
themselves.  This is useful for compatibility with other MUAs, and is
one of the few generally useful tags that can be synced through IMAP.

The 'unread' tag should also not be something that the user ever
consciously has to get rid of.  Notmuch MUAs should just get rid of it
automatically when the user actually views the message.  The notmuch
emacs UI is currently not handling this very well, but I think we can
tweak that fairly easily.

As for the 'inbox' tag, I think it should be kept because it has a
very particular, well understood, and useful semantic meaning for MUAs
that people already understand.  My mail inbox is actually what I'm
syncing via offlineimap and I would like the 'inbox' tag to correspond
directly to the messages that are in my actual maildir inbox.

This last point I think is very important because I think it
corresponds to the way that *most* people are or need to handle their
mail, and it's something that I think notmuch hasn't quite fully come
to grips with yet.

I'm going to follow up this message with another one that describes my
idea for a streamlined message flow in notmuch that would utilize
these tags better.

jamie.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: inbox/unread tags for new messages [was: Re: Thoughts on notmuch and Lua]

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 4830 bytes --]

On Sat, 16 Jan 2010 15:18:03 -0500, Jameson Rollins <jrollins@finestructure.net> wrote:
> Hey, Carl.  I would actually like to argue against replacing the
> 'inbox' and 'unread' tags with a single "new" tag.  Here's why:

Oh, well maybe I didn't make that very clear.

Personally, the only thing *I* would do with a "new" tag would be to
have my auto-tagger search for anything tagged as "new" and tag that as
"inbox" and "unread".

I definitely agree that it's important to separate these two
notions. And as you refer to below, the emacs UI makes plenty of
assumptions about these tags existing and manipulating them on behalf of
the user.

So the idea with "new" is really to just make the lower-level pieces of
notmuch, (the command-line interface and the library), to steal less of
the tag namespace. Specifically, the proposal is to reserve a single tag
("new") rather than two tags ("inbox" and "unread"). Then, it's a matter
of some higher-level layers (such as the emacs interface) to apply
special meaning to things like "inbox" and "unread".

> At first I didn't think I liked the fact that new messages were doubly
> tagged with 'inbox' and 'unread'.  But then I realized that what I was
> really not liking was the way that the emacs UI was handling those
> tags, and the more I thought about it I realized that those two tags
> are very useful and we just need to streamline how they're handled.

I agree some changes are needed.

> I think that the 'unread' tag should be tightly synced to the maildir
> 'S' flags, which indicate the read status on the maildir message files
> themselves.  This is useful for compatibility with other MUAs, and is
> one of the few generally useful tags that can be synced through IMAP.

The tight synchronization of this tag does seem useful. But I'm still
not sure what the precise plan here should be. We could change from having
the tag in the database be authoritative to instead have the character
in the filename be authoritative, but we would need answers for the
following questions:

  1. How do we deal with files that don't match maildir conventions?
     (Either not in a /cur or /new directory or not matching the
     filename convention.)

  2. How do we actually make the transition from the old scheme to the
     new? (Perhaps we do this with an increment in the database version
     number and transfer the data from the database to the filenames for
     all known files at the time of the upgrade?)

> The 'unread' tag should also not be something that the user ever
> consciously has to get rid of.  Notmuch MUAs should just get rid of it
> automatically when the user actually views the message.  The notmuch
> emacs UI is currently not handling this very well, but I think we can
> tweak that fairly easily.

I agree on all points. I've been sitting on an unfinished patch series
to implement some ideas I proposed here:

	id:877ht3hfh0.fsf@yoom.home.cworth.org

I plan to get back to finishing that soon. (I set all emacs interface
work aside for a while to do the rename support.) Some of the ideas I
outline there will make the handling of unread more sane I think. I'd be
glad to hear any further ideas.

> As for the 'inbox' tag, I think it should be kept because it has a
> very particular, well understood, and useful semantic meaning for MUAs
> that people already understand.  My mail inbox is actually what I'm
> syncing via offlineimap and I would like the 'inbox' tag to correspond
> directly to the messages that are in my actual maildir inbox.

Yes. If we change to "unread" simply being state that is controlled
entirely by the mailstore, then we would be to the point where "inbox"
is the only tag being claimed from the tag namespace by "notmuch
new". And at that point, I agree we could/should just leave it named
"inbox" rather than "new".

> This last point I think is very important because I think it
> corresponds to the way that *most* people are or need to handle their
> mail, and it's something that I think notmuch hasn't quite fully come
> to grips with yet.

I totally agree. One of the most-important reasons I had in mind when
starting notmuch was the desired to get to a point where I could
actually reason about, and explore more-improved mail flow. Everything
so far has been about getting some low-level plumbing working. I've got
a bunch of ideas for novel mail-handling techniques that will be enabled
by notmuch, and we definitely haven't scratched the surface yet. I'm
sure many readers of this list have similar (and different!) ideas and
I'm looking forward to seeing what we can come up with together.

> I'm going to follow up this message with another one that describes my
> idea for a streamlined message flow in notmuch that would utilize
> these tags better.

I'll look forward to that. Thanks again!

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: inbox/unread tags for new messages [was: Re: Thoughts on notmuch and Lua]

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 6141 bytes --]

On Sat, Jan 16, 2010 at 02:22:09PM -0800, Carl Worth wrote:
> So the idea with "new" is really to just make the lower-level pieces of
> notmuch, (the command-line interface and the library), to steal less of
> the tag namespace. Specifically, the proposal is to reserve a single tag
> ("new") rather than two tags ("inbox" and "unread"). Then, it's a matter
> of some higher-level layers (such as the emacs interface) to apply
> special meaning to things like "inbox" and "unread".

I personally don't really think that this is a good idea.  If this is
all left up to mail reader, and the mail readers all do something
different, then it would be very difficult to switch readers, or to
use different readers for different circumstances.  I really think
that all of this stuff should be handled uniformly by the notmuch CLI
in a very structured way.  If we make sane decisions at that level,
then things are nicely consistent.

What I would instead suggest is that there is a notmuch new hook,
handled at the notmuch program level, that would allow users to define
scripts or functions that would be applied to all new messages.  Then
users could do whatever they want to new messages.  As long as it's
done at the notmuch CLI level, instead of at the reader level, things
won't get fractured by divergent reader implementations.  This ideal
is also in line with the proposal I put forth in
id:20100116204955.GA858@finestructure.net.

> On Sat, 16 Jan 2010 15:18:03 -0500, Jameson Rollins <jrollins@finestructure.net> wrote: 
> > I think that the 'unread' tag should be tightly synced to the maildir
> > 'S' flags, which indicate the read status on the maildir message files
> > themselves.  This is useful for compatibility with other MUAs, and is
> > one of the few generally useful tags that can be synced through IMAP.
> 
> The tight synchronization of this tag does seem useful. But I'm still
> not sure what the precise plan here should be. We could change from having
> the tag in the database be authoritative to instead have the character
> in the filename be authoritative, but we would need answers for the
> following questions:
> 
>   1. How do we deal with files that don't match maildir conventions?
>      (Either not in a /cur or /new directory or not matching the
>      filename convention.)

I'm not sure this is the right question.  It seems to me that if
notmuch is going to be using maildir, then notmuch itself should do
everything to conform to the maildir convention, as well as assume
that anything else touching the same maildirs is doing so as well.
Notmuch should be moving files from new to cur as it processes them,
and it should be adding the S flag if a message has been seen/read.

If there are files in the maildir that don't conform, then they should
either be ignored with some sort of warning, or they should just be
added to the database with no special care at all (maybe with a
warning).

>   2. How do we actually make the transition from the old scheme to the
>      new? (Perhaps we do this with an increment in the database version
>      number and transfer the data from the database to the filenames for
>      all known files at the time of the upgrade?)

Is it really a scheme change that would require delicate handling?  I
would think that the change would be fairly straightforward:

- if notmuch sees that a message in the maildir has moved from new to
  cur and/or the S flag has been added then any 'unread' flags should
  be removed from the message.

- if a 'unread' tag is removed from a message, then the message S flag
  should be added.

It doesn't seem that any database changes need to be made.

> > The 'unread' tag should also not be something that the user ever
> > consciously has to get rid of.  Notmuch MUAs should just get rid of it
> > automatically when the user actually views the message.  The notmuch
> > emacs UI is currently not handling this very well, but I think we can
> > tweak that fairly easily.
> 
> I agree on all points. I've been sitting on an unfinished patch series
> to implement some ideas I proposed here:
> 
> 	id:877ht3hfh0.fsf@yoom.home.cworth.org
> 
> I plan to get back to finishing that soon. (I set all emacs interface
> work aside for a while to do the rename support.) Some of the ideas I
> outline there will make the handling of unread more sane I think. I'd be
> glad to hear any further ideas.

I really think this should just be done in the most stupid simple way
imaginable: if you view/open a message buffer, the 'unread' tag goes
away.  That simple.  I really don't think it's worth spending any time
trying to make it more nuanced or sophisticated than that.  Users
should not ever have to think about changing this tag, and if they
really want to, they can just use the tag function.

> I totally agree. One of the most-important reasons I had in mind when
> starting notmuch was the desired to get to a point where I could
> actually reason about, and explore more-improved mail flow. Everything
> so far has been about getting some low-level plumbing working. I've got
> a bunch of ideas for novel mail-handling techniques that will be enabled
> by notmuch, and we definitely haven't scratched the surface yet. I'm
> sure many readers of this list have similar (and different!) ideas and
> I'm looking forward to seeing what we can come up with together.

Yeah, I am totally on board with your vision for notmuch, Carl.  It's
a incredibly nice new way to handle mail.  You've done a great job
keeping things simple yet flexible.  I really hope we can continue in
that same vein.  I think that the proposal I put forward in my other
message will help further that goal (keep things simple and well
structured, but allow flexibility).

I have no doubt that there will be lots of very novel uses for notmuch
in the future.  I was already talking to a friend about how notmuch
should absolutely be used by mailing list handlers.  How sweet would
it be to have web interfaces to notmuch mail list archives.

jamie.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: inbox/unread tags for new messages [was: Re: Thoughts on notmuch and Lua]

[Ben Gamari]

Excerpts from Jameson Rollins's message of Sat Jan 16 18:38:40 -0500 2010:
> On Sat, Jan 16, 2010 at 02:22:09PM -0800, Carl Worth wrote:
> > So the idea with "new" is really to just make the lower-level pieces of
> > notmuch, (the command-line interface and the library), to steal less of
> > the tag namespace. Specifically, the proposal is to reserve a single tag
> > ("new") rather than two tags ("inbox" and "unread"). Then, it's a matter
> > of some higher-level layers (such as the emacs interface) to apply
> > special meaning to things like "inbox" and "unread".
> 
> I personally don't really think that this is a good idea.  If this is
> all left up to mail reader, and the mail readers all do something
> different, then it would be very difficult to switch readers, or to
> use different readers for different circumstances.  I really think
> that all of this stuff should be handled uniformly by the notmuch CLI
> in a very structured way.  If we make sane decisions at that level,
> then things are nicely consistent.
> 
I strongly disagree. The fact that mail readers _might_ behave
differently is not a good reason to enforce this on the notmuch layer.
In fact, it might be that mail readers purposefully want to handle
'inbox' and 'unread' differently. If that is what they want to do, then
they should be able to. Even if tagging conventions do differ, it would
be trivial to write a script to make the transition. This is because of
notmuch's simplicity and flexibility and why it is important to keep it
that way by avoiding adding unnecessary (even redundant) logic as you
propose below.

> What I would instead suggest is that there is a notmuch new hook,
> handled at the notmuch program level, that would allow users to define
> scripts or functions that would be applied to all new messages.  Then
> users could do whatever they want to new messages.  As long as it's
> done at the notmuch CLI level, instead of at the reader level, things
> won't get fractured by divergent reader implementations.  This ideal
> is also in line with the proposal I put forth in
> id:20100116204955.GA858@finestructure.net.
> 
I fail to see how this offers anything not provided by Carl's original
proposal. As far as I can see, it adds more code to notmuch while
providing no functionality not already attainable and while potentially
placing limits on what is possible.

In my opinion, notmuch should provide little more than a mail store.
There are much better places to put this sort of policy than in notmuch.

- Ben

Re: inbox/unread tags for new messages [was: Re: Thoughts on notmuch and Lua]

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1284 bytes --]

On Sat, 16 Jan 2010 18:38:40 -0500, Jameson Rollins <jrollins@finestructure.net> wrote:
> Is it really a scheme change that would require delicate handling?

Yes.

> I would think that the change would be fairly straightforward:
> 
> - if notmuch sees that a message in the maildir has moved from new to
>   cur and/or the S flag has been added then any 'unread' flags should
>   be removed from the message.
> 
> - if a 'unread' tag is removed from a message, then the message S flag
>   should be added.

I have thousands of messages that I have read with sup and notmuch. The
filenames have never changed since the mail was originally delivered by
maildrop. So these messages are all in "new" directories, do not have
the 'S' flag in the filename, and do not have an "unread" tag.

So the database and the filesystem are currently inconsistent and
neither of the above state changes will trigger for any of these
messages.

Things "work" fine for me now because I'm treating the database state as
authoritative and ignoring the names of the files. Your proposal is to
treat the filenames as authoritative, so will require an additional
step, (to copy the current state from the database to the file names),
besides what you've outlineed above.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: inbox/unread tags for new messages [was: Re: Thoughts on notmuch and Lua]

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 1627 bytes --]

On Sat, Jan 16, 2010 at 06:33:17PM -0800, Carl Worth wrote:
> I have thousands of messages that I have read with sup and notmuch. The
> filenames have never changed since the mail was originally delivered by
> maildrop. So these messages are all in "new" directories, do not have
> the 'S' flag in the filename, and do not have an "unread" tag.
> 
> So the database and the filesystem are currently inconsistent and
> neither of the above state changes will trigger for any of these
> messages.
> 
> Things "work" fine for me now because I'm treating the database state as
> authoritative and ignoring the names of the files. Your proposal is to
> treat the filenames as authoritative, so will require an additional
> step, (to copy the current state from the database to the file names),
> besides what you've outlineed above.

I guess my take on this is that notmuch is a mail processing program
with nominal support for maildir, and should therefore be following
the maildir specification.  I wouldn't say this necessarily means that
the filesystem is "authoritative", just that the specification is
being followed.  I could also see notmuch punting on that and just
offering something that indexes messages stored in individual,
uniquely named files.

I can definitely see, though, that switching to fully following the
maildir spec would require a bit more work and a transition.  My
personal opinion is that notmuch should follow the maildir
specification.  It would definitely facilitate interoperability with
other MTAs and MUAs that do properly follow the maildir specification.

jamie.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]