Some thoughts about notmuch sync with other agents

Some thoughts about notmuch sync with other agents

[Paul R]

Hi,

before going into details, I'd like to tell you how much I like the
notmuch workflow, thank you for producing this nice piece of software !

Like most potential users, I can not really fully jump into notmuch
because of the current synchronisation issues with others MUA. One may
or may not like IMAP for good reasons, the fact is that it is here and
has allowed users to read mails from various places and terminals,
keeping important information synced. So I think that notmuch will have
to live with that, and provide what is required to integrate smoothly
into this environment. Redefining a new mail retrieval protocol and
a mail storing format are both really exciting projects, but they are
projects on their own that could only distract notmuch from its most
beautiful goal : giving *today* users the power to process mail in an
efficient way.

As you see, I advocate a NotMuch <-> IMAP synchronisation ASAP :)

At the moment, notmuch input are mail-as-file + user-defined tags.
OfflineIMAP allows to do the IMAP <-> mail-as-file transition, in both
directions, mail-as-file being namely MailDir. So we can simply aim at
a NotMuch <-> MailDir synchronisation, offlineimap will take care of
IMAP itself.

Of course, my proposal does not require to implement any MailDir
specific logic inside NotMuch, but rather describes how should notmuch
evolve to allow easy third-party-tool jobs.


Preliminary thoughts :
----------------------

First of all, processing mail with MUA involves some simple logic that
is shared by most MUA. This is about receiving *new* mails, *reading*
mail, *replying* to mail and so on... IMHO, this really belongs to the
mail processing logic and not to the user logic. Hence my first
request :

  1: Don't use user tags space to store MUA flags.

     That means no more "seen", "unread", "replied" as tags. These are
     MUA processing *flags*, that must belong to an established set.
     Tags, on the other hand, are user-land information. So no more
     [seen, replied, grandma, important] tag sets :)

Once this is done, notmuch will know, in a robust a predictable way,
what happened to a mail. Simply put, NotMuch will store and expose MUA
flags (Passed, Replied, Seen, Trashed, Draft, and Flagged [1]). For each
<flag>, notmuch should associate a <flag>_synced flag. When changing
<flag> from notmuch, it should set the <flag>_synced bit to 0. These are
MUA mail flags.

Additionally, in a third « space », notmuch should store its « new »
bit, as well as a « missing » bit probably. Again, this is neither MUA
logic or user logic, so this should not interfer with user
classification facility provided by tags, nor with MUA flags. It,
really, is something else, let's name it "status". Once this is done,
the 'notmuch new' command should find new mails and set the 'new' bit
for them, and find the missing mails and set the 'missing' bit for them.
This will allow for robust external processing.

Finally, notmuch should provide a switch to output a list of filenames
to stdout and to process a list of filenames from stdin.


NotMuch <-> MailDir synchronisation :
-------------------------------------

Provided the behaviour described above, notmuch <-> MailDir
synchronisation could be done fully externally, by a 'notmuch-maildir'
adapter.

Here is some pseudo code, that could be wrapped into a single
'notmuch-sync' command. The | are unix stream pipes, and everything
should be on a single line.

# 1/ Sync from NotMuch to MailDir

    notmuch list flags:(seen and not seen_synced) 
      | notmuch-maildir --mark-mail seen
      | notmuch move --stdin
      | notmuch set flags:+seen_synced --stdin

The output of the first command would be a list of filenames for mails
'seen' since last sync. The second one, an external notmuch--maildir
helper, would propagate this logic to the MailDir store (easy, this is
simply a rename), and output the list of (old-name ! new-name). Then
notmuch would use this information, via a generic 'move' switch, to know
that mail has been moved, and would output the list of the new places.
Finaly, notmuch would set the seen_synced flag.

Same would apply for the Replied, Trashed, Flagged and Passed flags.

# 2/ Then lets do the MailDir <-> IMAP sync

     offlineimap

... done ! that was easy :)

# 3/ notmuch new

     notmuch new

At this point, notmuch should set the 'new' or the 'missing' status bit
to the mails. Let's forget how to deal with the missing bit, that should
be easy to do.

# 4/ Sync from MailDir to NotMuch

  notmuch list status:new 
   | notmuch-maildir --filter seen
   | notmuch set flags:+seen+seen_synced --stdin

First command outputs newly discovered mail. Second one reads stdin and
output only files that are already seen (again, this is as easy as
a filter based on a regular expression). Third one reads stdin and
applies the seen and seen_synced flags.

Same applies for the Replied, Trashed, Flagged and Passed flags.


Conclusion:
-----------

As we have seen, this would allow most IMAP <-> MailDir <-> NotMuch
synchronisation, without having to implement any kind of
MailDir-specific logic inside notmuch. In fact, this notmuch-maildir
helper would be a simple script, and we could imagine doing similar
script for other stores, without having to touch the core of notmuch.


That was a long mail indeed, thank you for reading ! I'm waiting for
your comments.



Footnotes: 
[1]  http://cr.yp.to/proto/maildir.html

-- 
  Paul

Re: Some thoughts about notmuch sync with other agents

[Michal Sojka]

On Wednesday 27 of January 2010 15:16:07 Paul R wrote:
> [...] 
> Conclusion:
> -----------
> 
> As we have seen, this would allow most IMAP <-> MailDir <-> NotMuch
> synchronisation, without having to implement any kind of
> MailDir-specific logic inside notmuch. In fact, this notmuch-maildir
> helper would be a simple script, and we could imagine doing similar
> script for other stores, without having to touch the core of notmuch.

Hi,

I mostly agree with you. I'd also like to use offlineimap with notmuch and 
have all "flags" synchronized. I'm only not sure whether it is reasonable to 
put all MailDir logic out of notmuch. Currently, there already is a piece of 
MailDir logic in notmuch-new.c in _entries_resemble_maildir().

I think there might be an option in .notmuch-config which enables MailDir 
handling in the way as you propose.

Michal

Re: Some thoughts about notmuch sync with other agents

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 4876 bytes --]

also sprach Paul R <paul.r.ml@gmail.com> [2010.01.28.0316 +1300]:
> As you see, I advocate a NotMuch <-> IMAP synchronisation ASAP :)

Given the limitations of IMAP (non-transactional, non-standard
keywords implementation, …), I think chances of this are rapidly
diminishing, but I'd love to be proven wrong.

> First of all, processing mail with MUA involves some simple logic that
> is shared by most MUA. This is about receiving *new* mails, *reading*
> mail, *replying* to mail and so on... IMHO, this really belongs to the
> mail processing logic and not to the user logic. Hence my first
> request :
> 
>   1: Don't use user tags space to store MUA flags.
> 
>      That means no more "seen", "unread", "replied" as tags. These are
>      MUA processing *flags*, that must belong to an established set.
>      Tags, on the other hand, are user-land information. So no more
>      [seen, replied, grandma, important] tag sets :)

I disagree.

The MUA actually doesn't (shouldn't) care at all about any of these
flags, at least not for core functionality. Sure, a MUA should
probably hide messages tagged 'deleted', and it would be nice if it
could be configured to highlight messages tagged 'important', but
none of the others — "seen", "unread", "replied", … — have any role
in the core functionality. They are purely user-specific.

They are leftovers from days when some MUA designer decided that
having these flags would be a useful way to organise e-mail
handling, but that person probably dealt with a dozen messages
a day, and didn't have half his/her life organised through
electronic mail.

Point being, times and needs have changed. And while we're in the
process of finding the technology that can suit those needs (in the
most generic way), we might just as well (and should) rid ourselves
from these leftovers.

Any solution must be generic enough so that if you rely on the
functionality provided by these tags, you can trivially make it
happen, e.g. with hooks to add/remove flags on certain actions (such
as sending a reply, or reading a message).

Neither IMAP nor Maildir is capable of storing an extensible,
freely-configurable set of tags (keywords). Therefore, we need a new
method anyway.

> Once this is done, notmuch will know, in a robust a predictable
> way, what happened to a mail. Simply put, NotMuch will store and
> expose MUA flags (Passed, Replied, Seen, Trashed, Draft, and
> Flagged [1]). For each <flag>, notmuch should associate
> a <flag>_synced flag. When changing <flag> from notmuch, it should
> set the <flag>_synced bit to 0. These are MUA mail flags.

I think the semantics would be clearer the other way around: setting
*_changed when a flag is changed.

> Additionally, in a third « space », notmuch should store its « new »
> bit, as well as a « missing » bit probably. Again, this is neither MUA
> logic or user logic, so this should not interfer with user
> classification facility provided by tags, nor with MUA flags. It,
> really, is something else, let's name it "status".

Or "lifetime".

> Once this is done, the 'notmuch new' command should find new mails
> and set the 'new' bit for them, and find the missing mails and set
> the 'missing' bit for them. This will allow for robust external
> processing.

Why would I want to keep around a record in the database when the
physical file is no longer present?

> # 1/ Sync from NotMuch to MailDir
> 
>     notmuch list flags:(seen and not seen_synced) 
>       | notmuch-maildir --mark-mail seen
>       | notmuch move --stdin
>       | notmuch set flags:+seen_synced --stdin

Have you seen/look at notmuchsync?

> The output of the first command would be a list of filenames for mails
> 'seen' since last sync. The second one, an external notmuch--maildir
> helper, would propagate this logic to the MailDir store (easy, this is
> simply a rename), and output the list of (old-name ! new-name). Then
> notmuch would use this information, via a generic 'move' switch, to know
> that mail has been moved, and would output the list of the new places.
> Finaly, notmuch would set the seen_synced flag.

What happens if notmuch-move gets killed due to out-of-memory half
way through?

> As we have seen, this would allow most IMAP <-> MailDir <-> NotMuch
> synchronisation, without having to implement any kind of
> MailDir-specific logic inside notmuch.

It would not take care of any tags beyond the very strictly defined
and limited set of tags you listed in your mail. My point is that we
need to solve this problem more generally anyway — why should
a "replied" tag be synchronised, but a "no-need-to-reply" tag not?

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
#include <signature.h>
 
spamtraps: madduck.bogus@madduck.net

[-- Attachment #2: Digital signature (see http://martin-krafft.net/gpg/) --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Some thoughts about notmuch sync with other agents

[Sebastian Spaeth]

[-- Attachment #1: Type: text/plain, Size: 1977 bytes --]

REPOSTING, as the previous reply went to Gmane

> One may
> or may not like IMAP for good reasons, the fact is that it is here and
> has allowed users to read mails from various places and terminals,
> keeping important information synced. So I think that notmuch will have
> to live with that, and provide what is required to integrate smoothly
> into this environment.

agreed

> First of all, processing mail with MUA involves some simple logic that
> is shared by most MUA. This is about receiving *new* mails, *reading*
> mail, *replying* to mail and so on... IMHO, this really belongs to the
> mail processing logic and not to the user logic. Hence my first
> request :
>   1: Don't use user tags space to store MUA flags.

I don't really see what you gain by this proposal. I don't necessarily
think it's bad either, but there is no advantage IMHO.

What you propose is basically that those tags are not directly seen by
the user and not modifiable by the user. What's the advantage over
having a small set of "reserved tag keywords"? Your notmuch-capable
editor could still chose to not display "unread" and "deleted" tags but
simply hide deleted mails and show unread mails in bold or so. I simply
don't think it's important how tags are stored and the current scheme is
dead simple which makes it pretty nice.
 
>      That means no more "seen", "unread", "replied" as tags. These are
>      MUA processing *flags*, that must belong to an established set.
>      Tags, on the other hand, are user-land information. So no more
>      [seen, replied, grandma, important] tag sets :)

See above, the mail editor might chose to hide thos MUA processing
flags, but I don't think it is important whether they are hidden by the
user or now. 

Other than that, I agree that "notmuch new" should set a "new" and/or
"modified"tag if it detects a new mail (or a modified one), rather than
the current set of fixed tags that aren't too helpful for 3rd party
scripts.

Sebastian 

[-- Attachment #2: Type: text/plain, Size: 78 bytes --]

Date: Mon, 01 Feb 2010 15:54:57 +0100
Message-ID: <871vh557lq.fsf@SSpaeth.de>