From Gnus to mu4e

From Gnus to mu4e

[Julien Cubizolles]

I'm considering moving from Gnus to mu4e, mostly because I spend too
much time tinkering with settings I don't really understand/need in
Gnus, and because the search feature seems promising. If some of you
have already made such a move, or are just happy users of mu4e, I'd
appreciate your input on the following issues.

* Can OfflineImap or whatever imap sync tool used exclude some folders
  in the main Maildir folder from being synced to the IMAP server ? I'd
  like to be able to archive some old mail without it being sent to
  Gmail.

* I'm using gmane to read a lot of mailing lists. I guess there's no way
  to do that with mu4e and I'm considering sticking with Gnus for that
  use. Does that seem like a reasonable solution ? 

* Can mu4e use the Gmail labels for search queries ?

* If a message is tagged with several labels on Gmail, does it appear in
  several folders in the local repository ? On a related note, if one
  were to move a message from a folder in a repository synced with Gmail
  to another folder not synced (for archiving purposes), I guess the
  message would remain at least in the [Gmail]/All Mail folder. Is there
  an easy way to remove it altogether from Gmail ?

* Finally, the mailing list for mu4e doesn't seem very active
  (especially when you compare it to the one for gnus). Are there any
  other channels to get some help ?

Thanks for your input.

Julien.

Re: From Gnus to mu4e

[Julien Cubizolles]

Somebody replied by mail to this message to give some advice, but my
gnus splitting isn't very fancy these days and I somehow lost it... I
just wanted to thank him.

Julien.

Re: From Gnus to mu4e

[Colin Yates]

I think that was me, here is the reply:

Not much to add as I am a newbie here, but https://github.com/skeeto/elfeed
is a similar tool for rss feeds.

I do use mu4e but I use mbsync which seems to have better google-karma than
offlineimap. It also allows you to choose which imap folders to pull down.

I am actually investigating notmuch as I think it has a getter story around
tagging emails; mu4e is folder based.  Actually, when I have more time I am
going to look into using org-mode and links for organising...

I wasn't being self-deprecating with the newbie warning so take the above
with a healthy dose of salt :-)

Re: From Gnus to mu4e

[Suvayu Ali]

Hi,

I think I can answer some of your queries.

On Thu, Aug 20, 2015 at 11:23:36PM +0200, Julien Cubizolles wrote:
> 
> * Can OfflineImap or whatever imap sync tool used exclude some folders
>   in the main Maildir folder from being synced to the IMAP server ? I'd
>   like to be able to archive some old mail without it being sent to
>   Gmail.

Yes, it is possible.  Just use a folderfilter.  Something like this:

folderfilter = lambda foldername: foldername in ['INBOX', 'mysent',
	       'mailinglist', 'org-mode', 'mutt', 'Emacs', 'Tux', 'XFCE',
	       'bash', 'notmuch', 'travel', '[Google Mail]/Drafts']

So Colin's statement about mbsync being better than OfflineIMAP because
of this reason, is inaccurate.  Although, I have heard mbsync performs
better than OfflineIMAP, I still have to test this though.

> * Can mu4e use the Gmail labels for search queries ?

Probably not directly, but if it allows for limiting by folder name, it
should be possible.  You can do this with notmuch (I use notmuch).

> * If a message is tagged with several labels on Gmail, does it appear in
>   several folders in the local repository ? On a related note, if one
>   were to move a message from a folder in a repository synced with Gmail
>   to another folder not synced (for archiving purposes), I guess the
>   message would remain at least in the [Gmail]/All Mail folder. Is there
>   an easy way to remove it altogether from Gmail ?

There are many subtleties when it comes to Gmail's implementation of the
IMAP interface.  I would recommend reading these two articles:

- https://support.google.com/mail/answer/77657?hl=en
- https://support.google.com/mail/answer/78755?hl=en

That said, in Gmail, you need something like this:

When a message is marked as deleted and expunged from the last visible IMAP folder:
- [ ] Archive the message (default)
- [X] Move the message to the Bin
- [ ] Immediately delete the message forever

You can also set the last option, choosing the default will leave it in
Gmail.  On top of this, you should make sure the All Mail folder is
_not_ accessible over IMAP.  This is important.  As long as the message
is present in at least one IMAP folder, Gmail will retain it.  This is
where it becomes important how you choose to delete it.  With notmuch,
when you delete a message the delete flag will be forwarded to all the
respective messages in the maildir, which in turn ensures it is deleted
from Gmail the next time you sync.  Maybe mu4e does something similar.

Hope this helps,

-- 
Suvayu

Open source is the future. It sets us free.

Re: From Gnus to mu4e

[Joost Kremers]

Julien Cubizolles wrote:
> * Can OfflineImap or whatever imap sync tool used exclude some folders
>   in the main Maildir folder from being synced to the IMAP server ? I'd
>   like to be able to archive some old mail without it being sent to
>   Gmail.

Yes, using `folderfilter`, which can be a lambda (offlineimaprc is
basically just Python). So I have something like:

folderfilter = lambda foldername: not re.search('(^Deleted|^Junk|^Tasks|^Calendar|^Notes|^Contacts|^Drafts)', foldername)

which excludes folders matching the regexp in re.search.

> * Can mu4e use the Gmail labels for search queries ?

There's bin quite some discussion about that on the mu4e mailing list,
and AFAIK there is a workable solution for it. But I don't use it
myself, so I can't really say. I'd check the archives and ask on the
mu4e mailing list if you still have questions.

> * If a message is tagged with several labels on Gmail, does it appear in
>   several folders in the local repository ? On a related note, if one
>   were to move a message from a folder in a repository synced with Gmail
>   to another folder not synced (for archiving purposes), I guess the
>   message would remain at least in the [Gmail]/All Mail folder. Is there
>   an easy way to remove it altogether from Gmail ?

This, too, has been discussed on the mailing list, IIRC.

> * Finally, the mailing list for mu4e doesn't seem very active
>   (especially when you compare it to the one for gnus). Are there any
>   other channels to get some help ?

Perhaps not compared to the gnus mailing list, but I'm subscribed and
can say that the list is active enough. Also, the mu/mu4e maintainer
reads the list and answers questions.



-- 
Joost Kremers                                   joostkremers@fastmail.fm
Selbst in die Unterwelt dringt durch Spalten Licht
EN:SiS(9)

Re: From Gnus to mu4e

[Bastien]

Julien Cubizolles <j.cubizolles@free.fr> writes:

> I'm considering moving from Gnus to mu4e, mostly because I spend too
> much time tinkering with settings I don't really understand/need in
> Gnus, and because the search feature seems promising.

After a *lot* of scratching and testing, here is my setup:

- Gnus for news and mails: because there are too many things I miss in
  other MUA.  A simple example: quote only the selected region when
  replying.  This cannot be implemented easily in mu and notmuch.

- notmuch as a search engine: mu is really great (I used it the last
  three months) but I cannot get used to mu4e.  And starting mu4e with
  a database of 100K emails takes two or three seconds.

- mbsync to sync imap folders as local maildirs: I used offlineimap
  for several years and mbsync is just faster.

- postfix with smtp_sender_dependent_authentication to use various
  smtp servers depending on the email I'm using.  I see people use
  msmtp and msmtp-mta, which I tried for a while, but I'm just too
  impatient with the .3s you have to wait when sending.  Will
  reconsider in a decade, when Emacs has more asynchronous power :)

This is what I'm sticking too now and I'm fine.

I was fine for years with a dummy fetchmail+nnml setup... many
groups, many posting styles, no complexity.

Then I obeyed to the social pressure and moved to all the dovecot,
offlineimap, stuff -- and yes, OT discussions on the orgmode mailing
lists pushed me in this directions!

I must say that I was very frustrated for months that I didn't stick
longer to this setup.  Especially because I don't use fancy phones and
I don't really need to go "mobile" or to sync many devices... but now
I'm a grown up and I can haz another device that I'll sync in peace.

Re: From Gnus to mu4e

[Ian Zimmerman]

On 2015-08-25 07:49 +0200, Bastien wrote:

> After a *lot* of scratching and testing, here is my setup:
> 
> - Gnus for news and mails: because there are too many things I miss in
>   other MUA.  A simple example: quote only the selected region when
>   replying.  This cannot be implemented easily in mu and notmuch.

I completely respect your choices - this one and the others.  But I
still would like to point out that this may also be an example of
something else.

Choices of this kind are always a tradeoff.  Great as Gnus is I don't
think it is perfect, or else I'd use it :-)  So, how bad are the
problems with Gnus, and how bad are the problems with the others?  The
missing feature you mention doesn't strike _me_ as a particularly
serious problem.  After all, you have to select the region first, so
it's about the same amount of work as deleting the unwanted material in
the reply.  

So if this were the _only_ problem with MUA X, it would not make me drop
it and use Gnus.  But YMMV!

-- 
Please *no* private copies of mailing list or newsgroup messages.
Rule 420: All persons more than eight miles high to leave the court.

Re: From Gnus to mu4e

[Bastien Guerry]

Ian Zimmerman <itz@buug.org> writes:

> So if this were the _only_ problem with MUA X, it would not make me drop
> it and use Gnus.  But YMMV!

Yes, sure, I'm not saying Gnus is better, just saying that I'm too
used to it.  The "switching cost" is too high for me.  Or maybe I'm
just too old! :)

Re: From Gnus to mu4e

[Julien Cubizolles]

Suvayu Ali <fatkasuvayu+linux@gmail.com> writes:

>> * If a message is tagged with several labels on Gmail, does it appear in
>>   several folders in the local repository ? On a related note, if one
>>   were to move a message from a folder in a repository synced with Gmail
>>   to another folder not synced (for archiving purposes), I guess the
>>   message would remain at least in the [Gmail]/All Mail folder. Is there
>>   an easy way to remove it altogether from Gmail ?
>
> There are many subtleties when it comes to Gmail's implementation of the
> IMAP interface.  I would recommend reading these two articles:
>
> - https://support.google.com/mail/answer/77657?hl=en
> - https://support.google.com/mail/answer/78755?hl=en
>
> That said, in Gmail, you need something like this:
>
> When a message is marked as deleted and expunged from the last visible IMAP folder:
> - [ ] Archive the message (default)
> - [X] Move the message to the Bin
> - [ ] Immediately delete the message forever
>
> You can also set the last option, choosing the default will leave it in
> Gmail.  On top of this, you should make sure the All Mail folder is
> _not_ accessible over IMAP.  This is important.  As long as the message
> is present in at least one IMAP folder, Gmail will retain it.  This is
> where it becomes important how you choose to delete it.  With notmuch,
> when you delete a message the delete flag will be forwarded to all the
> respective messages in the maildir, which in turn ensures it is deleted
> from Gmail the next time you sync.  Maybe mu4e does something similar.

Thanks a lot for this info: It will be useful regardless of the MUA I
use.

Julien.

Re: From Gnus to mu4e

[Julien Cubizolles]

Bastien <bzg@gnu.org> writes:

> Julien Cubizolles <j.cubizolles@free.fr> writes:
>
>> I'm considering moving from Gnus to mu4e, mostly because I spend too
>> much time tinkering with settings I don't really understand/need in
>> Gnus, and because the search feature seems promising.
>
> After a *lot* of scratching and testing, here is my setup:
>
> - Gnus for news and mails: because there are too many things I miss in
>   other MUA.  A simple example: quote only the selected region when
>   replying.  This cannot be implemented easily in mu and notmuch.
>
> - notmuch as a search engine: mu is really great (I used it the last
>   three months) but I cannot get used to mu4e.  And starting mu4e with
>   a database of 100K emails takes two or three seconds.
>
> - mbsync to sync imap folders as local maildirs: I used offlineimap
>   for several years and mbsync is just faster.

That seems very interesting, and I think I'll try to set it
up. offlineimap or mbsync seems to be the right way to:
  + have a fast access to my mail (nnimap from Gnus on a remote server
  is a bit slow sometimes)

  + have a complete access from my mobile phone

  + I can still choose whatever MUA I want on top of that if I want to
  try something else thant Gnus and still access my old nnml mail.

I have a few questions about some details of your setup:

  + regarding filtering and spam detection, do you rely on the
  possibilities offered by you mail provider (Gmail or other)?

  + you said you're running Dovecot. Gnus could also directly access the
  local maildir without a local imap server the way mu4e does, couldn't
  it ? What's the benefit of running it ?

> - postfix with smtp_sender_dependent_authentication to use various
>   smtp servers depending on the email I'm using.  I see people use
>   msmtp and msmtp-mta, which I tried for a while, but I'm just too
>   impatient with the .3s you have to wait when sending.  

  From looking at my fossilized .gnus.el, I'm using smtpmail, which
  would be equivalent to msmtp.

  + I'm gonna need to duplicate this setup on two machines (laptop and
  desktop). I guess running offline imap on the two machines and
  synchronizing the nnml, nnarchive mail by unison would work.

Thanks a lot for all this information. I've a much clearer idea of all
the possibilities now.

Julien.

Re: From Gnus to mu4e

[Rainer M Krug]

[-- Attachment #1: Type: text/plain, Size: 5700 bytes --]

Julien Cubizolles <j.cubizolles@free.fr> writes:

> Bastien <bzg@gnu.org> writes:
>
>> Julien Cubizolles <j.cubizolles@free.fr> writes:
>>
>>> I'm considering moving from Gnus to mu4e, mostly because I spend too
>>> much time tinkering with settings I don't really understand/need in
>>> Gnus, and because the search feature seems promising.
>>
>> After a *lot* of scratching and testing, here is my setup:
>>
>> - Gnus for news and mails: because there are too many things I miss in
>>   other MUA.  A simple example: quote only the selected region when
>>   replying.  This cannot be implemented easily in mu and notmuch.
>>
>> - notmuch as a search engine: mu is really great (I used it the last
>>   three months) but I cannot get used to mu4e.  And starting mu4e with
>>   a database of 100K emails takes two or three seconds.
>>
>> - mbsync to sync imap folders as local maildirs: I used offlineimap
>>   for several years and mbsync is just faster.

Sorry - I am late to this question.

I am using the same setup, only with offlineimap - it works, and speed
is not such a problem for me so that I would move to mbsync.

>
> That seems very interesting, and I think I'll try to set it
> up. offlineimap or mbsync seems to be the right way to:
>   + have a fast access to my mail (nnimap from Gnus on a remote server
>   is a bit slow sometimes)
>
>   + have a complete access from my mobile phone
>
>   + I can still choose whatever MUA I want on top of that if I want to
>   try something else thant Gnus and still access my old nnml mail.

Exactly - and you can easily make backups from your all your emails in a
format which will be supported for quite some time.

>
> I have a few questions about some details of your setup:
>
>   + regarding filtering and spam detection, do you rely on the
>   possibilities offered by you mail provider (Gmail or other)?

I am using gmail for this - thought for some minutes about implementing
it locally, but gmail works reliable.

>
>   + you said you're running Dovecot. Gnus could also directly access the
>   local maildir without a local imap server the way mu4e does, couldn't
>   it ? What's the benefit of running it ?

There is one program accessing the mail folder: dovecot - no conflicts, race
conditions, no slightly different formats, standards, ... And you can
also remotely access dovecot if you want to.

By the way: the dovecot daemon is not permanently running, as I am using
the shell access from gnus:

,----
| (nnimap-shell-program "/usr/local/opt/dovecot/libexec/dovecot/imap -o mail_location=maildir:$HOME/Maildir")))
`----

The same from offlineimap:

,----
| preauthtunnel = /usr/local/opt/dovecot/libexec/dovecot/imap -o mail_location=maildir:$HOME/Maildir
`----

and offlineimap only reads the emails from the maildir.

And I have found the following two functions to open the same message
which is open in gnus in notmuch and the other way around:

--8<---------------cut here---------------start------------->8---
** In notmuch jump to gnus folder
From [[http://roland.entierement.nu/blog/2010/09/08/gnus-dovecot-offlineimap-search-a-howto.html]]:
  But seeing just the message that matches is not enough, 
  sometimes we want the whole thread. Here's a snippet of 
  Lisp for your .gnus, based on Tassilo Horn's configuration. 
  Do a Notmuch search, enter one of the results, type C-c C-c, 
  you'll get transported to the folder where that message was, 
  with the context. Note this requires code from org-mode, 
  so you might need to install that.

#+begin_src emacs-lisp

(defun rmk/notmuch-file-to-group (file)
  "Calculate the Gnus group name from the given file name.
  "
  (let ((group (file-name-directory (directory-file-name (file-name-directory file)))))
    (setq group (replace-regexp-in-string ".*/Maildir/" "nnimap+Maildir:" group))
    (setq group (replace-regexp-in-string "/$" "" group))
    (if (string-match ":$" group)
        (concat group "INBOX")
      (replace-regexp-in-string ":\\." ":" group))))

(defun rmk/notmuch-goto-message-in-gnus ()
  "Open a summary buffer containing the current notmuch
  article."
  (interactive)
  (let ((group (rmk/notmuch-file-to-group (notmuch-show-get-filename)))
        (message-id (replace-regexp-in-string
                     "^id:" "" (notmuch-show-get-message-id))))
    (setq message-id (replace-regexp-in-string "\"" "" message-id))
    (if (and group message-id)
        (progn 
          (switch-to-buffer "*Group*")
          (org-gnus-follow-link group message-id))
      (message "Couldn't get relevant infos for switching to Gnus."))))

(define-key notmuch-show-mode-map (kbd "C-c C-c") 'rmk/notmuch-goto-message-in-gnus)
#+end_src
--8<---------------cut here---------------end--------------->8---

Hope this helps,

Rainer

>
>> - postfix with smtp_sender_dependent_authentication to use various
>>   smtp servers depending on the email I'm using.  I see people use
>>   msmtp and msmtp-mta, which I tried for a while, but I'm just too
>>   impatient with the .3s you have to wait when sending.  
>
>   From looking at my fossilized .gnus.el, I'm using smtpmail, which
>   would be equivalent to msmtp.
>
>   + I'm gonna need to duplicate this setup on two machines (laptop and
>   desktop). I guess running offline imap on the two machines and
>   synchronizing the nnml, nnarchive mail by unison would work.
>
> Thanks a lot for all this information. I've a much clearer idea of all
> the possibilities now.
>
> Julien.
>
>
>

-- 
Rainer M. Krug
email: Rainer<at>krugs<dot>de
PGP: 0x0F52F982

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 480 bytes --]

Re: From Gnus to mu4e

[Suvayu Ali]

On Wed, Aug 26, 2015 at 09:22:26AM +0200, Julien Cubizolles wrote:
> 
> I have a few questions about some details of your setup:
> 
>   + regarding filtering and spam detection, do you rely on the
>   possibilities offered by you mail provider (Gmail or other)?

I don't know about Bastien, but for me, since I use Gmail I heavily rely
on both filtering and spam detection.  I use Gmails filtering eventhough
it is a bit clunky to avoid setting up something on my end; I find it
simpler this way.  And the spam detection, I find Gmail's spam detection
to be very good, specially when you have been training it for a few
years by reporting spam and phishing messages, as appropriate.

>   + you said you're running Dovecot. Gnus could also directly access the
>   local maildir without a local imap server the way mu4e does, couldn't
>   it ? What's the benefit of running it ?

Probably Bastien has more up to date comments, but last time I looked, I
found Gnus's support for maildir rather poor.  There were a few issues,
it didn't respect the usual maildir flags and stored its own flags
withing the maildir.  So along with incorrect flag, the mail store took
about ~70% more disk space.  It was also a bit slow, despite everything
being local.  My impression is based on the status from about 2-3 years
back, so things might have improved a lot by now.

Hope this helps,

-- 
Suvayu

Open source is the future. It sets us free.

Re: From Gnus to mu4e

[Glenn Morris]

Suvayu Ali wrote:

> Probably Bastien has more up to date comments, but last time I looked, I
> found Gnus's support for maildir rather poor.  There were a few issues,
> it didn't respect the usual maildir flags and stored its own flags
> withing the maildir.  So along with incorrect flag, the mail store took
> about ~70% more disk space.  It was also a bit slow, despite everything
> being local.  My impression is based on the status from about 2-3 years
> back, so things might have improved a lot by now.

Flag handling at least is better since Emacs 24.3
(http://debbugs.gnu.org/8055).

Re: From Gnus to mu4e

[Bastien]

Julien Cubizolles <j.cubizolles@free.fr> writes:

>   + regarding filtering and spam detection, do you rely on the
>   possibilities offered by you mail provider (Gmail or other)?

For providers that do spam filtering, obviously yes.  For others, I
first fetch emails on a server using fetchmail/spamassassin, then
expose these emails using dovecot on the server.
>
>   + you said you're running Dovecot. Gnus could also directly access the
>   local maildir without a local imap server the way mu4e does, couldn't
>   it ? What's the benefit of running it ?

Sorry I wasn't clear.

I have

1. an address from my internet provider
2. an address from my own remote server

multiple addresses are collected on 2 (using fetchmail/spamassassin)
and dovecot is running on this remote server.

I use maildir for both 1 and 2 on my local machine, sync'ing all
mails with mbsync.

>> - postfix with smtp_sender_dependent_authentication to use various
>>   smtp servers depending on the email I'm using.  I see people use
>>   msmtp and msmtp-mta, which I tried for a while, but I'm just too
>>   impatient with the .3s you have to wait when sending.  
>
>   From looking at my fossilized .gnus.el, I'm using smtpmail, which
>   would be equivalent to msmtp.
>
>   + I'm gonna need to duplicate this setup on two machines (laptop and
>   desktop). I guess running offline imap on the two machines and
>   synchronizing the nnml, nnarchive mail by unison would work.

I guess so, yes.

> Thanks a lot for all this information. I've a much clearer idea of all
> the possibilities now.

You're welcome!

PS: I secretly love anything related to emails, this is fascinating.

-- 
 Bastien

Re: From Gnus to mu4e

[Bastien]

Hi Suvayu,

Suvayu Ali <fatkasuvayu+linux@gmail.com> writes:

>>   + you said you're running Dovecot. Gnus could also directly access the
>>   local maildir without a local imap server the way mu4e does, couldn't
>>   it ? What's the benefit of running it ?
>
> Probably Bastien has more up to date comments, but last time I looked, I
> found Gnus's support for maildir rather poor.  There were a few issues,
> it didn't respect the usual maildir flags and stored its own flags
> withing the maildir.  So along with incorrect flag, the mail store took
> about ~70% more disk space.  It was also a bit slow, despite everything
> being local.  My impression is based on the status from about 2-3 years
> back, so things might have improved a lot by now.

I find it reasonably OK right now, but this is very subjective of course.

-- 
 Bastien

Re: From Gnus to mu4e

[Bastien]

Hi Rainer,

Rainer M Krug <Rainer@krugs.de> writes:

> There is one program accessing the mail folder: dovecot - no conflicts, race
> conditions, no slightly different formats, standards, ... And you can
> also remotely access dovecot if you want to.
>
> By the way: the dovecot daemon is not permanently running, as I am using
> the shell access from gnus:
>
> ,----
> | (nnimap-shell-program "/usr/local/opt/dovecot/libexec/dovecot/imap -o mail_location=maildir:$HOME/Maildir")))
> `----
>
> The same from offlineimap:
>
> ,----
> | preauthtunnel = /usr/local/opt/dovecot/libexec/dovecot/imap -o mail_location=maildir:$HOME/Maildir
> `----

Interesting -- can you detail this parts of your configuration?

I am not using dovecot locally so far, just nnmaildir from Gnus.

But I've encountered some annoying race conditions, while sync'ing
through mbsync *and* reading/replying from Gnus, so I'm interested
in knowing how to avoid them -- ideally letting mbsync to run in
the background.

Thanks!

-- 
 Bastien

Re: From Gnus to mu4e

[Suvayu Ali]

On Thu, Aug 27, 2015 at 07:08:47PM +0200, Bastien wrote:
> 
> PS: I secretly love anything related to emails, this is fascinating.

Me too!

-- 
Suvayu

Open source is the future. It sets us free.

Re: From Gnus to mu4e

[Suvayu Ali]

On Thu, Aug 27, 2015 at 11:52:05AM -0400, Glenn Morris wrote:
> Suvayu Ali wrote:
> 
> > Probably Bastien has more up to date comments, but last time I looked, I
> > found Gnus's support for maildir rather poor.  There were a few issues,
> > it didn't respect the usual maildir flags and stored its own flags
> > withing the maildir.  So along with incorrect flag, the mail store took
> > about ~70% more disk space.  It was also a bit slow, despite everything
> > being local.  My impression is based on the status from about 2-3 years
> > back, so things might have improved a lot by now.
> 
> Flag handling at least is better since Emacs 24.3
> (http://debbugs.gnu.org/8055).
> 

Thanks Glenn.  Maybe one of these days I'll end up experimenting again.

:)

-- 
Suvayu

Open source is the future. It sets us free.

Re: From Gnus to mu4e

[Julien Cubizolles]

Rainer M Krug <Rainer@krugs.de> writes:


> There is one program accessing the mail folder: dovecot - no conflicts, race
> conditions, no slightly different formats, standards, ... And you can
> also remotely access dovecot if you want to.

Because otherwise, you would have Gnus and offlineimap competing for
access ?

> By the way: the dovecot daemon is not permanently running, as I am using
> the shell access from gnus:
>
> ,----
> | (nnimap-shell-program "/usr/local/opt/dovecot/libexec/dovecot/imap -o mail_location=maildir:$HOME/Maildir")))
> `----
>
> The same from offlineimap:
>
> ,----
> | preauthtunnel = /usr/local/opt/dovecot/libexec/dovecot/imap -o mail_location=maildir:$HOME/Maildir
> `----
>
> and offlineimap only reads the emails from the maildir.

That looks interesting, I'll try it.

> And I have found the following two functions to open the same message
> which is open in gnus in notmuch and the other way around:
>
> ** In notmuch jump to gnus folder

I think you forgot the other way around :-)

> Hope this helps,

I'm sure it will, thanks a lot.

Julien.

Re: From Gnus to mu4e

[Julien Cubizolles]

Bastien <bzg@gnu.org> writes:


> Sorry I wasn't clear.
>
> I have
>
> 1. an address from my internet provider
> 2. an address from my own remote server
>
> multiple addresses are collected on 2 (using fetchmail/spamassassin)
> and dovecot is running on this remote server.
>
> I use maildir for both 1 and 2 on my local machine, sync'ing all
> mails with mbsync.

And you access the maildir on your local machine directly with nnimap
from Gnus, without dovecot ?

> You're welcome!
>
> PS: I secretly love anything related to emails, this is fascinating.

I can see that ? As for me, I'm starting to be a little bit frightened
by all I'm gonna have to get running. So far I'm collecting info, and
will put it together when I have a few days free. This list has been, as
always a source of precious help, thanks a lot.

Julien.

Re: From Gnus to mu4e

[Bastien]

Julien Cubizolles <j.cubizolles@free.fr> writes:

> And you access the maildir on your local machine directly with nnimap
> from Gnus, without dovecot ?

No, I use the nnmaildir backend from Gnus directly.

-- 
 Bastien

Re: From Gnus to mu4e

[Julien Cubizolles]

Bastien <bzg@gnu.org> writes:

> Julien Cubizolles <j.cubizolles@free.fr> writes:
>
>> And you access the maildir on your local machine directly with nnimap
>> from Gnus, without dovecot ?
>
> No, I use the nnmaildir backend from Gnus directly.

My bad, I meant nnmaildir of course.

Re: From Gnus to mu4e

[Suvayu Ali]

On Thu, Aug 27, 2015 at 11:27:50PM +0200, Julien Cubizolles wrote:
> Rainer M Krug <Rainer@krugs.de> writes:
> 
> > There is one program accessing the mail folder: dovecot - no conflicts, race
> > conditions, no slightly different formats, standards, ... And you can
> > also remotely access dovecot if you want to.
> 
> Because otherwise, you would have Gnus and offlineimap competing for
> access ?

I do not think this is possible.  In fact AFAIK, any kind of race
condition in a maildir is not possible by design, as long as the
filesystem operations are atomic.

-- 
Suvayu

Open source is the future. It sets us free.

Re: From Gnus to mu4e

[Ian Zimmerman]

On 2015-08-28 01:34 +0200, Suvayu Ali wrote:

> > Because otherwise, you would have Gnus and offlineimap competing for
> > access ?

> I do not think this is possible.  In fact AFAIK, any kind of race
> condition in a maildir is not possible by design, as long as the
> filesystem operations are atomic.

I think this applies to the original delivery, but not necessarily to
subsequent manipulation of maildirs.

For example, the message flags are stored by appending them to the file
name.  If two MUAs concurrently scan the maildir for messages, then try
to set a flag on one message, a race is possible.

I observed wierdness when I tried to access my mail both by IMAP (the
with Dovecot as the IMAP server) and directly reading the underlying
maildirs with mutt.  That was part of the reason why I gave up on IMAP
entirely, and instead read over ssh and mutt locally on the server.

-- 
Please *no* private copies of mailing list or newsgroup messages.
Rule 420: All persons more than eight miles high to leave the court.

Re: From Gnus to mu4e

[Suvayu Ali]

On Thu, Aug 27, 2015 at 04:50:23PM -0700, Ian Zimmerman wrote:
> On 2015-08-28 01:34 +0200, Suvayu Ali wrote:
> 
> > > Because otherwise, you would have Gnus and offlineimap competing for
> > > access ?
> 
> > I do not think this is possible.  In fact AFAIK, any kind of race
> > condition in a maildir is not possible by design, as long as the
> > filesystem operations are atomic.
> 
> I think this applies to the original delivery, but not necessarily to
> subsequent manipulation of maildirs.
> 
> For example, the message flags are stored by appending them to the file
> name.  If two MUAs concurrently scan the maildir for messages, then try
> to set a flag on one message, a race is possible.
> 
> I observed wierdness when I tried to access my mail both by IMAP (the
> with Dovecot as the IMAP server) and directly reading the underlying
> maildirs with mutt.  That was part of the reason why I gave up on IMAP
> entirely, and instead read over ssh and mutt locally on the server.

No, it's not.  AFAIK on most (all?) *nix filesystems, basic operations
are atomic.  See for example this (outdated) list:

http://rcrowley.org/2010/01/06/things-unix-can-do-atomically.html

E.g., this breaks down when you talk about network mounts like NFS (not
sure about SSHFS), but it is a valid assumption as long as your storage
is local.

-- 
Suvayu

Open source is the future. It sets us free.

Re: From Gnus to mu4e

[Ian Zimmerman]

On 2015-08-28 02:14 +0200, Suvayu Ali wrote:

> No, it's not.  AFAIK on most (all?) *nix filesystems, basic operations
> are atomic.  See for example this (outdated) list:
> 
> http://rcrowley.org/2010/01/06/things-unix-can-do-atomically.html
> 
> E.g., this breaks down when you talk about network mounts like NFS (not
> sure about SSHFS), but it is a valid assumption as long as your storage
> is local.

Assume there's a message ~/Mail/inbox/cur/1440718280.10956_2.ahiker:2,S

MUA 1 wants to set the "tagged" flag, which means renaming

1440718280.10956_2.ahiker:2,S -> 1440718280.10956_2.ahiker:2,FS

MUA 2 wants to set the "replied" flag, which means renaming

1440718280.10956_2.ahiker:2,S -> 1440718280.10956_2.ahiker:2,RS

Only one of them can succeed, depending on the order they try.  And
final state also depends on that order.

-- 
Please *no* private copies of mailing list or newsgroup messages.
Rule 420: All persons more than eight miles high to leave the court.

Re: From Gnus to mu4e

[Rainer M Krug]

[-- Attachment #1: Type: text/plain, Size: 2213 bytes --]

Julien Cubizolles <j.cubizolles@free.fr> writes:

> Rainer M Krug <Rainer@krugs.de> writes:
>
>
>> There is one program accessing the mail folder: dovecot - no conflicts, race
>> conditions, no slightly different formats, standards, ... And you can
>> also remotely access dovecot if you want to.
>
> Because otherwise, you would have Gnus and offlineimap competing for
> access ?

I have not actually experienced this, but I was afraid tat, especially
simultaneous write access to the mailbox files cal lead to corruption or
errors due to simultaneous access. 

>
>> By the way: the dovecot daemon is not permanently running, as I am using
>> the shell access from gnus:
>>
>> ,----
>> | (nnimap-shell-program "/usr/local/opt/dovecot/libexec/dovecot/imap -o mail_location=maildir:$HOME/Maildir")))
>> `----
>>
>> The same from offlineimap:
>>
>> ,----
>> | preauthtunnel = /usr/local/opt/dovecot/libexec/dovecot/imap -o mail_location=maildir:$HOME/Maildir
>> `----
>>
>> and offlineimap only reads the emails from the maildir.
>
> That looks interesting, I'll try it.
>
>> And I have found the following two functions to open the same message
>> which is open in gnus in notmuch and the other way around:
>>
>> ** In notmuch jump to gnus folder
>
> I think you forgot the other way around :-)

Sorry - here is it:

,----
| ** In gnus jump to notmuch
| Thanks to [[notmuch:id:87fvn4j45h.fsf@thinkpad-t61.fritz.box][Email from Tassilo Horn: Re: How to get message-id from]]
| #+begin_src emacs-lisp
| (defun rmk/gnus-goto-message-in-notmuch ()
|   (interactive)
|   (unless (buffer-live-p gnus-summary-buffer)
|     (user-error "No open Gnus summary buffer!"))
|   (let ((message-id (substring (with-current-buffer gnus-summary-buffer
|                                  (mail-header-id (gnus-summary-article-header))) 1 -1)))
|     (message message-id)
|     (notmuch-tree (concatenate 'string "id:" message-id)) ;; Now do the notmuch stuff
|     ))
| #+end_src
`----

>
>> Hope this helps,
>
> I'm sure it will, thanks a lot.

Great.

Cheers,

Rainer

>
> Julien.
>
>

-- 
Rainer M. Krug
email: Rainer<at>krugs<dot>de
PGP: 0x0F52F982

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 480 bytes --]

Re: From Gnus to mu4e

[Rainer M Krug]

[-- Attachment #1: Type: text/plain, Size: 1446 bytes --]

Ian Zimmerman <itz@buug.org> writes:

> On 2015-08-28 02:14 +0200, Suvayu Ali wrote:
>
>> No, it's not.  AFAIK on most (all?) *nix filesystems, basic operations
>> are atomic.  See for example this (outdated) list:
>> 
>> http://rcrowley.org/2010/01/06/things-unix-can-do-atomically.html
>> 
>> E.g., this breaks down when you talk about network mounts like NFS (not
>> sure about SSHFS), but it is a valid assumption as long as your storage
>> is local.
>
> Assume there's a message ~/Mail/inbox/cur/1440718280.10956_2.ahiker:2,S
>
> MUA 1 wants to set the "tagged" flag, which means renaming
>
> 1440718280.10956_2.ahiker:2,S -> 1440718280.10956_2.ahiker:2,FS
>
> MUA 2 wants to set the "replied" flag, which means renaming
>
> 1440718280.10956_2.ahiker:2,S -> 1440718280.10956_2.ahiker:2,RS
>
> Only one of them can succeed, depending on the order they try.  And
> final state also depends on that order.

Interesting discussion - and I am learning that I do not know enough to
say that there *can* be race conditions, but I know that they are *bad*
when they occur and can lead to corruption or undefined behavior. And I
also know that I do not want this, particularly not in email.
So using dovecot to RW access the maildir is the safe way to go.
Notmuch is also accessing the maildir directly, but only read-only.

Cheers,

Rainer


-- 
Rainer M. Krug
email: Rainer<at>krugs<dot>de
PGP: 0x0F52F982

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 480 bytes --]

Re: From Gnus to mu4e

[Rainer M Krug]

[-- Attachment #1: Type: text/plain, Size: 2906 bytes --]

Bastien <bzg@gnu.org> writes:

> Hi Rainer,
>
> Rainer M Krug <Rainer@krugs.de> writes:
>
>> There is one program accessing the mail folder: dovecot - no conflicts, race
>> conditions, no slightly different formats, standards, ... And you can
>> also remotely access dovecot if you want to.
>>
>> By the way: the dovecot daemon is not permanently running, as I am using
>> the shell access from gnus:
>>
>> ,----
>> | (nnimap-shell-program "/usr/local/opt/dovecot/libexec/dovecot/imap -o mail_location=maildir:$HOME/Maildir")))
>> `----
>>
>> The same from offlineimap:
>>
>> ,----
>> | preauthtunnel = /usr/local/opt/dovecot/libexec/dovecot/imap -o mail_location=maildir:$HOME/Maildir
>> `----
>
> Interesting -- can you detail this parts of your configuration?

This is how I define my secondary select method in GNUS, which is my
local, with offlineimap synched, mail:

--8<---------------cut here---------------start------------->8---
(add-to-list 'gnus-secondary-select-methods          
                      ;; (nnimap-stream network)
                      ;; (nnimap-server-port 143)
                      ;; (nnimap-address "localhost")))
                      (nnimap-stream shell)
                      (nnimap-shell-program "/usr/local/opt/dovecot/libexec/dovecot/imap -o mail_location=maildir:$HOME/Maildir")))
--8<---------------cut here---------------end--------------->8---

To ose the daemon, you have to toggle the comments (I hope - haven't
done it in a long time...).

In offlineimap, I have the following for my local mail location:

--8<---------------cut here---------------start------------->8---
[Repository LocalRMKrugGMAIL]
type = IMAP
preauthtunnel = /usr/local/opt/dovecot/libexec/dovecot/imap -o mail_location=maildir:$HOME/Maildir
holdconnectionopen = yes
maxconnections = 3
--8<---------------cut here---------------end--------------->8---

So offlineimap uses the binary (.../imap) to connect to my maildir.

>
> I am not using dovecot locally so far, just nnmaildir from Gnus.
>
> But I've encountered some annoying race conditions, while sync'ing
> through mbsync *and* reading/replying from Gnus, so I'm interested
> in knowing how to avoid them -- ideally letting mbsync to run in
> the background.

Interesting.

I have seen that mbsync can also sync to a *running* imap server, but I
do not if you can use something like I use in gnus and offlineimap.

Thinking about the race conditions more, I do not know if my approach
actually avoids them, as there are two different processes of imap
running (one offlineimap, one gnus) accessing the same mail.
I can only say this worked perfectly for me so far.
If you really want to be sure, run everything through a local docecot imap
server.

Cheers,

Rainer


>
> Thanks!

-- 
Rainer M. Krug
email: Rainer<at>krugs<dot>de
PGP: 0x0F52F982

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 480 bytes --]

Re: From Gnus to mu4e

[Bastien]

Thanks again for the detailed config Rainer.

The way to go for me know is to discipline myself not retrieving email
too often.  An old "papa"-way to avoid race conditions.

-- 
 Bastien

Re: From Gnus to mu4e

[Bastien]

Thanks again for the detailed config Rainer.

The way to go for me know is to discipline myself not retrieving email
too often.  An old "papa"-way to avoid race conditions.

-- 
 Bastien

Re: From Gnus to mu4e

[Rainer M Krug]

[-- Attachment #1: Type: text/plain, Size: 827 bytes --]

Bastien <bzg@gnu.org> writes:

> Thanks again for the detailed config Rainer.
>
> The way to go for me know is to discipline myself not retrieving email
> too often.  An old "papa"-way to avoid race conditions.

True - offlineimap checks in my case every two minutes (offlineimap's
quick check), and an more detailed check every 10 minutes - I guess this
is often quite often.

Cheers,

Rainer

-- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa

Tel :       +33 - (0)9 53 10 27 44
Cell:       +33 - (0)6 85 62 59 98
Fax :       +33 - (0)9 58 10 27 44

Fax (D):    +49 - (0)3 21 21 25 22 44

email:      Rainer@krugs.de

Skype:      RMkrug

PGP: 0x0F52F982

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 480 bytes --]

Re: From Gnus to mu4e

[Rasmus]

Rainer M Krug <Rainer@krugs.de> writes:

> Bastien <bzg@gnu.org> writes:
>
>> Thanks again for the detailed config Rainer.
>>
>> The way to go for me know is to discipline myself not retrieving email
>> too often.  An old "papa"-way to avoid race conditions.
>
> True - offlineimap checks in my case every two minutes (offlineimap's
> quick check), and an more detailed check every 10 minutes - I guess this
> is often quite often.

Offlineimap supports IDLE.

Rasmus

-- 
Bang bang

Re: From Gnus to mu4e

[Suvayu Ali]

On Thu, Aug 27, 2015 at 05:24:33PM -0700, Ian Zimmerman wrote:
> On 2015-08-28 02:14 +0200, Suvayu Ali wrote:
> 
> > No, it's not.  AFAIK on most (all?) *nix filesystems, basic operations
> > are atomic.  See for example this (outdated) list:
> > 
> > http://rcrowley.org/2010/01/06/things-unix-can-do-atomically.html
> > 
> > E.g., this breaks down when you talk about network mounts like NFS (not
> > sure about SSHFS), but it is a valid assumption as long as your storage
> > is local.
> 
> Assume there's a message ~/Mail/inbox/cur/1440718280.10956_2.ahiker:2,S
> 
> MUA 1 wants to set the "tagged" flag, which means renaming
> 
> 1440718280.10956_2.ahiker:2,S -> 1440718280.10956_2.ahiker:2,FS
> 
> MUA 2 wants to set the "replied" flag, which means renaming
> 
> 1440718280.10956_2.ahiker:2,S -> 1440718280.10956_2.ahiker:2,RS
> 
> Only one of them can succeed, depending on the order they try.  And
> final state also depends on that order.

I think you are confusing failures on the MUA side with race conditions
leading to email corruption.  Take your example above, when MUA 1 flags
a message, and succeeds, atomicity of filesystem operation ensures
setting the reply flag by MUA 2 will fail.  It is up to MUA 2 to handle
this failure.  The maildir is guaranteed to be in a consistent state by
the atomicity of filesystem operations.

When MUA 2 fails, it is easily handled by reporting to the user and not
trying to commit the changes again.  E.g. in mutt, this is handled by
telling the user something like: file does not exist (I don't recall the
exact phrasing), and keeping the folder state as is.  The user now has
to do two things, mark the folder read-only (so mutt stops trying to
write state changes back to the maildir), and reread the present
maildir.  During all this, your maildir has _not_ been corrupted in any
way, and both MUAs are reporting things correctly.

It is an entirely different story however if your maildir is over NFS
(and possibly SSHFS, actually maybe this is true for any network based
filesystem).  In that case, say the MUAs are accessing the maildir from
different locations.  MUA 1 writes the "flagging" action, the NFS client
commits the change.  However the real files on disk have not been
changed, and when MUA 2 tries to write its "replying" action, it
succeeds!  Now you have a race condition, MUA 1 thinks the message is
flagged, MUA 2 thinks it's read.  And no one knows what it is in reality
other than the NFS daemon.  Technically the email is not corrupted, only
the meta information about the flag is.  This is easily recoverable by
quiting both MUAs.  And essentially this is the beauty of the maildir
format, data integrity.

Hopefully I have explained myself clearly enough.  Or maybe you were
trying to say something else?  In that case please feel free to stop me.

Cheers,

-- 
Suvayu

Open source is the future. It sets us free.

Re: From Gnus to mu4e

[Julien Cubizolles]

Rasmus <rasmus@gmx.us> writes:

> Rainer M Krug <Rainer@krugs.de> writes:
>
>> Bastien <bzg@gnu.org> writes:
>>
>>> Thanks again for the detailed config Rainer.
>>>
>>> The way to go for me know is to discipline myself not retrieving email
>>> too often.  An old "papa"-way to avoid race conditions.
>>
>> True - offlineimap checks in my case every two minutes (offlineimap's
>> quick check), and an more detailed check every 10 minutes - I guess this
>> is often quite often.
>
> Offlineimap supports IDLE.

Does this mean you don't need to run offlineimap periodically if you use
this mode ? It doesn't help with potential race conditions with Gnus
accessing the mailbox, does it ? 
>
> Rasmus

Re: From Gnus to mu4e

[Rainer M Krug]

[-- Attachment #1: Type: text/plain, Size: 1177 bytes --]

Julien Cubizolles <j.cubizolles@free.fr> writes:

> Rasmus <rasmus@gmx.us> writes:
>
>> Rainer M Krug <Rainer@krugs.de> writes:
>>
>>> Bastien <bzg@gnu.org> writes:
>>>
>>>> Thanks again for the detailed config Rainer.
>>>>
>>>> The way to go for me know is to discipline myself not retrieving email
>>>> too often.  An old "papa"-way to avoid race conditions.
>>>
>>> True - offlineimap checks in my case every two minutes (offlineimap's
>>> quick check), and an more detailed check every 10 minutes - I guess this
>>> is often quite often.
>>
>> Offlineimap supports IDLE.
>
> Does this mean you don't need to run offlineimap periodically if you use
> this mode ? It doesn't help with potential race conditions with Gnus
> accessing the mailbox, does it ? 

Yes - offlineimap is running permanently, but ony checks in specific
intervalls - the rest of the time it is idling, but can, if told so,
keep the connection open (and this is what I do in my config).

No - I don't think it has any impact on potential race conditions.

Cheers,

Rainer

>>
>> Rasmus
>
>
>

-- 
Rainer M. Krug
email: Rainer<at>krugs<dot>de
PGP: 0x0F52F982

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 480 bytes --]

Re: From Gnus to mu4e

[Rasmus]

Julien Cubizolles <j.cubizolles@free.fr> writes:

> Does this mean you don't need to run offlineimap periodically if you use
> this mode ?

Right.  It's what TB and k9 uses as well.  It works OK.  Sometimes it's
even able to handle disconnect → suspend → wake up on another network.
It's not, however, as good as the IDLE support of "proper" mailclients
such as K9 and TB.

You need to add something like

    idlefolders = ['INBOX']

To your offlineimap [remote REPOSITORY].

And maybe

    maxsyncaccounts = 5
    socktimeout = 60

To your [general].

Also, you probably need to add this to [account REPOSITORY]

      autorefresh = 28

There might be more, I added it to my offlineimap settings a while ago.

> It doesn't help with potential race conditions with Gnus
> accessing the mailbox, does it ? 

No idea.  I use dovecot for accesing the maildir, like this:

(add-to-list 'gnus-secondary-select-methods
               '(nnimap "mail"
                        (nnimap-stream shell)
                        (nnimap-shell-program
                         "MAIL=maildir:$HOME/mail /usr/lib/dovecot/imap")))

Maybe you can use postsynchook in [Account REPOSITORY] to your advantage.
E.g. I get popups via:

postsynchook = emacsclient -s gnus --eval "(with-timeout (2) (when (gnus-alive-p) (gnus-group-get-new-news 1)))"

Rasmus

-- 
When the facts change, I change my mind. What do you do, sir?

Re: From Gnus to mu4e

[Rasmus]

Rainer M Krug <Rainer@krugs.de> writes:

> Yes - offlineimap is running permanently, but ony checks in specific
> intervalls - the rest of the time it is idling, but can, if told so,
> keep the connection open (and this is what I do in my config).

This is incorrect.

    In email technology, IDLE is an IMAP feature described in RFC 2177
    that allows a client to indicate to the server that it is ready to
    accept real-time notifications.                        (Wikipedia)
           ^^^^^^^^^^^^^^^^^^^^^^^ 

What you are talking about is polling; like a chron job.  This is
inefficient.

Rasmus

-- 
m-mm-mmm-mmmm bacon!

Re: From Gnus to mu4e

[Rainer M Krug]

[-- Attachment #1: Type: text/plain, Size: 1181 bytes --]

Rasmus <rasmus@gmx.us> writes:

> Rainer M Krug <Rainer@krugs.de> writes:
>
>> Yes - offlineimap is running permanently, but ony checks in specific
>> intervalls - the rest of the time it is idling, but can, if told so,
>> keep the connection open (and this is what I do in my config).
>
> This is incorrect.
>
>     In email technology, IDLE is an IMAP feature described in RFC 2177
>     that allows a client to indicate to the server that it is ready to
>     accept real-time notifications.                        (Wikipedia)
>            ^^^^^^^^^^^^^^^^^^^^^^^ 

Sorry - mixed it up with running offlineimap in daemon mode with
checking periodically, controlled by the "autorefresh" option.

Didn't know about the idlefolders option.

>
> What you are talking about is polling; like a chron job.  This is
> inefficient.

Why is it inefficient, and inefficient in regards to local resources,
time, bandwidth? The connections are kept open, and it is just
polling every few minutes. With idlefolders, offlineimap is permanently
listening to the server?

Rainer

>
> Rasmus

-- 
Rainer M. Krug
email: Rainer<at>krugs<dot>de
PGP: 0x0F52F982

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 480 bytes --]

Re: From Gnus to mu4e

[Ian Zimmerman]

On 2015-08-28 11:06 +0200, Suvayu Ali wrote:

> And essentially this is the beauty of the maildir
> format, data integrity.

I agree, but that is not the same as absence of races.  See below.

> Hopefully I have explained myself clearly enough.  Or maybe you were
> trying to say something else?  In that case please feel free to stop me.

We have different definitions of "race".  To me, corruption is not a
necessary ingredient; a race is just a situation where two (or more)
threads or processes mutate data in ways that are independent
(ie. commutative) in the abstract, and yet the final state of the data
depends on the order of the changes.

A race can even be completely harmless if the two concrete
representations of the final state in fact represent the same abstract
state.  An extreme example of that would be a log structured system,
where the difference is simply in the order of the log records left by
the changes.  Yes, this is a race by my definition; but it's a feature,
not a bug :-)

> When MUA 2 fails, it is easily handled by reporting to the user and not
> trying to commit the changes again.  E.g. in mutt, this is handled by
> telling the user something like: file does not exist (I don't recall the
> exact phrasing), and keeping the folder state as is.

I don't know about you, but if I got this blurb from my MUA I would
first stare at the screen for minutes, repeating "wtf wtf", then (if I
was having a good day) investigate, and finally mutate the state of my
computer to ensure exclusive access :-) IOW, I don't want such weirdness
to happen, whether I call it "race" or not.

Lastly, it is not even clear what we should consider "corruption", or
the absence of it.  Is internal consistency enough?  Consider the
classic motivating example for database transactions, with two threads
trying to update an account balance datum, and only one succeeding.
Let's say there's no separate transactions table (a stretch, I know).
Then the data after the mixup is internally consistent; there just
aren't many ways that a single numerical value can be incosistent :-)
But it doesn't reflect reality, so maybe it is corrupt?

And to close the circle, this example in fact has analogy with our
maildir situation:  the MUA trying to set the "Replied" flag presumably
just sent off a reply.  So the failure to set the flag means the maildir
state, while internally consistent, doesn't reflect reality.

-- 
Please *no* private copies of mailing list or newsgroup messages.
Rule 420: All persons more than eight miles high to leave the court.

Re: From Gnus to mu4e

[Suvayu Ali]

On Fri, Aug 28, 2015 at 08:31:26AM -0700, Ian Zimmerman wrote:
> On 2015-08-28 11:06 +0200, Suvayu Ali wrote:
> 
> > When MUA 2 fails, it is easily handled by reporting to the user and not
> > trying to commit the changes again.  E.g. in mutt, this is handled by
> > telling the user something like: file does not exist (I don't recall the
> > exact phrasing), and keeping the folder state as is.
> 
> I don't know about you, but if I got this blurb from my MUA I would
> first stare at the screen for minutes, repeating "wtf wtf", then (if I
> was having a good day) investigate, and finally mutate the state of my
> computer to ensure exclusive access :-) IOW, I don't want such weirdness
> to happen, whether I call it "race" or not.

Okay, I found the error message, it's cryptic indeed.

  rename: No such file or directory (errno = 2)

IIRC, I did understand it quite immediately when I saw it the first
time.  But maybe it was easy for me because I was quite familiar with
the maildir format, having dealt with OfflineIMAP and I had been
experimenting with Gmail's IMAP implementation to understand how it
works.

Having said all that, I do not think there is any easy solution to this
issue.  This is inherrent in how we are approaching the problem the
moment we choose a solution that relies on syncing rather than being
live (IMAP access).

> And to close the circle, this example in fact has analogy with our
> maildir situation:  the MUA trying to set the "Replied" flag presumably
> just sent off a reply.  So the failure to set the flag means the maildir
> state, while internally consistent, doesn't reflect reality.

This is a very interesting case indeed.  However I do not see any
possible solutions as long as we choose to sync rather than use IMAP.
In this particular case, the inconsistent flags will be corrected on the
next sync, since the remote server has the correct flags.

Cheers,

-- 
Suvayu

Open source is the future. It sets us free.