notmuch "lost" email during power failure

notmuch "lost" email during power failure

[Antoine Beaupré]

Hi!

Today, in Canadaland(show.com), it's winter! There's snow everywhere,
it's beautiful but cold, which means we power up those heating system
and blow up breakers. Which means computers go down because those little
stupid things need power to keep going, especially when they don't have
UPS systems built in. Those backwards machines were previously known as
"desktops" but are now known as "what, you still have one of those? yes
I do leave me alone."

Anyways. Today my desktop lost power while I was writing a too long
email to dkg about internet standards. Then power went out halfway
through the email. When I rebooted, notmuch couldn't find the email and
I assumed it was dead and gone, wrongly, as it turns out.

I draw many conclusions from this:

 1. i shouldn't write long emails to dkg
 2. i shouldn't write long emails to replies of dkg to my long emails
 3. notmuch should never lose email drafts of long emails i write to dkg
 4. i should learn not to worry and love the bomb

Obviously, I come to you with conclusion #3.

Being the resourceful and friendly human being that he is, dkg pointed
out the `~/.emacs.d/auto-save-list/` directory. It contains a list of
files that have a list of file names in them (!?) pointing to various
auto-save files. I couldn't figure out where that thing comes from, but
it did lead me to a saved copy of the message.

Naturally, I only found dkg's message *after* I spent another hour
rewriting the damn message after the power failure, but it was accurate:
one of those files had a filename that pointed at:

~/Mail/drafts/#*message*-20191112-165309#

which contained the auto-save of the message! Hurray!

As it turns out, that location (~/Mail/drafts) is the default
`message-auto-save-directory`. That's great, but the problem is there's
no visibility in that directory from notmuch's perspective.

Even worse, message-mode leaves stray messages in there. While cleaning
it up, I even found an old message from 2018 that had a *password*
(*gasp*!) in it that I carelessly sent to other people. Interestingly,
that message was sent encrypted (with OpenPGP) but was still stored as a
tempfile in cleartext there.

Obviously, none of that so far is directly notmuch's fault: those are
all problems with message-mode.

*But* I would argue that notmuch should at least allow me to recover
from a power failure like this, as a MUA. It should "know" that
message-mode stores those messages there, and, why not, also store its
tempfiles there. And indeed, if I hit [control-x control-s] on this
message, it *does* get saved as a "draft" in that it gets written in:

~/Maildir/drafts/cur/1573596264.M156307P32312.curie:2,DF

That's a great improvement already. I don't remember exactly when or how
this happened, but that's great and I thank whoever did that for us
here.

I do remember hacking something like this together before that happened
however. I made message-mode write temporary messages directly in that
folder (by setting the auto-save-directory to ~/Maildir/draft/new),
which notmuch would somewhat pickup at the next automated `notmuch new`
run. But that made notmuch unhappy for various reasons:

 1. the autosave files don't have message IDs which would confuse notmuch

 2. the files wouldn't automatically be removed from notmuch's database
    even when (or if!) message-mode would actually clean them up

I had to write hacks like this to cleanup those files:

https://gitlab.com/anarcat/scripts/blob/master/notmuch-clear

It was a mess, so I reverted message-auto-save-directory back to its
default and totally forgot about it.

Which led me to writing a second long email to dkg.

Which made me kind of obstinately sad.

Which is very weird.

But that's probably standard IETF behavior by this point. Not sure which
draft, RFC or BCP that is, but that's probably irrelevant. ;)

To make a long story short, I think the following should happen:

 1. message-mode should automatically cleanup after itself a little
    better (not notmuch's job? yay double-negative, that means it's much
    job!)

 2. encrypted emails should *never* be written in cleartext on the
    filesystem (not notmuch job, which also means it's much job!)

 3. notmuch's draft subsystem should know about Emacs' autosave files
    and somehow show them in the UI

Sorry for the long email. My attempts at comedy in this one are probably
far from the previous one, but the topic was less happy and I was less
motivated. I promise to try again next time though, and thank you for
flying nutty anarcat.

Cheers,

A.
-- 
The problem is not a lack of highly educated workers, the problem is a
lack of highly educated workers willing to work for the minimum wage or
lower in the U.S. Costs are driving outsourcing, not the quality of
American schools.       - Scott Kirwin, IT Professionals Association

unifying and automating draft ("postpone/resume") behavior [was: Re: notmuch "lost" email during power failure]

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 3127 bytes --]

Hi Antoine--

[epic story and funny rnat trimmed down to the salient bug reports and
 feature requests because i'm a boring person]

On Tue 2019-11-12 17:19:05 -0500, Antoine Beaupré wrote:
> I would argue that notmuch should at least allow me to recover from a
> power failure like this, as a MUA. It should "know" that message-mode
> stores those messages there, and, why not, also store its tempfiles
> there.

This seems like a very specific ask.  I think what we want more
generally is a sensible unification of the notmuch-emacs drafting
behaviors such that message-mode autosaves are discoverable and
recoverable in the same way that deliberately-saved drafts are.

If that means that notmuch learns about message-mode drafts, that's one
solution.  But it could also mean that notmuch-emacs overrides
message-mode's autosave drafting approach, and fixes things there too.

or, we somehow fix message-mode?

> And indeed, if I hit [control-x control-s] on this message, it *does*
> get saved as a "draft" in that it gets written in:
>
> ~/Maildir/drafts/cur/1573596264.M156307P32312.curie:2,DF
>
> That's a great improvement already. I don't remember exactly when or how
> this happened, but that's great and I thank whoever did that for us
> here.

That was Mark Walters (in Cc), see the series from 2016 starting at:

     id:1479046130-2716-1-git-send-email-markwalters1009@gmail.com

Thank you, Mark!

> To make a long story short, I think the following should happen:
>
>  1. message-mode should automatically cleanup after itself a little
>     better (not notmuch's job? yay double-negative, that means it's much
>     job!)
>
>  2. encrypted emails should *never* be written in cleartext on the
>     filesystem (not notmuch job, which also means it's much job!)
>
>  3. notmuch's draft subsystem should know about Emacs' autosave files
>     and somehow show them in the UI

Much of the above sounds like message-mode cleanup work to me.  It might
be worth asking the message-mode folks to weigh on in this strategy.

Alternately, it might help to characterize the differences between
message-mode autosaving and notmuch explicit drafting.

 - message-mode autosaves happen without your knowledge.
   notmuch draft saving only happens when you explicitly C-x C-s
 
 - message-mode autosaves are automatically cleaned up from the
   filesystem after you send.  notmuch saved drafts persist in your
   mailstore until you remove them (though they are tagged with
   "delete")

 - message-mode autosaves appear to save in cleartext even when the
   message will be encrypted(‽).  if you try to C-x C-s on a notmuch
   draft, notmuch-emacs gives you a big ol' warning and won't let you
   save it in plaintext unless you agree (see
   notmuch-draft-save-plaintext).  (Maybe there's a third behavior which
   would be better than all of these, which is to explicitly save the
   draft of encrypted messages encrypted-to-self-only, but that's not
   implemented, afaict)

Does this all sound right?  There's a lot to unpack here.

   --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: unifying and automating draft ("postpone/resume") behavior [was: Re: notmuch "lost" email during power failure]

[Antoine Beaupré]

On 2019-11-12 18:03:58, Daniel Kahn Gillmor wrote:
> Hi Antoine--
>
> [epic story and funny rnat trimmed down to the salient bug reports and
>  feature requests because i'm a boring person]

Hi!

[I don't remember what RNAT (reverse network address translation) have to
do with anything, but it's true it was a rather long RANT so maybe I
forgot that I went through some weird addressing along the way. ;)]

> On Tue 2019-11-12 17:19:05 -0500, Antoine Beaupré wrote:
>> I would argue that notmuch should at least allow me to recover from a
>> power failure like this, as a MUA. It should "know" that message-mode
>> stores those messages there, and, why not, also store its tempfiles
>> there.
>
> This seems like a very specific ask.  I think what we want more
> generally is a sensible unification of the notmuch-emacs drafting
> behaviors such that message-mode autosaves are discoverable and
> recoverable in the same way that deliberately-saved drafts are.

Yeah. That is pretty much what I meant to say, but said in a more
useful, constructive and clever way. :p

> If that means that notmuch learns about message-mode drafts, that's one
> solution.  But it could also mean that notmuch-emacs overrides
> message-mode's autosave drafting approach, and fixes things there too.

I think the main problem right now is that "control-x control-s" does
something special in notmuch, in that it adds a message-id to the saved
message even if the message isn't saved yet.

But then when *message-mode* auto-saves those files, the message-id is
not present (just checked). For example this very message has 
References (hidden), From, To, Cc, Subject, In-Reply-To and Fcc headers
on disk, along with the fancy `--text follows this line--` magic blob on
disk, in ~/Mail/drafts/#...

While if I save this message to disk, it looks like a proper email
message. It has a (properly encoded) From header, To, Cc, Subject,
In-Reply-To, Fcc (so far pretty close to message-mode), but then also a
Message-ID, Date, X-Notmuch-Emacs-Draft, MIME-Version, Content-type and
Content-transfer-encoding. And of course, the content *is*
quoted-printable-encoded, which might be a problem for message-mode.

In my opinion, that's the fundamental conflict between the two modes:
message-mode expects buffers (and auto-saved files) to have this weird
exotic format to operate, but it's pretty much exactly the same on-disk
format, while notmuch expects a standard email message, which *differs*
from the actual buffer content. It will be fundamentally difficult to
reconcile those two.

> or, we somehow fix message-mode?

That could be a huge challenge. We'd need to teach it to parse real
RFC852 messages properly, without the magic separator, and to include
proper message IDs and other headers.

Alternatively, maybe message-mode could do the same translation magic
that notmuch does when it saves and loads those files, on write. By
moving the notmuch logic upwards in the stack, everyone could benefit.

But it's kind of a big change.

>> And indeed, if I hit [control-x control-s] on this message, it *does*
>> get saved as a "draft" in that it gets written in:
>>
>> ~/Maildir/drafts/cur/1573596264.M156307P32312.curie:2,DF
>>
>> That's a great improvement already. I don't remember exactly when or how
>> this happened, but that's great and I thank whoever did that for us
>> here.
>
> That was Mark Walters (in Cc), see the series from 2016 starting at:
>
>      id:1479046130-2716-1-git-send-email-markwalters1009@gmail.com
>
> Thank you, Mark!

Yeeeaay! thank you Mark!

>> To make a long story short, I think the following should happen:
>>
>>  1. message-mode should automatically cleanup after itself a little
>>     better (not notmuch's job? yay double-negative, that means it's much
>>     job!)
>>
>>  2. encrypted emails should *never* be written in cleartext on the
>>     filesystem (not notmuch job, which also means it's much job!)
>>
>>  3. notmuch's draft subsystem should know about Emacs' autosave files
>>     and somehow show them in the UI
>
> Much of the above sounds like message-mode cleanup work to me.  It might
> be worth asking the message-mode folks to weigh on in this strategy.

For what it's worth, I reviewed the original patchset that was merged
from Mark, and notmuch *does* the right thing with encrypted messages
(issue 2 above): it asks users (configurable) what to do. This, again,
could be moved up into message-mode as well...

> Alternately, it might help to characterize the differences between
> message-mode autosaving and notmuch explicit drafting.
>
>  - message-mode autosaves happen without your knowledge.
>    notmuch draft saving only happens when you explicitly C-x C-s
>  
>  - message-mode autosaves are automatically cleaned up from the
>    filesystem after you send.  notmuch saved drafts persist in your
>    mailstore until you remove them (though they are tagged with
>    "delete")

Those two are equivalent for me in notmuch, as "deleted" emails get
periodically cleaned (yes, I'm an heretic).

>  - message-mode autosaves appear to save in cleartext even when the
>    message will be encrypted(‽).  if you try to C-x C-s on a notmuch
>    draft, notmuch-emacs gives you a big ol' warning and won't let you
>    save it in plaintext unless you agree (see
>    notmuch-draft-save-plaintext).

That is correct.

>    (Maybe there's a third behavior which
>    would be better than all of these, which is to explicitly save the
>    draft of encrypted messages encrypted-to-self-only, but that's not
>    implemented, afaict)
>
> Does this all sound right?  There's a lot to unpack here.

It does! thanks for clarifying my novel. ;)

Fundamentally, I think message-mode should treat emails the same way
epg.el treats encrypted files: the data in the buffer isn't the same as
the on-disk data, and there should be translation between the two. At
the very least, the on-disk data could be a properly formed email
message. If it's marked as needing encryption, then it should be
encrypted.

Arguably, I haven't looked in details at how epg.el works: maybe it
suffers from the same flaw. But from what I can tell, it simply bypasses
the autosave functionality. If you have edit an ecrypted file called
`foo.gpg` (note the non-standard extension), the `#foo.gpg` file is just
a symlink, a lockfile instead of the normal autosave.

This is probably because it would be prohibitively expensive to
constantly call gpg every time a buffer changes to do the autosave. I
think that is actually a fairly reasonable tradeoff.

A.

-- 
Tu connaîtras la vérité de ton chemin à ce qui te rend heureux.
                        - Aristote

Re: unifying and automating draft ("postpone/resume") behavior [was: Re: notmuch "lost" email during power failure]

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1411 bytes --]

On Wed 2019-11-13 10:19:05 -0500, Antoine Beaupré wrote:
> Fundamentally, I think message-mode should treat emails the same way
> epg.el treats encrypted files: the data in the buffer isn't the same as
> the on-disk data, and there should be translation between the two. At
> the very least, the on-disk data could be a properly formed email
> message. If it's marked as needing encryption, then it should be
> encrypted.

i think it might be easier for notmuch-emacs to override message-mode
entirely, than to try to convince message-mode to do something so
radically different.

but i agree with you that it would be nice for the whole emacs MUA
ecosystem if they were more sensible there.

> Arguably, I haven't looked in details at how epg.el works: maybe it
> suffers from the same flaw. But from what I can tell, it simply bypasses
> the autosave functionality. If you have edit an ecrypted file called
> `foo.gpg` (note the non-standard extension), the `#foo.gpg` file is just
> a symlink, a lockfile instead of the normal autosave.
>
> This is probably because it would be prohibitively expensive to
> constantly call gpg every time a buffer changes to do the autosave. I
> think that is actually a fairly reasonable tradeoff.

if it's prohibitively expensive to encrypt-when-saving on a modern
machine, then we should be using a different encryption tool.

         --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: unifying and automating draft ("postpone/resume") behavior [was: Re: notmuch "lost" email during power failure]

[Antoine Beaupré]

On 2019-11-14 03:16:51, Daniel Kahn Gillmor wrote:
> On Wed 2019-11-13 10:19:05 -0500, Antoine Beaupré wrote:
>> Fundamentally, I think message-mode should treat emails the same way
>> epg.el treats encrypted files: the data in the buffer isn't the same as
>> the on-disk data, and there should be translation between the two. At
>> the very least, the on-disk data could be a properly formed email
>> message. If it's marked as needing encryption, then it should be
>> encrypted.
>
> i think it might be easier for notmuch-emacs to override message-mode
> entirely, than to try to convince message-mode to do something so
> radically different.

Possibly.

> but i agree with you that it would be nice for the whole emacs MUA
> ecosystem if they were more sensible there.

I would just hate to think of notmuch as a (yet another) Emacs MUA that
reinvents everything for itself, with nice little "edges" compared to
the other clients. But I guess that's the easier way to move forward and
I would certainly prefer that to not having a solution for this. :)

>> Arguably, I haven't looked in details at how epg.el works: maybe it
>> suffers from the same flaw. But from what I can tell, it simply bypasses
>> the autosave functionality. If you have edit an ecrypted file called
>> `foo.gpg` (note the non-standard extension), the `#foo.gpg` file is just
>> a symlink, a lockfile instead of the normal autosave.
>>
>> This is probably because it would be prohibitively expensive to
>> constantly call gpg every time a buffer changes to do the autosave. I
>> think that is actually a fairly reasonable tradeoff.
>
> if it's prohibitively expensive to encrypt-when-saving on a modern
> machine, then we should be using a different encryption tool.

The individual calls are not what's prohibitively expensive. The problem
is latency. Even if it just takes 100ms[1] to save the file to the disk,
that's a huge delay in terms of human interface, because the entire
Emacs UI will *freeze* for that amount of time whenever it thinks it
should do an autosave.

So yes, that's prohibitively expensive, and I'm not sure it's realistic
to blame gpg for this here. We are, after all, shelling out to a
different process and most runtimes have at least *some* overhead on
started (e.g. cPython 2 is ~13ms here and cPython 3 is around 25ms, both
noticeable latencies).

In other words, maybe the right solution for autosave is to skip it in
some cases.

A.

[1] anarcat@curie:~(master)$ multitime -n 10 -r 'rm -f motd.gpg' -v gpg -e -r anarcat@debian.org motd
===> multitime results
1: -r "rm -f motd.gpg" gpg -e -r anarcat@debian.org motd
            Mean        Std.Dev.    Min         Median      Max
real        0.096       0.007       0.088       0.099       0.104       
user        0.062       0.011       0.041       0.061       0.080       
sys         0.023       0.008       0.012       0.023       0.037       


-- 
There are two ways of constructing a software design: One way is to
make it so simple that there are obviously no deficiencies, and the
other way is to make it so complicated that there are no obvious
deficiencies. The first method is far more difficult.
                        - C.A.R. Hoare

Re: unifying and automating draft ("postpone/resume") behavior [was: Re: notmuch "lost" email during power failure]

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 715 bytes --]

On Fri 2019-11-15 10:08:02 -0500, Antoine Beaupré wrote:
> The individual calls are not what's prohibitively expensive. The problem
> is latency. Even if it just takes 100ms[1] to save the file to the disk,
> that's a huge delay in terms of human interface, because the entire
> Emacs UI will *freeze* for that amount of time whenever it thinks it
> should do an autosave.

Surely the problem there is that the Emacs UI is freezing and blocking
to do something that has nothing to do with interactive work!

Is it not possible in 2019 to have emacs handle its autosave
functionality in either a background thread, or in a non-blocking
asynchronous interaction with a co-process?

             --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]