Re: Data loss

Re: Data loss

[Fulvio Pizzigoni]

Hi Carl e thanks for your prompt answer.

As you suggested, I add notmuch@notmuchmail.org email as well.

This is what I did:
fulvio@linux:~$ notmuch setup 
Your full name [fulvio]: 
Your primary email address [my address]: return
Additional email address [Press 'Enter' if none]: return
Top-level directory of your email archive [/home/fulvio/.mutt]: return
Tags to apply to all new messages (separated by spaces) [unread inbox]: return
Tags to exclude when searching messages (separated by spaces) [deleted spam]: return
fulvio@linux:~$

After this my .mutt directory (~ 4 GB di mail-boxess) appears so:
fulvio@linux:~$ ll .mutt
totale 12
drwxr-xr-x 2 fulvio fulvio 4096 18 feb 20.32 cur
drwxr-xr-x 2 fulvio fulvio 4096 18 feb 20.32 new
drwxr-xr-x 2 fulvio fulvio 4096 18 feb 20.32 tmp
fulvio@linux:~$ du -h .mutt
4,0K	.mutt/new
4,0K	.mutt/tmp
4,0K	.mutt/cur
16K	.mutt

That's all.

Terrible!

What happened?

Thanks in advance.

Fulvio


On Mon, Apr 03, 2023 at 09:27:22AM -0700, Carl Worth wrote:
> Hi Fulvio,
> 
> I've never used notmuch-mutt.
> 
> But notmuch itself doesn't delete any mail. It's really paranoid that
> way, (knowing how valuable mail is).
> 
> I would suggest you write an email to the notmuch@notmuchmail.org
> mailing list where you will be able to reach more people likely to have
> experience with all of the software you were using.
> 
> And if you could provide more details on the actual steps you used, that
> would be useful. For example, you said "configuration process" and "at
> the end". But what actual commands were you running on those steps, for
> example?
> 
> -Carl
> 
> On Mon, Apr 03 2023, Fulvio Pizzigoni wrote:
> > Hi.
> >
> > I have installed packages notmuch and notmuch-mutt.
> >
> > During configuration process I have indicated the directory used from 
> > Mutt for your mail-boxes.
> >
> > The configuration process have not indicated any allert.
> >
> > At the end the mail-boxes in that directory was removed; 3 new
> > empty sub-directory were created: cur, new, tmp.
> >
> > Was the data irremediably lost?
> >
> > Thanks in advance.
> >
> > Fulvio Pizzigoni

Re: Data loss

[David Bremner]

Fulvio Pizzigoni <fulviopizzigoni@libero.it> writes:
>
> After this my .mutt directory (~ 4 GB di mail-boxess) appears so:
> fulvio@linux:~$ ll .mutt
> totale 12
> drwxr-xr-x 2 fulvio fulvio 4096 18 feb 20.32 cur
> drwxr-xr-x 2 fulvio fulvio 4096 18 feb 20.32 new
> drwxr-xr-x 2 fulvio fulvio 4096 18 feb 20.32 tmp

Hi Fulvio;

I don't understand your situation yet, but I have a preliminary
question.

Is the date on your computer more or less correct? You mention the
directories cur, new, and tmp being created by running notmuch setup; on
the other hand your listing above shows they already existed in
February.

d

Re: Data loss

[Michael J Gruber]

Am Mo., 3. Apr. 2023 um 20:17 Uhr schrieb Fulvio Pizzigoni
<fulviopizzigoni@libero.it>:
>
>
>
>
>
>
> Hi Carl e thanks for your prompt answer.
>
> As you suggested, I add notmuch@notmuchmail.org email as well.
>
> This is what I did:
> fulvio@linux:~$ notmuch setup
> Your full name [fulvio]:
> Your primary email address [my address]: return
> Additional email address [Press 'Enter' if none]: return
> Top-level directory of your email archive [/home/fulvio/.mutt]: return
> Tags to apply to all new messages (separated by spaces) [unread inbox]: return
> Tags to exclude when searching messages (separated by spaces) [deleted spam]: return
> fulvio@linux:~$
>
> After this my .mutt directory (~ 4 GB di mail-boxess) appears so:
> fulvio@linux:~$ ll .mutt
> totale 12
> drwxr-xr-x 2 fulvio fulvio 4096 18 feb 20.32 cur
> drwxr-xr-x 2 fulvio fulvio 4096 18 feb 20.32 new
> drwxr-xr-x 2 fulvio fulvio 4096 18 feb 20.32 tmp
> fulvio@linux:~$ du -h .mutt
> 4,0K    .mutt/new
> 4,0K    .mutt/tmp
> 4,0K    .mutt/cur
> 16K     .mutt
>
> That's all.
>
> Terrible!
>
> What happened?
>
> Thanks in advance.
>
> Fulvio

First of all: Do you have a back-up?

notmuch itself does not delete mails, as Carl pointed out. In addition
to notmuch you mentioned notmuch-mutt. Did you run that manually or
using some mutt config? It creates a maildir of symlinks with search
results. In order to do so, it deletes the maildir ... Usually this
sits in a cache dir, though.

Michael

> On Mon, Apr 03, 2023 at 09:27:22AM -0700, Carl Worth wrote:
> > Hi Fulvio,
> >
> > I've never used notmuch-mutt.
> >
> > But notmuch itself doesn't delete any mail. It's really paranoid that
> > way, (knowing how valuable mail is).
> >
> > I would suggest you write an email to the notmuch@notmuchmail.org
> > mailing list where you will be able to reach more people likely to have
> > experience with all of the software you were using.
> >
> > And if you could provide more details on the actual steps you used, that
> > would be useful. For example, you said "configuration process" and "at
> > the end". But what actual commands were you running on those steps, for
> > example?
> >
> > -Carl
> >
> > On Mon, Apr 03 2023, Fulvio Pizzigoni wrote:
> > > Hi.
> > >
> > > I have installed packages notmuch and notmuch-mutt.
> > >
> > > During configuration process I have indicated the directory used from
> > > Mutt for your mail-boxes.
> > >
> > > The configuration process have not indicated any allert.
> > >
> > > At the end the mail-boxes in that directory was removed; 3 new
> > > empty sub-directory were created: cur, new, tmp.
> > >
> > > Was the data irremediably lost?
> > >
> > > Thanks in advance.
> > >
> > > Fulvio Pizzigoni
> _______________________________________________
> notmuch mailing list -- notmuch@notmuchmail.org
> To unsubscribe send an email to notmuch-leave@notmuchmail.org

Re: Data loss

[Fulvio Pizzigoni]

On Mon, Apr 03, 2023 at 03:38:21PM -0300, David Bremner wrote:
> Fulvio Pizzigoni <fulviopizzigoni@libero.it> writes:
> >
> > After this my .mutt directory (~ 4 GB di mail-boxess) appears so:
> > fulvio@linux:~$ ll .mutt
> > totale 12
> > drwxr-xr-x 2 fulvio fulvio 4096 18 feb 20.32 cur
> > drwxr-xr-x 2 fulvio fulvio 4096 18 feb 20.32 new
> > drwxr-xr-x 2 fulvio fulvio 4096 18 feb 20.32 tmp
> 
> Hi Fulvio;

Hi David.

> 
> I don't understand your situation yet, but I have a preliminary
> question.
> 
> Is the date on your computer more or less correct? 

Yes, it is.

> You mention the
> directories cur, new, and tmp being created by running notmuch setup;

It is correct.
 
> on the other hand your listing above shows they already existed in
> February.

It is the date when I did the setup.

Thanks in advance.

Fulvio

> 
> d

Re: Data loss

[Fulvio Pizzigoni]

On Mon, Apr 03, 2023 at 08:39:02PM +0200, Michael J Gruber wrote:
> Am Mo., 3. Apr. 2023 um 20:17 Uhr schrieb Fulvio Pizzigoni
> <fulviopizzigoni@libero.it>:
> >
> >
> >
> >
> >
> >
> > Hi Carl e thanks for your prompt answer.
> >
> > As you suggested, I add notmuch@notmuchmail.org email as well.
> >
> > This is what I did:
> > fulvio@linux:~$ notmuch setup
> > Your full name [fulvio]:
> > Your primary email address [my address]: return
> > Additional email address [Press 'Enter' if none]: return
> > Top-level directory of your email archive [/home/fulvio/.mutt]: return
> > Tags to apply to all new messages (separated by spaces) [unread inbox]: return
> > Tags to exclude when searching messages (separated by spaces) [deleted spam]: return
> > fulvio@linux:~$
> >
> > After this my .mutt directory (~ 4 GB di mail-boxess) appears so:
> > fulvio@linux:~$ ll .mutt
> > totale 12
> > drwxr-xr-x 2 fulvio fulvio 4096 18 feb 20.32 cur
> > drwxr-xr-x 2 fulvio fulvio 4096 18 feb 20.32 new
> > drwxr-xr-x 2 fulvio fulvio 4096 18 feb 20.32 tmp
> > fulvio@linux:~$ du -h .mutt
> > 4,0K    .mutt/new
> > 4,0K    .mutt/tmp
> > 4,0K    .mutt/cur
> > 16K     .mutt
> >
> > That's all.
> >
> > Terrible!
> >
> > What happened?
> >
> > Thanks in advance.
> >
> > Fulvio

Hi Michael.

> 
> First of all: Do you have a back-up? 

GRRRR, no :-((

> 
> notmuch itself does not delete mails, as Carl pointed out. In addition
> to notmuch you mentioned notmuch-mutt. Did you run that manually or
> using some mutt config? 

Yes I did this manually:
fulvio@linux:~$ notmuch-mutt -o .mutt/<mail-box> search <SEARCH-TERM>

> It creates a maildir of symlinks with search
> results. In order to do so, it deletes the maildir ... 

I think that's the cause.

... And I think it's irreparable. Am I wrong?

Thanks in advance.

Fulvio

> Usually this  sits in a cache dir, though.
> 
> Michael
> 
> > On Mon, Apr 03, 2023 at 09:27:22AM -0700, Carl Worth wrote:
> > > Hi Fulvio,
> > >
> > > I've never used notmuch-mutt.
> > >
> > > But notmuch itself doesn't delete any mail. It's really paranoid that
> > > way, (knowing how valuable mail is).
> > >
> > > I would suggest you write an email to the notmuch@notmuchmail.org
> > > mailing list where you will be able to reach more people likely to have
> > > experience with all of the software you were using.
> > >
> > > And if you could provide more details on the actual steps you used, that
> > > would be useful. For example, you said "configuration process" and "at
> > > the end". But what actual commands were you running on those steps, for
> > > example?
> > >
> > > -Carl
> > >
> > > On Mon, Apr 03 2023, Fulvio Pizzigoni wrote:
> > > > Hi.
> > > >
> > > > I have installed packages notmuch and notmuch-mutt.
> > > >
> > > > During configuration process I have indicated the directory used from
> > > > Mutt for your mail-boxes.
> > > >
> > > > The configuration process have not indicated any allert.
> > > >
> > > > At the end the mail-boxes in that directory was removed; 3 new
> > > > empty sub-directory were created: cur, new, tmp.
> > > >
> > > > Was the data irremediably lost?
> > > >
> > > > Thanks in advance.
> > > >
> > > > Fulvio Pizzigoni
> > _______________________________________________
> > notmuch mailing list -- notmuch@notmuchmail.org
> > To unsubscribe send an email to notmuch-leave@notmuchmail.org

Re: Data loss

[Carl Worth]

Ouch.

It's really unfortunate if notmuch-mutt makes it that easy to throw away
your email.

That sounds like a nasty bug that should be fixed in that program.

As for recovering, I suppose there _is_ a fair amount of detail in your
notmuch index from all of the position-indexed terms, (as long as you
haven't run "notmuch new" again since the mail was deleted).

If you still have a large Xapian database in the notmuch database
directory, it would be theoretically possible to recover a lot of the
email content. But I don't know that anyone has ever written a recovery
tool to help with that process.

-Carl

On Mon, Apr 03 2023, Fulvio Pizzigoni wrote:
> On Mon, Apr 03, 2023 at 08:39:02PM +0200, Michael J Gruber wrote:
>> Am Mo., 3. Apr. 2023 um 20:17 Uhr schrieb Fulvio Pizzigoni
>> <fulviopizzigoni@libero.it>:
>> >
>> >
>> >
>> >
>> >
>> >
>> > Hi Carl e thanks for your prompt answer.
>> >
>> > As you suggested, I add notmuch@notmuchmail.org email as well.
>> >
>> > This is what I did:
>> > fulvio@linux:~$ notmuch setup
>> > Your full name [fulvio]:
>> > Your primary email address [my address]: return
>> > Additional email address [Press 'Enter' if none]: return
>> > Top-level directory of your email archive [/home/fulvio/.mutt]: return
>> > Tags to apply to all new messages (separated by spaces) [unread inbox]: return
>> > Tags to exclude when searching messages (separated by spaces) [deleted spam]: return
>> > fulvio@linux:~$
>> >
>> > After this my .mutt directory (~ 4 GB di mail-boxess) appears so:
>> > fulvio@linux:~$ ll .mutt
>> > totale 12
>> > drwxr-xr-x 2 fulvio fulvio 4096 18 feb 20.32 cur
>> > drwxr-xr-x 2 fulvio fulvio 4096 18 feb 20.32 new
>> > drwxr-xr-x 2 fulvio fulvio 4096 18 feb 20.32 tmp
>> > fulvio@linux:~$ du -h .mutt
>> > 4,0K    .mutt/new
>> > 4,0K    .mutt/tmp
>> > 4,0K    .mutt/cur
>> > 16K     .mutt
>> >
>> > That's all.
>> >
>> > Terrible!
>> >
>> > What happened?
>> >
>> > Thanks in advance.
>> >
>> > Fulvio
>
> Hi Michael.
>
>> 
>> First of all: Do you have a back-up? 
>
> GRRRR, no :-((
>
>> 
>> notmuch itself does not delete mails, as Carl pointed out. In addition
>> to notmuch you mentioned notmuch-mutt. Did you run that manually or
>> using some mutt config? 
>
> Yes I did this manually:
> fulvio@linux:~$ notmuch-mutt -o .mutt/<mail-box> search <SEARCH-TERM>
>
>> It creates a maildir of symlinks with search
>> results. In order to do so, it deletes the maildir ... 
>
> I think that's the cause.
>
> ... And I think it's irreparable. Am I wrong?
>
> Thanks in advance.
>
> Fulvio
>
>> Usually this  sits in a cache dir, though.
>> 
>> Michael
>> 
>> > On Mon, Apr 03, 2023 at 09:27:22AM -0700, Carl Worth wrote:
>> > > Hi Fulvio,
>> > >
>> > > I've never used notmuch-mutt.
>> > >
>> > > But notmuch itself doesn't delete any mail. It's really paranoid that
>> > > way, (knowing how valuable mail is).
>> > >
>> > > I would suggest you write an email to the notmuch@notmuchmail.org
>> > > mailing list where you will be able to reach more people likely to have
>> > > experience with all of the software you were using.
>> > >
>> > > And if you could provide more details on the actual steps you used, that
>> > > would be useful. For example, you said "configuration process" and "at
>> > > the end". But what actual commands were you running on those steps, for
>> > > example?
>> > >
>> > > -Carl
>> > >
>> > > On Mon, Apr 03 2023, Fulvio Pizzigoni wrote:
>> > > > Hi.
>> > > >
>> > > > I have installed packages notmuch and notmuch-mutt.
>> > > >
>> > > > During configuration process I have indicated the directory used from
>> > > > Mutt for your mail-boxes.
>> > > >
>> > > > The configuration process have not indicated any allert.
>> > > >
>> > > > At the end the mail-boxes in that directory was removed; 3 new
>> > > > empty sub-directory were created: cur, new, tmp.
>> > > >
>> > > > Was the data irremediably lost?
>> > > >
>> > > > Thanks in advance.
>> > > >
>> > > > Fulvio Pizzigoni
>> > _______________________________________________
>> > notmuch mailing list -- notmuch@notmuchmail.org
>> > To unsubscribe send an email to notmuch-leave@notmuchmail.org

Re: Data loss

[Michael J Gruber]

Am Di., 4. Apr. 2023 um 00:54 Uhr schrieb Carl Worth <cworth@debian.org>:
>
> Ouch.
>

Yes, we're feeling with you, Fulvio :|

> It's really unfortunate if notmuch-mutt makes it that easy to throw away
> your email.
>
> That sounds like a nasty bug that should be fixed in that program.

To be fair to notmuch-mutt:
```
--output-dir DIR

Store search results as (symlink) messages under maildir DIR. Beware: DIR will
be overwritten. (Default: F<~/.cache/notmuch/mutt/results/>)
```
So the default is safe, and the warning is there. It would be a bug
only if notmuch-mutt descended up and deleted a root dir or such.
Maybe naming the option `scratch-dir` or `cache-dir` or such could
have helped, but otoh the name of the default is there.

Note that neomutt incorporates notmuch-mutt's functionality, it might
be a better choice for this and other reasons.

> As for recovering, I suppose there _is_ a fair amount of detail in your
> notmuch index from all of the position-indexed terms, (as long as you
> haven't run "notmuch new" again since the mail was deleted).
>
> If you still have a large Xapian database in the notmuch database
> directory, it would be theoretically possible to recover a lot of the
> email content. But I don't know that anyone has ever written a recovery
> tool to help with that process.
>

That is an interesting question for the insiders, indeed - can e-mail
bodies be recovered fully (minus capitalisation, stemming)?

In Fulvio's case I would try a file-system dump (if it's not too late)
and a recovery tool depending on the fs type.

Michael

Re: Data loss

[Carl Worth]

On Tue, Apr 04 2023, Michael J. Gruber wrote:
>> It's really unfortunate if notmuch-mutt makes it that easy to throw away
>> your email.
>>
>> That sounds like a nasty bug that should be fixed in that program.
>
> To be fair to notmuch-mutt:
> ```
> --output-dir DIR
>
> Store search results as (symlink) messages under maildir DIR. Beware: DIR will
> be overwritten. (Default: F<~/.cache/notmuch/mutt/results/>)
> ```
> So the default is safe, and the warning is there. It would be a bug
> only if notmuch-mutt descended up and deleted a root dir or such.

OK. It's definitely acting as documented. So maybe "bug" was a little
harsh in my description.

It could still be made safer, and I think it would be a good idea to
change it.

For example, notmuch-mutt currently happily creates a directory that
doesn't already exists. That's safe and good.

What it could also do is create some placeholder file like
.NOTMUCH-MUTT-OUTPUT-DIR when it writes its output. And then it could
check for that file's existence before removing a directory.

-Carl

Re: Data loss

[David Bremner]

Carl Worth <cworth@cworth.org> writes:
>
> What it could also do is create some placeholder file like
> .NOTMUCH-MUTT-OUTPUT-DIR when it writes its output. And then it could
> check for that file's existence before removing a directory.
>

Sounds reasonable to me. Would someone like to make a patch for
notmuch-mutt?

d

[PATCH] notmuch-mutt: check that the search cache Maildir is not a real Maildir WAS: Re: Data loss

[Paul Wise]

[-- Attachment #1.1.1: Type: text/plain, Size: 891 bytes --]

David Bremner wrote:

> Carl Worth <cworth@cworth.org> writes:
> 
> > What it could also do is create some placeholder file like
> > .NOTMUCH-MUTT-OUTPUT-DIR when it writes its output. And then it could
> > check for that file's existence before removing a directory.
> 
> Sounds reasonable to me. Would someone like to make a patch for
> notmuch-mutt?

This has the problem that the migration from existing search cache
Maildirs to the new setup needs users to do manual work and that they
or something else could later modify the migrated Maildir by adding
real mail data or subfolders containing real data.

I think a better option is to detect if the search cache Maildir is
actually a Maildir and doesn't contain any data that could be lost.

I have attached three patches I created to improve notmuch-mutt safety.

-- 
bye,
pabs

https://bonedaddy.net/pabs3/

[-- Attachment #1.1.2: 0001-notmuch-mutt-clarify-the-empty-Maildir-function-oper.patch --]
[-- Type: text/x-patch, Size: 1694 bytes --]

From 22dad230dc368ff09cef2fd1eddd10ba89d41d16 Mon Sep 17 00:00:00 2001
From: Paul Wise <pabs3@bonedaddy.net>
Date: Wed, 24 May 2023 08:26:38 +0800
Subject: [PATCH 1/3] notmuch-mutt: clarify the empty Maildir function operates
 on search caches

Rename the function and adjust the documentation comment.
---
 contrib/notmuch-mutt/notmuch-mutt | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/contrib/notmuch-mutt/notmuch-mutt b/contrib/notmuch-mutt/notmuch-mutt
index d1e2c084..eef1dea5 100755
--- a/contrib/notmuch-mutt/notmuch-mutt
+++ b/contrib/notmuch-mutt/notmuch-mutt
@@ -27,8 +27,8 @@ $xdg_cache_dir = $ENV{XDG_CACHE_HOME} if $ENV{XDG_CACHE_HOME};
 my $cache_dir = "$xdg_cache_dir/notmuch/mutt";
 
 
-# create an empty maildir (if missing) or empty an existing maildir"
-sub empty_maildir($) {
+# create an empty search cache maildir (if missing) or empty existing one
+sub empty_search_cache_maildir($) {
     my ($maildir) = (@_);
     rmtree($maildir) if (-d $maildir);
     my $folder = new Mail::Box::Maildir(folder => $maildir,
@@ -46,7 +46,7 @@ sub search($$$) {
     push @args, "--duplicate=1" if $remove_dups;
     push @args, $query;
 
-    empty_maildir($maildir);
+    empty_search_cache_maildir($maildir);
     open my $pipe, '-|', @args or die "Running @args failed: $!\n";
     while (<$pipe>) {
 	chomp;
@@ -121,7 +121,7 @@ sub thread_action($$@) {
 
     my $mid = get_message_id();
     if (! defined $mid) {
-	empty_maildir($results_dir);
+	empty_search_cache_maildir($results_dir);
 	die "notmuch-mutt: cannot find Message-Id, abort.\n";
     }
     my $search_cmd = 'notmuch search --output=threads ' . shell_quote("id:$mid");
-- 
2.40.1


[-- Attachment #1.1.3: 0002-notmuch-mutt-do-not-clear-search-cache-Maildir-when-.patch --]
[-- Type: text/x-patch, Size: 918 bytes --]

From 4cb6a83e634b4c2dbbaa1099811cf84d65627029 Mon Sep 17 00:00:00 2001
From: Paul Wise <pabs3@bonedaddy.net>
Date: Wed, 24 May 2023 08:24:02 +0800
Subject: [PATCH 2/3] notmuch-mutt: do not clear search cache Maildir when
 nothing is found

The previous results might be useful to the user but
an empty directory definitely isn't useful.
---
 contrib/notmuch-mutt/notmuch-mutt | 1 -
 1 file changed, 1 deletion(-)

diff --git a/contrib/notmuch-mutt/notmuch-mutt b/contrib/notmuch-mutt/notmuch-mutt
index eef1dea5..9badc59c 100755
--- a/contrib/notmuch-mutt/notmuch-mutt
+++ b/contrib/notmuch-mutt/notmuch-mutt
@@ -121,7 +121,6 @@ sub thread_action($$@) {
 
     my $mid = get_message_id();
     if (! defined $mid) {
-	empty_search_cache_maildir($results_dir);
 	die "notmuch-mutt: cannot find Message-Id, abort.\n";
     }
     my $search_cmd = 'notmuch search --output=threads ' . shell_quote("id:$mid");
-- 
2.40.1


[-- Attachment #1.1.4: 0003-notmuch-mutt-check-that-the-search-cache-Maildir-is-.patch --]
[-- Type: text/x-patch, Size: 3153 bytes --]

From 46036801970d43c4aee684a46ff0b0e0649e7cfd Mon Sep 17 00:00:00 2001
From: Paul Wise <pabs3@bonedaddy.net>
Date: Wed, 24 May 2023 06:22:01 +0800
Subject: [PATCH 3/3] notmuch-mutt: check that the search cache Maildir is not
 a real Maildir

This prevents data loss when users configure the search cache Maildir to be a
real Maildir containing their real mail data, since the search cache Maildir
is expected to contain only symlinks to the real mail data.

Prevents: <ZCsQBNmbzwkvbpHA@localhost.localdomain>
---
 contrib/notmuch-mutt/notmuch-mutt | 46 +++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/contrib/notmuch-mutt/notmuch-mutt b/contrib/notmuch-mutt/notmuch-mutt
index 9badc59c..2569d055 100755
--- a/contrib/notmuch-mutt/notmuch-mutt
+++ b/contrib/notmuch-mutt/notmuch-mutt
@@ -13,6 +13,7 @@ use warnings;
 
 use File::Path;
 use File::Basename;
+use File::Find;
 use Getopt::Long qw(:config no_getopt_compat);
 use Mail::Header;
 use Mail::Box::Maildir;
@@ -26,10 +27,55 @@ my $xdg_cache_dir = "$ENV{HOME}/.cache";
 $xdg_cache_dir = $ENV{XDG_CACHE_HOME} if $ENV{XDG_CACHE_HOME};
 my $cache_dir = "$xdg_cache_dir/notmuch/mutt";
 
+sub die_dir($$) {
+    my ($maildir, $error) = @_;
+    die "notmuch-mutt: search cache maildir $maildir $error\n".
+        "Please ensure that the notmuch-mutt search cache Maildir\n".
+        "contains no subfolders or real mail data, only symlinks to mail\n";
+}
+
+sub die_subdir($$$) {
+    my ($maildir, $subdir, $error) = @_;
+    die_dir($maildir, "subdir $subdir $error");
+}
+
+# check that the search cache maildir is that and not a real maildir
+# otherwise there could be data loss when the search cache is emptied
+sub check_search_cache_maildir($) {
+    my ($maildir) = (@_);
+
+    return unless -e $maildir;
+
+    -d $maildir or die_dir($maildir, 'is not a directory');
+
+    opendir(my $mdh, $maildir) or die_dir($maildir, "cannot be opened: $!");
+    my @contents = grep { !/^\.\.?$/ } readdir $mdh;
+    closedir $mdh;
+
+    my @required = ('cur', 'new', 'tmp');
+    foreach my $d (@required) {
+        -l "$maildir/$d" and die_dir($maildir, "contains symlink $d");
+        -e "$maildir/$d" or die_subdir($maildir, $d, 'is missing');
+        -d "$maildir/$d" or die_subdir($maildir, $d, 'is not a directory');
+        find(sub {
+            $_ eq '.' and return;
+            $_ eq '..' and return;
+            -l $_ or die_subdir($maildir, $d, "contains non-symlink $_");
+        }, "$maildir/$d");
+    }
+
+    my %required = map { $_ => 1 } @required;
+    foreach my $d (@contents) {
+        -l "$maildir/$d" and die_dir( $maildir, "contains symlink $d");
+        -d "$maildir/$d" or die_dir( $maildir, "contains non-directory $d");
+        exists($required[$d]) or die_dir( $maildir, "contains directory $d");
+    }
+}
 
 # create an empty search cache maildir (if missing) or empty existing one
 sub empty_search_cache_maildir($) {
     my ($maildir) = (@_);
+    check_search_cache_maildir($maildir);
     rmtree($maildir) if (-d $maildir);
     my $folder = new Mail::Box::Maildir(folder => $maildir,
 					create => 1);
-- 
2.40.1


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [PATCH] notmuch-mutt: check that the search cache Maildir is not a real Maildir WAS: Re: Data loss

[David Bremner]

Paul Wise <pabs3@bonedaddy.net> writes:

> I think a better option is to detect if the search cache Maildir is
> actually a Maildir and doesn't contain any data that could be lost.
>
> I have attached three patches I created to improve notmuch-mutt safety.

If possible, please use git-send-email for future series. 

> +    my %required = map { $_ => 1 } @required;
> +    foreach my $d (@contents) {
> +        -l "$maildir/$d" and die_dir( $maildir, "contains symlink $d");
> +        -d "$maildir/$d" or die_dir( $maildir, "contains non-directory $d");
> +        exists($required[$d]) or die_dir( $maildir, "contains directory $d");
> +    }
> +}

This seems to have introduced a warning, and I think there's a bug.

% notmuch mutt search tag:inbox 
Argument "new" isn't numeric in array or hash lookup at /usr/bin/notmuch-mutt line 70.
Argument "cur" isn't numeric in array or hash lookup at /usr/bin/notmuch-mutt line 70.
Argument "tmp" isn't numeric in array or hash lookup at /usr/bin/notmuch-mutt line 70.

If I remember correctly you need to use {} for hash references in Perl.

>  
>  # create an empty search cache maildir (if missing) or empty existing one
>  sub empty_search_cache_maildir($) {
>      my ($maildir) = (@_);
> +    check_search_cache_maildir($maildir);
>      rmtree($maildir) if (-d $maildir);
>      my $folder = new Mail::Box::Maildir(folder => $maildir,
>  					create => 1);

It might clearer to use the return value from (a possibly renamed
check_search_cache (then the -d can be eliminated / folded into the new
function).

Re: [PATCH] notmuch-mutt: check that the search cache Maildir is not a real Maildir WAS: Re: Data loss

[Paul Wise]

[-- Attachment #1.1: Type: text/plain, Size: 2238 bytes --]

On Sat, 2023-05-27 at 21:10 -0300, David Bremner wrote:

> If possible, please use git-send-email for future series. 

Will do, I forgot about the --cover-letter option.

> This seems to have introduced a warning, and I think there's a bug.
> 
> % notmuch mutt search tag:inbox 
> Argument "new" isn't numeric in array or hash lookup at /usr/bin/notmuch-mutt line 70.
> Argument "cur" isn't numeric in array or hash lookup at /usr/bin/notmuch-mutt line 70.
> Argument "tmp" isn't numeric in array or hash lookup at /usr/bin/notmuch-mutt line 70.
> 
> If I remember correctly you need to use {} for hash references in Perl.

Woops! Thanks, fixed in the v2 patch series.

> It might clearer to use the return value from (a possibly renamed
> check_search_cache (then the -d can be eliminated / folded into the new
> function).

When the checks fail, the Maildir creation and Maildir symlink
additions shouldn't happen either, otherwise Maildirs with real data
may get mail files overwritten with symlinks back to themselves. So
there needs to be a die() or return before the creation and the search
anyway and so checks may as well just die() when the first check fails.

I experimented with returning anyway and it makes things more complex:

File::Find cannot abort the search except via die() and so the calling
code would need to use eval in order to catch the die() and return it.

Threading return values all the way back up to search_action adds lots
of error checks to every function along the way.

The search_action interactive mode doesn't do more than one search
anyway so it would still just need to die() when there is an error.

If the code were to grow a multi-command interactive prompt, that could
use eval to catch and print the die() errors. Such a prompt would need
to do that anyway, because the non-search actions call die() already.

So I think that throwing exceptions via die() is the simpler method.

Since checking that the Maildir is suitable is really a separate action
to clearing the previous search results, I've moved the check function
from the empty function to the search function in the v2 patch series.

-- 
bye,
pabs

https://bonedaddy.net/pabs3/

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

[PATCH v2 1/3] notmuch-mutt: clarify the empty Maildir function operates on search caches

[Paul Wise]

Rename the function and adjust the documentation comment.
---
 contrib/notmuch-mutt/notmuch-mutt | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/contrib/notmuch-mutt/notmuch-mutt b/contrib/notmuch-mutt/notmuch-mutt
index 01db6908..0f7379ac 100755
--- a/contrib/notmuch-mutt/notmuch-mutt
+++ b/contrib/notmuch-mutt/notmuch-mutt
@@ -26,8 +26,8 @@ $xdg_cache_dir = $ENV{XDG_CACHE_HOME} if $ENV{XDG_CACHE_HOME};
 my $cache_dir = "$xdg_cache_dir/notmuch/mutt";
 
 
-# create an empty maildir (if missing) or empty an existing maildir"
-sub empty_maildir($) {
+# create an empty search cache maildir (if missing) or empty existing one
+sub empty_search_cache_maildir($) {
     my ($maildir) = (@_);
     rmtree($maildir) if (-d $maildir);
     my $folder = new Mail::Box::Maildir(folder => $maildir,
@@ -45,7 +45,7 @@ sub search($$$) {
     push @args, "--duplicate=1" if $remove_dups;
     push @args, $query;
 
-    empty_maildir($maildir);
+    empty_search_cache_maildir($maildir);
     open my $pipe, '-|', @args or die "Running @args failed: $!\n";
     while (<$pipe>) {
 	chomp;
@@ -120,7 +120,7 @@ sub thread_action($$@) {
 
     my $mid = get_message_id();
     if (! defined $mid) {
-	empty_maildir($results_dir);
+	empty_search_cache_maildir($results_dir);
 	die "notmuch-mutt: cannot find Message-Id, abort.\n";
     }
 
-- 
2.40.1

[PATCH v2 2/3] notmuch-mutt: do not clear search cache Maildir when nothing is found

[Paul Wise]

The previous results might be useful to the user but
an empty directory definitely isn't useful.
---
 contrib/notmuch-mutt/notmuch-mutt | 1 -
 1 file changed, 1 deletion(-)

diff --git a/contrib/notmuch-mutt/notmuch-mutt b/contrib/notmuch-mutt/notmuch-mutt
index 0f7379ac..875fd032 100755
--- a/contrib/notmuch-mutt/notmuch-mutt
+++ b/contrib/notmuch-mutt/notmuch-mutt
@@ -120,7 +120,6 @@ sub thread_action($$@) {
 
     my $mid = get_message_id();
     if (! defined $mid) {
-	empty_search_cache_maildir($results_dir);
 	die "notmuch-mutt: cannot find Message-Id, abort.\n";
     }
 
-- 
2.40.1

[PATCH v2 3/3] notmuch-mutt: check that the search cache Maildir is not a real Maildir

[Paul Wise]

This prevents data loss when users configure the search cache Maildir to be a
real Maildir containing their real mail data, since the search cache Maildir
is expected to contain only symlinks to the real mail data.

Prevents: <ZCsQBNmbzwkvbpHA@localhost.localdomain>
---
 contrib/notmuch-mutt/notmuch-mutt | 46 +++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/contrib/notmuch-mutt/notmuch-mutt b/contrib/notmuch-mutt/notmuch-mutt
index 875fd032..1ac68065 100755
--- a/contrib/notmuch-mutt/notmuch-mutt
+++ b/contrib/notmuch-mutt/notmuch-mutt
@@ -13,6 +13,7 @@ use warnings;
 
 use File::Path;
 use File::Basename;
+use File::Find;
 use Getopt::Long qw(:config no_getopt_compat);
 use Mail::Header;
 use Mail::Box::Maildir;
@@ -25,6 +26,50 @@ my $xdg_cache_dir = "$ENV{HOME}/.cache";
 $xdg_cache_dir = $ENV{XDG_CACHE_HOME} if $ENV{XDG_CACHE_HOME};
 my $cache_dir = "$xdg_cache_dir/notmuch/mutt";
 
+sub die_dir($$) {
+    my ($maildir, $error) = @_;
+    die "notmuch-mutt: search cache maildir $maildir $error\n".
+        "Please ensure that the notmuch-mutt search cache Maildir\n".
+        "contains no subfolders or real mail data, only symlinks to mail\n";
+}
+
+sub die_subdir($$$) {
+    my ($maildir, $subdir, $error) = @_;
+    die_dir($maildir, "subdir $subdir $error");
+}
+
+# check that the search cache maildir is that and not a real maildir
+# otherwise there could be data loss when the search cache is emptied
+sub check_search_cache_maildir($) {
+    my ($maildir) = (@_);
+
+    return unless -e $maildir;
+
+    -d $maildir or die_dir($maildir, 'is not a directory');
+
+    opendir(my $mdh, $maildir) or die_dir($maildir, "cannot be opened: $!");
+    my @contents = grep { !/^\.\.?$/ } readdir $mdh;
+    closedir $mdh;
+
+    my @required = ('cur', 'new', 'tmp');
+    foreach my $d (@required) {
+        -l "$maildir/$d" and die_dir($maildir, "contains symlink $d");
+        -e "$maildir/$d" or die_subdir($maildir, $d, 'is missing');
+        -d "$maildir/$d" or die_subdir($maildir, $d, 'is not a directory');
+        find(sub {
+            $_ eq '.' and return;
+            $_ eq '..' and return;
+            -l $_ or die_subdir($maildir, $d, "contains non-symlink $_");
+        }, "$maildir/$d");
+    }
+
+    my %required = map { $_ => 1 } @required;
+    foreach my $d (@contents) {
+        -l "$maildir/$d" and die_dir( $maildir, "contains symlink $d");
+        -d "$maildir/$d" or die_dir( $maildir, "contains non-directory $d");
+        exists($required[$d]) or die_dir( $maildir, "contains directory $d");
+    }
+}
 
 # create an empty search cache maildir (if missing) or empty existing one
 sub empty_search_cache_maildir($) {
@@ -45,6 +90,7 @@ sub search($$$) {
     push @args, "--duplicate=1" if $remove_dups;
     push @args, $query;
 
+    check_search_cache_maildir($maildir);
     empty_search_cache_maildir($maildir);
     open my $pipe, '-|', @args or die "Running @args failed: $!\n";
     while (<$pipe>) {
-- 
2.40.1

[PATCH v3 3/3] notmuch-mutt: check that the search cache Maildir is not a real Maildir

[Paul Wise]

This prevents data loss when users configure the search cache Maildir to be a
real Maildir containing their real mail data, since the search cache Maildir
is expected to contain only symlinks to the real mail data.

Prevents: <ZCsQBNmbzwkvbpHA@localhost.localdomain>
---
 contrib/notmuch-mutt/notmuch-mutt | 46 +++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/contrib/notmuch-mutt/notmuch-mutt b/contrib/notmuch-mutt/notmuch-mutt
index 875fd032..b81252c8 100755
--- a/contrib/notmuch-mutt/notmuch-mutt
+++ b/contrib/notmuch-mutt/notmuch-mutt
@@ -13,6 +13,7 @@ use warnings;
 
 use File::Path;
 use File::Basename;
+use File::Find;
 use Getopt::Long qw(:config no_getopt_compat);
 use Mail::Header;
 use Mail::Box::Maildir;
@@ -25,6 +26,50 @@ my $xdg_cache_dir = "$ENV{HOME}/.cache";
 $xdg_cache_dir = $ENV{XDG_CACHE_HOME} if $ENV{XDG_CACHE_HOME};
 my $cache_dir = "$xdg_cache_dir/notmuch/mutt";
 
+sub die_dir($$) {
+    my ($maildir, $error) = @_;
+    die "notmuch-mutt: search cache maildir $maildir $error\n".
+        "Please ensure that the notmuch-mutt search cache Maildir\n".
+        "contains no subfolders or real mail data, only symlinks to mail\n";
+}
+
+sub die_subdir($$$) {
+    my ($maildir, $subdir, $error) = @_;
+    die_dir($maildir, "subdir $subdir $error");
+}
+
+# check that the search cache maildir is that and not a real maildir
+# otherwise there could be data loss when the search cache is emptied
+sub check_search_cache_maildir($) {
+    my ($maildir) = (@_);
+
+    return unless -e $maildir;
+
+    -d $maildir or die_dir($maildir, 'is not a directory');
+
+    opendir(my $mdh, $maildir) or die_dir($maildir, "cannot be opened: $!");
+    my @contents = grep { !/^\.\.?$/ } readdir $mdh;
+    closedir $mdh;
+
+    my @required = ('cur', 'new', 'tmp');
+    foreach my $d (@required) {
+        -l "$maildir/$d" and die_dir($maildir, "contains symlink $d");
+        -e "$maildir/$d" or die_subdir($maildir, $d, 'is missing');
+        -d "$maildir/$d" or die_subdir($maildir, $d, 'is not a directory');
+        find(sub {
+            $_ eq '.' and return;
+            $_ eq '..' and return;
+            -l $_ or die_subdir($maildir, $d, "contains non-symlink $_");
+        }, "$maildir/$d");
+    }
+
+    my %required = map { $_ => 1 } @required;
+    foreach my $d (@contents) {
+        -l "$maildir/$d" and die_dir( $maildir, "contains symlink $d");
+        -d "$maildir/$d" or die_dir( $maildir, "contains non-directory $d");
+        exists($required{$d}) or die_dir( $maildir, "contains directory $d");
+    }
+}
 
 # create an empty search cache maildir (if missing) or empty existing one
 sub empty_search_cache_maildir($) {
@@ -45,6 +90,7 @@ sub search($$$) {
     push @args, "--duplicate=1" if $remove_dups;
     push @args, $query;
 
+    check_search_cache_maildir($maildir);
     empty_search_cache_maildir($maildir);
     open my $pipe, '-|', @args or die "Running @args failed: $!\n";
     while (<$pipe>) {
-- 
2.40.1

Re: [PATCH v3 3/3] notmuch-mutt: check that the search cache Maildir is not a real Maildir

[David Bremner]

Paul Wise <pabs3@bonedaddy.net> writes:

> This prevents data loss when users configure the search cache Maildir to be a
> real Maildir containing their real mail data, since the search cache Maildir
> is expected to contain only symlinks to the real mail data.
>
> Prevents: <ZCsQBNmbzwkvbpHA@localhost.localdomain>

v2 applied to master

Re: [PATCH v3 3/3] notmuch-mutt: check that the search cache Maildir is not a real Maildir

[David Bremner]

Paul Wise <pabs3@bonedaddy.net> writes:

> This prevents data loss when users configure the search cache Maildir to be a
> real Maildir containing their real mail data, since the search cache Maildir
> is expected to contain only symlinks to the real mail data.
>
> Prevents: <ZCsQBNmbzwkvbpHA@localhost.localdomain>

Was there a change from V2? I might have missed it if so.

d

[PATCH] notmuch-mutt: fix Perl syntax of hash index lookups

[Paul Wise]

Fixes: commit 239fdbbbf0cbd6cd6ebafb87e88cdb3cded75364
---
 contrib/notmuch-mutt/notmuch-mutt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/contrib/notmuch-mutt/notmuch-mutt b/contrib/notmuch-mutt/notmuch-mutt
index 1ac68065..b81252c8 100755
--- a/contrib/notmuch-mutt/notmuch-mutt
+++ b/contrib/notmuch-mutt/notmuch-mutt
@@ -67,7 +67,7 @@ sub check_search_cache_maildir($) {
     foreach my $d (@contents) {
         -l "$maildir/$d" and die_dir( $maildir, "contains symlink $d");
         -d "$maildir/$d" or die_dir( $maildir, "contains non-directory $d");
-        exists($required[$d]) or die_dir( $maildir, "contains directory $d");
+        exists($required{$d}) or die_dir( $maildir, "contains directory $d");
     }
 }
 
-- 
2.42.0

Re: [PATCH] notmuch-mutt: fix Perl syntax of hash index lookups

[Tomi Ollila]

On Thu, Oct 12 2023, Paul Wise wrote:

> Fixes: commit 239fdbbbf0cbd6cd6ebafb87e88cdb3cded75364
> ---
>  contrib/notmuch-mutt/notmuch-mutt | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/contrib/notmuch-mutt/notmuch-mutt b/contrib/notmuch-mutt/notmuch-mutt
> index 1ac68065..b81252c8 100755
> --- a/contrib/notmuch-mutt/notmuch-mutt
> +++ b/contrib/notmuch-mutt/notmuch-mutt
> @@ -67,7 +67,7 @@ sub check_search_cache_maildir($) {
>      foreach my $d (@contents) {
>          -l "$maildir/$d" and die_dir( $maildir, "contains symlink $d");
>          -d "$maildir/$d" or die_dir( $maildir, "contains non-directory $d");
> -        exists($required[$d]) or die_dir( $maildir, "contains directory $d");
> +        exists($required{$d}) or die_dir( $maildir, "contains directory $d");

LGTM. (for sure that is hash referenced)

Tomi

Re: [PATCH] notmuch-mutt: fix Perl syntax of hash index lookups

[David Bremner]

Paul Wise <pabs3@bonedaddy.net> writes:

> Fixes: commit 239fdbbbf0cbd6cd6ebafb87e88cdb3cded75364
> ---
>  contrib/notmuch-mutt/notmuch-mutt | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/contrib/notmuch-mutt/notmuch-mutt b/contrib/notmuch-mutt/notmuch-mutt
> index 1ac68065..b81252c8 100755
> --- a/contrib/notmuch-mutt/notmuch-mutt
> +++ b/contrib/notmuch-mutt/notmuch-mutt
> @@ -67,7 +67,7 @@ sub check_search_cache_maildir($) {
>      foreach my $d (@contents) {
>          -l "$maildir/$d" and die_dir( $maildir, "contains symlink $d");
>          -d "$maildir/$d" or die_dir( $maildir, "contains non-directory $d");
> -        exists($required[$d]) or die_dir( $maildir, "contains directory $d");
> +        exists($required{$d}) or die_dir( $maildir, "contains directory $d");
>      }
>  }
>  
> -- 
> 2.42.0
>
> _______________________________________________
> notmuch mailing list -- notmuch@notmuchmail.org
> To unsubscribe send an email to notmuch-leave@notmuchmail.org

applied to release and master (will be part of 0.38.1)

d