Mail to self

Mail to self

[Leo Gaspard]

Hello,

I'm looking for a way to have notmuch consider, when it finds both an
unread and a read mail with the same Message-Id (eg. sending a mail to
self from a phone), that the mail is unread (because I'd rather have too
many mails unread than miss important mail, and it's also useful for
tests).

Do you see any way of having notmuch do this?

Cheers,
Leo

Re: Mail to self

[David Bremner]

Leo Gaspard <leo@gaspard.io> writes:

> Hello,
>
> I'm looking for a way to have notmuch consider, when it finds both an
> unread and a read mail with the same Message-Id (eg. sending a mail to
> self from a phone), that the mail is unread (because I'd rather have too
> many mails unread than miss important mail, and it's also useful for
> tests).
>
> Do you see any way of having notmuch do this?

After a discussion on IRC, my understanding is the following. The
underlying behaviour that Leo is objecting to is that if any one file
with a given message-id has the ,S maildir flag, then when notmuch syncs
maildir flags to tags, it will remove the unread tag from that
message-id. After some back and forth and thinking about it, I think
notmuch's current behaviour is actually correct (given the constraint
that tags attach to message-ids). You could argue for different ways of
resolving conflicts for maildir flags in general, but the ",S" or "seen"
flag has fairly natural common sense semantics.

d

Re: Mail to self

[Leo Gaspard]

> After a discussion on IRC, my understanding is the following. The
> underlying behaviour that Leo is objecting to is that if any one file
> with a given message-id has the ,S maildir flag, then when notmuch syncs
> maildir flags to tags, it will remove the unread tag from that
> message-id. After some back and forth and thinking about it, I think
> notmuch's current behaviour is actually correct (given the constraint
> that tags attach to message-ids). You could argue for different ways of
> resolving conflicts for maildir flags in general, but the ",S" or "seen"
> flag has fairly natural common sense semantics.

I can confirm this is my issue. My thinking is that if any file does not
have the ,S maildir flag, then I would prefer the mail to be marked as
unread, as a read mail that is spuriously marked as unread is a small
inconvenience, while an unread mail that is spuriously marked as read
can have huge consequences.

Actually, I just noticed while writing this mail that I had setup a
sieve filter for duplicate email that put them in a “Duplicate” folder
and automatically marks them as “read”, so that at the same time they
wouldn't bother me in the thunderbird interface but I could still check
dovecot's duplicate detection didn't have false positives.

This behaviour of notmuch thus made me miss 9 emails in the 2 weeks I've
been using it, the oldest being 12 days ago, without any warning. Well,
now I know about it and can try to change my setup (except I can't
really touch this sieve filter as other people rely on it), but…

If this is a willing choice, I'd be glad were it revisited :)

Re: Mail to self

[Gregor Zattler]

Hi Leo, notmuch users and developers,

I too filter emails into different folders.  And I used to filter
some emails in several folders in order to find them in different
contexts. 

* Leo Gaspard <notmuchmail.org@leo.gaspard.io> [2018-10-28; 16:31]:
> I can confirm this is my issue. My thinking is that if any file does not
> have the ,S maildir flag, then I would prefer the mail to be marked as
> unread, as a read mail that is spuriously marked as unread is a small
> inconvenience, while an unread mail that is spuriously marked as read
> can have huge consequences.

For me, when I filtered emails in several folders and read them
with mutt (1) it was the other way around: I wished for the
possibility to mark them as read in all folders if read in one.

I think your reasoning is also sound but only iff tags would
belong to files, but they belong to messages.

> Actually, I just noticed while writing this mail that I had setup a
> sieve filter for duplicate email that put them in a "Duplicate" folder
> and automatically marks them as "read", so that at the same time they
> wouldn't bother me in the thunderbird interface but I could still check
> dovecot's duplicate detection didn't have false positives.
>
> This behaviour of notmuch thus made me miss 9 emails in the 2 weeks I've
> been using it, the oldest being 12 days ago, without any warning. Well,
> now I know about it and can try to change my setup (except I can't
> really touch this sieve filter as other people rely on it), but...

But why would you tag a message (as opposed to a file) as
"unread" because it's a duplicate?

If it's somehow important to you to know that a certain email
your read via notmuch is a duplicate, why not tag it as
"duplicate" instead?

I suppose you do this filtering in case you read those emails
with a file/folders based client.  But would you actually "read"
emails in a duplicate folder?  I too have a duplicate folder and
it was useful some times in order to investigate problems with my
processing of incoming emails, but I do not read them in order to
understand the meaning of some message a human sent me.

Since I also filter emails in folders my searches heavily relay
on the path the message is in with the "path:" search prefix.
These is true for all mailing lists, for instance.

For the specific case of the duplicates folder: It's simply not
in the same maildir notmuch indexes and therefore there is no
issue with them for me.  My procmail scripts first do a backup of
an incoming email, later they filter duplicates in a duplicates
folder and when an email actually is delivered in the maildir
hierarchy notmuch indexes, then it is also backed up in a third
backup folder for post-processed emails.  These three special
folders are directories under my ~/.procmail directory and are
not indexed by notmuch.


HTH, Gregor
-- 
 -... --- .-. . -.. ..--.. ...-.-

Re: Mail to self

[Leo Gaspard]

Hi Gregor,

Gregor Zattler <telegraph@gmx.net> writes:
> For me, when I filtered emails in several folders and read them
> with mutt (1) it was the other way around: I wished for the
> possibility to mark them as read in all folders if read in one.

This isn't incompatible, and I'm not wishing that notmuch stop marking
as read per-message.

However, I *am* wishing that when some other tool marked one of the
message files as read and not the other, notmuch takes the safe route of
marking the overall message as unread in its database.

> But why would you tag a message (as opposed to a file) as
> "unread" because it's a duplicate?

Because usual folder-based applications display an unread marker with
the number of unread messages near the folder, and this is a useless
visual disturbance for the people for whom I administer the mail server.

> If it's somehow important to you to know that a certain email
> your read via notmuch is a duplicate, why not tag it as
> "duplicate" instead?

This is not a problem in notmuch (notmuch anyway just assumes that
duplicates with the same Message-ID will never happen, which in my
opinion is a bit optimistic, but I guess this design point is not going
to change anyway and so am not going to argue further), but it is a
problem with folder-based applications.

When using a folder-based application, I don't want a Duplicates folder
always popping up in my folder list with new messages until I read them,
I want a Duplicates folder only to debug when things go weird and
messages appear missing.

> For the specific case of the duplicates folder: [...]

Yes, now that I'm aware of this issue I can adapt my setup, likely by
not indexing the duplicates folder indeed.

However, I still think this is a big footgun, and take my missing 9
emails in the first 2 weeks of notmuch as proof of it.

Just to repeat it at the end so my objective is clear:

What I am wishing for is that when some tool outside of notmuch marked
one of the message files as read and not the other, upon
`synchronize_flags` notmuch takes the safe route of marking the overall
message as unread in its database.

Re: Mail to self

[Leo Gaspard]

(sorry for sending twice, had the wrong From: line once again, hopefully
I have fixed my client by adding this address to `user.other_email` now)

Hi Gregor,

Gregor Zattler <telegraph@gmx.net> writes:
> For me, when I filtered emails in several folders and read them
> with mutt (1) it was the other way around: I wished for the
> possibility to mark them as read in all folders if read in one.

This isn't incompatible, and I'm not wishing that notmuch stop marking
as read per-message.

However, I *am* wishing that when some other tool marked one of the
message files as read and not the other, notmuch takes the safe route of
marking the overall message as unread in its database.

> But why would you tag a message (as opposed to a file) as
> "unread" because it's a duplicate?

Because usual folder-based applications display an unread marker with
the number of unread messages near the folder, and this is a useless
visual disturbance for the people for whom I administer the mail server.

> If it's somehow important to you to know that a certain email
> your read via notmuch is a duplicate, why not tag it as
> "duplicate" instead?

This is not a problem in notmuch (notmuch anyway just assumes that
duplicates with the same Message-ID will never happen, which in my
opinion is a bit optimistic, but I guess this design point is not going
to change anyway and so am not going to argue further), but it is a
problem with folder-based applications.

When using a folder-based application, I don't want a Duplicates folder
always popping up in my folder list with new messages until I read them,
I want a Duplicates folder only to debug when things go weird and
messages appear missing.

> For the specific case of the duplicates folder: [...]

Yes, now that I'm aware of this issue I can adapt my setup, likely by
not indexing the duplicates folder indeed.

However, I still think this is a big footgun, and take my missing 9
emails in the first 2 weeks of notmuch as proof of it.

Just to repeat it at the end so my objective is clear:

What I am wishing for is that when some tool outside of notmuch marked
one of the message files as read and not the other, upon
`synchronize_flags` notmuch takes the safe route of marking the overall
message as unread in its database.

Re: Mail to self

[Leo Gaspard]

(sorry for sending twice, had the wrong From: line once again, hopefully
I have fixed my client by adding this address to `user.other_email` now…
need some time to adapt, sorry)

Hi Gregor,

Gregor Zattler <telegraph@gmx.net> writes:
> For me, when I filtered emails in several folders and read them
> with mutt (1) it was the other way around: I wished for the
> possibility to mark them as read in all folders if read in one.

This isn't incompatible, and I'm not wishing that notmuch stop marking
as read per-message.

However, I *am* wishing that when some other tool marked one of the
message files as read and not the other, notmuch takes the safe route of
marking the overall message as unread in its database.

> But why would you tag a message (as opposed to a file) as
> "unread" because it's a duplicate?

Because usual folder-based applications display an unread marker with
the number of unread messages near the folder, and this is a useless
visual disturbance for the people for whom I administer the mail server.

> If it's somehow important to you to know that a certain email
> your read via notmuch is a duplicate, why not tag it as
> "duplicate" instead?

This is not a problem in notmuch (notmuch anyway just assumes that
duplicates with the same Message-ID will never happen, which in my
opinion is a bit optimistic, but I guess this design point is not going
to change anyway and so am not going to argue further), but it is a
problem with folder-based applications.

When using a folder-based application, I don't want a Duplicates folder
always popping up in my folder list with new messages until I read them,
I want a Duplicates folder only to debug when things go weird and
messages appear missing.

> For the specific case of the duplicates folder: [...]

Yes, now that I'm aware of this issue I can adapt my setup, likely by
not indexing the duplicates folder indeed.

However, I still think this is a big footgun, and take my missing 9
emails in the first 2 weeks of notmuch as proof of it.

Just to repeat it at the end so my objective is clear:

What I am wishing for is that when some tool outside of notmuch marked
one of the message files as read and not the other, upon
`synchronize_flags` notmuch takes the safe route of marking the overall
message as unread in its database.

[PATCH 0/1] notmuch: be conservative and prefer marking too many messages as unread than too few

[notmuchmail.org]

From: Léo Gaspard <notmuchmail.org@leo.gaspard.io>

Here a patch that does what I want, hopefully this makes what I expected from
notmuch to be more explicit.

Note: it is not ready for integration yet, as the “Removing info from filename
leaves tags unchanged” test should likely not add “unread” back.

If I hear something positive about it that makes me think it has a chance of
going upstream I will try to adapt it, otherwise I will just continue running my
local notmuch instance with it applied.

Cheers :)
Leo

Léo Gaspard (1):
  notmuch: be conservative and prefer marking too many messages as
    unread than too few

 lib/message.cc            | 18 +++++++++++++++++-
 test/T340-maildir-sync.sh | 29 +++++++++++++++++++----------
 2 files changed, 36 insertions(+), 11 deletions(-)

-- 
2.18.1

[PATCH 1/1] notmuch: be conservative and prefer marking too many messages as unread than too few

[notmuchmail.org]

From: Léo Gaspard <notmuchmail.org@leo.gaspard.io>

When two maildir flags have different flags, currently the behavior is
to take the union of the all the flags.

For unread, this means that if an email has duplicate files, that it is
read, and that then on eg. a mobile device one of the files is marked as
unread (but not the other), then notmuch won't catch this and keep the
message as read.

This patch makes notmuch consider that when a message has both a file
marked read and a file marked unread from maildir, then it will be
treated as unread for notmuch.
---
 lib/message.cc            | 18 +++++++++++++++++-
 test/T340-maildir-sync.sh | 29 +++++++++++++++++++----------
 2 files changed, 36 insertions(+), 11 deletions(-)

diff --git a/lib/message.cc b/lib/message.cc
index 6f2f6345..90bbb32a 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -1,6 +1,7 @@
 /* message.cc - Results of message-based searches from a notmuch database
  *
  * Copyright © 2009 Carl Worth
+ * Copyright © 2018 Leo Gaspard
  *
  * This program is free software: you can redistribute it and/or modify
  * it under the terms of the GNU General Public License as published by
@@ -15,7 +16,8 @@
  * You should have received a copy of the GNU General Public License
  * along with this program.  If not, see https://www.gnu.org/licenses/ .
  *
- * Author: Carl Worth <cworth@cworth.org>
+ * Authors: Carl Worth <cworth@cworth.org>
+ *          Leo Gaspard <leo@gaspard.io>
  */
 
 #include "notmuch-private.h"
@@ -1682,6 +1684,7 @@ notmuch_message_maildir_flags_to_tags (notmuch_message_t *message)
 {
     notmuch_status_t status;
     unsigned i;
+    int num_read_flags;
 
     _ensure_maildir_flags (message, true);
     /* If none of the filenames have any maildir info field (not even
@@ -1695,6 +1698,8 @@ notmuch_message_maildir_flags_to_tags (notmuch_message_t *message)
 	return status;
 
     for (i = 0; i < ARRAY_SIZE(flag2tag); i++) {
+	if (flag2tag[i].flag == 'S')
+	    continue;
 	if ((strchr (message->maildir_flags, flag2tag[i].flag) != NULL)
 	    ^
 	    flag2tag[i].inverse)
@@ -1706,6 +1711,17 @@ notmuch_message_maildir_flags_to_tags (notmuch_message_t *message)
 	if (status)
 	    return status;
     }
+    num_read_flags = 0;
+    for (i = 0; message->maildir_flags[i] != '\0'; ++i)
+	if (message->maildir_flags[i] == 'S')
+	    ++num_read_flags;
+    if (notmuch_message_count_files (message) == num_read_flags)
+	status = notmuch_message_remove_tag (message, "unread");
+    else
+	status = notmuch_message_add_tag (message, "unread");
+    if (status)
+	return status;
+
     status = notmuch_message_thaw (message);
 
     return status;
diff --git a/test/T340-maildir-sync.sh b/test/T340-maildir-sync.sh
index 7fece5f2..88acab3b 100755
--- a/test/T340-maildir-sync.sh
+++ b/test/T340-maildir-sync.sh
@@ -121,16 +121,16 @@ test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; R
 No new mail. Detected 1 file rename.
 thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Removing S flag (inbox unread)"
 
-test_begin_subtest "Removing info from filename leaves tags unchanged"
+test_begin_subtest "Removing info from filename leaves tags unchanged (but re-adds unread)"
 add_message [subject]='"Message to lose maildir info"' [filename]='message-to-lose-maildir-info' [dir]=cur
-notmuch tag -unread subject:"Message to lose maildir info"
-mv "$MAIL_DIR/cur/message-to-lose-maildir-info:2,S" "$MAIL_DIR/cur/message-without-maildir-info"
+notmuch tag -unread +replied subject:"Message to lose maildir info"
+mv "$MAIL_DIR/cur/message-to-lose-maildir-info:2,RS" "$MAIL_DIR/cur/message-without-maildir-info"
 output=$(NOTMUCH_NEW)
 output+="
 "
 output+=$(notmuch search subject:"Message to lose maildir info" | notmuch_search_sanitize)
 test_expect_equal "$output" "No new mail. Detected 1 file rename.
-thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Message to lose maildir info (inbox)"
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Message to lose maildir info (inbox replied unread)"
 
 test_begin_subtest "Can remove unread tag from message in non-maildir directory"
 add_message [subject]='"Non-maildir message"' [dir]=notmaildir [filename]='non-maildir-message'
@@ -155,27 +155,36 @@ notmuch restore --input=dump.txt
 output=$(ls $MAIL_DIR/cur)
 test_expect_equal "$output" "$expected"
 
-test_begin_subtest 'Adding flags to duplicate message tags the mail'
+test_begin_subtest 'Adding flags to duplicate message tags the mail (except for unread)'
 add_message [subject]='"Duplicated message"' [dir]=cur [filename]='duplicated-message:2,'
 cp "$MAIL_DIR/cur/duplicated-message:2," "$MAIL_DIR/cur/duplicated-message-copy:2,RS"
 NOTMUCH_NEW > output
 notmuch search subject:"Duplicated message" | notmuch_search_sanitize >> output
 test_expect_equal "$(< output)" "No new mail.
+thread:XXX   2001-01-05 [1/1(2)] Notmuch Test Suite; Duplicated message (inbox replied unread)"
+
+test_begin_subtest 'Adding flags to all duplicate messages tags the mail'
+add_message [subject]='"Duplicated message"' [dir]=cur [filename]='duplicated-message:2,'
+cp "$MAIL_DIR/cur/duplicated-message:2," "$MAIL_DIR/cur/duplicated-message-copy:2,RS"
+mv "$MAIL_DIR/cur/duplicated-message:2," "$MAIL_DIR/cur/duplicated-message:2,S"
+NOTMUCH_NEW > output
+notmuch search subject:"Duplicated message" | notmuch_search_sanitize >> output
+test_expect_equal "$(< output)" "No new mail. Detected 1 file rename.
 thread:XXX   2001-01-05 [1/1(2)] Notmuch Test Suite; Duplicated message (inbox replied)"
 
-test_begin_subtest "Adding duplicate message without flags does not remove tags"
+test_begin_subtest "Adding duplicate message without flags does not remove tags (but adds unread)"
 cp "$MAIL_DIR/cur/duplicated-message-copy:2,RS" "$MAIL_DIR/cur/duplicated-message-another-copy:2,"
 NOTMUCH_NEW > output
 notmuch search subject:"Duplicated message" | notmuch_search_sanitize >> output
 test_expect_equal "$(< output)" "No new mail.
-thread:XXX   2001-01-05 [1/1(3)] Notmuch Test Suite; Duplicated message (inbox replied)"
+thread:XXX   2001-01-05 [1/1(3)] Notmuch Test Suite; Duplicated message (inbox replied unread)"
 
 test_begin_subtest "Tag changes modify flags of multiple files"
 notmuch tag -replied subject:"Duplicated message"
 (cd $MAIL_DIR/cur/; ls duplicated*) > actual
-test_expect_equal "$(< actual)"  "duplicated-message-another-copy:2,S
-duplicated-message-copy:2,S
-duplicated-message:2,S"
+test_expect_equal "$(< actual)"  "duplicated-message-another-copy:2,
+duplicated-message-copy:2,
+duplicated-message:2,"
 
 test_begin_subtest "Synchronizing tag changes preserves unsupported maildir flags"
 add_message [subject]='"Unsupported maildir flags"' [dir]=cur [filename]='unsupported-maildir-flags:2,FSZxyz'
-- 
2.18.1