Tagging messages sometimes causes them to be marked as old

Tagging messages sometimes causes them to be marked as old

[Andreas Kusalananda Kähäri]

Greetings from a fairly new notmuch user.

I'm using notmuch 0.37 on OpenBSD.  I tag my incoming mail with a
"post-new" hook shell script that looks like this:

#!/bin/sh

tr -s '\t' ' ' <<'TAG_BATCH' | notmuch tag --batch
+inbox          -- folder:/INBOX/
+sent           -- folder:/Sent/
+trash          -- folder:/Trash/
-archive        -- not folder:/Archive/
-junk           -- not folder:/Junk/
-unsorted       -- not folder:/Unsorted/
TAG_BATCH

This tags the mail correctly in my local mail store, which is
"~/Mail/inboxes" with a subdirectory for each of the five accounts that
I receive mail for (so I have "~/Mail/inboxes/work-account-1/INBOX"
etc.)

However, since I often do notmuch searches on the command line, I'd like
to see what account the search results come from, which I would see if
the messages were tagged with the corresponding account name.

I therefore added tagging like this:

notmuch tag +account-workaccount1 -- 'path:work-account-1/**'

(and similarly for the other accounts; it does not matter if I do a
"path:" or modify it into a "folder:" search).  This works and tags the
messages correctly, however, for some reason, any unread mail are also
marked as "old" in mutt (the mail file is moved from the "new" to the
"cur" directory in the maildir).  This does not happen with the tagging
done by the original "post-new" hook shown above.  It *does* happen if I
add the tagging to the hook script.

If anyone could enlighten me as to why this happens, I'd be grateful.
It may be as simple as not understading the search or tag semantics, or
missing some obvious implication of tagging like this.

In my configuration, apart from the initial default things, I have:

[new]
tags=inbox;unread

[maildir]
syncronize_flags=true

[index]
header.List=List-Id

(the "post-new" hook is picked up from the ".notmuch/hooks" directory in the
mail store, so it's not mentioned in the configuration).

-- 
Andreas (Kusalananda) Kähäri
SciLifeLab, NBIS, ICM
Uppsala University, Sweden

.\r

Re: Tagging messages sometimes causes them to be marked as old

[David Bremner]

Andreas Kusalananda Kähäri <andreas.kahari@abc.se> writes:


> (and similarly for the other accounts; it does not matter if I do a
> "path:" or modify it into a "folder:" search).  This works and tags the
> messages correctly, however, for some reason, any unread mail are also
> marked as "old" in mutt (the mail file is moved from the "new" to the
> "cur" directory in the maildir).  This does not happen with the tagging
> done by the original "post-new" hook shown above.  It *does* happen if I
> add the tagging to the hook script.
>

Short answer: the mutt concept of "old" conflicts with the maildir
definition. If you want any maildir flags, your message cannot be in
new/, which iirc is what mutt means by "not old". You could turn off
maildir flag synching (but of course that has side effects).\r

Re: Tagging messages sometimes causes them to be marked as old

[Andreas Kusalananda Kähäri]

On Thu, Aug 31, 2023 at 01:24:25PM +0000, David Bremner wrote:
> Andreas Kusalananda Kähäri <andreas.kahari@abc.se> writes:
> 
> 
> > (and similarly for the other accounts; it does not matter if I do a
> > "path:" or modify it into a "folder:" search).  This works and tags the
> > messages correctly, however, for some reason, any unread mail are also
> > marked as "old" in mutt (the mail file is moved from the "new" to the
> > "cur" directory in the maildir).  This does not happen with the tagging
> > done by the original "post-new" hook shown above.  It *does* happen if I
> > add the tagging to the hook script.
> >
> 
> Short answer: the mutt concept of "old" conflicts with the maildir
> definition. If you want any maildir flags, your message cannot be in
> new/, which iirc is what mutt means by "not old". You could turn off
> maildir flag synching (but of course that has side effects).

Mutt shows messages without the "S" flag in "cur" as "old", and messages
without the "S" flag in "new" as "new".  Messages with the "S" flag are
shown as read.  This is the default behaviour of mutt.  I don't really
care if this is breaking conventions or not, as this conversation is not
really about mutt at all.

What's so special about the new tags that I'm adding that makes the
unread messages move from "new" to "cur"?  I can *see* them getting
moved when I run the tagging.  Why is notmuch moving them?  The "unread"
tag never changes.

	$ ls "$notmuch_dbdir"/abc/INBOX/new
	1693487404_0.62384.harpo.local,U=352958,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
	1693488604_2.29495.harpo.local,U=352962,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
	1693488604_4.29495.harpo.local,U=352964,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
	1693488604_5.29495.harpo.local,U=352965,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
	1693488604_6.29495.harpo.local,U=352966,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
	1693488604_7.29495.harpo.local,U=352967,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
	1693488604_8.29495.harpo.local,U=352968,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
	1693489804_0.76572.harpo.local,U=352969,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
	1693489804_1.76572.harpo.local,U=352970,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
	1693489804_2.76572.harpo.local,U=352971,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
	1693489804_3.76572.harpo.local,U=352972,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
	1693489804_4.76572.harpo.local,U=352973,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
	1693489804_5.76572.harpo.local,U=352974,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
	$ notmuch tag +test -- 'path:abc/INBOX/**'
	$ ls "$notmuch_dbdir"/abc/INBOX/new
	(no output)

With

	notmuch search --output=files -- tag:unread and 'path:abc/INBOX/**'

I now see that they were moved into the neighbouring "cur" directory.


-- 
Andreas (Kusalananda) Kähäri
SciLifeLab, NBIS, ICM
Uppsala University, Sweden

.\r

Re: Tagging messages sometimes causes them to be marked as old

[David Bremner]

Andreas Kusalananda Kähäri <andreas.kahari@abc.se> writes:

> Mutt shows messages without the "S" flag in "cur" as "old", and messages
> without the "S" flag in "new" as "new".  Messages with the "S" flag are
> shown as read.  This is the default behaviour of mutt.  I don't really
> care if this is breaking conventions or not, as this conversation is not
> really about mutt at all.

I thought you were changing some tag synched to a maildir flags, but
apparently not. Notmuch makes some effort not to move files from new to
cur until they need to be, but I guess there could be a bug in that
code.

\r

Re: Tagging messages sometimes causes them to be marked as old

[David Bremner]

Andreas Kusalananda Kähäri <andreas.kahari@abc.se> writes:

> What's so special about the new tags that I'm adding that makes the
> unread messages move from "new" to "cur"?  I can *see* them getting
> moved when I run the tagging.  Why is notmuch moving them?  The "unread"
> tag never changes.
>
> 	$ ls "$notmuch_dbdir"/abc/INBOX/new
> 	1693487404_0.62384.harpo.local,U=352958,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
> 	1693488604_2.29495.harpo.local,U=352962,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
> 	1693488604_4.29495.harpo.local,U=352964,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
> 	1693488604_5.29495.harpo.local,U=352965,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,

Apparently this is intended behaviour (from test/T340-maildir-sync.sh)

    test_begin_subtest "Message in new with maildir info is moved to cur on any tag change"
    add_message [filename]='message-with-info-to-be-moved-to-cur:2,' [dir]=new
    notmuch tag +anytag id:$gen_msg_id
    output=$(cd "$MAIL_DIR"; ls */message-with-info-to-be-moved-to-cur*)
    test_expect_equal "$output" "cur/message-with-info-to-be-moved-to-cur:2,"

That behaviour is about 10 years old. It seems to be based on the idea
that files in new/ should not have maildir info. According to the
standard [1] (such as it is)

        When you move a file from new to cur, you have to change its
        name from uniq to uniq:info. Make sure to preserve the uniq
        string, so that separate messages can't bump into each other.

So in a sense notmuch is "fixing" the maildir. Whether that intervention
is needed notmuch to work correctly, I'm not sure.

I'd be a bit nervous about changing such an established behaviour of
notmuch, but I guess can't completely rule it out without knowing what
would break.

Maybe you can convince whatever puts files in new/ not to add :2, ?

[1]: http://cr.yp.to/proto/maildir.html

Re: Tagging messages sometimes causes them to be marked as old

[Andreas Kusalananda Kähäri]

On Fri, Sep 01, 2023 at 12:16:58PM +0000, David Bremner wrote:
> Andreas Kusalananda Kähäri <andreas.kahari@abc.se> writes:
> 
> > What's so special about the new tags that I'm adding that makes the
> > unread messages move from "new" to "cur"?  I can *see* them getting
> > moved when I run the tagging.  Why is notmuch moving them?  The "unread"
> > tag never changes.
> >
> > 	$ ls "$notmuch_dbdir"/abc/INBOX/new
> > 	1693487404_0.62384.harpo.local,U=352958,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
> > 	1693488604_2.29495.harpo.local,U=352962,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
> > 	1693488604_4.29495.harpo.local,U=352964,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
> > 	1693488604_5.29495.harpo.local,U=352965,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
> 
> Apparently this is intended behaviour (from test/T340-maildir-sync.sh)
> 
>     test_begin_subtest "Message in new with maildir info is moved to cur on any tag change"
>     add_message [filename]='message-with-info-to-be-moved-to-cur:2,' [dir]=new
>     notmuch tag +anytag id:$gen_msg_id
>     output=$(cd "$MAIL_DIR"; ls */message-with-info-to-be-moved-to-cur*)
>     test_expect_equal "$output" "cur/message-with-info-to-be-moved-to-cur:2,"
> 
> That behaviour is about 10 years old. It seems to be based on the idea
> that files in new/ should not have maildir info. According to the
> standard [1] (such as it is)
> 
>         When you move a file from new to cur, you have to change its
>         name from uniq to uniq:info. Make sure to preserve the uniq
>         string, so that separate messages can't bump into each other.
> 
> So in a sense notmuch is "fixing" the maildir. Whether that intervention
> is needed notmuch to work correctly, I'm not sure.
> 
> I'd be a bit nervous about changing such an established behaviour of
> notmuch, but I guess can't completely rule it out without knowing what
> would break.
> 
> Maybe you can convince whatever puts files in new/ not to add :2, ?
> 
> [1]: http://cr.yp.to/proto/maildir.html

Thanks for the explanation.  It's offlineimap that puts the files in
"new" and it seems to add them with the ":2," suffix.

I don't think I can change this behaviour in offlineimap.

However, note that I have a "post-new" hook for notmuch which already
adds certain tags to messages.  I'll include it again (was part of my
initial message to the list):

#!/bin/sh

tr -s '\t' ' ' <<'TAG_BATCH' | notmuch tag --batch
+inbox          -- folder:/INBOX/
+sent           -- folder:/Sent/
+trash          -- folder:/Trash/
-archive        -- not folder:/Archive/
-junk           -- not folder:/Junk/
-unsorted       -- not folder:/Unsorted/
TAG_BATCH

This hook does not cause any issues for me, and the newly delivered
mesages are still in "new" after it has run.  It does not matter if I
run the hook manually or if it is run automatically by notmuch.

Running "notmuch tag" as I showed in a previous message then moves the
mesages to the "cur" directory.

What I also noted in my initial mail was that if I add the extra tagging
*to the hook script*, the issue appears.  It is as if using a "path:" or
"folder:" query that matches a directory that is not a maildir causes
the messages to be moved from "new" to "cur".

I store mails in five account subdirectories under the "~Mail/inboxes"
top-level directory (the notmuch database.path directory).  An account
subdirectory may contain several maildirs, such as "INBOX".  The
offending tagging all tries to match directory names (account names)
directly under "~Mail/inboxes".



-- 
Andreas (Kusalananda) Kähäri
SciLifeLab, NBIS, ICM
Uppsala University, Sweden

.\r

Re: Tagging messages sometimes causes them to be marked as old

[Andreas Kusalananda Kähäri]

On Fri, Sep 01, 2023 at 02:42:01PM +0200, Kusalananda Kähäri wrote:
> On Fri, Sep 01, 2023 at 12:16:58PM +0000, David Bremner wrote:
> > Andreas Kusalananda Kähäri <andreas.kahari@abc.se> writes:
> > 
> > > What's so special about the new tags that I'm adding that makes the
> > > unread messages move from "new" to "cur"?  I can *see* them getting
> > > moved when I run the tagging.  Why is notmuch moving them?  The "unread"
> > > tag never changes.
> > >
> > > 	$ ls "$notmuch_dbdir"/abc/INBOX/new
> > > 	1693487404_0.62384.harpo.local,U=352958,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
> > > 	1693488604_2.29495.harpo.local,U=352962,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
> > > 	1693488604_4.29495.harpo.local,U=352964,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
> > > 	1693488604_5.29495.harpo.local,U=352965,FMD5=7e33429f656f1e6e9d79b29c3f82c57e:2,
> > 
> > Apparently this is intended behaviour (from test/T340-maildir-sync.sh)
> > 
> >     test_begin_subtest "Message in new with maildir info is moved to cur on any tag change"
> >     add_message [filename]='message-with-info-to-be-moved-to-cur:2,' [dir]=new
> >     notmuch tag +anytag id:$gen_msg_id
> >     output=$(cd "$MAIL_DIR"; ls */message-with-info-to-be-moved-to-cur*)
> >     test_expect_equal "$output" "cur/message-with-info-to-be-moved-to-cur:2,"
> > 
> > That behaviour is about 10 years old. It seems to be based on the idea
> > that files in new/ should not have maildir info. According to the
> > standard [1] (such as it is)
> > 
> >         When you move a file from new to cur, you have to change its
> >         name from uniq to uniq:info. Make sure to preserve the uniq
> >         string, so that separate messages can't bump into each other.
> > 
> > So in a sense notmuch is "fixing" the maildir. Whether that intervention
> > is needed notmuch to work correctly, I'm not sure.
> > 
> > I'd be a bit nervous about changing such an established behaviour of
> > notmuch, but I guess can't completely rule it out without knowing what
> > would break.
> > 
> > Maybe you can convince whatever puts files in new/ not to add :2, ?
> > 
> > [1]: http://cr.yp.to/proto/maildir.html
> 
> Thanks for the explanation.  It's offlineimap that puts the files in
> "new" and it seems to add them with the ":2," suffix.
> 
> I don't think I can change this behaviour in offlineimap.
> 

Sorry, but I just found this: 

	https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=789625;msg=2

It appear someone else has already had this problem and that one might
possibly want to fix it in offlineimap.  Mutt is not at fault here, but
I still think notmuch is a bit pedantic when it moves the files.

I will contact the package maintainer of offlineimap on my system
(OpenBSD) and see if this is something that can be fixed there with the
simple patch includud in the abovementioned bug report.

Thanks!


-- 
Andreas (Kusalananda) Kähäri
SciLifeLab, NBIS, ICM
Uppsala University, Sweden

.\r