Difficulty understanding maildir.synchronize_flags behavior

Difficulty understanding maildir.synchronize_flags behavior

[Andrew Todd]

Hello,

I'm trying to set up notmuch, and I think that the behavior I want 
should be possible (and is desirable), but I can't seem to make it work. 
I also can't find any reference online suggesting that it *shouldn't* 
work, and I would have thought that it's a very common use case for 
people using multiple mail clients with one IMAP server.

Running Arch Linux, so on latest (currently 0.38.1). Working with the 
CLI and Emacs frontend.

I'm synchronizing mail from a remote IMAP server to local maildir using 
isync/mbsync.

What I want is for messages that do not have maildir files ending in 
':2,S' to be tagged as 'unread' in notmuch.

In other words, I want notmuch to respect the read/unread state of 
messages from the IMAP server and not make any other changes.

My understanding is that this is the purpose of the 
maildir.synchronize_flags option. I have explicitly set this to true in 
my config:

[maildir]
synchronize_flags=true

Moreover, I have also turned off all default tags when running notmuch new:

[new]
tags=

What I see is:

1) When I pull in messages with `mbsync`, they end up in the `new/` 
directory as expected.
2) When I run `notmuch new`, the messages remain in the `new/` 
directory. I guess this is correct. I don't really mind where they are.
3) When I open the Emacs interface, I see that the message is not tagged 
as `unread`. Moreover, when I read the message, it is not moved from 
`new/` to `cur/`, and the filename does not change from ':2,' to ':2,S'.

This is the first part of the problem.

The second part is that when I try to implement tagging in a post-new 
hook, all messages are immediately marked as read, e.g. they are moved 
from `new/` to `cur/` and their filename ending changes from ':2,' to 
':2,S'.

This is the only tag operation I'm performing:

+inbox -- folder:current/Inbox

My understanding of what the behavior should be is that:

1) `notmuch new` should automatically tag messages in `new/` and `cur/` 
that do not have 'S' in their filename as 'unread'.
2) `notmuch tag` should not mark an unread message as read; maybe this 
is happening because `notmuch new` didn't add the `unread` tag beforehand?

On the other hand, I have confirmed that if I go into the emacs 
interface and add the 'unread' tag to messages, their maildir files do 
get the 'S' removed from them. So the tag synchronization is working in 
the opposite direction.

I feel like I must be missing something fundamental in my understanding, 
or there is a fundamental bug in maildir.synchronize_flags. I would 
appreciate any advice.

Thank you,
Andrew Todd
at@auspicacious.org

Re: Difficulty understanding maildir.synchronize_flags behavior

[Michael J Gruber]

[-- Attachment #1.1: Type: text/plain, Size: 1696 bytes --]

Am Di., 12. Dez. 2023 um 10:58 Uhr schrieb Andrew Todd <at@auspicacious.org
>:

> Hello,
>
> I'm trying to set up notmuch, and I think that the behavior I want
> should be possible (and is desirable), but I can't seem to make it work.
> I also can't find any reference online suggesting that it *shouldn't*
> work, and I would have thought that it's a very common use case for
> people using multiple mail clients with one IMAP server.
>
> Running Arch Linux, so on latest (currently 0.38.1). Working with the
> CLI and Emacs frontend.
>
> I'm synchronizing mail from a remote IMAP server to local maildir using
> isync/mbsync.
>
> What I want is for messages that do not have maildir files ending in
> ':2,S' to be tagged as 'unread' in notmuch.
>
> In other words, I want notmuch to respect the read/unread state of
> messages from the IMAP server and not make any other changes.
>
> My understanding is that this is the purpose of the
> maildir.synchronize_flags option. I have explicitly set this to true in
> my config:
>
> [maildir]
> synchronize_flags=true
>
> Moreover, I have also turned off all default tags when running notmuch new:
>
> [new]
> tags=
>
> The default is `unread;inbox`. I understand you don't want the inbox tag,
but have you tried with `unread` here?

The manual is not overly clear about the interaction between this config
and flag sync, but AFAIU, `notmuch new` sets `unread` and then possibly
unsets it during the flag sync (maybe in one atomic db write, i.e. you
don't see this). By removing `unread` from `new.tags` you keep `notmuch
new` from doing that first step.

And just in case this feels like a "duh moment" to you - we all have them ;)

Cheers,
Michael

[-- Attachment #1.2: Type: text/html, Size: 2265 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Difficulty understanding maildir.synchronize_flags behavior

[Andrew Todd]

On 12/12/23 20:25, Michael J Gruber wrote:
> The manual is not overly clear about the interaction between this config 
> and flag sync, but AFAIU, `notmuch new` sets `unread` and then possibly 
> unsets it during the flag sync (maybe in one atomic db write, i.e. you 
> don't see this). By removing `unread` from `new.tags` you keep `notmuch 
> new` from doing that first step.

Thanks for your help; it pushed me to explore a bit more. I have 
realized that something much more subtle is going on. In fact, you do 
not have to set `tags=unread` in the configuration for this to work.

First, I noticed that other maildir flags, such as D/draft and 
R/replied, were synchronizing correctly.

Then, I realized that I was doing my testing by sending emails to 
myself. This created duplicate copies of the same message, with the same 
message ID, in both `Inbox` and `Sent`.

However, the copy of the message in `Sent` was, of course, already 
marked as read. notmuch's synchronization was working as intended, but 
since notmuch considers both copies to be the same message, it 
presumably synchronized the copy in `Sent` after the copy in `Inbox` and 
therefore marked both copies as already read.

With `tags=` (empty), I sent a message from another email account to my 
test account. Synchronization of S not present/`unread` worked correctly.

I'm not sure if this behavior is considered a bug, given notmuch's model 
for dealing with duplicate messages. I can live with this corner case, 
anyway. Also, for the record, I understand that I will probably have to 
switch to `tags=new` or something in the future to avoid reprocessing 
old mail. (Although emacs seems to treat `new` as a special tag? Maybe 
I'll use something else).

> And just in case this feels like a "duh moment" to you - we all have them ;)

All the same, from reading the documentation, I felt like the `tags` and 
`synchronize_flags` options operate independently of each other, which 
seems to be mostly correct. However, some more details of how things are 
processed and how it interacts with duplicate messages would help build 
a stronger mental model on which to make decisions.

I also found this while searching around, which was helpful, but I can't 
figure out how to access it from the notmuch homepage. Maybe it's out of 
date?:

https://github.com/notmuch/notmuch-wiki/blob/master/special-tags.mdwn

Thanks,
Andrew Todd\r