Forcing a sync of maildir flags?

Forcing a sync of maildir flags?

[Sean Whitton]

Hello,

I have this in my post-new hook:

    notmuch tag -unread -- folder:sent

The idea is that copies of my sent mail which get returned to me and
stored in my inbox, e.g. by mailing lists, would get maildir Seen tags
added to them.  But that does not happen.

This works:

    notmuch tag +unread -- folder:sent
    notmuch tag -unread -- folder:sent

Is there some more elegant way to force a sync of maildir flags?  Am I
the only one that has copies of my own mail in my inbox folder, not
marked as read from the point of view of anything except notmuch?

Please CC me on replies.

-- 
Sean Whitton

Re: Forcing a sync of maildir flags?

[David Bremner]

Sean Whitton <spwhitton@spwhitton.name> writes:

> I have this in my post-new hook:
>
>     notmuch tag -unread -- folder:sent
>
> The idea is that copies of my sent mail which get returned to me and
> stored in my inbox, e.g. by mailing lists, would get maildir Seen tags
> added to them.  But that does not happen.

I guess this is most likely a bug. It would be nice to have a test in
T340-maildir-sync.sh that duplicated it. I suspect the problem is
related to messages in the new/ subdirectory. Some non-notmuch clients
(prominently mutt) interpret being in new/ has having user-visible
semantics, so notmuch tries not to move files out of there
unnecessarily, On the other hand the maildir spec says that files in
new/ cannot have flags, so they cannot have their unread tag removed
without moving the file to cur/

> This works:
>
>     notmuch tag +unread -- folder:sent
>     notmuch tag -unread -- folder:sent
>
> Is there some more elegant way to force a sync of maildir flags?  Am I
> the only one that has copies of my own mail in my inbox folder, not
> marked as read from the point of view of anything except notmuch?

I guess my workflow doesn't expose this problem as I leave many messages
unread (relying on the inbox flag). I'm not sure about the locking
implications, but having sent messages delivered to cur/ instead of to
new/ would probably workaround the problem.

d

Re: Forcing a sync of maildir flags?

[David Bremner]

David Bremner <david@tethera.net> writes:

>> This works:
>>
>>     notmuch tag +unread -- folder:sent
>>     notmuch tag -unread -- folder:sent
>>
>> Is there some more elegant way to force a sync of maildir flags?  Am I
>> the only one that has copies of my own mail in my inbox folder, not
>> marked as read from the point of view of anything except notmuch?
>
> I guess my workflow doesn't expose this problem as I leave many messages
> unread (relying on the inbox flag). I'm not sure about the locking
> implications, but having sent messages delivered to cur/ instead of to
> new/ would probably workaround the problem.

I guess the other thing is that I have

[new]
tags=unread;inbox;new

in .notmuch-config, which means that "notmuch new" moves things into
cur/. Maybe you can use that effect?

d

Re: Forcing a sync of maildir flags?

[Sean Whitton]

Hello David,

Thank you for your reply.

On Thu 20 Feb 2020 at 08:22AM -04, David Bremner wrote:

> Sean Whitton <spwhitton@spwhitton.name> writes:
>
>> I have this in my post-new hook:
>>
>>     notmuch tag -unread -- folder:sent
>>
>> The idea is that copies of my sent mail which get returned to me and
>> stored in my inbox, e.g. by mailing lists, would get maildir Seen tags
>> added to them.  But that does not happen.
>
> I guess this is most likely a bug. It would be nice to have a test in
> T340-maildir-sync.sh that duplicated it. I suspect the problem is
> related to messages in the new/ subdirectory. Some non-notmuch clients
> (prominently mutt) interpret being in new/ has having user-visible
> semantics, so notmuch tries not to move files out of there
> unnecessarily, On the other hand the maildir spec says that files in
> new/ cannot have flags, so they cannot have their unread tag removed
> without moving the file to cur/

So, your hypothesis is that notmuch is opting not to add the Seen flag
in order to avoid having to move the message from new to cur, for the
sake of mutt, but that's a bug because I've explicitly requested adding
Seen, so it ought to go ahead and move it from new to cur?

>> This works:
>>
>>     notmuch tag +unread -- folder:sent
>>     notmuch tag -unread -- folder:sent
>>
>> Is there some more elegant way to force a sync of maildir flags?  Am I
>> the only one that has copies of my own mail in my inbox folder, not
>> marked as read from the point of view of anything except notmuch?
>
> I guess my workflow doesn't expose this problem as I leave many messages
> unread (relying on the inbox flag). I'm not sure about the locking
> implications, but having sent messages delivered to cur/ instead of to
> new/ would probably workaround the problem.

In this case it's mbsync downloading messages from IMAP, and there
doesn't seem to be a way to configure it to deliver to cur instead of
new.

On Thu 20 Feb 2020 at 09:21AM -04, David Bremner wrote:

> I guess the other thing is that I have
>
> [new]
> tags=unread;inbox;new
>
> in .notmuch-config, which means that "notmuch new" moves things into
> cur/. Maybe you can use that effect?

To be honest that doesn't seem really better than the workaround I'm
currently using.

-- 
Sean Whitton

Re: Forcing a sync of maildir flags?

[David Bremner]

Sean Whitton <spwhitton@spwhitton.name> writes:

> So, your hypothesis is that notmuch is opting not to add the Seen flag
> in order to avoid having to move the message from new to cur, for the
> sake of mutt, but that's a bug because I've explicitly requested adding
> Seen, so it ought to go ahead and move it from new to cur?

Yeah, I think that's the case. This is based only vague memories of when
I wrote 0082a557.

d

Re: Forcing a sync of maildir flags?

[Sean Whitton]

Hello,

On Thu 20 Feb 2020 at 08:06PM -04, David Bremner wrote:

> Sean Whitton <spwhitton@spwhitton.name> writes:
>
>> So, your hypothesis is that notmuch is opting not to add the Seen flag
>> in order to avoid having to move the message from new to cur, for the
>> sake of mutt, but that's a bug because I've explicitly requested adding
>> Seen, so it ought to go ahead and move it from new to cur?
>
> Yeah, I think that's the case. This is based only vague memories of when
> I wrote 0082a557.

Thanks.  Let me record in this thread what I will believe it will take a
reproduce this in a test:

1) inbox and sent are Maildirs

1) Compose mail to a mailing list which will return copies of
   submissions, with `Fcc: sent -unread`

2) notmuch new, and in post-new hook, notmuch tag -unread -- folder:sent

Expected result: copy of message in both inbox and sent has Seen flag

Actual result: only copy of message in sent has Seen flag

-- 
Sean Whitton

Re: Forcing a sync of maildir flags?

[Sean Whitton]

Hello David,

On Thu 20 Feb 2020 at 08:22AM -04, David Bremner wrote:

> Sean Whitton <spwhitton@spwhitton.name> writes:
>
>> I have this in my post-new hook:
>>
>>     notmuch tag -unread -- folder:sent
>>
>> The idea is that copies of my sent mail which get returned to me and
>> stored in my inbox, e.g. by mailing lists, would get maildir Seen tags
>> added to them.  But that does not happen.
>
> I guess this is most likely a bug. It would be nice to have a test in
> T340-maildir-sync.sh that duplicated it. I suspect the problem is
> related to messages in the new/ subdirectory. Some non-notmuch clients
> (prominently mutt) interpret being in new/ has having user-visible
> semantics, so notmuch tries not to move files out of there
> unnecessarily, On the other hand the maildir spec says that files in
> new/ cannot have flags, so they cannot have their unread tag removed
> without moving the file to cur/

I am seeing this bug, or a closely related one, a whole lot right now.
Messages are coming back as unread over and over again.  I recently made
some changes to my notmuch cronjobs, so that probably has something to
do with it, but I have no guesses as to what the problem is.

As a first thing to try, I am going to add something to my pre-new hook
to perform the new/ -> cur/ move as specified by maildir(5) on all my
synced maildirs, so that notmuch never sees messages in new/ except when
it writes new drafts and sent mail there (and they'll get moved on the
next sync).

I'll drop the 'notmuch tag -unread -- folder:sent' workaround at the
same time to see what happens.

-- 
Sean Whitton

Re: Forcing a sync of maildir flags?

[Sean Whitton]

Hello,

On Tue 22 Mar 2022 at 12:44pm -07, Sean Whitton wrote:

> I am seeing this bug, or a closely related one, a whole lot right now.
> Messages are coming back as unread over and over again.  I recently made
> some changes to my notmuch cronjobs, so that probably has something to
> do with it, but I have no guesses as to what the problem is.
>
> As a first thing to try, I am going to add something to my pre-new hook
> to perform the new/ -> cur/ move as specified by maildir(5) on all my
> synced maildirs, so that notmuch never sees messages in new/ except when
> it writes new drafts and sent mail there (and they'll get moved on the
> next sync).
>
> I'll drop the 'notmuch tag -unread -- folder:sent' workaround at the
> same time to see what happens.

Seems to be working well.

-- 
Sean Whitton

Re: Forcing a sync of maildir flags?

[Gregor Zattler]

Hi Sean,
* Sean Whitton <spwhitton@spwhitton.name> [2022-04-04; 21:48]:
> On Tue 22 Mar 2022 at 12:44pm -07, Sean Whitton wrote:
>
>> I am seeing this bug, or a closely related one, a whole lot right now.
>> Messages are coming back as unread over and over again.

me too.  And it's not only messages from me, but also
messages from other people appear as unread of which I'm
quite certain I already read them.

>> I recently made some changes to my notmuch cronjobs, so
>> that probably has something to do with it, but I have no
>> guesses as to what the problem is.
>>
>> As a first thing to try, I am going to add something to my pre-new hook
>> to perform the new/ -> cur/ move as specified by maildir(5) on all my
>> synced maildirs, so that notmuch never sees messages in new/ except when
>> it writes new drafts and sent mail there (and they'll get moved on the
>> next sync).

Would you please disclose your workaround?  I'm very much
interested.


Ciao; Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

Re: Forcing a sync of maildir flags?

[David Bremner]

Sean Whitton <spwhitton@spwhitton.name> writes:
>
> Thanks.  Let me record in this thread what I will believe it will take a
> reproduce this in a test:
>
> 1) inbox and sent are Maildirs
>
> 1) Compose mail to a mailing list which will return copies of
>    submissions, with `Fcc: sent -unread`

Since you mention Fcc, are you using notmuch insert?

>
> 2) notmuch new, and in post-new hook, notmuch tag -unread -- folder:sent
>
> Expected result: copy of message in both inbox and sent has Seen flag
>
> Actual result: only copy of message in sent has Seen flag

I tried to simulate this in the form of a test. I'm not using notmuch
insert here, but I manage to trigger similar looking behaviour:

test_begin_subtest "duplicate file with seen flag"
test_subtest_known_broken
mkdir -p "${MAIL_DIR}"/sent/{cur,new,tmp}
notmuch config set maildir.synchronize_flags true
add_message '[subject]="seen message"'  '[filename]=fcc-file:2,S' '[dir]=sent/cur'
notmuch new
# for debugging
notmuch search subject:"seen message"
mkdir -p "${MAIL_DIR}"/inbox/{cur,new,tmp}
cp "${gen_msg_filename}" "${MAIL_DIR}"/inbox/new/copied-file:2,
notmuch new
# for debugging
notmuch search subject:"seen message"
# next line is a no-op, because it already doesn't have the unread tag
notmuch tag -unread folder:sent
count=$(notmuch search --output=files subject:"seen message" | grep -c ',S$')
# for debugging:
notmuch search --output=files subject:"seen message"
test_expect_equal "${count}" "2"

This has the output

    BROKEN duplicate file with seen flag
            --- T050-new.41.expected	2022-05-01 22:51:45.644747316 +0000
            +++ T050-new.41.output	2022-05-01 22:51:45.644747316 +0000
            @@ -1 +1 @@
            -2
            +1
    No new mail.
    thread:0000000000000011   2001-01-05 [1/1] Notmuch Test Suite; seen message (inbox)
    Processed 1 file in almost no time.
    No new mail.
    thread:0000000000000011   2001-01-05 [1/1(2)] Notmuch Test Suite; seen message (inbox)
    /home/bremner/software/upstream/notmuch/test/tmp.T050-new/mail/sent/cur/fcc-file:2,S
    /home/bremner/software/upstream/notmuch/test/tmp.T050-new/mail/inbox/new/copied-file:2,

The key point is that from notmuch's point of view the message never has
the unread tag, so there is no change for "notmuch tag" to sync with
maildir flags.

It doesn't seem to make a difference if I put the copy in inbox/new or
inbox/cur, so I don't think it is related to the previous efforts not to
prematurely move files out of new/.

As far as I can tell, notmuch-new (unlike notmuch-insert) does not call
notmuch_message_tags_to_maildir_flags, so the maildir flags on the newly
discovered copy are not updated. Perhaps it should, but that seems like
a pretty big change, so I want to proceed with caution.

Re: Forcing a sync of maildir flags?

[Sean Whitton]

Hello David,

On Sun 01 May 2022 at 08:23PM -03, David Bremner wrote:

> Sean Whitton <spwhitton@spwhitton.name> writes:
>>
>> Thanks.  Let me record in this thread what I will believe it will take a
>> reproduce this in a test:
>>
>> 1) inbox and sent are Maildirs
>>
>> 1) Compose mail to a mailing list which will return copies of
>>    submissions, with `Fcc: sent -unread`
>
> Since you mention Fcc, are you using notmuch insert?

notmuch-maildir-use-notmuch-insert is t so I think I am?

> # next line is a no-op, because it already doesn't have the unread tag
> notmuch tag -unread folder:sent

Seems also worth noting that to my mind it ultimately shouldn't be
necessary to run that command -- notmuch should notice that one copy of
the message has different maildir flags to the other in a way that's out
of sync with the notmuch tags it has.

> The key point is that from notmuch's point of view the message never has
> the unread tag, so there is no change for "notmuch tag" to sync with
> maildir flags.
>
> It doesn't seem to make a difference if I put the copy in inbox/new or
> inbox/cur, so I don't think it is related to the previous efforts not to
> prematurely move files out of new/.
>
> As far as I can tell, notmuch-new (unlike notmuch-insert) does not call
> notmuch_message_tags_to_maildir_flags, so the maildir flags on the newly
> discovered copy are not updated. Perhaps it should, but that seems like
> a pretty big change, so I want to proceed with caution.

It makes sense to me for notmuch-new to call that function too, fwiw.

-- 
Sean Whitton

Re: Forcing a sync of maildir flags?

[David Bremner]

Sean Whitton <spwhitton@spwhitton.name> writes:

> Hello David,
>> # next line is a no-op, because it already doesn't have the unread tag
>> notmuch tag -unread folder:sent
>
> Seems also worth noting that to my mind it ultimately shouldn't be
> necessary to run that command -- notmuch should notice that one copy of
> the message has different maildir flags to the other in a way that's out
> of sync with the notmuch tags it has.
>

Right, that's probably a useful way to think about the problem.

>> The key point is that from notmuch's point of view the message never has
>> the unread tag, so there is no change for "notmuch tag" to sync with
>> maildir flags.
>>
>> It doesn't seem to make a difference if I put the copy in inbox/new or
>> inbox/cur, so I don't think it is related to the previous efforts not to
>> prematurely move files out of new/.
>>
>> As far as I can tell, notmuch-new (unlike notmuch-insert) does not call
>> notmuch_message_tags_to_maildir_flags, so the maildir flags on the newly
>> discovered copy are not updated. Perhaps it should, but that seems like
>> a pretty big change, so I want to proceed with caution.
>
> It makes sense to me for notmuch-new to call that function too, fwiw.

I have the same intuition. Unfortunately it is not as simple as adding
in a couple calls to this function. The complications I am aware of so
far are

1) We need to distinguish between when a newly discovered file should
update tags, and when the newly discovered file should have it's flags
updated.

2) Renaming files in the middle of notmuch new needs to be done
carefully in order that notmuch doesn't lose track of the files. They
are only lost until the next run of notmuch-new, but it's not ideal.

d

Re: Forcing a sync of maildir flags?

[Sean Whitton]

Hello,

On Tue 03 May 2022 at 09:14AM -03, David Bremner wrote:

> I have the same intuition. Unfortunately it is not as simple as adding
> in a couple calls to this function. The complications I am aware of so
> far are
>
> 1) We need to distinguish between when a newly discovered file should
> update tags, and when the newly discovered file should have it's flags
> updated.
>
> 2) Renaming files in the middle of notmuch new needs to be done
> carefully in order that notmuch doesn't lose track of the files. They
> are only lost until the next run of notmuch-new, but it's not ideal.

Okay, thanks for the info.  Nice to have it all recorded in this thread.

-- 
Sean Whitton