Is there a reason why the trashed flag is not synced?

Is there a reason why the trashed flag is not synced?

[Felipe Contreras]

Hi,

I just noticed that a message I untagged in Gmail was not deleted nor
tagged as deleted locally.

I understand deleting files is complex, but what's wrong with simply
tagging the T (trashed) messages as 'deleted'?

I'm using synchronize_flags=true.

Cheers.

-- 
Felipe Contreras

Re: Is there a reason why the trashed flag is not synced?

[David Bremner]

Felipe Contreras <felipe.contreras@gmail.com> writes:

> I just noticed that a message I untagged in Gmail was not deleted nor
> tagged as deleted locally.
>
> I understand deleting files is complex, but what's wrong with simply
> tagging the T (trashed) messages as 'deleted'?
>
> I'm using synchronize_flags=true.
>

The current lack of synchronization is intentional, with the reasoning
explained in the commit message of [1].  I don't know if a one way sync
(only from T -> deleted) has been discussed.

https://git.notmuchmail.org/git/notmuch/commit/2c262042ac174d7bc96d6035ab9c88bd0abe7f35

Re: Is there a reason why the trashed flag is not synced?

[Felipe Contreras]

On Sun, May 9, 2021 at 4:38 AM David Bremner <david@tethera.net> wrote:
> Felipe Contreras <felipe.contreras@gmail.com> writes:

> > I understand deleting files is complex, but what's wrong with simply
> > tagging the T (trashed) messages as 'deleted'?
> >
> > I'm using synchronize_flags=true.
>
> The current lack of synchronization is intentional, with the reasoning
> explained in the commit message of [1].  I don't know if a one way sync
> (only from T -> deleted) has been discussed.
> https://git.notmuchmail.org/git/notmuch/commit/2c262042ac174d7bc96d6035ab9c88bd0abe7f35

This rationale makes sense to me, and I agree with the conclusion that
such behavior "can be potentially dangerous".

But the fact that something is potentially dangerous doesn't mean that
it necessarily is.

It makes sense that by default synchronize_flags=true doesn't sync the
trash flag, but what's wrong with a new configuration
synchronize_flags=all that does?

That way it's the responsibility of the user to ensure that such
potentially dangerous behavior can't happen before doing
synchronize_flags=all.

Cheers.

-- 
Felipe Contreras

Re: Is there a reason why the trashed flag is not synced?

[David Bremner]

Felipe Contreras <felipe.contreras@gmail.com> writes:

>
> It makes sense that by default synchronize_flags=true doesn't sync the
> trash flag, but what's wrong with a new configuration
> synchronize_flags=all that does?
>

That seems plausible to me, but I'd like more input in case I missed
something.

d

Re: Is there a reason why the trashed flag is not synced?

[Michael J Gruber]

David Bremner venit, vidit, dixit 2021-05-10 14:04:27:
> Felipe Contreras <felipe.contreras@gmail.com> writes:
> 
> >
> > It makes sense that by default synchronize_flags=true doesn't sync the
> > trash flag, but what's wrong with a new configuration
> > synchronize_flags=all that does?
> >
> 
> That seems plausible to me, but I'd like more input in case I missed
> something.

What happens if you have multiple copies of the same message, some "unseen"
or "replied" etc.? notmuch synchronizes them with the flag on the
message and in turn with all message copies, and this works "as
expected" because, for example, there is no "unseen" flag but a "seen"
flag.

Well, partially. What happens if you mark a message unread (clear the
"seen" flag) in one account? Notmuch will see a changed message file for
one copy - will it reread all files for this mid and reconstruct the
flags (thereby overwriting the unseen with the seen from the other one)?
(Honest question, I don't know.)

Deleting by chance is more harmful, yes. But is it more typical to clear
out duplicate files using "T" or to delete a message (i.e. all it's file
copies)? I'd even argue for the latter, at least for supporting
"maildir.synchronize_deleted" in both directions. To be super-safe, one
could special case this to take an AND of all message delete flags:
"unread" already acts as if it was an AND of the unread flags of all
copies (because it's the negation of an OR of seen flags).

A one-sync, however, can be confusing, I'm afraid.

Personally, I use trash folders synced with flags (through mbsync+afew,
gmailieer) togther with the typical 30day clean-up. Has worked well so
far.

Cheers
Michael

Re: Is there a reason why the trashed flag is not synced?

[omgitsaheadcrab]

[-- Attachment #1.1: Type: text/plain, Size: 2239 bytes --]

My use is similar to Michael's.

That being said, I think giving users the option, assuming they are aware
of the risks,
is a good idea.

I do however think that this should be a separate flag as I dislike
extra options being bolted on to booleans.

On Mon, May 10, 2021 at 2:32 PM Michael J Gruber <git@grubix.eu> wrote:

> David Bremner venit, vidit, dixit 2021-05-10 14:04:27:
> > Felipe Contreras <felipe.contreras@gmail.com> writes:
> >
> > >
> > > It makes sense that by default synchronize_flags=true doesn't sync the
> > > trash flag, but what's wrong with a new configuration
> > > synchronize_flags=all that does?
> > >
> >
> > That seems plausible to me, but I'd like more input in case I missed
> > something.
>
> What happens if you have multiple copies of the same message, some "unseen"
> or "replied" etc.? notmuch synchronizes them with the flag on the
> message and in turn with all message copies, and this works "as
> expected" because, for example, there is no "unseen" flag but a "seen"
> flag.
>
> Well, partially. What happens if you mark a message unread (clear the
> "seen" flag) in one account? Notmuch will see a changed message file for
> one copy - will it reread all files for this mid and reconstruct the
> flags (thereby overwriting the unseen with the seen from the other one)?
> (Honest question, I don't know.)
>
> Deleting by chance is more harmful, yes. But is it more typical to clear
> out duplicate files using "T" or to delete a message (i.e. all it's file
> copies)? I'd even argue for the latter, at least for supporting
> "maildir.synchronize_deleted" in both directions. To be super-safe, one
> could special case this to take an AND of all message delete flags:
> "unread" already acts as if it was an AND of the unread flags of all
> copies (because it's the negation of an OR of seen flags).
>
> A one-sync, however, can be confusing, I'm afraid.
>
> Personally, I use trash folders synced with flags (through mbsync+afew,
> gmailieer) togther with the typical 30day clean-up. Has worked well so
> far.
>
> Cheers
> Michael
> _______________________________________________
> notmuch mailing list -- notmuch@notmuchmail.org
> To unsubscribe send an email to notmuch-leave@notmuchmail.org
>

[-- Attachment #1.2: Type: text/html, Size: 3052 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Is there a reason why the trashed flag is not synced?

[Reto]

On Mon, May 10, 2021 at 03:31:52PM +0200, Michael J Gruber wrote:
> Deleting by chance is more harmful, yes. But is it more typical to clear
> out duplicate files using "T" or to delete a message (i.e. all it's file
> copies)?

You have to be a bit careful here... imap is a relatively strange protocol.
Consider a provider like gmail that uses a label system over normal imap folders.
If you assign a message "inbox" and "vacation" and maybe "archive" and then want
to remove the message from inbox and vacation folders you execute delete instructions.
That however, certainly shouldn't propagate to the "archive" copy.

So I'd argue that in fact no, what you mention is not the "typical" thing to do.
At least not with those providers.
Not all MUAs behave the same in regards to whether or not they set the trash flag
first.
Yes, you can tell neomutt to do almost anything, but if setting the Trash flag
on one message starts deleting all other copies I'd be very much surprised.

Re: Is there a reason why the trashed flag is not synced?

[Michael J Gruber]

Reto venit, vidit, dixit 2021-05-11 08:23:21:
> On Mon, May 10, 2021 at 03:31:52PM +0200, Michael J Gruber wrote:
> > Deleting by chance is more harmful, yes. But is it more typical to clear
> > out duplicate files using "T" or to delete a message (i.e. all it's file
> > copies)?
> 
> You have to be a bit careful here... imap is a relatively strange protocol.
> Consider a provider like gmail that uses a label system over normal imap folders.
> If you assign a message "inbox" and "vacation" and maybe "archive" and then want
> to remove the message from inbox and vacation folders you execute delete instructions.
> That however, certainly shouldn't propagate to the "archive" copy.
> 
> So I'd argue that in fact no, what you mention is not the "typical" thing to do.
> At least not with those providers.
> Not all MUAs behave the same in regards to whether or not they set the trash flag
> first.
> Yes, you can tell neomutt to do almost anything, but if setting the Trash flag
> on one message starts deleting all other copies I'd be very much surprised.

If you sync gmail labels to exact copies in different folders then you're not
holding it right, sorry ;)

Gmail does not "execute a delete instruction" when it removes a label
from a message, and if you sync 1 e-mail to several copies with the same
message id then they are in fact not copies but identical from the point
of view of any programme that identifies messages by message id - such
as notmuch.

Gmail is not an IMAP service; it has an IMAP API which exposes labels as
folders, with all the caveats which this implies. That's why there are
better ways to sync Gmail with a notmuch mail store (gmailieer).

I don't think notmuch should break its design principle (1 message id, 1
message) just to work around a problem caused by a wrong sync procedure:
that "delete instruction" is a result (merely: artifact) of speaking IMAP to Gmail.

Michael

Re: Is there a reason why the trashed flag is not synced?

[Reto]

On Tue, May 11, 2021 at 08:48:45AM +0200, Michael J Gruber wrote:
> If you sync gmail labels to exact copies in different folders then you're not
> holding it right, sorry ;)

> Gmail is not an IMAP service; it has an IMAP API which exposes labels as
> folders, with all the caveats which this implies. That's why there are
> better ways to sync Gmail with a notmuch mail store (gmailieer).

Better is always a relative term. I prefer a standardized protocol over some
homebrew stuff.

> I don't think notmuch should break its design principle (1 message id, 1
> message) just to work around a problem caused by a wrong sync procedure:
> that "delete instruction" is a result (merely: artifact) of speaking IMAP to Gmail.

It's not only gmail, really any IMAP server.
If you have a message in multiple folders at the same time (which is entirely
reasonable, think "todo" + where it actually should go after processing).
They all have the same message ID.

Only because you delete the copy from todo doesn't mean that you want it gone
from other places as well.

Sure, you could move instead of copying...
But that means that you may now need to hunt through multiple folders instead of
being able to grab the message from the actual mbox it should be in.

IMAP doesn't impose a workflow, just because yours happens to be a certain way
doesn't make it the gold standard ;)

Cheers,
Reto

Re: Is there a reason why the trashed flag is not synced?

[Michael J Gruber]

Reto venit, vidit, dixit 2021-05-12 07:10:13:
> On Tue, May 11, 2021 at 08:48:45AM +0200, Michael J Gruber wrote:
> > If you sync gmail labels to exact copies in different folders then you're not
> > holding it right, sorry ;)
> 
> > Gmail is not an IMAP service; it has an IMAP API which exposes labels as
> > folders, with all the caveats which this implies. That's why there are
> > better ways to sync Gmail with a notmuch mail store (gmailieer).
> 
> Better is always a relative term. I prefer a standardized protocol over some
> homebrew stuff.
 
???

The question is whether you talk Gmail API to Gmail or IMAP to Gmail.
What is homebrew here?

> > I don't think notmuch should break its design principle (1 message id, 1
> > message) just to work around a problem caused by a wrong sync procedure:
> > that "delete instruction" is a result (merely: artifact) of speaking IMAP to Gmail.
> 
> It's not only gmail, really any IMAP server.
> If you have a message in multiple folders at the same time (which is entirely
> reasonable, think "todo" + where it actually should go after processing).
> They all have the same message ID.
> 
> Only because you delete the copy from todo doesn't mean that you want it gone
> from other places as well.
> 
> Sure, you could move instead of copying...
> But that means that you may now need to hunt through multiple folders instead of
> being able to grab the message from the actual mbox it should be in.
> 
> IMAP doesn't impose a workflow, just because yours happens to be a certain way
> doesn't make it the gold standard ;)

Exactly, those arguments are turning into the ridiculous.

The point is that folders are neither labels nor tags.

You use an IMAP-sync tool to bolt on the tag concept onto a folder
structure. This creates problems which only the sync tool can solve, not
the indexer (notmuch), because it is the sync tool which translates the
IMAP folder structure (and IMAP flags) into a maildir store (and maildir
flags).

Indeed, IMAP even has extended flags but support on the IMAP client side is
typically suboptimal, so people don't use them at all or not to a great
extent; they would be a good match for notmuch tags.

Now, the only point at which notmuch is involved is indexing the
maildir. This includes synchronizing the maildir flags to notmuch tags -
and currently notmuch fails to synchronize one flag from the maildir
standard. Felipe and I suggest to reinstate that.

Note that since notmuch identifies messages by mid and one mid can correspond
to multiple files (e.g. due to folder copies, list copies, sent/cc) it
has to decide which file to index. Currently, notmuch assumes the
content to be identical (it indexes the content of one file) but takes a
union of reference headers and flags from all files.

"Union of [boolean] flags" is the real issue here: AND or OR? Currently
it's OR for all flags, and this happens to work for most workflows
because the tag "unread" does not correspond to a maildir flag - rather,
maildir has "S" (seen) so that OR of seen maildir flags decides whether
notmuch does not tag the message "unread".

Notice how this imposes a "gold standard workflow" on how you use 
read/unread on multiple copies of the same message?
On the IMAP side, marking 1 copy of your message read or unread does not
change the other one (much to the dismay of users who want to use
automatic filters for sorting). Sync that (mbsync or such) two your
maildir store and you one message file with "S", the other one without.
Index with notmuch and *the* message will not be "unread" (even though
one copy is not "Seen" per maildir flag). Toggle the "unread" tag with
notmuch and all copies or none will have "S"; syncing back to the IMAP
side will correspondingly mark all copies read or all unread.

So, unless you force notmuch to sync tags back to maildir flags (e.g. by
toggling them) it does not even mark all copies S even though the tag
(absence of "unread") would warrant that.

But whether this is the correct representation of "S" as "not unread"
really depends on your workflow, that is your usage of copies und your
usage of that flag. And yet, notmuch does sync it rather than bailing
out because it requires a decision.

In the very same way as marking one copy read marks all read (on the
notmuch side as opposed to the IMAP side), why shouldn't the same be
true for deleted? Because users with an IMAP mindset will complain about
one but not the other, even though they are analogous...

That's why I suggest the softer approach of treating the union of "T" as
and AND: tag a message as "trashed" if all its copies have maildir flag
"T". That way, users with multiple copies don't shoot themselves in the
foot (and don't complain to us). If you tag a message as "trashed", all
copies get "T", of course. This should give the best user experience:
Once an IMAP server or IMAP sync tool expunges "T" messages notmuch will
merely notices a missing file, unless it was the last one in which case
it notices message deletion (and in which case it showed "trashed"
before because it was the only one and had "T").

But really, if you use folders as a tag substitute you should think
about moving files to a trash folder rather than marking them trashed,
in which case we don't have to worry about "T". (Notmuch's lack of
T-syncing made me do this and it works fine.)

Michael

Re: Is there a reason why the trashed flag is not synced?

[Felipe Contreras]

On Wed, May 12, 2021 at 3:37 AM Michael J Gruber <git@grubix.eu> wrote:

> But really, if you use folders as a tag substitute you should think
> about moving files to a trash folder rather than marking them trashed,
> in which case we don't have to worry about "T". (Notmuch's lack of
> T-syncing made me do this and it works fine.)

In my case trashed doesn't mean deleted, it means it's no longer
reachable from any of the directories I'm synchronizing.

Right now I'm only tracking a single label. If I remove that label
from the message in Gmail, that would translate as a trashed flag in
IMAP.

I'm not actually deleting anything, I just don't want to see those
messages in my notmuch client (thanks to excluded_tags).

-- 
Felipe Contreras