How do I write labels back to the source Maildir?

How do I write labels back to the source Maildir?

[Josh Steadmon]

Hi folks,

This ended up being a longer email than I wanted, so to summarize, my
main question is: is it possible to have lei write labels created with
`lei-tag` back to the source Maildir which was indexed with `lei-index`?
Additional details below:


I'm a Git developer, and I am subscribed to git@vger.kernel.org. I sync
that mail from my corp Gmail account to a local Maildir with mbsync. I
would like that single Maildir to be the canonical source of mail, which
I generally read with Mutt.

Within that Maildir, I would like to be able to quickly find emails
about unit tests. I can find these on lore.kernel.org with a query like:
"dfn:t/unit-tests OR s:unit OR ((nq:bug OR nq:regression) AND nq:unit)"

Ideally, I'd like some sort of automated lei setup to tag messages
matching the above query, and then I can use Mutt's built-in filtering
to show me only those threads.

Since I want the mbsync-ed Maildir to be the canonical source of mail,
it sounds like `lei-index` is what I want. I ran
`lei index ~/.mail/gmail/git-external` and it seems to be OK.

Next I want to see if I can search those indexed mails, so I run
`lei q dfn:t/unit-tests OR s:unit OR \(\(nq:bug OR nq:regression\) AND nq:unit\)`
and I get some JSON containing email metadata matching the query. So
far, so good.

Now I want to label those matching messages. After a bit of
experimentation, I come up with the following combination of lei-q and
lei-tag:
```
lei q -f mboxrd dfn:t/unit-tests OR s:unit OR \(\(nq:bug OR nq:regression\) AND nq:unit\) \
  | lei tag -F mboxrd - +L:git-unit-tests
```

To make sure the label was applied, I run `lei q L:git-unit-tests` and I
get back the same set of emails I expect. Great.

However, now I'm stuck trying to get that label metadata back into my
Maildir. I've tried both `lei refresh-mail-sync` and `lei export-kw`
with various combinations of options, but nothing is updating the
Maildir. I thought that at least one of these commands would update the
matching emails with an "X-Label: git-unit-tests" header, although now I
can't find any reference to X-Label in the lei docs, so I probably
confused myself somehow about that.


I *can* get lei-q to write these matching emails to a separate Maildir
using `lei q -o /tmp/git-unit-tests dfn:t/unit-tests ...`, but this
workflow is more annoying than I'd hoped for, as metadata is not
automatically synced between the various Maildirs, and it's a bit slower
in Mutt to switch Maildirs than to filter the currently open one.

So to repeat my original question, is there any way to get lei to
somehow update the source Maildir to reflect the labels on these tagged
messages?

Thanks in advance for any advice,
Josh

Re: How do I write labels back to the source Maildir?

[Eric Wong]

Josh Steadmon <steadmon@google.com> wrote:
> Hi folks,
> 
> This ended up being a longer email than I wanted, so to summarize, my
> main question is: is it possible to have lei write labels created with
> `lei-tag` back to the source Maildir which was indexed with `lei-index`?

TL;DR: not yet...

<snip all correct observations and usages>

> However, now I'm stuck trying to get that label metadata back into my
> Maildir. I've tried both `lei refresh-mail-sync` and `lei export-kw`
> with various combinations of options, but nothing is updating the
> Maildir. I thought that at least one of these commands would update the
> matching emails with an "X-Label: git-unit-tests" header, although now I
> can't find any reference to X-Label in the lei docs, so I probably
> confused myself somehow about that.

Correct.  I've been considering X-Label, but it also makes it
more complicated to:

1) to deduplicate messages when exposing lei/store blobs contents
   (for lei import users) as a Maildir via FUSE3 (planned)

2) to retrieve blobs for lei-index users; since blob contents end
   up getting invalidated on the FS

Of course (as you know), blobs tend to stay forever in git
(but hopefully deltafication can help reduce the space needed)

Good news is 2.0 (out soon, hopefully :x) no longer requires
Linux::Inotify2 to get inotify support on most architectures,
so inotify should work more easily out-of-the-box.

Of course, X-Label is also non-standard and I haven't done much
research into it...

* do other MUAs support it? (and differences from mutt)
* does mutt support it for mbox, MH, IMAP, etc..?
* character restrictions in supported MUAs? (vs. what chars Xapian allows)

Fwiw, I've never used X-Label despite using mutt for decades.

There should also be a way to filter out X-Label (and any other
arbitrary headers) when converting/exporting messages for
public consumption.

> I *can* get lei-q to write these matching emails to a separate Maildir
> using `lei q -o /tmp/git-unit-tests dfn:t/unit-tests ...`, but this
> workflow is more annoying than I'd hoped for, as metadata is not
> automatically synced between the various Maildirs, and it's a bit slower
> in Mutt to switch Maildirs than to filter the currently open one.

Yeah, that's the current way.  The goal of the (future) FUSE3
backend is avoiding duplication of blob contents onto disk and
maybe easier synchronization of both kw and label metadata.

Being able to auto-label (at least in Xapian) messages which
enter a certain directory may also be a good idea.

> So to repeat my original question, is there any way to get lei to
> somehow update the source Maildir to reflect the labels on these tagged
> messages?

Again, not yet...

I've been distracted with the web UI for coderepo integration
(to support lei-rediff in WWW) and also more distracted
investigating ways to improve performance :x

Re: How do I write labels back to the source Maildir?

[Josh Steadmon]

On 2024.01.25 00:26, Eric Wong wrote:
> Josh Steadmon <steadmon@google.com> wrote:
> > Hi folks,
> > 
> > This ended up being a longer email than I wanted, so to summarize, my
> > main question is: is it possible to have lei write labels created with
> > `lei-tag` back to the source Maildir which was indexed with `lei-index`?
> 
> TL;DR: not yet...
> 
> <snip all correct observations and usages>

Thanks for the explanation!

> > However, now I'm stuck trying to get that label metadata back into my
> > Maildir. I've tried both `lei refresh-mail-sync` and `lei export-kw`
> > with various combinations of options, but nothing is updating the
> > Maildir. I thought that at least one of these commands would update the
> > matching emails with an "X-Label: git-unit-tests" header, although now I
> > can't find any reference to X-Label in the lei docs, so I probably
> > confused myself somehow about that.
> 
> Correct.  I've been considering X-Label, but it also makes it
> more complicated to:
> 
> 1) to deduplicate messages when exposing lei/store blobs contents
>    (for lei import users) as a Maildir via FUSE3 (planned)
> 
> 2) to retrieve blobs for lei-index users; since blob contents end
>    up getting invalidated on the FS
> 
> Of course (as you know), blobs tend to stay forever in git
> (but hopefully deltafication can help reduce the space needed)
> 
> Good news is 2.0 (out soon, hopefully :x) no longer requires
> Linux::Inotify2 to get inotify support on most architectures,
> so inotify should work more easily out-of-the-box.

Understood. I'll keep an eye out for the release.

> Of course, X-Label is also non-standard and I haven't done much
> research into it...
> 
> * do other MUAs support it? (and differences from mutt)
> * does mutt support it for mbox, MH, IMAP, etc..?
> * character restrictions in supported MUAs? (vs. what chars Xapian allows)
> 
> Fwiw, I've never used X-Label despite using mutt for decades.

Yeah, mutt has a pattern modifier (~y) for X-Label. I am unsure about
other MUAs.

> There should also be a way to filter out X-Label (and any other
> arbitrary headers) when converting/exporting messages for
> public consumption.
> 
> > I *can* get lei-q to write these matching emails to a separate Maildir
> > using `lei q -o /tmp/git-unit-tests dfn:t/unit-tests ...`, but this
> > workflow is more annoying than I'd hoped for, as metadata is not
> > automatically synced between the various Maildirs, and it's a bit slower
> > in Mutt to switch Maildirs than to filter the currently open one.
> 
> Yeah, that's the current way.  The goal of the (future) FUSE3
> backend is avoiding duplication of blob contents onto disk and
> maybe easier synchronization of both kw and label metadata.
>
> Being able to auto-label (at least in Xapian) messages which
> enter a certain directory may also be a good idea.
> 
> > So to repeat my original question, is there any way to get lei to
> > somehow update the source Maildir to reflect the labels on these tagged
> > messages?
> 
> Again, not yet...
> 
> I've been distracted with the web UI for coderepo integration
> (to support lei-rediff in WWW) and also more distracted
> investigating ways to improve performance :x