make menuconfig interface for lei / grok-pull

make menuconfig interface for lei / grok-pull

[Luis Chamberlain]

Greetings,

I unsubscribed to most mailing lists thanks to public-inbox. Thanks
to public-inbox my inbox went on quite a bit of diet.

Now, the way I work though is... I like to hop onto different systems
every now and then, and not rely on one central place for everything.
This is because when I've done that before I get outages / network
outages / etc.

So I often have to start from scratch.

My use case is I'm subscribed to a few kernel mailign lists and I use
mutt with Maildir. I had configured recently pi-piper and grokmirror
so that I get only the last 1 year of email from a few set of mailing
lists. For this I needed to know the commit IDs for those emails on
the public-inbox git mirror for each mailing list.

To start from scratch I could go into my logs and figure out what that
was again, and start from scratch... but I'm inpsired to just provide
a 'make menuconfig' interface for this so that all I have to do is
select the mailing list I want to git clone and then dump into Maildir
the last 1 year of content for each. Since this time frame of 1 year
can be variable, it would seem a good candidate would be to use kconfig
to allow those interested to associate the content year by year to git
tree and respective commit IDs on each mailing list epoch/git tree.

I was hinted using lei would be better though. But I'm stuck:

lei init
# The next two don't work
lei import import https://lore.kernel.org/linux-fsdevel/git/0.git
lei add-external git://lore.kernel.org/linux-fsdevel/git/0.git

I can use nttp but jeesh, really?

Anyway, once the above works, if one didn't want to have the git
entire git tree around I gather that I can use something like

lei index <same-path-as-above>

Then in so far as the date idea I mentioned, I take it the way to go
there might be after this to use something like:

lei q --only=linux-fsdevel <search-term>

but its not clear if a date is supported as a search term.

Also how about doing updates?

Is my intended use case better with grok-pull or is there
hopes I can end up using lei for all what I have intentions for?
My vision is, kernel deveoper would 'make menuconfig' and select
a few mailing lists they want to import, then when one is selected
new options appear so you can pick the starting year from where
you want the import to occur for. Then for each mailing list there
is also a refresh thing, so that we can grok-pull differently
for each mailing list, which will be handled by systemd timers
and a service for each. The reason for differnt times for refresh
is updating linux-kernel is not a priority to me, but linux-fsdevel
and linux-block are, so I want linux-fsdevel and linux-block to
refresh once every 10 minutes, while linux-kernel can update... 
I don't know once every 8 hours. Anyway, this should all be
configurable. And my hope is that developers would not have to
know what's going on behind the scenes. All they need to know
is that their ~Mail/linux/ directory will eventually be stuffed
with Maildir directories for the mailing lists they enabled and
they'll be updated as often as specified in their configuration.

 Luis

Re: make menuconfig interface for lei / grok-pull

[Eric Wong]

Luis Chamberlain <mcgrof@kernel.org> wrote:
> # The next two don't work
> lei import import https://lore.kernel.org/linux-fsdevel/git/0.git
> lei add-external git://lore.kernel.org/linux-fsdevel/git/0.git

Apologies, I need to document this better.  This should work:

  lei add-external --mirror https://lore.kernel.org/linux-fsdevel SOME_FS_PATH

> Also how about doing updates?

  public-inbox-fetch -C SOME_FS_PATH
  public-inbox-index SOME_FS_PATH

And I just posted some patches to simplify updates to:

  make -C SOME_FS_PATH

In case you reorganize your file system:

  lei forget-external OLD_FS_PATH
  lei add-external NEW_FS_PATH

I'll read the rest and reply later.  "lei index" is still
half-baked, but the above should be fine.

Re: make menuconfig interface for lei / grok-pull

[Luis Chamberlain]

On Wed, Sep 15, 2021 at 09:44:20PM +0000, Eric Wong wrote:
> Luis Chamberlain <mcgrof@kernel.org> wrote:
> > # The next two don't work
> > lei import import https://lore.kernel.org/linux-fsdevel/git/0.git
> > lei add-external git://lore.kernel.org/linux-fsdevel/git/0.git
> 
> Apologies, I need to document this better.  This should work:
> 
>   lei add-external --mirror https://lore.kernel.org/linux-fsdevel SOME_FS_PATH

Groovy thanks! I'll start with that!

Any reason we need to specify SOME_FS_PATH ? Can't we assume a sensible
default somewhere in .local/lei or whatever in the future?

> > Also how about doing updates?
> 
>   public-inbox-fetch -C SOME_FS_PATH
>   public-inbox-index SOME_FS_PATH

Neat!

> And I just posted some patches to simplify updates to:
> 
>   make -C SOME_FS_PATH

Oh, pretty nice.

> In case you reorganize your file system:
> 
>   lei forget-external OLD_FS_PATH
>   lei add-external NEW_FS_PATH
> 
> I'll read the rest and reply later.  "lei index" is still
> half-baked, but the above should be fine.

Thanks!

  Luis

Re: make menuconfig interface for lei / grok-pull

[Kyle Meyer]

Eric Wong writes:

> Luis Chamberlain <mcgrof@kernel.org> wrote:
>> # The next two don't work
>> lei import import https://lore.kernel.org/linux-fsdevel/git/0.git
>> lei add-external git://lore.kernel.org/linux-fsdevel/git/0.git
>
> Apologies, I need to document this better.  This should work:

A good chunk of the current documentation was fumbled together by me, so
I'm sure there are many spots for improvement but...

>   lei add-external --mirror https://lore.kernel.org/linux-fsdevel SOME_FS_PATH

... hey, the equivalent of the snippet above is one of the few things
that made it into lei-overview :)

Re: make menuconfig interface for lei / grok-pull

[Eric Wong]

Luis Chamberlain <mcgrof@kernel.org> wrote:
> Greetings,
> 
> I unsubscribed to most mailing lists thanks to public-inbox. Thanks
> to public-inbox my inbox went on quite a bit of diet.
> 
> Now, the way I work though is... I like to hop onto different systems
> every now and then, and not rely on one central place for everything.
> This is because when I've done that before I get outages / network
> outages / etc.
> 
> So I often have to start from scratch.

Same here, this goes for project code repos, as well...

At the same time, I don't want too much synchronization since
the systems I hop on to vary hugely in storage capacity.

<snip replied parts>

> lei q --only=linux-fsdevel <search-term>
> 
> but its not clear if a date is supported as a search term.

All the search prefixes documented in
https://public-inbox.org/meta/_/text/help/ work for lei.

dt: (date-time), d: (date (mairix-compatible) have been there a while,
rt: (Received-time) is new.

They gained support for git approxidate recently, so "rt:last.week.."
gives you all the messages that were received from last week til now.
Anything w/o spaces which git understands should work.

> Is my intended use case better with grok-pull or is there
> hopes I can end up using lei for all what I have intentions for?

I think introducing support for "lei update-external [--all]"
could be useful combined with my previous note about
add-external.

> My vision is, kernel deveoper would 'make menuconfig' and select
> a few mailing lists they want to import, then when one is selected
> new options appear so you can pick the starting year from where
> you want the import to occur for. Then for each mailing list there
> is also a refresh thing, so that we can grok-pull differently
> for each mailing list, which will be handled by systemd timers
> and a service for each. The reason for differnt times for refresh
> is updating linux-kernel is not a priority to me, but linux-fsdevel
> and linux-block are, so I want linux-fsdevel and linux-block to
> refresh once every 10 minutes, while linux-kernel can update... 
> I don't know once every 8 hours. Anyway, this should all be
> configurable. And my hope is that developers would not have to
> know what's going on behind the scenes. All they need to know
> is that their ~Mail/linux/ directory will eventually be stuffed
> with Maildir directories for the mailing lists they enabled and
> they'll be updated as often as specified in their configuration.

Interesting...  I suppose it could work.  I don't know much
about curses or interactive UIs, but "make menuconfig" isn't
a bad UI :)  I've been halfway envisioning something else
that would use curses (more on that some other time).

Does lore.kernel.org run public-inbox-imapd?
Well, at least nntp works for lore:

   # populate ~/.cache/lei/uri_folder/.sqlite3
   lei ls-mail-source nntp://nntp.lore.kernel.org

   # if you're using bash:
   . contrib/completion/lei-completion.bash

   lei import <TAB><TAB>

lei import will download the entire newsgroup, which may get
huge, unfortunately.  It should support ranges (e.g.
nntp://example.com/inbox.foo.example/900-1000) but the URI
package doesn't support it, yet...

public-inbox-imapd IMAP mailbox slices are limited to 50K
messages since typical MUAs can't handle much more, so it's
easier to import just the most recent .[0-9]+ slices.

If you have Tor (on port 9050), libio-socket-socks (IO::Socket::Socks)

  # long Tor v3 URL :<
  url=imap://rskvuqcfnfizkjg6h5jvovwb3wkikzcwskf54lfpymus6mxrzw67b5ad.onion

  # configure IMAP to use Tor on port 9050:
  lei config imap.$url.proxy socks5h://0:9050

  # git 2.26+ can use a wildcard for all .onions:
  # lei config 'imap.imap://*.onion.proxy' socks5h://0:9050

  # I also suggest enabling compression for public-inbox-imapd
  # (some IMAP servers can't handle it, our -imapd does):
  lei config imap.$url.compress 1

  # Same as above with NNTP:
  lei ls-mail-source $url
  lei import <TAB-COMPLETE>

For public-inbox IMAP folders, make sure the \.[0-9]+ is at the
end of the mailbox to get the slice you want (there isn't
recursive import, yet).

Right now, "lei import" can be run periodically.  The plan is to
support IMAP IDLE and auto-imports.  For read/write IMAP
folders, there'd be auto keyword (FLAG) synchronization
(currently manual: lei export-kw imaps://...)
It mostly works for Maildirs at the moment...

...Which reminds me, I need to add an imapserver config variable
so the WWW interface can advertise IMAP servers like it does
NNTP servers

Re: make menuconfig interface for lei / grok-pull

[Eric Wong]

Luis Chamberlain <mcgrof@kernel.org> wrote:
> On Wed, Sep 15, 2021 at 09:44:20PM +0000, Eric Wong wrote:
> > Luis Chamberlain <mcgrof@kernel.org> wrote:
> > > # The next two don't work
> > > lei import import https://lore.kernel.org/linux-fsdevel/git/0.git
> > > lei add-external git://lore.kernel.org/linux-fsdevel/git/0.git
> > 
> > Apologies, I need to document this better.  This should work:
> > 
> >   lei add-external --mirror https://lore.kernel.org/linux-fsdevel SOME_FS_PATH
> 
> Groovy thanks! I'll start with that!
> 
> Any reason we need to specify SOME_FS_PATH ? Can't we assume a sensible
> default somewhere in .local/lei or whatever in the future?

Probably OK to have a default path for --mirror, the
add-external w/o --mirror works for existing inboxes (which I
also serve via -httpd/-imapd/-nntpd).

I think ~/.local/public-inbox/ would be a better default location
for mirrors, since I assume anything mirrored is public (and
therefor to rehost, as well).

Re: make menuconfig interface for lei / grok-pull

[Eric Wong]

Kyle Meyer <kyle@kyleam.com> wrote:
> A good chunk of the current documentation was fumbled together by me, so
> I'm sure there are many spots for improvement but...

No worries, my brain is a disorganized mess (always has been,
but pandemic has made it worse) and I've realized a lot of my
design choices around public-inbox + lei is based around that :x

Btw, I'm thinking about replacing prune-mail-sync with
"refresh-mail-sync [--prune]", so probably no point
in documenting it just yet....

Re: make menuconfig interface for lei / grok-pull

[Eric Wong]

Btw, without having any externals on disk:

  lei q [OPTIONS] --only https://lore.kernel.org/linux-fsdevel/ SEARCH_TERMS...

works, and I use it often.
Perl^Wlei: there's more than one way to do it :>

Re: make menuconfig interface for lei / grok-pull

[Konstantin Ryabitsev]

On Wed, Sep 15, 2021 at 02:34:40PM -0700, Luis Chamberlain wrote:
> My use case is I'm subscribed to a few kernel mailign lists and I use
> mutt with Maildir. I had configured recently pi-piper and grokmirror
> so that I get only the last 1 year of email from a few set of mailing
> lists. For this I needed to know the commit IDs for those emails on
> the public-inbox git mirror for each mailing list.

The pi-piper bit was really only useful for this until lei showed up. I wrote
it mostly so we could pipe things to patchwork straight from public-inbox git
archives.

> I was hinted using lei would be better though. But I'm stuck:

FYI, I'm giving a talk about that on Monday.
https://linuxplumbersconf.org/event/11/contributions/983/

Assuming I finish the prep work by then.

Hopefully, you don't live on the US West Coast and don't have to wake up at
7AM to attend.

-K

Re: make menuconfig interface for lei / grok-pull

[Konstantin Ryabitsev]

On Wed, Sep 15, 2021 at 11:06:05PM +0000, Eric Wong wrote:
> Does lore.kernel.org run public-inbox-imapd?

I'm still not convinced it's useful for huge collections, especially
considering how chatty IMAP is. Is there any benefit to enable it for lei use?

-K

IMAP vs NNTP [was: make menuconfig interface for lei / grok-pull]

[Eric Wong]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Wed, Sep 15, 2021 at 11:06:05PM +0000, Eric Wong wrote:
> > Does lore.kernel.org run public-inbox-imapd?
> 
> I'm still not convinced it's useful for huge collections, especially
> considering how chatty IMAP is. Is there any benefit to enable it for lei use?

lei tries to treat IMAP and NNTP the same for (import|tag).

One benefit of IMAP over NNTP is it doesn't add Xref or
Newsgroups headers that could throw off OID-based deduplication.
The other is compression (see below).

lei will be able to use IMAP IDLE to detect new messages for
auto-import; AFAIK there's nothing like it in NNTP.

In general, client-side support for read-only IMAP folders is
lacking, but lei isn't too different in that regard.

On the Perl side (which affects lei):

* the Mail::IMAPClient maintainer has been responsive to
  bugfixes I've sent, while the Net::NNTP (libnet) maintainer
  hasn't been.

* as a result, Net::NNTP doesn't have DEFLATE support, yet
  https://rt.cpan.org/Ticket/Display.html?id=129966
  https://rt.cpan.org/Ticket/Display.html?id=129967

* Net::NNTP is bundled with Perl, but Mail::IMAPClient is a
  separate package (but widely-packaged by distros)

lei import on epochs [was: make menuconfig interface for lei / grok-pull]

[Eric Wong]

Luis Chamberlain <mcgrof@kernel.org> wrote:
> # The next two don't work
> lei import import https://lore.kernel.org/linux-fsdevel/git/0.git

Btw, extra "import" aside; I never intended to support the above
case.  However, your post got me thinking we could and probably
should... :>

(epochs are git storage with a loose limit of ~1G packed data,
 "slices" are public-inbox-imapd folders capped at 50K to
 accomodate limitations of existing clients/filesystems)

Re: lei import on epochs [was: make menuconfig interface for lei / grok-pull]

[Luis Chamberlain]

On Thu, Sep 16, 2021 at 09:09:02PM +0000, Eric Wong wrote:
> Luis Chamberlain <mcgrof@kernel.org> wrote:
> > # The next two don't work
> > lei import import https://lore.kernel.org/linux-fsdevel/git/0.git
> 
> Btw, extra "import" aside; I never intended to support the above
> case.  However, your post got me thinking we could and probably
> should... :>
> 
> (epochs are git storage with a loose limit of ~1G packed data,
>  "slices" are public-inbox-imapd folders capped at 50K to
>  accomodate limitations of existing clients/filesystems)

So... by default all epochs are pulled? If so, yeah geesh.
I don't want to pull all linux-kernel epochs. Only if doing
R&D on that list would I need it.

Maybe a --query-epochs and then a --use-epoch 0  or whatever
as well?

  Luis

Re: make menuconfig interface for lei / grok-pull

[Luis Chamberlain]

On Thu, Sep 16, 2021 at 01:38:32PM -0400, Konstantin Ryabitsev wrote:
> On Wed, Sep 15, 2021 at 02:34:40PM -0700, Luis Chamberlain wrote:
> > My use case is I'm subscribed to a few kernel mailign lists and I use
> > mutt with Maildir. I had configured recently pi-piper and grokmirror
> > so that I get only the last 1 year of email from a few set of mailing
> > lists. For this I needed to know the commit IDs for those emails on
> > the public-inbox git mirror for each mailing list.
> 
> The pi-piper bit was really only useful for this until lei showed up. I wrote
> it mostly so we could pipe things to patchwork straight from public-inbox git
> archives.

OK got it! In lei we trust.

> > I was hinted using lei would be better though. But I'm stuck:
> 
> FYI, I'm giving a talk about that on Monday.
> https://linuxplumbersconf.org/event/11/contributions/983/
> 
> Assuming I finish the prep work by then.
> 
> Hopefully, you don't live on the US West Coast and don't have to wake up at
> 7AM to attend.

I do.. but hey I sometimes have 5am meetings so I might be able to make it!

  Luis

Re: lei import on epochs [was: make menuconfig interface for lei / grok-pull]

[Eric Wong]

Luis Chamberlain <mcgrof@kernel.org> wrote:
> On Thu, Sep 16, 2021 at 09:09:02PM +0000, Eric Wong wrote:
> > Luis Chamberlain <mcgrof@kernel.org> wrote:
> > > # The next two don't work
> > > lei import import https://lore.kernel.org/linux-fsdevel/git/0.git
> > 
> > Btw, extra "import" aside; I never intended to support the above
> > case.  However, your post got me thinking we could and probably
> > should... :>
> > 
> > (epochs are git storage with a loose limit of ~1G packed data,
> >  "slices" are public-inbox-imapd folders capped at 50K to
> >  accomodate limitations of existing clients/filesystems)
> 
> So... by default all epochs are pulled? If so, yeah geesh.
> I don't want to pull all linux-kernel epochs. Only if doing
> R&D on that list would I need it.

Right now, "lei import" doesn't read nor understand git repos at all.

If you meant add-external, yes, all epochs are pulled when using
"lei add-external --mirror" (and public-inbox-clone).  Yes,
there should be a way to do partial mirrors, just haven't
gotten around to it...

"lei import" currently reads IMAP, NNTP, Maildir, mbox*, etc.
It also understands various https?://*/(raw,t.mbox.gz) URLs used by
PublicInbox::WWW, since they're (gzipped) mboxrd.
But none of that is reading from git repos directly...

> Maybe a --query-epochs and then a --use-epoch 0  or whatever
> as well?

Since epochs expose the gittyness of lei, perhaps a limited form
of commitish-like ranges used by git would work:

  lei add-external --mirror $URL --epoch=0..9
  lei add-external --mirror $URL --epoch=5..
  lei add-external --mirror $URL --epoch=~2..  # like HEAD~2..

That only works for contiguous ranges, though; but I'm also
unsure if discontiguous ranges make sense to anyone.

There's other syntaxes (e.g. (sed|Perl)-like), of course,
but I think git-like here would be most natural.

Thoughts?

Re: lei import on epochs [was: make menuconfig interface for lei / grok-pull]

[Luis Chamberlain]

On Thu, Sep 16, 2021 at 5:22 PM Eric Wong <e@80x24.org> wrote:
> Since epochs expose the gittyness of lei, perhaps a limited form
> of commitish-like ranges used by git would work:
>
>   lei add-external --mirror $URL --epoch=0..9
>   lei add-external --mirror $URL --epoch=5..
>   lei add-external --mirror $URL --epoch=~2..  # like HEAD~2..

That looks sexy.

I mentioned the query option so that I could end up displaying on
'make menuconfig' the epochs dynamically, it would be a script that
wants the list of epochs, and I'd use this to dynamically populate
kconfig. Dynamic population of kconfig is not a yet well understood
thing, but I've figured out a way to do it :)

> That only works for contiguous ranges, though; but I'm also
> unsure if discontiguous ranges make sense to anyone.
>
> There's other syntaxes (e.g. (sed|Perl)-like), of course,
> but I think git-like here would be most natural.
>
> Thoughts?

I think the above would suffice for sure!

  Luis

Re: make menuconfig interface for lei / grok-pull

[Eric Wong]

Eric Wong <e@80x24.org> wrote:
> Luis Chamberlain <mcgrof@kernel.org> wrote:
>    # populate ~/.cache/lei/uri_folder/.sqlite3
>    lei ls-mail-source nntp://nntp.lore.kernel.org

	lei ls-mail-source -l nntp://nntp.lore.kernel.org

I just made "-l" show pretty JSON output in the terminal
to help find NNTP article ranges.

>    # if you're using bash:
>    . contrib/completion/lei-completion.bash
> 
>    lei import <TAB><TAB>
> 
> lei import will download the entire newsgroup, which may get
> huge, unfortunately.  It should support ranges (e.g.
> nntp://example.com/inbox.foo.example/900-1000) but the URI
> package doesn't support it, yet...

I was wrong about the URI package not supporting ranges,
it does; and forgot lei even seemed to have (untested)
support for it.  It should be working, now, just pushed out
commit 67fe4d8d90ac77419c8fc41457c849aa7d366a9d
("net_reader: fix single NNTP article fetch, test ranges")
<https://public-inbox.org/meta/20210919125035.6331-11-e@80x24.org/>

Neither Net::NNTP nor Mail::IMAPClient support pipelining
out-of-the-box, though; so network latency is still a problem.

I'm still not sure about doing an interactive UI (my brain
tends to struggle with that stuff :x); but I think extending
tab-completion for epochs and NNTP article ranges would be
good (I'm not particularly good with tab-completion, either :x)