Re: Large delays in mailing list delivery?

Re: Large delays in mailing list delivery?

[Ævar Arnfjörð Bjarmason]

On Fri, Dec 03 2021, Konstantin Ryabitsev wrote:

> On Fri, Dec 03, 2021 at 09:02:48PM +0100, Ævar Arnfjörð Bjarmason wrote:
>> When I've experienced delays (sometimes of half a day or more) both
>> https://public-inbox.org/git/ and https://lore.kernel.org/git/ have been
>> updated.
>
> Btw, you can source lore.kernel.org straight into your gmail inbox. :)
>
>     https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
>     https://people.kernel.org/monsieuricon/lore-lei-part-2-now-with-imap
>
> Or, you can read it via nntp://nntp.lore.kernel.org/.

[CC'd meta@public-inbox.org, probably best to move this thread over
there sooner than later, but CC'ing git@ still in case this is
interesting to others]

I poked a bit at setting this up but couldn't find from building
public-inbox.org & trying to page through the docs how I'd get from an
existing public-inbox.org/git/ checkout to a local Maildir.

If you could share some recipe or a pointer to the right docs for that
that would be much appreciated. Thanks!

Re: Large delays in mailing list delivery?

[Eric Wong]

Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
> On Fri, Dec 03 2021, Konstantin Ryabitsev wrote:
> 
> > On Fri, Dec 03, 2021 at 09:02:48PM +0100, Ævar Arnfjörð Bjarmason wrote:
> >> When I've experienced delays (sometimes of half a day or more) both
> >> https://public-inbox.org/git/ and https://lore.kernel.org/git/ have been
> >> updated.
> >
> > Btw, you can source lore.kernel.org straight into your gmail inbox. :)
> >
> >     https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
> >     https://people.kernel.org/monsieuricon/lore-lei-part-2-now-with-imap
> >
> > Or, you can read it via nntp://nntp.lore.kernel.org/.
> 
> [CC'd meta@public-inbox.org, probably best to move this thread over
> there sooner than later, but CC'ing git@ still in case this is
> interesting to others]
> 
> I poked a bit at setting this up but couldn't find from building
> public-inbox.org & trying to page through the docs how I'd get from an
> existing public-inbox.org/git/ checkout to a local Maildir.

Existing, public-inboxes can be set as "externals" and managed
via {add,forget,ls}-external sub-commands:

	# for locally-cloned inboxes:
	public-inbox-index /path/to/existing/inbox
	lei add-external /path/to/existing/inbox

	# relies on curl, memoizes data downloaded for each search:
	lei add-external https://yhbt.net/lore/git

Local externals will be included by every "lei q" invocation;
HTTP(S) ones require "lei q --remote"

If you only want to use an external as a one-off without adding
it, the -I/--include and -O/--only flags are useful:

  lei q -O https://yhbt.net/lore/git -o /tmp/results SEARCH_TERMS

> If you could share some recipe or a pointer to the right docs for that
> that would be much appreciated. Thanks!

lei-overview(7) manpage documents some things, at least:
https://public-inbox.org/lei-overview.html  Patches welcome :>

IMHO lei still kinda sucks, and I probably won't have time to
work on it for a bit :<

Using public-inbox+lei+Emacs+mu+mu4e (was: Large delays in mailing list delivery?)

[Ævar Arnfjörð Bjarmason]

On Mon, Dec 06 2021, Eric Wong wrote:

> Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
>> On Fri, Dec 03 2021, Konstantin Ryabitsev wrote:
>> 
>> > On Fri, Dec 03, 2021 at 09:02:48PM +0100, Ævar Arnfjörð Bjarmason wrote:
>> >> When I've experienced delays (sometimes of half a day or more) both
>> >> https://public-inbox.org/git/ and https://lore.kernel.org/git/ have been
>> >> updated.
>> >
>> > Btw, you can source lore.kernel.org straight into your gmail inbox. :)
>> >
>> >     https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
>> >     https://people.kernel.org/monsieuricon/lore-lei-part-2-now-with-imap
>> >
>> > Or, you can read it via nntp://nntp.lore.kernel.org/.
>> 
>> [CC'd meta@public-inbox.org, probably best to move this thread over
>> there sooner than later, but CC'ing git@ still in case this is
>> interesting to others]
>> 
>> I poked a bit at setting this up but couldn't find from building
>> public-inbox.org & trying to page through the docs how I'd get from an
>> existing public-inbox.org/git/ checkout to a local Maildir.
>
> Existing, public-inboxes can be set as "externals" and managed
> via {add,forget,ls}-external sub-commands:
>
> 	# for locally-cloned inboxes:
> 	public-inbox-index /path/to/existing/inbox
> 	lei add-external /path/to/existing/inbox
>
> 	# relies on curl, memoizes data downloaded for each search:
> 	lei add-external https://yhbt.net/lore/git
>
> Local externals will be included by every "lei q" invocation;
> HTTP(S) ones require "lei q --remote"
>
> If you only want to use an external as a one-off without adding
> it, the -I/--include and -O/--only flags are useful:
>
>   lei q -O https://yhbt.net/lore/git -o /tmp/results SEARCH_TERMS
>
>> If you could share some recipe or a pointer to the right docs for that
>> that would be much appreciated. Thanks!
>
> lei-overview(7) manpage documents some things, at least:
> https://public-inbox.org/lei-overview.html  Patches welcome :>
>
> IMHO lei still kinda sucks, and I probably won't have time to
> work on it for a bit :<

Thanks. I finally got around to setting this up.

The above instructions didn't quite work for me, but here's what I did
(in the form of a script lifted from a screen(1) config I've
got). Indented with un-indented comments. The "stuff" is screen's way of
"run this command" (or well, input these characters):

I had a ~/g/git-ml clone already, but this makes one:
    
    stuff "cd ~/g/git-ml || git clone https://public-inbox.org/git ~/g/git-ml^M"

The initial index:

    ## This will create a .git/publici-inbox in ~/g/git-ml. Takes a while
    ## the first time.
    stuff "time public-inbox-index -v \$PWD^M"

I fiddled with this for a bit because it refused to work, turns out it
was missing the .git at the end, i.e. it expected a bare repo[1]:

    ## When we add the lei external it *must* have the ".git" part,
    ## because it'll try to find the "public-inbox" folder at wherever we
    ## point it.
    stuff "test -d ~/.config/lei || lei add-external ~/g/git-ml/.git^M"

A bit of a UX wart not to be able to specify no --limit, or maybe I'm
missing a way:

    ## The one-off massive import of the Git ML. TODO: No way to specify
    ## an infinite limit? Not --no-limit or --limit=0.
    stuff "test -d ~/Maildir/lei-q-git-ml || time lei q --limit=999999999 -v -o ~/Maildir/lei-q-git-ml l:git.vger.kernel.org^M"

The initial indexing:

    ## After the one-off import this will take forever *the first time*
    ## (or around 20m), but subsequent invocations will be fast:
    stuff "time lei up ~/Maildir/lei-q-git-ml^M"

Runs an ad-hoc script to keep it up-to-date, which is quoted below:

    ## Run it in a loop
    stuff "public-inbox-lei-pull-index^M"

That script (which I whipped up just now. Is there a better/more
standard way? to keep a public-inbox+lei pair up-to-date with
sleep/backoff etc?
	
	#!/bin/sh
	set -xe
	
	repo=~/g/git-ml
	while true
	do
		oid=$(git -C $repo rev-parse HEAD)
		git -C $repo pull
		noid=$(git -C $repo rev-parse HEAD)
		if test "$oid" = "$noid"
		then
			echo Nothing to update
			sleep 60
			continue
		fi
		(
			cd $repo &&
			public-inbox-index -v "$PWD"
		)
		lei up ~/Maildir/lei-q-git-ml
		sleep 1
	done

I use Emacs+mu4e for my E-Mail. And since I index ~/Maildir having these
files dropped in there will be added to its index. Then I just changed
my saved search to also look through that maildir (I guess the entire
first condition could be dropped, but whatever):

    "(maildir:/personal-gmail/* OR maildir:/lei-q-git-ml/*) AND list:git.vger.kernel.org OR recip:git@vger.kernel.org OR recip:git-packagers@googlegroups.com"

Because "mu" is generally good about de-duplicating stuff I've now got
an inbox with mixed messages I can sync from GMail (including my "Sent"
folder), and I get up-to-the-minute ML traffic now (it's bee 10hrs-4day
delayed for 3-4 months at least).

So new messages are generally from the "lei" directory, but when I send
one it'll be dropped in the personal-gmail.

I still need to check if it's doing the wrong thing with e.g. "read"
flags if I read a mail synced via lei that later arrives in GMail. But I
mostly don't use "read" statuses anyway...

1.  Maybe this "I only tested if it complied" patch would make sense to catch that?

diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 2958d3f9..be49621f 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -613,7 +613,7 @@ sub add_uri {
 	}
 }
 
-sub prepare_external {
+sub _prepare_external {
 	my ($self, $loc, $boost) = @_; # n.b. already ordered by boost
 	if (ref $loc) { # already a URI, or PublicInbox::Inbox-like object
 		return add_uri($self, $loc) if $loc->can('scheme');
@@ -638,6 +638,14 @@ sub prepare_external {
 	push @{$self->{locals}}, $loc;
 }
 
+sub prepare_external {
+	my ($self, $loc, $boost) = @_;
+	my $ret = _prepare_external($self, $loc, $boost);
+	warn "W: we got nothing from $loc, did you mean $loc/.git?"
+		if !$ret && -e "$loc/.git";
+	return $ret;
+}
+
 sub _lcat_i { # LeiMailSync->each_src iterator callback
 	my ($oidbin, $id, $each_smsg) = @_;
 	$each_smsg->({blob => unpack('H*', $oidbin), pct => 100});

Re: Using public-inbox+lei+Emacs+mu+mu4e (was: Large delays in mailing list delivery?)

[Eric Wong]

Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
> On Mon, Dec 06 2021, Eric Wong wrote:
> > Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
> >> On Fri, Dec 03 2021, Konstantin Ryabitsev wrote:
> >> 
> >> > On Fri, Dec 03, 2021 at 09:02:48PM +0100, Ævar Arnfjörð Bjarmason wrote:
> >> >> When I've experienced delays (sometimes of half a day or more) both
> >> >> https://public-inbox.org/git/ and https://lore.kernel.org/git/ have been
> >> >> updated.
> >> >
> >> > Btw, you can source lore.kernel.org straight into your gmail inbox. :)
> >> >
> >> >     https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
> >> >     https://people.kernel.org/monsieuricon/lore-lei-part-2-now-with-imap
> >> >
> >> > Or, you can read it via nntp://nntp.lore.kernel.org/.
> >> 
> >> [CC'd meta@public-inbox.org, probably best to move this thread over
> >> there sooner than later, but CC'ing git@ still in case this is
> >> interesting to others]
> >> 
> >> I poked a bit at setting this up but couldn't find from building
> >> public-inbox.org & trying to page through the docs how I'd get from an
> >> existing public-inbox.org/git/ checkout to a local Maildir.
> >
> > Existing, public-inboxes can be set as "externals" and managed
> > via {add,forget,ls}-external sub-commands:
> >
> > 	# for locally-cloned inboxes:
> > 	public-inbox-index /path/to/existing/inbox
> > 	lei add-external /path/to/existing/inbox
> >
> > 	# relies on curl, memoizes data downloaded for each search:
> > 	lei add-external https://yhbt.net/lore/git
> >
> > Local externals will be included by every "lei q" invocation;
> > HTTP(S) ones require "lei q --remote"
> >
> > If you only want to use an external as a one-off without adding
> > it, the -I/--include and -O/--only flags are useful:
> >
> >   lei q -O https://yhbt.net/lore/git -o /tmp/results SEARCH_TERMS
> >
> >> If you could share some recipe or a pointer to the right docs for that
> >> that would be much appreciated. Thanks!
> >
> > lei-overview(7) manpage documents some things, at least:
> > https://public-inbox.org/lei-overview.html  Patches welcome :>
> >
> > IMHO lei still kinda sucks, and I probably won't have time to
> > work on it for a bit :<
> 
> Thanks. I finally got around to setting this up.
> 
> The above instructions didn't quite work for me, but here's what I did
> (in the form of a script lifted from a screen(1) config I've
> got). Indented with un-indented comments. The "stuff" is screen's way of
> "run this command" (or well, input these characters):
> 
> I had a ~/g/git-ml clone already, but this makes one:
>     
>     stuff "cd ~/g/git-ml || git clone https://public-inbox.org/git ~/g/git-ml^M"

Wait, lack of --mirror or --bare for git-clone on any
public-inbox is a big mistake in terms of inode+disk use

> The initial index:
> 
>     ## This will create a .git/publici-inbox in ~/g/git-ml. Takes a while
>     ## the first time.
>     stuff "time public-inbox-index -v \$PWD^M"
> 
> I fiddled with this for a bit because it refused to work, turns out it
> was missing the .git at the end, i.e. it expected a bare repo[1]:
> 
>     ## When we add the lei external it *must* have the ".git" part,
>     ## because it'll try to find the "public-inbox" folder at wherever we
>     ## point it.
>     stuff "test -d ~/.config/lei || lei add-external ~/g/git-ml/.git^M"

Yeah, public-inbox has never supported non-bare usage.

> A bit of a UX wart not to be able to specify no --limit, or maybe I'm
> missing a way:
> 
>     ## The one-off massive import of the Git ML. TODO: No way to specify
>     ## an infinite limit? Not --no-limit or --limit=0.
>     stuff "test -d ~/Maildir/lei-q-git-ml || time lei q --limit=999999999 -v -o ~/Maildir/lei-q-git-ml l:git.vger.kernel.org^M"

Oops.  --no-limit or --limit=0 or --limit=-1 support will be
added at some point...

> The initial indexing:
> 
>     ## After the one-off import this will take forever *the first time*
>     ## (or around 20m), but subsequent invocations will be fast:
>     stuff "time lei up ~/Maildir/lei-q-git-ml^M"
> 
> Runs an ad-hoc script to keep it up-to-date, which is quoted below:
> 
>     ## Run it in a loop
>     stuff "public-inbox-lei-pull-index^M"
> 
> That script (which I whipped up just now. Is there a better/more
> standard way? to keep a public-inbox+lei pair up-to-date with
> sleep/backoff etc?

Not yet, unfortunately.  I would like to have some sort of
long-polling support (similar to IDLE in IMAP) because I hate
sleep/backoff.  But I don't think I'm physically nor mentally
capable of doing that or much of anything, anymore.

> 	#!/bin/sh
> 	set -xe
> 	
> 	repo=~/g/git-ml
> 	while true
> 	do
> 		oid=$(git -C $repo rev-parse HEAD)
> 		git -C $repo pull
> 		noid=$(git -C $repo rev-parse HEAD)
> 		if test "$oid" = "$noid"
> 		then
> 			echo Nothing to update
> 			sleep 60
> 			continue
> 		fi
> 		(
> 			cd $repo &&
> 			public-inbox-index -v "$PWD"
> 		)
> 		lei up ~/Maildir/lei-q-git-ml
> 		sleep 1
> 	done
> 
> I use Emacs+mu4e for my E-Mail. And since I index ~/Maildir having these
> files dropped in there will be added to its index. Then I just changed
> my saved search to also look through that maildir (I guess the entire
> first condition could be dropped, but whatever):
> 
>     "(maildir:/personal-gmail/* OR maildir:/lei-q-git-ml/*) AND list:git.vger.kernel.org OR recip:git@vger.kernel.org OR recip:git-packagers@googlegroups.com"
> 
> Because "mu" is generally good about de-duplicating stuff I've now got
> an inbox with mixed messages I can sync from GMail (including my "Sent"
> folder), and I get up-to-the-minute ML traffic now (it's bee 10hrs-4day
> delayed for 3-4 months at least).
> 
> So new messages are generally from the "lei" directory, but when I send
> one it'll be dropped in the personal-gmail.
> 
> I still need to check if it's doing the wrong thing with e.g. "read"
> flags if I read a mail synced via lei that later arrives in GMail. But I
> mostly don't use "read" statuses anyway...

lei can export flags from its internal DBs to IMAP via "lei export-kw"
see lei-mail-sync-overview(7) for some details, but usability is still
rough...

> 1.  Maybe this "I only tested if it complied" patch would make sense to catch that?
> 
> diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
> index 2958d3f9..be49621f 100644
> --- a/lib/PublicInbox/LeiXSearch.pm
> +++ b/lib/PublicInbox/LeiXSearch.pm
> @@ -613,7 +613,7 @@ sub add_uri {
>  	}
>  }
>  
> -sub prepare_external {
> +sub _prepare_external {
>  	my ($self, $loc, $boost) = @_; # n.b. already ordered by boost
>  	if (ref $loc) { # already a URI, or PublicInbox::Inbox-like object
>  		return add_uri($self, $loc) if $loc->can('scheme');
> @@ -638,6 +638,14 @@ sub prepare_external {
>  	push @{$self->{locals}}, $loc;
>  }
>  
> +sub prepare_external {
> +	my ($self, $loc, $boost) = @_;
> +	my $ret = _prepare_external($self, $loc, $boost);
> +	warn "W: we got nothing from $loc, did you mean $loc/.git?"
> +		if !$ret && -e "$loc/.git";
> +	return $ret;
> +}

Probably add a note saying non-bare repos are a terrible idea,
anyways, especially for v1 public-inboxes like public-inbox.org/git