unofficial mirror of meta@public-inbox.org
 help / color / mirror / Atom feed
* [PATCH 01/64] inbox: add uidvalidity method
@ 2020-10-16  6:59 Eric Wong
  2020-10-16  6:59 ` [PATCH 02/64] git: ensure ->destroy clobbers check_async read buffer Eric Wong
                   ` (4 more replies)
  0 siblings, 5 replies; 6+ messages in thread
From: Eric Wong @ 2020-10-16  6:59 UTC (permalink / raw)
  To: meta

This will make it easier to deal with ExtSearchIdx, which
won't have msgmap.
---
 lib/PublicInbox/DummyInbox.pm | 4 ++--
 lib/PublicInbox/IMAPD.pm      | 6 +++---
 lib/PublicInbox/Inbox.pm      | 2 ++
 3 files changed, 7 insertions(+), 5 deletions(-)

diff --git a/lib/PublicInbox/DummyInbox.pm b/lib/PublicInbox/DummyInbox.pm
index 69b0b683..02426f13 100644
--- a/lib/PublicInbox/DummyInbox.pm
+++ b/lib/PublicInbox/DummyInbox.pm
@@ -7,13 +7,13 @@
 package PublicInbox::DummyInbox;
 use strict;
 
-sub created_at { 0 } # Msgmap::created_at
+sub uidvalidity { 0 } # Msgmap::created_at
 sub mm { shift }
 sub uid_range { [] } # Over::uid_range
 sub subscribe_unlock { undef };
 
 no warnings 'once';
-*max = \&created_at;
+*max = \&uidvalidity;
 *query_xover = \&uid_range;
 *over = \&mm;
 *search = *unsubscribe_unlock =
diff --git a/lib/PublicInbox/IMAPD.pm b/lib/PublicInbox/IMAPD.pm
index 3c211ee1..bb705136 100644
--- a/lib/PublicInbox/IMAPD.pm
+++ b/lib/PublicInbox/IMAPD.pm
@@ -38,13 +38,13 @@ sub imapd_refresh_ibx { # pi_config->each_inbox cb
 	}
 	$ibx->over or return;
 	$ibx->{over} = undef;
-	my $mm = $ibx->mm or return;
-	$ibx->{mm} = undef;
 
 	# RFC 3501 2.3.1.1 -  "A good UIDVALIDITY value to use in
 	# this case is a 32-bit representation of the creation
 	# date/time of the mailbox"
-	defined($ibx->{uidvalidity} = $mm->created_at) or return;
+	eval { $ibx->uidvalidity };
+	my $mm = delete($ibx->{mm}) or return;
+	defined($ibx->{uidvalidity}) or return;
 	PublicInbox::IMAP::ensure_slices_exist($imapd, $ibx, $mm->max // 0);
 
 	# preload to avoid fragmentation:
diff --git a/lib/PublicInbox/Inbox.pm b/lib/PublicInbox/Inbox.pm
index b0894a7d..cbb95b8d 100644
--- a/lib/PublicInbox/Inbox.pm
+++ b/lib/PublicInbox/Inbox.pm
@@ -428,4 +428,6 @@ sub on_unlock {
 	}
 }
 
+sub uidvalidity  { $_[0]->{uidvalidity} //= $_[0]->mm->created_at }
+
 1;

^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH 02/64] git: ensure ->destroy clobbers check_async read buffer
  2020-10-16  6:59 [PATCH 01/64] inbox: add uidvalidity method Eric Wong
@ 2020-10-16  6:59 ` Eric Wong
  2020-10-16  6:59 ` [PATCH 03/64] git: *_async: support nested callback invocations Eric Wong
                   ` (3 subsequent siblings)
  4 siblings, 0 replies; 6+ messages in thread
From: Eric Wong @ 2020-10-16  6:59 UTC (permalink / raw)
  To: meta

It's currently not a problem as ->destroy doesn't
happen for no reason, we'll need to ensure future uses of
->destroy correctly discard the check_async buffer.
---
 lib/PublicInbox/Git.pm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/PublicInbox/Git.pm b/lib/PublicInbox/Git.pm
index 2323cecc..449223ec 100644
--- a/lib/PublicInbox/Git.pm
+++ b/lib/PublicInbox/Git.pm
@@ -239,7 +239,7 @@ sub check_async_step ($$) {
 	my ($self, $inflight_c) = @_;
 	die 'BUG: inflight empty or odd' if scalar(@$inflight_c) < 3;
 	my ($req, $cb, $arg) = splice(@$inflight_c, 0, 3);
-	my $rbuf = delete($self->{rbuf_c}) // \(my $new = '');
+	my $rbuf = delete($self->{chk_rbuf}) // \(my $new = '');
 	chomp(my $line = my_readline($self->{in_c}, $rbuf));
 	my ($hex, $type, $size) = split(/ /, $line);
 
@@ -253,7 +253,7 @@ sub check_async_step ($$) {
 	}
 	eval { $cb->($hex, $type, $size, $arg, $self) };
 	warn "E: check($req) $@\n" if $@;
-	$self->{rbuf_c} = $rbuf if $$rbuf ne '';
+	$self->{chk_rbuf} = $rbuf if $$rbuf ne '';
 }
 
 sub check_async_wait ($) {

^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH 03/64] git: *_async: support nested callback invocations
  2020-10-16  6:59 [PATCH 01/64] inbox: add uidvalidity method Eric Wong
  2020-10-16  6:59 ` [PATCH 02/64] git: ensure ->destroy clobbers check_async read buffer Eric Wong
@ 2020-10-16  6:59 ` Eric Wong
  2020-10-16  6:59 ` [PATCH 04/64] git: async: loop inflight checks for nested callbacks Eric Wong
                   ` (2 subsequent siblings)
  4 siblings, 0 replies; 6+ messages in thread
From: Eric Wong @ 2020-10-16  6:59 UTC (permalink / raw)
  To: meta

For external indices, we'll need to support nested cat_async
invocations to deduplicate cross-posted messages.

Thus we need to ensure we do not clobber the {inflight*} queues
while stepping through and ensure {cat_rbuf} is stored before
invoking callbacks.

This fixes the ->cat_async-only case, but does not yet
account for the mix of ->check_async interspersed with
->cat_async calls, yet.  More work will be needed on that
front at a later date.
---
 lib/PublicInbox/Git.pm | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/lib/PublicInbox/Git.pm b/lib/PublicInbox/Git.pm
index 449223ec..eb5de159 100644
--- a/lib/PublicInbox/Git.pm
+++ b/lib/PublicInbox/Git.pm
@@ -204,14 +204,14 @@ sub cat_async_step ($$) {
 	} else {
 		$self->fail("Unexpected result from async git cat-file: $head");
 	}
-	eval { $cb->($bref, $oid, $type, $size, $arg) };
 	$self->{cat_rbuf} = $rbuf if $$rbuf ne '';
+	eval { $cb->($bref, $oid, $type, $size, $arg) };
 	warn "E: $oid: $@\n" if $@;
 }
 
 sub cat_async_wait ($) {
 	my ($self) = @_;
-	my $inflight = delete $self->{inflight} or return;
+	my $inflight = $self->{inflight} or return;
 	while (scalar(@$inflight)) {
 		cat_async_step($self, $inflight);
 	}
@@ -251,14 +251,14 @@ sub check_async_step ($$) {
 		my $ret = my_read($self->{in_c}, $rbuf, $type + 1);
 		fail($self, defined($ret) ? 'read EOF' : "read: $!") if !$ret;
 	}
+	$self->{chk_rbuf} = $rbuf if $$rbuf ne '';
 	eval { $cb->($hex, $type, $size, $arg, $self) };
 	warn "E: check($req) $@\n" if $@;
-	$self->{chk_rbuf} = $rbuf if $$rbuf ne '';
 }
 
 sub check_async_wait ($) {
 	my ($self) = @_;
-	my $inflight_c = delete $self->{inflight_c} or return;
+	my $inflight_c = $self->{inflight_c} or return;
 	while (scalar(@$inflight_c)) {
 		check_async_step($self, $inflight_c);
 	}
@@ -318,13 +318,15 @@ sub _destroy {
 
 sub cat_async_abort ($) {
 	my ($self) = @_;
-	if (my $inflight = delete $self->{inflight}) {
+	if (my $inflight = $self->{inflight}) {
 		while (@$inflight) {
 			my ($req, $cb, $arg) = splice(@$inflight, 0, 3);
 			$req =~ s/ .*//; # drop git_dir for Gcf2Client
 			eval { $cb->(undef, $req, undef, undef, $arg) };
 			warn "E: $req: $@ (in abort)\n" if $@;
 		}
+		delete $self->{cat_rbuf};
+		delete $self->{inflight};
 	}
 	cleanup($self);
 }
@@ -357,6 +359,8 @@ sub cleanup {
 	delete $self->{async_cat};
 	check_async_wait($self);
 	cat_async_wait($self);
+	delete $self->{inflight};
+	delete $self->{inflight_c};
 	_destroy($self, qw(cat_rbuf in out pid));
 	_destroy($self, qw(chk_rbuf in_c out_c pid_c err_c));
 	!!($self->{pid} || $self->{pid_c});

^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH 04/64] git: async: loop inflight checks for nested callbacks
  2020-10-16  6:59 [PATCH 01/64] inbox: add uidvalidity method Eric Wong
  2020-10-16  6:59 ` [PATCH 02/64] git: ensure ->destroy clobbers check_async read buffer Eric Wong
  2020-10-16  6:59 ` [PATCH 03/64] git: *_async: support nested callback invocations Eric Wong
@ 2020-10-16  6:59 ` Eric Wong
  2020-10-16  7:02 ` oops, :x was supposed to be 1/3 for git: stuff Eric Wong
  2020-10-16  7:03 ` [PATCH 01/64] inbox: add uidvalidity method Eric Wong
  4 siblings, 0 replies; 6+ messages in thread
From: Eric Wong @ 2020-10-16  6:59 UTC (permalink / raw)
  To: meta

We need to loop the inflight check for nested callback
invocations to ensure we don't clog the pipe that feeds
`git cat-file'.

This bug was obscured by the fact that we're already
accounting for 64-char git OIDs with SHA-256 in the
pipe space calculation; perhaps we shouldn't do that.
---
 lib/PublicInbox/Git.pm | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/lib/PublicInbox/Git.pm b/lib/PublicInbox/Git.pm
index eb5de159..e3a2bcb8 100644
--- a/lib/PublicInbox/Git.pm
+++ b/lib/PublicInbox/Git.pm
@@ -275,7 +275,7 @@ sub check_async_begin ($) {
 sub check_async ($$$$) {
 	my ($self, $oid, $cb, $arg) = @_;
 	my $inflight_c = $self->{inflight_c} // check_async_begin($self);
-	if (scalar(@$inflight_c) >= MAX_INFLIGHT) {
+	while (scalar(@$inflight_c) >= MAX_INFLIGHT) {
 		check_async_step($self, $inflight_c);
 	}
 	print { $self->{out_c} } $oid, "\n" or fail($self, "write error: $!");
@@ -420,10 +420,9 @@ sub cat_async_begin {
 sub cat_async ($$$;$) {
 	my ($self, $oid, $cb, $arg) = @_;
 	my $inflight = $self->{inflight} // cat_async_begin($self);
-	if (scalar(@$inflight) >= MAX_INFLIGHT) {
+	while (scalar(@$inflight) >= MAX_INFLIGHT) {
 		cat_async_step($self, $inflight);
 	}
-
 	print { $self->{out} } $oid, "\n" or fail($self, "write error: $!");
 	push(@$inflight, $oid, $cb, $arg);
 }

^ permalink raw reply	[flat|nested] 6+ messages in thread

* oops, :x was supposed to be 1/3 for git: stuff
  2020-10-16  6:59 [PATCH 01/64] inbox: add uidvalidity method Eric Wong
                   ` (2 preceding siblings ...)
  2020-10-16  6:59 ` [PATCH 04/64] git: async: loop inflight checks for nested callbacks Eric Wong
@ 2020-10-16  7:02 ` Eric Wong
  2020-10-16  7:03 ` [PATCH 01/64] inbox: add uidvalidity method Eric Wong
  4 siblings, 0 replies; 6+ messages in thread
From: Eric Wong @ 2020-10-16  7:02 UTC (permalink / raw)
  To: meta

Fat-fingered and sent the wrong directory.  Anyways, some git
async fixes which seems to be working well enough and ironed out
by yet-to-be-published extsearch indexing code.

AFAIK none of the current code in public-inbox.git is affected
by these chagnes.

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH 01/64] inbox: add uidvalidity method
  2020-10-16  6:59 [PATCH 01/64] inbox: add uidvalidity method Eric Wong
                   ` (3 preceding siblings ...)
  2020-10-16  7:02 ` oops, :x was supposed to be 1/3 for git: stuff Eric Wong
@ 2020-10-16  7:03 ` Eric Wong
  4 siblings, 0 replies; 6+ messages in thread
From: Eric Wong @ 2020-10-16  7:03 UTC (permalink / raw)
  To: meta

Eric Wong <e@80x24.org> wrote:
> This will make it easier to deal with ExtSearchIdx, which
> won't have msgmap.

I was going to send this as a standalone patch (not part of any
series, but it's fine, here).

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2020-10-16  7:03 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-10-16  6:59 [PATCH 01/64] inbox: add uidvalidity method Eric Wong
2020-10-16  6:59 ` [PATCH 02/64] git: ensure ->destroy clobbers check_async read buffer Eric Wong
2020-10-16  6:59 ` [PATCH 03/64] git: *_async: support nested callback invocations Eric Wong
2020-10-16  6:59 ` [PATCH 04/64] git: async: loop inflight checks for nested callbacks Eric Wong
2020-10-16  7:02 ` oops, :x was supposed to be 1/3 for git: stuff Eric Wong
2020-10-16  7:03 ` [PATCH 01/64] inbox: add uidvalidity method Eric Wong

unofficial mirror of meta@public-inbox.org

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://yhetil.org/meta

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 meta meta/ https://yhetil.org/meta \
		meta@public-inbox.org
	public-inbox-index meta

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.yhetil.org/yhetil.mail.public-inbox.meta
	nntp://news.public-inbox.org/inbox.mail.public-inbox.meta
	nntp://news.gmane.io/gmane.mail.public-inbox.general


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git