* [PATCH 02/64] git: ensure ->destroy clobbers check_async read buffer
2020-10-16 6:59 [PATCH 01/64] inbox: add uidvalidity method Eric Wong
@ 2020-10-16 6:59 ` Eric Wong
2020-10-16 6:59 ` [PATCH 03/64] git: *_async: support nested callback invocations Eric Wong
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Eric Wong @ 2020-10-16 6:59 UTC (permalink / raw)
To: meta
It's currently not a problem as ->destroy doesn't
happen for no reason, we'll need to ensure future uses of
->destroy correctly discard the check_async buffer.
---
lib/PublicInbox/Git.pm | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/Git.pm b/lib/PublicInbox/Git.pm
index 2323cecc..449223ec 100644
--- a/lib/PublicInbox/Git.pm
+++ b/lib/PublicInbox/Git.pm
@@ -239,7 +239,7 @@ sub check_async_step ($$) {
my ($self, $inflight_c) = @_;
die 'BUG: inflight empty or odd' if scalar(@$inflight_c) < 3;
my ($req, $cb, $arg) = splice(@$inflight_c, 0, 3);
- my $rbuf = delete($self->{rbuf_c}) // \(my $new = '');
+ my $rbuf = delete($self->{chk_rbuf}) // \(my $new = '');
chomp(my $line = my_readline($self->{in_c}, $rbuf));
my ($hex, $type, $size) = split(/ /, $line);
@@ -253,7 +253,7 @@ sub check_async_step ($$) {
}
eval { $cb->($hex, $type, $size, $arg, $self) };
warn "E: check($req) $@\n" if $@;
- $self->{rbuf_c} = $rbuf if $$rbuf ne '';
+ $self->{chk_rbuf} = $rbuf if $$rbuf ne '';
}
sub check_async_wait ($) {
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH 03/64] git: *_async: support nested callback invocations
2020-10-16 6:59 [PATCH 01/64] inbox: add uidvalidity method Eric Wong
2020-10-16 6:59 ` [PATCH 02/64] git: ensure ->destroy clobbers check_async read buffer Eric Wong
@ 2020-10-16 6:59 ` Eric Wong
2020-10-16 6:59 ` [PATCH 04/64] git: async: loop inflight checks for nested callbacks Eric Wong
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Eric Wong @ 2020-10-16 6:59 UTC (permalink / raw)
To: meta
For external indices, we'll need to support nested cat_async
invocations to deduplicate cross-posted messages.
Thus we need to ensure we do not clobber the {inflight*} queues
while stepping through and ensure {cat_rbuf} is stored before
invoking callbacks.
This fixes the ->cat_async-only case, but does not yet
account for the mix of ->check_async interspersed with
->cat_async calls, yet. More work will be needed on that
front at a later date.
---
lib/PublicInbox/Git.pm | 14 +++++++++-----
1 file changed, 9 insertions(+), 5 deletions(-)
diff --git a/lib/PublicInbox/Git.pm b/lib/PublicInbox/Git.pm
index 449223ec..eb5de159 100644
--- a/lib/PublicInbox/Git.pm
+++ b/lib/PublicInbox/Git.pm
@@ -204,14 +204,14 @@ sub cat_async_step ($$) {
} else {
$self->fail("Unexpected result from async git cat-file: $head");
}
- eval { $cb->($bref, $oid, $type, $size, $arg) };
$self->{cat_rbuf} = $rbuf if $$rbuf ne '';
+ eval { $cb->($bref, $oid, $type, $size, $arg) };
warn "E: $oid: $@\n" if $@;
}
sub cat_async_wait ($) {
my ($self) = @_;
- my $inflight = delete $self->{inflight} or return;
+ my $inflight = $self->{inflight} or return;
while (scalar(@$inflight)) {
cat_async_step($self, $inflight);
}
@@ -251,14 +251,14 @@ sub check_async_step ($$) {
my $ret = my_read($self->{in_c}, $rbuf, $type + 1);
fail($self, defined($ret) ? 'read EOF' : "read: $!") if !$ret;
}
+ $self->{chk_rbuf} = $rbuf if $$rbuf ne '';
eval { $cb->($hex, $type, $size, $arg, $self) };
warn "E: check($req) $@\n" if $@;
- $self->{chk_rbuf} = $rbuf if $$rbuf ne '';
}
sub check_async_wait ($) {
my ($self) = @_;
- my $inflight_c = delete $self->{inflight_c} or return;
+ my $inflight_c = $self->{inflight_c} or return;
while (scalar(@$inflight_c)) {
check_async_step($self, $inflight_c);
}
@@ -318,13 +318,15 @@ sub _destroy {
sub cat_async_abort ($) {
my ($self) = @_;
- if (my $inflight = delete $self->{inflight}) {
+ if (my $inflight = $self->{inflight}) {
while (@$inflight) {
my ($req, $cb, $arg) = splice(@$inflight, 0, 3);
$req =~ s/ .*//; # drop git_dir for Gcf2Client
eval { $cb->(undef, $req, undef, undef, $arg) };
warn "E: $req: $@ (in abort)\n" if $@;
}
+ delete $self->{cat_rbuf};
+ delete $self->{inflight};
}
cleanup($self);
}
@@ -357,6 +359,8 @@ sub cleanup {
delete $self->{async_cat};
check_async_wait($self);
cat_async_wait($self);
+ delete $self->{inflight};
+ delete $self->{inflight_c};
_destroy($self, qw(cat_rbuf in out pid));
_destroy($self, qw(chk_rbuf in_c out_c pid_c err_c));
!!($self->{pid} || $self->{pid_c});
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH 04/64] git: async: loop inflight checks for nested callbacks
2020-10-16 6:59 [PATCH 01/64] inbox: add uidvalidity method Eric Wong
2020-10-16 6:59 ` [PATCH 02/64] git: ensure ->destroy clobbers check_async read buffer Eric Wong
2020-10-16 6:59 ` [PATCH 03/64] git: *_async: support nested callback invocations Eric Wong
@ 2020-10-16 6:59 ` Eric Wong
2020-10-16 7:02 ` oops, :x was supposed to be 1/3 for git: stuff Eric Wong
2020-10-16 7:03 ` [PATCH 01/64] inbox: add uidvalidity method Eric Wong
4 siblings, 0 replies; 6+ messages in thread
From: Eric Wong @ 2020-10-16 6:59 UTC (permalink / raw)
To: meta
We need to loop the inflight check for nested callback
invocations to ensure we don't clog the pipe that feeds
`git cat-file'.
This bug was obscured by the fact that we're already
accounting for 64-char git OIDs with SHA-256 in the
pipe space calculation; perhaps we shouldn't do that.
---
lib/PublicInbox/Git.pm | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/Git.pm b/lib/PublicInbox/Git.pm
index eb5de159..e3a2bcb8 100644
--- a/lib/PublicInbox/Git.pm
+++ b/lib/PublicInbox/Git.pm
@@ -275,7 +275,7 @@ sub check_async_begin ($) {
sub check_async ($$$$) {
my ($self, $oid, $cb, $arg) = @_;
my $inflight_c = $self->{inflight_c} // check_async_begin($self);
- if (scalar(@$inflight_c) >= MAX_INFLIGHT) {
+ while (scalar(@$inflight_c) >= MAX_INFLIGHT) {
check_async_step($self, $inflight_c);
}
print { $self->{out_c} } $oid, "\n" or fail($self, "write error: $!");
@@ -420,10 +420,9 @@ sub cat_async_begin {
sub cat_async ($$$;$) {
my ($self, $oid, $cb, $arg) = @_;
my $inflight = $self->{inflight} // cat_async_begin($self);
- if (scalar(@$inflight) >= MAX_INFLIGHT) {
+ while (scalar(@$inflight) >= MAX_INFLIGHT) {
cat_async_step($self, $inflight);
}
-
print { $self->{out} } $oid, "\n" or fail($self, "write error: $!");
push(@$inflight, $oid, $cb, $arg);
}
^ permalink raw reply related [flat|nested] 6+ messages in thread* oops, :x was supposed to be 1/3 for git: stuff
2020-10-16 6:59 [PATCH 01/64] inbox: add uidvalidity method Eric Wong
` (2 preceding siblings ...)
2020-10-16 6:59 ` [PATCH 04/64] git: async: loop inflight checks for nested callbacks Eric Wong
@ 2020-10-16 7:02 ` Eric Wong
2020-10-16 7:03 ` [PATCH 01/64] inbox: add uidvalidity method Eric Wong
4 siblings, 0 replies; 6+ messages in thread
From: Eric Wong @ 2020-10-16 7:02 UTC (permalink / raw)
To: meta
Fat-fingered and sent the wrong directory. Anyways, some git
async fixes which seems to be working well enough and ironed out
by yet-to-be-published extsearch indexing code.
AFAIK none of the current code in public-inbox.git is affected
by these chagnes.
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH 01/64] inbox: add uidvalidity method
2020-10-16 6:59 [PATCH 01/64] inbox: add uidvalidity method Eric Wong
` (3 preceding siblings ...)
2020-10-16 7:02 ` oops, :x was supposed to be 1/3 for git: stuff Eric Wong
@ 2020-10-16 7:03 ` Eric Wong
4 siblings, 0 replies; 6+ messages in thread
From: Eric Wong @ 2020-10-16 7:03 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> This will make it easier to deal with ExtSearchIdx, which
> won't have msgmap.
I was going to send this as a standalone patch (not part of any
series, but it's fine, here).
^ permalink raw reply [flat|nested] 6+ messages in thread