unofficial mirror of meta@public-inbox.org
 help / color / mirror / Atom feed
From: Eric Wong <e@80x24.org>
To: Chris Brannon <chris@the-brannons.com>
Cc: meta@public-inbox.org
Subject: Re: public-inbox-convert hangs on systems using musl libc
Date: Wed, 21 Dec 2022 21:11:14 +0000	[thread overview]
Message-ID: <20221221211114.M412849@dcvr> (raw)
In-Reply-To: <87mt7glc23.fsf@the-brannons.com>

Chris Brannon <chris@the-brannons.com> wrote:
> Eric Wong <e@80x24.org> writes:
> 
> > Are you able to isolate whether $PIPE_BUFSIZ or MAX_INFLIGHT on
> > it's own fixes it?
> 
> MAX_INFLIGHT is sufficient.  In fact, $PIPE_BUFSIZ had no effect at all
> when I tried just now.

OK, good to know.  Thanks so far for your other testing and
reports.

> > Capping MAX_INFLIGHT to a smaller value is probably fine (maybe
> > 10 can work).
> 
> I tried with 10 just now, and that was fine also.

Oops, so POSIX::PIPE_BUF is actually 4096 on Linux.
(`perl -MPOSIX -E 'say POSIX::PIPE_BUF'` shows 4096 on the
Alpine cfarm machine)

But I'm still confused as to why the original code didn't
work on Linux since the pipe can't be smaller than 4096...

The following changes MAX_INFLIGHT to 23 for all platforms (and
explains things better).  I also wonder if you can find a
threshold for when things start failing (e.g. hardcoding 24
should fail on FreeBSD)

diff --git a/lib/PublicInbox/Git.pm b/lib/PublicInbox/Git.pm
index 882a9a4a..a1af776b 100644
--- a/lib/PublicInbox/Git.pm
+++ b/lib/PublicInbox/Git.pm
@@ -28,8 +28,10 @@ our $in_cleanup;
 our $RDTIMEO = 60_000; # milliseconds
 our $async_warn; # true in read-only daemons
 
-use constant MAX_INFLIGHT => (POSIX::PIPE_BUF * 3) /
-	65; # SHA-256 hex size + "\n" in preparation for git using non-SHA1
+# 512: POSIX PIPE_BUF minimum (see pipe(7))
+# 3: @$inflight is flattened [ $OID, $cb, $arg ]
+# 65: SHA-256 hex size + "\n" in preparation for git using non-SHA1
+use constant MAX_INFLIGHT => 512 * 3 / 65;
 
 my %GIT_ESC = (
 	a => "\a",

  reply	other threads:[~2022-12-21 21:11 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-12-21 11:28 public-inbox-convert hangs on systems using musl libc Chris Brannon
2022-12-21 12:21 ` Eric Wong
2022-12-21 13:46   ` Chris Brannon
2022-12-21 19:48     ` Eric Wong
2022-12-21 20:46       ` Chris Brannon
2022-12-21 21:11         ` Eric Wong [this message]
2022-12-21 22:17           ` Chris Brannon
2022-12-21 23:22             ` [PATCH] git: cap MAX_INFLIGHT value to POSIX minimum Eric Wong
2022-12-21 23:57               ` Chris Brannon
2023-01-04  3:49                 ` [PATCH] git: fix asynchronous batching for deep pipelines Eric Wong
2023-01-05  1:08                   ` Chris Brannon
2023-01-05  1:44                     ` [PATCH] git: write_all: remove leftover debug messages Eric Wong
2023-01-05  7:32                       ` Chris Brannon

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20221221211114.M412849@dcvr \
    --to=e@80x24.org \
    --cc=chris@the-brannons.com \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).