From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.1 required=3.0 tests=ALL_TRUSTED,AWL,BAYES_00, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id F28DF1F852; Wed, 21 Dec 2022 23:22:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=80x24.org; s=selector1; t=1671664931; bh=jqax/0mGmn/ClWJwm+TG3PcPG4CjIgXpBxBm5mEzNcg=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=kGb/Oj59xGLrzj3xeo04xejPBByxy8jevUKW0cSV0XXM2fqwQLDlB1MlYQi0k3142 aNq57cXjlTT+dYF1JaOZTrid9jSNvl5i601No3tPneAKZ9s2N/hnCrRqIuaqAizAdb oa08AQCwScTIausDQeLXfvyZvOZ0dWDljNC4WV5s= Date: Wed, 21 Dec 2022 23:22:10 +0000 From: Eric Wong To: Chris Brannon Cc: meta@public-inbox.org Subject: [PATCH] git: cap MAX_INFLIGHT value to POSIX minimum Message-ID: <20221221232210.M865188@dcvr> References: <875ye5m1wo.fsf@the-brannons.com> <20221221122102.M600156@dcvr> <871qosna30.fsf@the-brannons.com> <20221221194855.GA5179@dcvr> <87mt7glc23.fsf@the-brannons.com> <20221221211114.M412849@dcvr> <87edssl7u0.fsf@the-brannons.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <87edssl7u0.fsf@the-brannons.com> List-Id: Chris Brannon wrote: > Eric Wong writes: > > > The following changes MAX_INFLIGHT to 23 for all platforms (and > > explains things better). I also wonder if you can find a > > threshold for when things start failing (e.g. hardcoding 24 > > should fail on FreeBSD) > > I did a binary search. On my Linux system with musl, deadlocks start at > MAX_INFLIGHT >= 115. Thanks. I'll push the following out and intend to keep it at <= 23 going forward. However, it still worries me greatly that I have no explanation to why MAX_INFLIGHT >= 115 fails. This stuff is my bread and butter, really... I'll wait on a cfarm admins to test. Also, can I assume just running 'public-inbox-index' on any freshly-cloned inbox also fails for you independently of -convert? ------------8<---------- Subject: [PATCH] git: cap MAX_INFLIGHT value to POSIX minimum This ensures we get consistent pipelining behavior across platforms. Furthermore, a smaller value is probably more reasonable since "git cat-file" can usually outpace indexing and lower values allow us to react to user interaction (e.g. Ctrl-C) more quickly. The previous value based on Linux PIPE_BUF (4096) allowed a value of 189 which worked fine on non-musl Linux systems, but failed on musl-based Void and Alpine Linux. Mysteriously, this works on musl up to a value of 114 and starts locking up at 115. The reason for this failure is currently unexplained and will hopefully be discovered soon. Regardless, capping the value to 23 based on the universal PIPE_BUF minimum (512) seems reasonable, anyways. Reported-by: Chris Brannon Tested-by: Chris Brannon Link: https://public-inbox.org/meta/87edssl7u0.fsf@the-brannons.com/T/ --- lib/PublicInbox/Git.pm | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/lib/PublicInbox/Git.pm b/lib/PublicInbox/Git.pm index 882a9a4a..a1af776b 100644 --- a/lib/PublicInbox/Git.pm +++ b/lib/PublicInbox/Git.pm @@ -28,8 +28,10 @@ our $in_cleanup; our $RDTIMEO = 60_000; # milliseconds our $async_warn; # true in read-only daemons -use constant MAX_INFLIGHT => (POSIX::PIPE_BUF * 3) / - 65; # SHA-256 hex size + "\n" in preparation for git using non-SHA1 +# 512: POSIX PIPE_BUF minimum (see pipe(7)) +# 3: @$inflight is flattened [ $OID, $cb, $arg ] +# 65: SHA-256 hex size + "\n" in preparation for git using non-SHA1 +use constant MAX_INFLIGHT => 512 * 3 / 65; my %GIT_ESC = ( a => "\a",