From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS63949 45.79.64.0/19 X-Spam-Status: No, score=-3.3 required=3.0 tests=AWL,BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 Received: from ms.lwn.net (ms.lwn.net [45.79.88.28]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 4E5D61F4C8 for ; Tue, 5 Nov 2024 23:29:26 +0000 (UTC) Authentication-Results: dcvr.yhbt.net; dkim=pass (2048-bit key; unprotected) header.d=lwn.net header.i=@lwn.net header.a=rsa-sha256 header.s=20201203 header.b=IftS/hHc; dkim-atps=neutral DKIM-Filter: OpenDKIM Filter v2.11.0 ms.lwn.net F012F42C0D DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lwn.net; s=20201203; t=1730849365; bh=a9TVHFJqIjQY5s/qfuTMKPJ1oAwCi1bkjklFrj2ZNuk=; h=From:To:Cc:Subject:In-Reply-To:References:Date:From; b=IftS/hHcZbxQkr1GUnKJMjgZV+ZpERRxP/W6kXo7d/VmEO1tP5pm+n6UPZW2Q/NIH 1aOp/QcBT6osGZonsgzo+mTJtJlAImxVw6wiksVJN4ne3cHRJQ9GMm0l1iNw9DavVB 9tKM1vURiRJa5rLo9t/aUYrf3dtPEPV2ng9mu5Rd0g6rQVzqqfcVn8El0xAY2Lopow +8J5x2UpOAxkhtROuvC5Guhlze003qKjog1R/+fZuw93KEFlcGoxZFx22ZigvgFokV jXCkYyJH6HlI8eNBQ6uvjWTjh0l4V4l6+djDeeDDgrWw9BqTOl0PRWQthg6zL3xo/X LvlEl8BOrErJg== Received: from localhost (unknown [IPv6:2601:280:5e00:625::1fe]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ms.lwn.net (Postfix) with ESMTPSA id F012F42C0D; Tue, 5 Nov 2024 23:29:24 +0000 (UTC) From: Jonathan Corbet To: Eric Wong Cc: meta@public-inbox.org Subject: Re: Occasional public-inbox-httpd flakiness In-Reply-To: <20241105232445.M291444@dcvr> References: <875xp15n3o.fsf@trenco.lwn.net> <20241105232445.M291444@dcvr> Date: Tue, 05 Nov 2024 16:29:24 -0700 Message-ID: <87wmhh45uz.fsf@trenco.lwn.net> MIME-Version: 1.0 Content-Type: text/plain List-Id: Eric Wong writes: > Jonathan Corbet wrote: >> The LWN archive server is running Debian's 1.9.0 public-inbox package. >> Every now and then, usually after at least a week of operation, HTTP >> requests will start returning empty messages; I find stuff like this in >> the log: >> >> Nov 05 15:24:37 archive2.lwn.net public-inbox-httpd[1267166]: Use of uninitialized value in index at /usr/share/perl5/PublicInbox/Eml.pm line 109. >> Nov 05 15:24:37 archive2.lwn.net public-inbox-httpd[1267166]: Use of uninitialized value in index at /usr/share/perl5/PublicInbox/Eml.pm line 109. >> Nov 05 15:24:37 archive2.lwn.net public-inbox-httpd[1267166]: Use of uninitialized value in pattern match (m//) at /usr/share/perl5/PublicInbox/Eml.pm line 115. >> Nov 05 15:24:37 archive2.lwn.net public-inbox-httpd[1267166]: Use of uninitialized value in pattern match (m//) at /usr/share/perl5/PublicInbox/Eml.pm line 120. >> Nov 05 15:24:37 archive2.lwn.net public-inbox-httpd[1267166]: Use of uninitialized value in pattern match (m//) at /usr/share/perl5/PublicInbox/Eml.pm line 127. >> Nov 05 15:24:37 archive2.lwn.net public-inbox-httpd[1267166]: Use of uninitialized value in substitution (s///) at /usr/share/perl5/PublicInbox/Mbox.pm line 115. > > Definitely something that's popped up in my recollection; but > hasn't happened in a while for Eml.pm and Mbox.pm (yeah, it's > been a while since v1.9 :x). > > Are the git cat-file (or Gcf2) processes still running? Are any > successful responses returned for requests to mail messages? As for the first, I'll have to look the next time it happens - I know of no way to force that, so it's a matter of waiting. There are still definitely successful responses; my guess has always been that one of the public-inbox-httpd processes has gone weird while the other still work. >> The pattern is pretty much always the same. Restarting >> public-inbox-httpd makes the problem go away again. > > Usually, uninitialized value errors are isolated to a single > request (e.g. broken emails) and there shouldn't be a > need to restart unless every request is failing. It's not the email that is the issue - a specific URL that fails before the restart will work afterward. >> Is this a problem that anybody else has seen, or am I especially >> lucky...? > > I've had some similar problems from inboxes/coderepos getting > removed; also there's OOMs on my HW causing git processes to fail. The system as a whole is far from any sort of OOM state; that was one of the first things I looked for. Thanks, jon