Re: Occasional public-inbox-httpd flakiness

[Jonathan Corbet <corbet@lwn.net>]

Eric Wong <e@80x24.org> writes:

> Jonathan Corbet <corbet@lwn.net> wrote:
>> The LWN archive server is running Debian's 1.9.0 public-inbox package.
>> Every now and then, usually after at least a week of operation, HTTP
>> requests will start returning empty messages; I find stuff like this in
>> the log:
>> 
>> Nov 05 15:24:37 archive2.lwn.net public-inbox-httpd[1267166]: Use of uninitialized value in index at /usr/share/perl5/PublicInbox/Eml.pm line 109.
>> Nov 05 15:24:37 archive2.lwn.net public-inbox-httpd[1267166]: Use of uninitialized value in index at /usr/share/perl5/PublicInbox/Eml.pm line 109.
>> Nov 05 15:24:37 archive2.lwn.net public-inbox-httpd[1267166]: Use of uninitialized value in pattern match (m//) at /usr/share/perl5/PublicInbox/Eml.pm line 115.
>> Nov 05 15:24:37 archive2.lwn.net public-inbox-httpd[1267166]: Use of uninitialized value in pattern match (m//) at /usr/share/perl5/PublicInbox/Eml.pm line 120.
>> Nov 05 15:24:37 archive2.lwn.net public-inbox-httpd[1267166]: Use of uninitialized value in pattern match (m//) at /usr/share/perl5/PublicInbox/Eml.pm line 127.
>> Nov 05 15:24:37 archive2.lwn.net public-inbox-httpd[1267166]: Use of uninitialized value in substitution (s///) at /usr/share/perl5/PublicInbox/Mbox.pm line 115.
>
> Definitely something that's popped up in my recollection; but
> hasn't happened in a while for Eml.pm and Mbox.pm (yeah, it's
> been a while since v1.9 :x).
>
> Are the git cat-file (or Gcf2) processes still running?  Are any
> successful responses returned for requests to mail messages?

As for the first, I'll have to look the next time it happens - I know of
no way to force that, so it's a matter of waiting.

There are still definitely successful responses; my guess has always
been that one of the public-inbox-httpd processes has gone weird while
the other still work.

>> The pattern is pretty much always the same.  Restarting
>> public-inbox-httpd makes the problem go away again.
>
> Usually, uninitialized value errors are isolated to a single
> request (e.g. broken emails) and there shouldn't be a
> need to restart unless every request is failing.

It's not the email that is the issue - a specific URL that fails before
the restart will work afterward.

>> Is this a problem that anybody else has seen, or am I especially
>> lucky...?
>
> I've had some similar problems from inboxes/coderepos getting
> removed; also there's OOMs on my HW causing git processes to fail.

The system as a whole is far from any sort of OOM state; that was one of
the first things I looked for.

Thanks,

jon