Re: trying to figure out 100% CPU usage in nntpd...

[Konstantin Ryabitsev <konstantin@linuxfoundation.org>]

On Tue, Sep 10, 2019 at 06:12:24PM +0000, Eric Wong wrote:
> > It does seem like there's perhaps a leak somewhere?
> 
> Probably.  Not seeing any of that on my (smaller) instances;
> but those -httpd haven't been restarted in weeks/months.
> 
> The "PerlIO_" prefix is created from open(..., '+>', undef), so
> it either has to be for:
> 
> 1. POST bodies for git http-backend
> 
> 1. git cat-file --batch-check stderr
> 
> 3. experimental ViewVCS|SolverGit which isn't configured on lore :)
> 
> = git-http-backend
> 
> Looking at the size (984), PerlIO_* prefix and proximity of FD
> numbers, I think those lines above are git-http-backend POST body.

Pretty sure that's the culprit. This is how we replicate between
lore.kernel.org to erol.kernel.org:

- once a minute, two nodes that are behind erol.kernel.org grab the
  newest manifest.js.gz
- if there are changes, each updated repository is pulled from
  lore.kernel.org, so if there were 5 repository updates, there would be
  10 "git pull" requests

I switched the replication nodes to pull once every 5 minutes instead of
once every minute and I see a direct correlation between when those
processes run and the number of broken pipes and "/tmp/PerlIO_*
(deleted)" processes showing up and hanging around. Not every run
produces these, but increase spikes come in roughly 5-minute intervals.
On the first run after public-inbox-httpd restart, the correlation is
direct:

this is from one of the mirroring nodes:

[82807] 2019-09-11 09:30:02,044 - INFO - Updating 18 repos from https://lore.kernel.org

this is on lore.kernel.org after the run is completed:

# ls -al /proc/{16212,16213,16214,16215}/fd | grep deleted | wc -l
36

> Any git-http-backend stuck from people fetching/cloning?

No, all git processes seem to exit cleanly on both ends.

> This is -httpd writing to varnish, still, right?

We bypass varnish for git requests, since this is not generally useful.
Nginx goes straight to public-inbox-httpd for those.

I did run some updates on lore.kernel.org on Thursday, including kernel
(3.10.0-957.27.2), nginx (1.16.1) and public-inbox updates. For the
latter, it went from f4f0a3be to what was latest master at the time
(d327141c).

Hope this helps, and thanks for your help!

-K