Re: trying to figure out 100% CPU usage in nntpd...

[Eric Wong <e@80x24.org>]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> There also was a weird problem a couple of days ago where one of the
> httpd daemons started returning "Internal Server Error" to all requests.
> Restarting public-inbox-httpd fixed the problem, but I am not sure how I
> would troubleshoot the causes if it happens next time -- is there a way
> to enable error logging for the httpd daemon?

That's a new one... I haven't seen any problems from -httpd myself
in ages.  So -httpd could not handle requests at all?

The daemons already spits errors to stderr which typically ends up
in syslog via systemd.  So, that's the first place to look
(also "systemctl status $SERVICE"); anything in there?

I can usually figure everything out from strace/lsof on a worker
process and hitting it with some requests (SIGTTOU to decrement
workers down to one).

That said, out-of-FD/memory conditions might not always be
logged correctly to stderr and we need to fix that.  Also, right
now the code considers git-cat-file to be reliable, but I guess
it wouldn't be the case in disk failures and perhaps timeouts
will be necessary.

Maybe nginx/varnish logs would have something, too; but
more likely syslog.


Also, would be curious how memory usage improves for you with
some of the new changes.  I don't think I've exceeded 100MB/worker
this year, but Email::MIME can be a pig with giant messages.