unofficial mirror of meta@public-inbox.org
 help / color / mirror / Atom feed
From: Eric Wong <e@80x24.org>
To: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Cc: meta@public-inbox.org
Subject: Re: trying to figure out 100% CPU usage in nntpd...
Date: Mon, 9 Sep 2019 17:53:41 +0000	[thread overview]
Message-ID: <20190909175340.u5aq4ztfzukko7zb@dcvr> (raw)
In-Reply-To: <20190909100500.GA9452@pure.paranoia.local>

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> There also was a weird problem a couple of days ago where one of the
> httpd daemons started returning "Internal Server Error" to all requests.
> Restarting public-inbox-httpd fixed the problem, but I am not sure how I
> would troubleshoot the causes if it happens next time -- is there a way
> to enable error logging for the httpd daemon?

That's a new one... I haven't seen any problems from -httpd myself
in ages.  So -httpd could not handle requests at all?

The daemons already spits errors to stderr which typically ends up
in syslog via systemd.  So, that's the first place to look
(also "systemctl status $SERVICE"); anything in there?

I can usually figure everything out from strace/lsof on a worker
process and hitting it with some requests (SIGTTOU to decrement
workers down to one).

That said, out-of-FD/memory conditions might not always be
logged correctly to stderr and we need to fix that.  Also, right
now the code considers git-cat-file to be reliable, but I guess
it wouldn't be the case in disk failures and perhaps timeouts
will be necessary.

Maybe nginx/varnish logs would have something, too; but
more likely syslog.


Also, would be curious how memory usage improves for you with
some of the new changes.  I don't think I've exceeded 100MB/worker
this year, but Email::MIME can be a pig with giant messages.

  reply	other threads:[~2019-09-09 17:53 UTC|newest]

Thread overview: 21+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-09-08 10:45 [PATCH] nntp: regexp always consumes rbuf if "\n" exists Eric Wong
2019-09-08 10:52 ` trying to figure out 100% CPU usage in nntpd Eric Wong
2019-09-09 10:05   ` Konstantin Ryabitsev
2019-09-09 17:53     ` Eric Wong [this message]
2019-09-10  8:38       ` Konstantin Ryabitsev
2019-09-10 18:12         ` Eric Wong
2019-09-11  2:22           ` httpd 502s [was: trying to figure out 100% CPU usage in nntpd...] Eric Wong
2019-09-11 10:24             ` Konstantin Ryabitsev
2019-09-11 17:12               ` Eric Wong
2019-09-11 17:36                 ` Konstantin Ryabitsev
2019-09-12  0:05                   ` Eric Wong
2019-09-12  2:49                     ` Eric Wong
2019-09-12  8:35                       ` Eric Wong
2019-09-12 11:37                         ` Konstantin Ryabitsev
2019-09-13  3:12                           ` Eric Wong
2019-09-13  7:03                             ` Eric Wong
2019-09-13  9:01                             ` Eric Wong
2019-09-13 18:07                             ` Konstantin Ryabitsev
2019-09-14  5:25                               ` Eric Wong
2019-09-11  9:44           ` trying to figure out 100% CPU usage in nntpd Konstantin Ryabitsev
2019-09-11 17:12             ` Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190909175340.u5aq4ztfzukko7zb@dcvr \
    --to=e@80x24.org \
    --cc=konstantin@linuxfoundation.org \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).