Re: public-inbox skipping new inboxes or many mails

["Robin H. Johnson" <robbat2@gentoo.org>]

[-- Attachment #1: Type: text/plain, Size: 2538 bytes --]

On Tue, Jul 16, 2024 at 07:05:50PM +0000, Eric Wong wrote:
> > It's definitely very busy after scan, but I can't tell if it's the full
> > set.
> OK, I think adding stderr diagnostic messages for full scans
> shouldn't be too noisy.
Thanks. I think it will be needed...

> > At an admin level, is there a way to dump out all of the paths it's
> > indexes, to compare against the paths on disk?
> No, path information isn't stored for public-facing inboxes
> since it's too unstable.  It should be possible to reverse map
> things at real-time and add better diagnostic tools, but the
> philosophy has always been to store||index as little as possible
> and be able to infer/regenerate needed data on-the-fly to avoid
> data consistency problems.
Can I easily dump out every message-id at least? I can compare that
against the files, other than the old messages with no message-ids.

> > > How many Maildirs are you watching?  I wonder if it's hitting
> > > RLIMIT_NOFILE... (errors should be logged to stderr).
> > 6774 Maildirs right now.
> > I should probably improve the OpenRC script for it, I think we're
> > throwing away stderr right now for -watch.
> 
> Yeah, watch stderr is important for diagnosing problems.
> 
> Fwiw, I run it inside a screen(1) session on one system,
> and rely on systemd to redirect stderr to syslog on another
I hacked in stderr: but bad luck, it doesn't dump anything useful before
it seems to vanish. Nothing in dmesg either, so a mundane crash.

> > > Curious, which architecture is that and is it using
> > > Linux::Inotify2 or inotify via the `syscall' perlop?
> > > (I expect 3 is the inotify FD).
> > x86-64, dev-perl/Linux-Inotify2 is installed on the host, but I can't
> > tell at a glance if -watch used perlop or package.
> 
> Probably, yes; but it can/should favor the pure Perl version
> soon.  Since it's Gentoo I trust it's up-to-date with broadcast
> and overflow support?
dev-perl/Linux-Inotify2 will be up to date. No guarantees that the
kernel is up to date - some legacy boxes at sponsors are pretty crufty
and have been unsafe to reboot to new kernels when we lack any OOB
management access: to that end, public-inbox's responsiveness is amazing
even running on a 10+ year old RAID1 HDD spinner setup.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation President & Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1113 bytes --]