Re: public-inbox skipping new inboxes or many mails

[Eric Wong <e@80x24.org>]

"Robin H. Johnson" <robbat2@gentoo.org> wrote:
> On Tue, Jul 16, 2024 at 07:05:50PM +0000, Eric Wong wrote:
> > > It's definitely very busy after scan, but I can't tell if it's the full
> > > set.
> > OK, I think adding stderr diagnostic messages for full scans
> > shouldn't be too noisy.
> Thanks. I think it will be needed...

OK, will add...

> > > At an admin level, is there a way to dump out all of the paths it's
> > > indexes, to compare against the paths on disk?
> > No, path information isn't stored for public-facing inboxes
> > since it's too unstable.  It should be possible to reverse map
> > things at real-time and add better diagnostic tools, but the
> > philosophy has always been to store||index as little as possible
> > and be able to infer/regenerate needed data on-the-fly to avoid
> > data consistency problems.
> Can I easily dump out every message-id at least? I can compare that
> against the files, other than the old messages with no message-ids.

$ sqlite3 /path/to/msgmap.sqlite3 'SELECT mid FROM msgmap'

For v2, old messages without Message-IDs or recycled+conflicting
Message-IDs will have Message-IDs synthesized
(<YYYYmmddHHMMSS.$base64_digest@z>) as allowed by RFC 3977.

> > > > How many Maildirs are you watching?  I wonder if it's hitting
> > > > RLIMIT_NOFILE... (errors should be logged to stderr).
> > > 6774 Maildirs right now.
> > > I should probably improve the OpenRC script for it, I think we're
> > > throwing away stderr right now for -watch.
> > 
> > Yeah, watch stderr is important for diagnosing problems.
> > 
> > Fwiw, I run it inside a screen(1) session on one system,
> > and rely on systemd to redirect stderr to syslog on another
> I hacked in stderr: but bad luck, it doesn't dump anything useful before
> it seems to vanish. Nothing in dmesg either, so a mundane crash.

Not having anything in stderr on errors is really bad :x

Any fast_import_crash_* files in the [0-9]+\.git dirs?

-watch really shouldn't just vanish...  I'm not familiar with
OpenRC, does/can it wait on processes so it can report exit codes?

> > > > Curious, which architecture is that and is it using
> > > > Linux::Inotify2 or inotify via the `syscall' perlop?
> > > > (I expect 3 is the inotify FD).
> > > x86-64, dev-perl/Linux-Inotify2 is installed on the host, but I can't
> > > tell at a glance if -watch used perlop or package.
> > 
> > Probably, yes; but it can/should favor the pure Perl version
> > soon.  Since it's Gentoo I trust it's up-to-date with broadcast
> > and overflow support?
> dev-perl/Linux-Inotify2 will be up to date. No guarantees that the
> kernel is up to date - some legacy boxes at sponsors are pretty crufty
> and have been unsafe to reboot to new kernels when we lack any OOB
> management access: to that end, public-inbox's responsiveness is amazing
> even running on a 10+ year old RAID1 HDD spinner setup.

OK.  The kernel shouldn't be a problem for inotify, just the
older XS versions lacked some things and the pure Perl version
reduces mmap||vm.max_map_count pressure.  But I also noticed a
bug where we were favoring the XS :x.

Fwiw, I've actually struggled a lot with HDDs w/ Xapian||SQLite
but glad it's working out for you.  I'm mainly working ~15 year
old systems with SSDs that replaced dead HDDs.  Still have
numerous performance and memory optimizations planned :>