Re: public-inbox skipping new inboxes or many mails

[Eric Wong <e@80x24.org>]

"Robin H. Johnson" <robbat2@gentoo.org> wrote:
> TL;DR: kill -USR1 seems to have triggered the import now, whereas even a
> restart didn't before.

OK, good to know.

> On Mon, Jul 15, 2024 at 09:03:40PM +0000, Eric Wong wrote:
> > Currently, public-inbox-watch is the only public-inbox-* tool which
> > works directly with Maildirs.
> HMm, I had excluded public-inbox-watch initially because it didn't seem
> to be doing anything after the very long startup. I'm thinking that the
> inotify is not working as expected, maybe relating to the huge number of
> folders we watch.

-watch is (or should be) doing a full scan every startup, but it
switches between inboxes every few messages and tries to
prioritize new messages from inotify.   Curious to see the
strace immediately after startup to see if it's indeed doing the
full scan.  I should probably add a stderr diagnostic for full
scan completion...

How many Maildirs are you watching?  I wonder if it's hitting
RLIMIT_NOFILE... (errors should be logged to stderr).

In retrospect, it'd probably be better to go one Maildir
at a time to reduce open FDs...  I'll work on that.

> Terminal 1:
> # strace -p $(pidof /usr/bin/public-inbox-watch)  -ff 
> strace: Process 93260 attached
> pselect6(8, [3 4], NULL, NULL, NULL, NULL
> (nothing more)

Curious, which architecture is that and is it using
Linux::Inotify2 or inotify via the `syscall' perlop?
(I expect 3 is the inotify FD).

You may also be running into fs.inotify.max_user_watches or
max_queued_events sysctl limits; each Maildir costs 2 inotify
watches; and it looks like I forgot to handle IN_Q_OVERFLOW
properly :x