From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS3701 140.211.0.0/16 X-Spam-Status: No, score=-3.5 required=3.0 tests=AWL,BAYES_00,RCVD_IN_DNSWL_HI, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 Received: from smtp.gentoo.org (woodpecker.gentoo.org [140.211.166.183]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id A7F031F572 for ; Tue, 16 Jul 2024 05:45:19 +0000 (UTC) Received: from grubbs.orbis-terrarum.net (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp.gentoo.org (Postfix) with ESMTPS id 957FB335D63 for ; Tue, 16 Jul 2024 05:45:18 +0000 (UTC) Received: from grubbs.orbis-terrarum.net (localhost [127.0.0.1]) by grubbs.orbis-terrarum.net (Postfix) with ESMTP id 8B913260656 for ; Tue, 16 Jul 2024 05:45:17 +0000 (UTC) Received: (qmail 568204 invoked by uid 10000); 16 Jul 2024 05:45:17 -0000 Date: Tue, 16 Jul 2024 05:45:17 +0000 From: "Robin H. Johnson" To: Eric Wong Cc: "Robin H. Johnson" , meta@public-inbox.org, infra@gentoo.org Subject: Re: public-inbox skipping new inboxes or many mails Message-ID: References: <20240715210340.M929931@dcvr> <20240715235808.M590962@dcvr> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="0RhYPBpcNtsj5R5H" Content-Disposition: inline In-Reply-To: <20240715235808.M590962@dcvr> List-Id: --0RhYPBpcNtsj5R5H Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jul 15, 2024 at 11:58:08PM +0000, Eric Wong wrote: > > On Mon, Jul 15, 2024 at 09:03:40PM +0000, Eric Wong wrote: > > > Currently, public-inbox-watch is the only public-inbox-* tool which > > > works directly with Maildirs. > > HMm, I had excluded public-inbox-watch initially because it didn't seem > > to be doing anything after the very long startup. I'm thinking that the > > inotify is not working as expected, maybe relating to the huge number of > > folders we watch. >=20 > -watch is (or should be) doing a full scan every startup, but it > switches between inboxes every few messages and tries to > prioritize new messages from inotify. Curious to see the > strace immediately after startup to see if it's indeed doing the > full scan. I should probably add a stderr diagnostic for full > scan completion... It's definitely very busy after scan, but I can't tell if it's the full set. At an admin level, is there a way to dump out all of the paths it's indexes, to compare against the paths on disk? > How many Maildirs are you watching? I wonder if it's hitting > RLIMIT_NOFILE... (errors should be logged to stderr). 6774 Maildirs right now. I should probably improve the OpenRC script for it, I think we're throwing away stderr right now for -watch. RLIMIT_NOFILE 1M sysctl fs.file-max 16M > > Terminal 1: > > # strace -p $(pidof /usr/bin/public-inbox-watch) -ff=20 > > strace: Process 93260 attached > > pselect6(8, [3 4], NULL, NULL, NULL, NULL > > (nothing more) > Curious, which architecture is that and is it using > Linux::Inotify2 or inotify via the `syscall' perlop? > (I expect 3 is the inotify FD). x86-64, dev-perl/Linux-Inotify2 is installed on the host, but I can't tell at a glance if -watch used perlop or package. fd 3 is inotify & fd 4 is signalfd. >=20 > You may also be running into fs.inotify.max_user_watches or > max_queued_events sysctl limits; each Maildir costs 2 inotify > watches; and it looks like I forgot to handle IN_Q_OVERFLOW > properly :x Already set quite high: fs.inotify.max_queued_events =3D 16384 fs.inotify.max_user_instances =3D 1024 fs.inotify.max_user_watches =3D 65536 --=20 Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation President & Treasurer E-Mail : robbat2@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 --0RhYPBpcNtsj5R5H Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 Comment: Robbat2 @ Orbis-Terrarum Networks - The text below is a digital signature. If it doesn't make any sense to you, ignore it. iQKTBAABCgB9FiEEveu2pS8Vb98xaNkRGTlfI8WIJsQFAmaWCOtfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEJE RUJCNkE1MkYxNTZGREYzMTY4RDkxMTE5Mzk1RjIzQzU4ODI2QzQACgkQGTlfI8WI JsQu2A//QkiyYYj8diD9xQDa/0gLJSPCVkuESpG903mINOrR2x7DEP5jFWm5ytYc wIUk0reLaTF3RnOuTiE0rWciH5zQtjpu0lu/f5I+vhqYTENcTLzPQ3gXbvkNJoHE sIngJLKkVS5wlmgfzSt+g7agjmfxvpd8AIJxtlPRJ9Fe5n4imrH135M9wSj58V0h ismuG1gBHtgPe8HP7aLQluobKnCOmfc3vlxVbH/kLQzvbBSJ6MZb+68/jKWhBeiB qYKWTZRDjEFX11sj/TuRVeF19+ol4Dl8Hczkce917IpGStahFK3fk+I6nrJNkP31 FTQWdBfqzaRjIEaz59P5sPkPfhJcYoTde+XuaWM9LCJW1spOd2AuqejblC2wpGut 3CHXMP4IhDbGPeO9JxWZa48p0yVaAYlzRAlSaRTyHR5NbDD6cHnGVUX/dG30+W3l jywDnnto0FtBZZ7XxhNW5kdqIVF41v0dSkDiZ0xUn3QEmkAYVPCxHrD10BCRMjB9 2TlcYOGOQefez7B6DEa8uRFUl60R16gxdMtkcuvprFXcE6A7deazW2iMdKwUpA4N rXH1TfhXGqJgMfg2rMAPRAlSptQNUrcRpkEaF1ujf5aWdVhubc2OyvqPYvXFGaNP wBsCHFbrglWWdpSgiOed7hT5+jMkH3+0IGDB0lSnLxsmGY2lE78= =2Gix -----END PGP SIGNATURE----- --0RhYPBpcNtsj5R5H--