From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS3701 140.211.0.0/16 X-Spam-Status: No, score=-3.4 required=3.0 tests=AWL,BAYES_00, RCVD_IN_DNSWL_MED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS, SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 Received: from smtp.gentoo.org (woodpecker.gentoo.org [140.211.166.183]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 2EF5F1F572 for ; Wed, 17 Jul 2024 03:04:36 +0000 (UTC) Received: from grubbs.orbis-terrarum.net (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp.gentoo.org (Postfix) with ESMTPS id 071E034072E for ; Wed, 17 Jul 2024 03:04:35 +0000 (UTC) Received: from grubbs.orbis-terrarum.net (localhost [127.0.0.1]) by grubbs.orbis-terrarum.net (Postfix) with ESMTP id BA5A3260657 for ; Wed, 17 Jul 2024 03:04:33 +0000 (UTC) Received: (qmail 741498 invoked by uid 10000); 17 Jul 2024 03:04:33 -0000 Date: Wed, 17 Jul 2024 03:04:33 +0000 From: "Robin H. Johnson" To: Eric Wong Cc: "Robin H. Johnson" , meta@public-inbox.org, infra@gentoo.org Subject: Re: public-inbox skipping new inboxes or many mails Message-ID: References: <20240715210340.M929931@dcvr> <20240715235808.M590962@dcvr> <20240716190550.M242334@dcvr> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="3Q3ggfV8kF7By0hW" Content-Disposition: inline In-Reply-To: <20240716190550.M242334@dcvr> List-Id: --3Q3ggfV8kF7By0hW Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jul 16, 2024 at 07:05:50PM +0000, Eric Wong wrote: > > It's definitely very busy after scan, but I can't tell if it's the full > > set. > OK, I think adding stderr diagnostic messages for full scans > shouldn't be too noisy. Thanks. I think it will be needed... > > At an admin level, is there a way to dump out all of the paths it's > > indexes, to compare against the paths on disk? > No, path information isn't stored for public-facing inboxes > since it's too unstable. It should be possible to reverse map > things at real-time and add better diagnostic tools, but the > philosophy has always been to store||index as little as possible > and be able to infer/regenerate needed data on-the-fly to avoid > data consistency problems. Can I easily dump out every message-id at least? I can compare that against the files, other than the old messages with no message-ids. > > > How many Maildirs are you watching? I wonder if it's hitting > > > RLIMIT_NOFILE... (errors should be logged to stderr). > > 6774 Maildirs right now. > > I should probably improve the OpenRC script for it, I think we're > > throwing away stderr right now for -watch. >=20 > Yeah, watch stderr is important for diagnosing problems. >=20 > Fwiw, I run it inside a screen(1) session on one system, > and rely on systemd to redirect stderr to syslog on another I hacked in stderr: but bad luck, it doesn't dump anything useful before it seems to vanish. Nothing in dmesg either, so a mundane crash. > > > Curious, which architecture is that and is it using > > > Linux::Inotify2 or inotify via the `syscall' perlop? > > > (I expect 3 is the inotify FD). > > x86-64, dev-perl/Linux-Inotify2 is installed on the host, but I can't > > tell at a glance if -watch used perlop or package. >=20 > Probably, yes; but it can/should favor the pure Perl version > soon. Since it's Gentoo I trust it's up-to-date with broadcast > and overflow support? dev-perl/Linux-Inotify2 will be up to date. No guarantees that the kernel is up to date - some legacy boxes at sponsors are pretty crufty and have been unsafe to reboot to new kernels when we lack any OOB management access: to that end, public-inbox's responsiveness is amazing even running on a 10+ year old RAID1 HDD spinner setup. --=20 Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation President & Treasurer E-Mail : robbat2@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 --3Q3ggfV8kF7By0hW Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 Comment: Robbat2 @ Orbis-Terrarum Networks - The text below is a digital signature. If it doesn't make any sense to you, ignore it. iQKTBAABCgB9FiEEveu2pS8Vb98xaNkRGTlfI8WIJsQFAmaXNL9fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEJE RUJCNkE1MkYxNTZGREYzMTY4RDkxMTE5Mzk1RjIzQzU4ODI2QzQACgkQGTlfI8WI JsQuWhAAgbDV/cYM+7yl+QqztYht5POkM3lS7Ng2TCMVYw+ifv4YQ1E8inNPBC5z +cNTKnlpqxNee/lY1CwvX25REi3ESufsappUe1GW55Qb5UlkPX9R8a5EckdVPq+y r08dAyCpfMJVdlGxQprCF73Ok6+GLby6Ph7pM2ettJ+dVgf9u44iPfWx/aGxhPWK WvhGm2SM25mrSI9CzVQs8zpstIyb0yAUYyNUXfN9TA7DQ+EHfGfh1RYTOGltzzsx GFeyQH+3+23CZxnRNFsa7PKNdY6VI1MnBTnE3qVYEthueym96Q+IV8TmL0b9D7uT yRf6R58TfWW9qObYQvoAhXKihjh+/dVV29f5Y2d3OV2MKVZdvXEMKL/eyO52ChlW idI/ICq4AoQPa2G8S4BbkrfAEBZ9matJ7q2nzwcXbYCldWAojT9nakEL8mcM0EAo r52HckDc2GrOVeY9U5diu3sMwS7fDyNwQ8adEmvFUYSumPeC0fGQtqMmBjO3WfRM n9haapiLnfJa/nc66o/AZuZeFG+UzvhgmleLksfnhcZ7dcONdbofEW3KzoSI7Vsn kYRQ64qQQobvhsPpLoRQ8aLRyNajJq8BvVIdcuxyjCj6Ty3+XEWnHJaPfwsakkU4 Xdmyc1BETNKI7uxbz2FENOuumQQ5Q0TdWWFichMtKo5Toej2l0Q= =IzFR -----END PGP SIGNATURE----- --3Q3ggfV8kF7By0hW--