From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS3701 140.211.0.0/16 X-Spam-Status: No, score=-3.4 required=3.0 tests=AWL,BAYES_00, RCVD_IN_DNSWL_MED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS, SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 Received: from smtp.gentoo.org (woodpecker.gentoo.org [140.211.166.183]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id B6E571F572 for ; Thu, 18 Jul 2024 00:02:58 +0000 (UTC) Received: from grubbs.orbis-terrarum.net (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp.gentoo.org (Postfix) with ESMTPS id B45B233BED4 for ; Thu, 18 Jul 2024 00:02:57 +0000 (UTC) Received: from grubbs.orbis-terrarum.net (localhost [127.0.0.1]) by grubbs.orbis-terrarum.net (Postfix) with ESMTP id A346F260182 for ; Thu, 18 Jul 2024 00:02:56 +0000 (UTC) Received: (qmail 952320 invoked by uid 10000); 18 Jul 2024 00:02:56 -0000 Date: Thu, 18 Jul 2024 00:02:56 +0000 From: "Robin H. Johnson" To: Eric Wong Cc: "Robin H. Johnson" , meta@public-inbox.org, infra@gentoo.org Subject: Re: public-inbox skipping new inboxes or many mails Message-ID: References: <20240715210340.M929931@dcvr> <20240715235808.M590962@dcvr> <20240716190550.M242334@dcvr> <20240717232532.M125694@dcvr> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="Li+QOFDRBZsnzgoA" Content-Disposition: inline In-Reply-To: <20240717232532.M125694@dcvr> List-Id: --Li+QOFDRBZsnzgoA Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Jul 17, 2024 at 11:25:32PM +0000, Eric Wong wrote: > > Can I easily dump out every message-id at least? I can compare that > > against the files, other than the old messages with no message-ids. >=20 > $ sqlite3 /path/to/msgmap.sqlite3 'SELECT mid FROM msgmap' >=20 > For v2, old messages without Message-IDs or recycled+conflicting > Message-IDs will have Message-IDs synthesized > () as allowed by RFC 3977. Thanks. > > I hacked in stderr: but bad luck, it doesn't dump anything useful before > > it seems to vanish. Nothing in dmesg either, so a mundane crash. > Not having anything in stderr on errors is really bad :x >=20 > Any fast_import_crash_* files in the [0-9]+\.git dirs? No crash files either. > -watch really shouldn't just vanish... I'm not familiar with > OpenRC, does/can it wait on processes so it can report exit codes? Not by default. > OK. The kernel shouldn't be a problem for inotify, just the > older XS versions lacked some things and the pure Perl version > reduces mmap||vm.max_map_count pressure. But I also noticed a > bug where we were favoring the XS :x. >=20 > Fwiw, I've actually struggled a lot with HDDs w/ Xapian||SQLite > but glad it's working out for you. I'm mainly working ~15 year > old systems with SSDs that replaced dead HDDs. Still have > numerous performance and memory optimizations planned :> I came up with a good hack for now: I split the config file by list, and I'm running 116 instances of public-inbox-watch, with different config files (and httpd has the giant config file). Taking a listname as an arg would have been cleaner, but this is working for now. It was also finally able to hit the IO limits of the HDDs by doing this, so there's a lot of low-hanging optimization fruit clearly. --=20 Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation President & Treasurer E-Mail : robbat2@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 --Li+QOFDRBZsnzgoA Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 Comment: Robbat2 @ Orbis-Terrarum Networks - The text below is a digital signature. If it doesn't make any sense to you, ignore it. iQKTBAABCgB9FiEEveu2pS8Vb98xaNkRGTlfI8WIJsQFAmaYW65fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEJE RUJCNkE1MkYxNTZGREYzMTY4RDkxMTE5Mzk1RjIzQzU4ODI2QzQACgkQGTlfI8WI JsSA6A/8CZcfZC1OEGRdvE2wDWqC1A3CXY/de0w2wLn0stbRRY+66BY0edFx5fuQ buN2T67SJAgy6QIOH7LIDPrjpxUOdMpXfUTUBFbbTckuUmYPwsPZD3vGBbb6dGAI qEL5sN1YIK/1ZqdMEWZOISfyUyYFfuVtocA1flih43cRk0e+i4hs43OQ+tXxUd/E 0ys1B11CLsfQ3660Odi1ZiMvfKe9qjkmgAu+VZhh264urdwaaDTZVgUYWxhrchz/ Y8wI/E+jGm6zDdkY9ZvhAq67rru9rL2P4IngX+0TGFhvsu2bjIsKmIWsKb9Se/Pn qBb9dAhMpJEt5vGSMzU8F/9kJXPJxS6keNdhbYDA/+uACLPiTKA7AmVEcR0RvJWo jk8oo35GG3d88SmVewaFs9V4RPlkKWBFx+XrcmatcosDI0Q64YykD+f0SuQel0sk 4L0w/xk+zpQQtvPamzye+gDW2q9CUlzI1cz9+PNYHtim/9zGTWJYA0KWeb80hedO 1/Kwg+TSBdlR5GJmdTzLHrbr8MyPhdzgWMhZLpWAsE73z0vTzrEYR0ZXdPNF77M8 yUOPedv6dD7BwyugOM6gin3tge8S7h6RUdeX5/8VnQ0bRwsCeiKhC3wcmmSDGcGu 79sjjNRSYmEsWG59nwE3Vr8DCC7Kup+hywJbRlRlvg6WHybQ06c= =wZPL -----END PGP SIGNATURE----- --Li+QOFDRBZsnzgoA--