From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.2 required=3.0 tests=ALL_TRUSTED,AWL,BAYES_00, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 56EB41F572; Wed, 17 Jul 2024 23:25:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=80x24.org; s=selector1; t=1721258732; bh=LaCDvR9bWLKgHg0IricLStklwXuE/8pXYtmztRTsOmU=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=y3mt9iTOZ0dvFopI2AXPkeY98DApHGee/VlTSNCBEea/Z5nXT90h74mhNCagv4aky 6eAvXrv1D2ykZm2XnkXX9eyHGOSUSk7y1iaMl2iT+pDxgXFGjGGFYFV1eWRlnTGa+e YVCc0lkXbEjriCnFGR4hG4EcOLvtsiMU9t5QKtuw= Date: Wed, 17 Jul 2024 23:25:32 +0000 From: Eric Wong To: "Robin H. Johnson" Cc: meta@public-inbox.org, infra@gentoo.org Subject: Re: public-inbox skipping new inboxes or many mails Message-ID: <20240717232532.M125694@dcvr> References: <20240715210340.M929931@dcvr> <20240715235808.M590962@dcvr> <20240716190550.M242334@dcvr> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: List-Id: "Robin H. Johnson" wrote: > On Tue, Jul 16, 2024 at 07:05:50PM +0000, Eric Wong wrote: > > > It's definitely very busy after scan, but I can't tell if it's the full > > > set. > > OK, I think adding stderr diagnostic messages for full scans > > shouldn't be too noisy. > Thanks. I think it will be needed... OK, will add... > > > At an admin level, is there a way to dump out all of the paths it's > > > indexes, to compare against the paths on disk? > > No, path information isn't stored for public-facing inboxes > > since it's too unstable. It should be possible to reverse map > > things at real-time and add better diagnostic tools, but the > > philosophy has always been to store||index as little as possible > > and be able to infer/regenerate needed data on-the-fly to avoid > > data consistency problems. > Can I easily dump out every message-id at least? I can compare that > against the files, other than the old messages with no message-ids. $ sqlite3 /path/to/msgmap.sqlite3 'SELECT mid FROM msgmap' For v2, old messages without Message-IDs or recycled+conflicting Message-IDs will have Message-IDs synthesized () as allowed by RFC 3977. > > > > How many Maildirs are you watching? I wonder if it's hitting > > > > RLIMIT_NOFILE... (errors should be logged to stderr). > > > 6774 Maildirs right now. > > > I should probably improve the OpenRC script for it, I think we're > > > throwing away stderr right now for -watch. > > > > Yeah, watch stderr is important for diagnosing problems. > > > > Fwiw, I run it inside a screen(1) session on one system, > > and rely on systemd to redirect stderr to syslog on another > I hacked in stderr: but bad luck, it doesn't dump anything useful before > it seems to vanish. Nothing in dmesg either, so a mundane crash. Not having anything in stderr on errors is really bad :x Any fast_import_crash_* files in the [0-9]+\.git dirs? -watch really shouldn't just vanish... I'm not familiar with OpenRC, does/can it wait on processes so it can report exit codes? > > > > Curious, which architecture is that and is it using > > > > Linux::Inotify2 or inotify via the `syscall' perlop? > > > > (I expect 3 is the inotify FD). > > > x86-64, dev-perl/Linux-Inotify2 is installed on the host, but I can't > > > tell at a glance if -watch used perlop or package. > > > > Probably, yes; but it can/should favor the pure Perl version > > soon. Since it's Gentoo I trust it's up-to-date with broadcast > > and overflow support? > dev-perl/Linux-Inotify2 will be up to date. No guarantees that the > kernel is up to date - some legacy boxes at sponsors are pretty crufty > and have been unsafe to reboot to new kernels when we lack any OOB > management access: to that end, public-inbox's responsiveness is amazing > even running on a 10+ year old RAID1 HDD spinner setup. OK. The kernel shouldn't be a problem for inotify, just the older XS versions lacked some things and the pure Perl version reduces mmap||vm.max_map_count pressure. But I also noticed a bug where we were favoring the XS :x. Fwiw, I've actually struggled a lot with HDDs w/ Xapian||SQLite but glad it's working out for you. I'm mainly working ~15 year old systems with SSDs that replaced dead HDDs. Still have numerous performance and memory optimizations planned :>