Re: notmuch and public-inbox

[Eric Wong <e@80x24.org>]

Carl Worth <cworth@cworth.org> wrote:
> On Sat, May 01 2021, Eric Wong wrote:
> > I never had the interest in using notmuch since Maildirs are a
> > non-starter with millions of messages with current FSes/OSes.
> 
> What bottleneck are you seeing here?
> 
> I don't have million(s) of messages but I'm getting close with 1.48M
> messages in my current notmuch index.
> 
> I'm not seeing any problematic performance from the filesystem or OS
> myself, so I'm curious what problem you're referring to here.

I assume you have several Maildirs and not just one with 1.48M?

Since I never actually used notmuch myself; most of my aversion
comes from years of using Maildir sync tools (mbsync,
offlineimap, rsync).  They all struggle with many inodes
and syscalls + cache required to walk them.

It's the same reason git puts old objects in packfiles rather
than having millions of loose objects.

Furthermore, my MUA (mutt) struggles on a single Maildir when
its size goes over ~50K.  Maildir is fine as a dumping ground
for mairix search results (typically a few dozen/hundred results).

Maildir is better nowadays on FSes with compression and
checksums; but lack of compression and checksumming were also
points against it; though syscalls are also more expensive with
CPU vulnerability mitigations.

I've always gzipped my archival mboxes for compression and CRC.

My local mirror of all the messages on lore.kernel.org/* is over
14.6M(*) and growing...  (LKML is 4M of that).


(*) 14.6M in the new combined "extindex" format that should be on
    lore.kernel.org, soon.  For now, I have an experimental
    instance on https://yhbt.net/lore/all/