From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:bcc0::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id 0EA0FenfjGA7ngAAgWs5BA (envelope-from ) for ; Sat, 01 May 2021 06:58:17 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id 6qTxEOnfjGBTEwAA1q6Kng (envelope-from ) for ; Sat, 01 May 2021 04:58:17 +0000 Received: from mail.notmuchmail.org (nmbug.tethera.net [144.217.243.247]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id D544BBC3E for ; Sat, 1 May 2021 06:58:15 +0200 (CEST) Received: from nmbug.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id 687B52716D; Sat, 1 May 2021 00:58:11 -0400 (EDT) Received: from dcvr.yhbt.net (dcvr.yhbt.net [64.71.152.64]) by mail.notmuchmail.org (Postfix) with ESMTPS id CD29F1FBD2 for ; Sat, 1 May 2021 00:58:08 -0400 (EDT) Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 3B2C91F5AE; Sat, 1 May 2021 04:58:07 +0000 (UTC) Date: Sat, 1 May 2021 04:58:07 +0000 From: Eric Wong To: Carl Worth Subject: Re: notmuch and public-inbox Message-ID: <20210501045807.GA6409@dcvr> References: <608c9185115fd_251d208e0@natae.notmuch> <20210501000500.GA20219@dcvr> <87lf8zs2by.fsf@wobbit.home.cworth.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <87lf8zs2by.fsf@wobbit.home.cworth.org> Message-ID-Hash: FLCDZXBNQW6M5QDAA2NKAJ4LQBYZ2YYR X-Message-ID-Hash: FLCDZXBNQW6M5QDAA2NKAJ4LQBYZ2YYR X-MailFrom: e@80x24.org X-Mailman-Rule-Hits: nonmember-moderation X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0 CC: notmuch@notmuchmail.org, "W. Trevor King" X-Mailman-Version: 3.2.1 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1619845096; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=GgXD7lz3SG7sXYh5NlIm2s1bRuBAjuNczuUV6tZtOPQ=; b=XvVyjQR+s+FZZ13Up+2gi0Ifqkwy25maaUhniOFGA4sZsj32chVoobnTct1kfoVD6CVz+j jvXNb9wNUgCxZC38BOsBZpExLi2JNQn1ihgFc8vpHySeZbRj9tkK5upSD9RNGJ6gJISVDV 0bPWU5ZYmKqna+cMg9zmS/LRb3qlGB2rIzNLGMbO8yM8Ry4ZwFbcid6/5fIMgmoHqYfpzl Wd0IlufkvKjxS0ZzvbFh+E6jkvNb/im9qTwJ75ZtGEjWEU8dW4Y+IQ24yq2zDfjDsF3050 phHzmw6jlFtHoRHwDkCGvhHKycmaqrILZWK9yPgnIgsQwGGmP99F5GCsC2Px1g== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1619845096; a=rsa-sha256; cv=none; b=NTsPtJO8HXLx+NUG3OBbZZ2go5Ft29a9lDDnqt7syq/pdX7A+7a4J7ub/L0pJ2xkwL/Bdg N4V5F7penDo04OeGs2/oL2wtAgESdMHhkMOsgMibcC0mxwOsOSIGz/yDhH5fl1eAdl5UAg CXmfHwic+cP/1N8bzTQejKtyBx1kTJUlwYgqmRyvaw3GezfHX9aA1J7fW0RSG56iC9ckPt BBLwJPGrd6bLFtAvBY73g2dHev1vP1W20QseEy2WfIEsaZotY0gcc+En807DmIj15mgCio 30HCSYuLwirgFgnYQajlpKQV7rEy5quXoTudrgBTr98tWgnLMXDxYeAGGC4Zdg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 144.217.243.247 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Spam-Score: -1.60 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 144.217.243.247 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Queue-Id: D544BBC3E X-Spam-Score: -1.60 X-Migadu-Scanner: scn0.migadu.com X-TUID: wgdZP6L5zrRH Carl Worth wrote: > On Sat, May 01 2021, Eric Wong wrote: > > I never had the interest in using notmuch since Maildirs are a > > non-starter with millions of messages with current FSes/OSes. > > What bottleneck are you seeing here? > > I don't have million(s) of messages but I'm getting close with 1.48M > messages in my current notmuch index. > > I'm not seeing any problematic performance from the filesystem or OS > myself, so I'm curious what problem you're referring to here. I assume you have several Maildirs and not just one with 1.48M? Since I never actually used notmuch myself; most of my aversion comes from years of using Maildir sync tools (mbsync, offlineimap, rsync). They all struggle with many inodes and syscalls + cache required to walk them. It's the same reason git puts old objects in packfiles rather than having millions of loose objects. Furthermore, my MUA (mutt) struggles on a single Maildir when its size goes over ~50K. Maildir is fine as a dumping ground for mairix search results (typically a few dozen/hundred results). Maildir is better nowadays on FSes with compression and checksums; but lack of compression and checksumming were also points against it; though syscalls are also more expensive with CPU vulnerability mitigations. I've always gzipped my archival mboxes for compression and CRC. My local mirror of all the messages on lore.kernel.org/* is over 14.6M(*) and growing... (LKML is 4M of that). (*) 14.6M in the new combined "extindex" format that should be on lore.kernel.org, soon. For now, I have an experimental instance on https://yhbt.net/lore/all/