From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id pfiHAuk7PWR6pAAASxT56A (envelope-from ) for ; Mon, 17 Apr 2023 14:30:33 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id oA6MAek7PWT6MwEA9RJhRA (envelope-from ) for ; Mon, 17 Apr 2023 14:30:33 +0200 Received: from mail.notmuchmail.org (yantan.tethera.net [IPv6:2a01:4f9:c011:7a79::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id CA0E010281 for ; Mon, 17 Apr 2023 14:30:27 +0200 (CEST) Received: from yantan.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id 1EB235F386; Mon, 17 Apr 2023 12:30:25 +0000 (UTC) Received: from mail-oa1-x2c.google.com (mail-oa1-x2c.google.com [IPv6:2001:4860:4864:20::2c]) by mail.notmuchmail.org (Postfix) with ESMTPS id E23955F35B for ; Mon, 17 Apr 2023 12:30:21 +0000 (UTC) Received: by mail-oa1-x2c.google.com with SMTP id 586e51a60fabf-1880110ebe5so369424fac.3 for ; Mon, 17 Apr 2023 05:30:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681734620; x=1684326620; h=content-transfer-encoding:mime-version:subject:references :in-reply-to:message-id:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=fSqyCCZ0e/aph/3EMluAjys0fpLs6n49Nuzo1rEqXC0=; b=e7CqcY/cJ65KQH/IRa8Ez7zY94AeultqAc3izf+7gpHyzyeIOWNJ7H1TBteNK+jQ0y lgAdldyD6JLFS2RN0TkBPAiL+5C1cBJHhkXNy6t/BICw6spFTBwLDuanpl2SsWBRbzhg PhTtAbBqKC6heAJ35C+NBpHTmcnw+7F4rj99Q6L7vIdQwt3Lw8g+hoKyNnPVO10L2oJ+ jgc5ugepxETy03CgZLkwfkfgekFEuyUsw+Tj6wwhcQRBhbZ5FagrwjVdM+8qACTGPPr+ Nkw2R+ynWGJ7pv73sirRBWC9xnhna1XQehgaa7hXoFC+KBuVo0UvrRMizOHLBOYnODC0 onBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681734620; x=1684326620; h=content-transfer-encoding:mime-version:subject:references :in-reply-to:message-id:cc:to:from:date:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=fSqyCCZ0e/aph/3EMluAjys0fpLs6n49Nuzo1rEqXC0=; b=aKYvq9m3Rp9ldOf+Bp7mRWQBmSPZK6tM/J2CNtz7zoph028wZ7IMd4noKMyIjdKyo0 Tk8CdWxCeZGZWeiIIyHP6kz8Ug6pZjZLWLWH722Uhb2AFW/2DS/q2ewZCpLOyw5214s8 6f+udy/g1bhhc01CTMZ1Q/PhKRFO0SUNCPw3LlnoxY9hqKPruTLugchagrvH6Y8CKZ8K 3BoBOB4tlbCROwQnx1VhYqHaMNC+N+2hteWw0+Evzi/4ukP84juaq474TbII1WMv9RK7 PqDhbMmaYMmXQRsyABbQHKdQBoreQuwtL38wf8IZikg+Icoe6Vat3YNeFdKs552154gz caSA== X-Gm-Message-State: AAQBX9f5XPFUCqWyB4W/NTfLLVOMu/k98yJh01fgCFt4SQ87wSXzGiRs 186p3k4p7GsWWSgdPdy5X3I= X-Google-Smtp-Source: AKy350ateIXDo7rGtrg1RWiALUBl4gravA4qA2VFKxCXEipFLDDeF3wAT8AWf5+6wRODQBVm9VFmOw== X-Received: by 2002:a05:6870:24a6:b0:187:836f:d36a with SMTP id s38-20020a05687024a600b00187836fd36amr8117069oaq.0.1681734620448; Mon, 17 Apr 2023 05:30:20 -0700 (PDT) Received: from localhost ([2806:2f0:4060:fff1:4ae7:daff:fe31:3285]) by smtp.gmail.com with ESMTPSA id i18-20020a056830011200b006a44d90de05sm2793418otp.69.2023.04.17.05.30.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Apr 2023 05:30:19 -0700 (PDT) Date: Mon, 17 Apr 2023 06:30:18 -0600 From: Felipe Contreras To: Michael J Gruber , Felipe Contreras Message-ID: <643d3bdae7ae6_751a29486@chronos.notmuch> In-Reply-To: References: <643ca044d2acc_13ca5294f@chronos.notmuch> Subject: Re: inbox-update: new competition of notmuch-lore Mime-Version: 1.0 Message-ID-Hash: GCUZPJ7V7BCNAOM5PLCPTIBTC45PYH7T X-Message-ID-Hash: GCUZPJ7V7BCNAOM5PLCPTIBTC45PYH7T X-MailFrom: felipe.contreras@gmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: notmuch@notmuchmail.org, Tobias Waldekranz X-Mailman-Version: 3.3.3 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Migadu-Country: DE X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1681734628; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-owner:list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=jWm3o4hvjUqBdRrumCe03ilbxUNookVVTZpwuIVUVz8=; b=Z9E7sJN8FYpbt986MDkA3DuqD4cZCR1GUPAAhSgqxXO5z5V/KnqxlxtMcOGSDJ41OEAxJU VYdVBqIfQEBUHFrKd90bhrzni+Xaqui8awGysdaQ/nJgptRv1AMIVdlu6HcTPDSv4z4bi1 Hr8UjYx4fh2c3//KtzP/Qq0SGOo3AtERWJ4VpONLyuM4kdkeOjzgHqqjlGiCz6VMKY4yF4 1bYvfO8iZ18+tcGlRuEpJLhMSXFhWwJG2XnX3gJkVEXTZf2Q0HoANXGZn1/+sjzYJQiU+9 965hLhV3wIENCeYC5bIJdm3kXGKPoDWqTMhurAkq7szN255rH9OU6leD/sFw6g== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=gmail.com header.s=20221208 header.b="e7CqcY/c"; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 2a01:4f9:c011:7a79::1 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none) ARC-Seal: i=1; s=key1; d=yhetil.org; t=1681734628; a=rsa-sha256; cv=none; b=UcveScAwa+BLI1WfE4+lWIP2bg0cnr0kAUDoun6DY3c6DUlmqVVbHb86Xx/aTT16sa8YW8 mfQ7HqCx3JsEhphW4dKsHOw4+LzLUSzUOZ9gL8XKlLr1FNfhNec3SIwbcyRuaoORCr1Pvl 6TpFHcnNh9TL7NXXsFfUKfZWSg9tLlkLah/wsxtxzcD+Nq+5kQ/iQ3BtBfNaZToL/v+roa OEsuebXyWSfZRQNOhQC1vUuhkTYmnvP9juhpWFQsYR6dUz9gD8hkhW6Rh1TsWf2hoLL5Hb 8EYgKKXrTfaiMFU8U4YngEDouUQ2e1Y1FpgD1YTn6uWTWUSVcBPFXqiXyaYk3g== Authentication-Results: aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=gmail.com header.s=20221208 header.b="e7CqcY/c"; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 2a01:4f9:c011:7a79::1 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none) X-Migadu-Spam-Score: 1.12 X-Spam-Score: 1.12 X-Migadu-Queue-Id: CA0E010281 X-Migadu-Scanner: scn1.migadu.com X-TUID: dao7uRDTXSfS Michael J Gruber wrote: > > I'm moving from mbsync to public-inbox and I find there aren't many tools to > > make it work with notmuch. > > Looking at that, too. > > > I gave a try to notmuch-lore [1] but I found it too slow and had a couple of > > issues. > > > > So I wrote my own script to convert public-inbox mailing lists to Maildir > > format: notmuch-tools/inbox-update [2]. > > > > It's much faster at the initial clone, it deals with deleted mails, and YAML is > > a much better configuration format. > > Looking at both scripts: Is the speed-up mainly due to `git cat-file` > vs. `git show`? My guess is that it's due to using `git cat-file` in batch mode, so it's called only once, instead of thousands of times. Presumably this can be done in notmuch-lore as well, with something like: git rev-list | sed -e /$/:m/ | git cat-file --batch But this still has the issue that some commits remove mail, don't add. > > Also, you can configure which epochs you want to fetch (notmuch-lore fetches > > all of them). > > > > One thing it doesn't yet do is trim the repository once the mails have been > > converted, but that's probably easy to add later on. > > What kind of trimming are you thinking about here? Partial history? Same as notmuch-lore does: just the last commit. Once the mails have been extracted there's no need for those commits. > I guess this shows that public-inbox's repo format is simply not the > best choice for the purpose of mail readers. It is optimised for other > uses, and I always wondered why they use a non-bare repo at all. That > single file path m at the root creates absolutely meaningless diffs. > And the commit message doubles the info which is present in the blob. > notes-ref could have served better for inspiration of public-inbox. > (Barking up the wrong tree, I know.) I don't know if there's a better format, git stores shapshots anyway, so as long as the information is retrivable in some way, I think that' fine. And I clone the public-inbox repositories as bare (mirror, actually), that's something for the client to decide. > There are even tools in the public-inbox eco system which feed that > info into a xapian db, though not notmuch-like, as if notmuch hadn't > existed already. > > What I'm dreaming of is a notmuch "storage backend" which is git > object db based rather than maildir based, and compatible with > public-inbox (at least with the use case, i.e. v3 or v4...). I mean - > why do we need a checkout of basically immutable files which are > stored in blobs already, just so that notmuch can index them? Yeap, that's exactly what I want as well. It should not be that difficult to decouple notmuch from physical files and feed some virtual content. > We need them for the MUAs, I know, and we would need a solution for > them, too. Or simply a tree in public-inbox which allows clients to > use a mere checkout ... 99% of the time the content is not needed for the MUAs. So perhaps there could be a way to request the body of the message through libnotmuch, and some provider of virtual messages retrives it on demand. Maildir seems like a cumbersome intermediary to me, at the moment. -- Felipe Contreras