From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id eLSnK0IePWRpHwAASxT56A (envelope-from ) for ; Mon, 17 Apr 2023 12:24:02 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id sCKiK0IePWQ6NgEAauVa8A (envelope-from ) for ; Mon, 17 Apr 2023 12:24:02 +0200 Received: from mail.notmuchmail.org (yantan.tethera.net [135.181.149.255]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 2DFEBA2D0 for ; Mon, 17 Apr 2023 12:24:02 +0200 (CEST) Received: from yantan.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id 83D6D5F386; Mon, 17 Apr 2023 10:23:59 +0000 (UTC) Received: from mail-ej1-x633.google.com (mail-ej1-x633.google.com [IPv6:2a00:1450:4864:20::633]) by mail.notmuchmail.org (Postfix) with ESMTPS id 2A58B5F35B for ; Mon, 17 Apr 2023 10:23:56 +0000 (UTC) Received: by mail-ej1-x633.google.com with SMTP id dm2so62838043ejc.8 for ; Mon, 17 Apr 2023 03:23:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681727035; x=1684319035; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=8qrx6r7MBrtTZiu+JahQIuxtE/Tj74wVgZPUdg8G+vU=; b=HFZz+emjlXe9DbtwWB7qGnoDbYimaoEV4rKo8a2SyMaV8LjmrqlpQg314LOOnWknZO GyJ2AwmW+AuCVgJxZZivdVF6WfIEgXPrDZ7jJzQ3HSpRNZgfmGuRY/yDWu/Kx7aOp3S3 VKFbfHxdzNJO1oPVsLZoQw8J+HhWAZKwr6Kvc6prD3nRM5q2mOzD/S5Iz7xtFkC8I4+/ phIDLLg5GyezCfyLdrtnhJEo/JmWLoiJKnNZNZZtPZpK6L1WRQs1KM4diynXjTuHdGI/ FN6MS9z4bcJ8/rRPiVBt5AaTkmWnqssiQ04CjpZ1DlaOLYbRZhxxpHL12g7bBN3A4ZGt /iAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681727035; x=1684319035; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=8qrx6r7MBrtTZiu+JahQIuxtE/Tj74wVgZPUdg8G+vU=; b=BGgtDEov2rBRVFtGRjwnGhRHjAQurTbvLA/tc0XZUGQDye9AG9j6/UPxogN9kLn+38 OhKQ+HHdQf17YoDoBeDBAP60e4NJoaRM9IYGCgAVQ+oVBXVeVMXNScApgos9epPWzUae IcEZVA3JKjpySyzsvvkhfM19npCa44TMcYEHF9UcBvvBaK9D/srQo3BONpwUr7cxYro9 IO5XBZQurZehoDjlZfJna1Bi14uzOAeZTKUYw9iPoQcr0TS86gO76weAvbtq/hd5BkJf 9dZ/e43Bx3sJaFa40FPIyRGDblraQ1ort0MR12PX94xmTih2ym+XDPHQz6WKZLDBNhaA 2phw== X-Gm-Message-State: AAQBX9eb0h8X2pgHOdw924qmf8JFp4KNDWskjR0csh498aLRoZiuv+MA 431F7Xdq2EfgK1hdcEAihiToxDouqQr5hS01uM4= X-Google-Smtp-Source: AKy350ZoanYmHbRzs3/1o2dNhhK/3PA+ILPeu5ciz2eTkFXzK47pkcW53uo+Be/fB7d6K/WeIGqIFo5JyN7khFqpUqg= X-Received: by 2002:a17:906:e247:b0:8f1:4c6a:e72 with SMTP id gq7-20020a170906e24700b008f14c6a0e72mr2828712ejb.0.1681727035160; Mon, 17 Apr 2023 03:23:55 -0700 (PDT) MIME-Version: 1.0 References: <643ca044d2acc_13ca5294f@chronos.notmuch> In-Reply-To: <643ca044d2acc_13ca5294f@chronos.notmuch> From: Michael J Gruber Date: Mon, 17 Apr 2023 12:23:43 +0200 Message-ID: Subject: Re: inbox-update: new competition of notmuch-lore To: Felipe Contreras Message-ID-Hash: DY7IX3DDAFATTWXJF3MMIVFCRLIP7FCX X-Message-ID-Hash: DY7IX3DDAFATTWXJF3MMIVFCRLIP7FCX X-MailFrom: michaeljgruber@gmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: notmuch@notmuchmail.org, Tobias Waldekranz X-Mailman-Version: 3.3.3 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Migadu-Country: DE X-Migadu-Flow: FLOW_IN ARC-Seal: i=1; s=key1; d=yhetil.org; t=1681727042; a=rsa-sha256; cv=none; b=ucQ4DXeiB57nwVLEdmlCh6aG7GC6N1XuFMh8JPJ2nggURkljCOVywqVxflMtx6zZOyljHE BPXBn1DcNwcgytkItiNNtxVdAxG743BSBQtTwI6ZOZQR8Ry37IKbQoOVRH/gXaVD43s37G B1VpkjZpGTMo2Dz4SGq8sMHoMP6LZ+LRV9E0FNtbywHtTx6LF2x/i6OIYmNBugH7hplxAY QfRnBCZxe9TIcDhbJw3hTpUtpm5+YCxaVwGLYrKvcb1BSHfXb06UH6SR75KS4A1ceEVznX 0zaLrtNyzB2elpbgLs43x2ItLouJiWvkKM2x39/OO2SjfBjQmiXqE97kE72rJQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=gmail.com header.s=20221208 header.b=HFZz+emj; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 135.181.149.255 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1681727042; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-owner:list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=ttHxDMX+E0Vr7wcWIlRHD0Q8QrWoVbVRoPyr3MsvTzs=; b=gAgo0wuth3f4Oi3PVdYKMLjpN7hFeugCGB7oQXO1hPSEYElCKeZKb3XvBAKMsqFYk+Yiwk +DnEiq1gufC4d97Ok1j1t2tva9EV6dyoiluQmpLNa9TjvTpHnSC7qi97lIBzB6jHFG2Osq lJftFBWwFK+vqsX9vPAI8Em3SiRoP8TrTX2vshI2fKuDILq81/Dno71zqjqFs+TqCrx16A Ahocx+bPM9qQn05jiXqqRvARya0Tuk8QmMGPUgtaz0+FlNNPxjlnzcAVmMJtUNefq9Sz2a nbdZqc/nMWI1j7IeWn/DBLsQnHmGOgQ6hWR5uIqCNkL7utv13mZAt1AmKSBRMg== X-Migadu-Scanner: scn1.migadu.com X-Migadu-Spam-Score: -1.03 X-Spam-Score: -1.03 X-Migadu-Queue-Id: 2DFEBA2D0 Authentication-Results: aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=gmail.com header.s=20221208 header.b=HFZz+emj; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 135.181.149.255 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none) X-TUID: yACXq3pUQE0I Hi Felipe > I'm moving from mbsync to public-inbox and I find there aren't many tools to > make it work with notmuch. Looking at that, too. > I gave a try to notmuch-lore [1] but I found it too slow and had a couple of > issues. > > So I wrote my own script to convert public-inbox mailing lists to Maildir > format: notmuch-tools/inbox-update [2]. > > It's much faster at the initial clone, it deals with deleted mails, and YAML is > a much better configuration format. Looking at both scripts: Is the speed-up mainly due to `git cat-file` vs. `git show`? > Also, you can configure which epochs you want to fetch (notmuch-lore fetches > all of them). > > One thing it doesn't yet do is trim the repository once the mails have been > converted, but that's probably easy to add later on. What kind of trimming are you thinking about here? Partial history? I guess this shows that public-inbox's repo format is simply not the best choice for the purpose of mail readers. It is optimised for other uses, and I always wondered why they use a non-bare repo at all. That single file path m at the root creates absolutely meaningless diffs. And the commit message doubles the info which is present in the blob. notes-ref could have served better for inspiration of public-inbox. (Barking up the wrong tree, I know.) There are even tools in the public-inbox eco system which feed that info into a xapian db, though not notmuch-like, as if notmuch hadn't existed already. What I'm dreaming of is a notmuch "storage backend" which is git object db based rather than maildir based, and compatible with public-inbox (at least with the use case, i.e. v3 or v4...). I mean - why do we need a checkout of basically immutable files which are stored in blobs already, just so that notmuch can index them? We need them for the MUAs, I know, and we would need a solution for them, too. Or simply a tree in public-inbox which allows clients to use a mere checkout ... Cheers, Michael