From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-Status: No, score=-3.3 required=3.0 tests=AWL,BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-qt1-x82d.google.com (mail-qt1-x82d.google.com [IPv6:2607:f8b0:4864:20::82d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 339431F4B4 for ; Wed, 7 Oct 2020 21:42:20 +0000 (UTC) Received: by mail-qt1-x82d.google.com with SMTP id c5so3364686qtw.3 for ; Wed, 07 Oct 2020 14:42:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=date:from:to:subject:message-id:mail-followup-to:mime-version :content-disposition; bh=0IYRn4WPvGW/hEH7sy6GZrTJkLiKJaFYyovHlb8R3eg=; b=VUWyEiMk16f74rWjevDrtwMpJIMOc0I7rO7/VwgHhT78t+Y4PPxFUhK4D3/tjv3GuM fV3brfayDddJ7KJ0lVSbJn+YFcsPsmv3lTn1xwNxDSWmGXscb2FJ8y0tzPKaQQBx/YBO Fa5ONH1T5+7gd95ZfsL68AxrJRvU2DYmZ8yyI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:subject:message-id:mail-followup-to :mime-version:content-disposition; bh=0IYRn4WPvGW/hEH7sy6GZrTJkLiKJaFYyovHlb8R3eg=; b=FdlHJq6qClqaF2GMiqsFFYB/t6DjPTfRHya0Fb5N9+x2OeEsTQvBQUcWZ45cJuHVev DSyV1XQZASSDMbqotnTuPq31kw44/iw269RRETgANa466NWKly8MyFJuVNjTAC9ij5hr HXpubbXyBsxxUzIqwG2ATEISD/+3qRs4HKiJA7tRJRUlOwhMHPU8IvR+1l+0Hab2K86d GorAZigGnxzhz+4Wj4tsJQbS/wxBMPUWE2dMmxXSMfqkYHULdW/mWiXLuL7IWhecgRO3 QJdgGOkaMtoTBg4BB8HGcS/CjNvD/vKkk35m2I1vUXDJD3FjDnAhIOrhTuiS4jKDnucX hLiA== X-Gm-Message-State: AOAM531w5So7ksaoz6pseCTxxQ8LH09BcwcFN0y1JqSoxER0JqpCsP/V DDSssodpFR8six+CsA+hpCBgZixdpFOByVWU X-Google-Smtp-Source: ABdhPJxczJGKQdD91Xp/jyyOOCenLcXCek7zBxXOqNru+Z+ystzyO3uSspxJSxZXoQef1a0XSwm9HA== X-Received: by 2002:ac8:5948:: with SMTP id 8mr5175164qtz.215.1602106938403; Wed, 07 Oct 2020 14:42:18 -0700 (PDT) Received: from chatter.i7.local ([89.36.78.230]) by smtp.gmail.com with ESMTPSA id f14sm2357410qkh.134.2020.10.07.14.42.17 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Oct 2020 14:42:17 -0700 (PDT) Date: Wed, 7 Oct 2020 17:42:16 -0400 From: Konstantin Ryabitsev To: meta@public-inbox.org Subject: Subscribing to public-inbox lists using grokmirror + procmail Message-ID: <20201007214216.ztns57g22rowlccu@chatter.i7.local> Mail-Followup-To: meta@public-inbox.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline List-Id: Hi, all: I needed a way to pipe public-inbox straight into patchwork.kernel.org without having to manage yet another list subscription with postfix pipe integration. Then I realized that it's generally useful as a way to deliver straight from public-inbox archives into local inboxes -- similar to how l2md does it, but with some extra perks: - we use manifest.js.gz provided by public-inbox to poll the remote manifest instead of running "git remote update" in each mirrored list - we automatically recognize when remote epochs roll over - if configured with `shallow=yes`, we can automatically re-shallow local git copies after each successful delivery, so they don't take up any space You can read the details here: https://people.kernel.org/monsieuricon/subscribing-to-lore-lists-with-grokmirror it's also available below. Hope this is useful to others! -K ---- # Subscribing to lore lists with grokmirror You can have lore.kernel.org mailing lists delivered right into your inbox straight from the git archive (in fact, this will work for any public-inbox server, not just for lore.kernel.org). It's efficient and (optionally) preserves a full copy of entire list archives on your system -- should you wish to keep them. Note: this requires grokmirror-2.0.2+, as earlier versions do not come with the grok-pi-piper utility. ## Installing grokmirror-2.0 Easiest is to install from pip: pip install --user grokmirror~=2.0.2 You may have grokmirror available from your distro packages, too, but make sure it's version 2.0.2 or above. ## Installing procmail Procmail should be available with your distribution, so install it like any other package. ## Configuring procmail Procmail configuration can be a book in itself, but if you just want to have messages delivered into your inbox, all you have to do is create a `~/.procmailrc` with the following contents: DEFAULT=$HOME/Mail/ # Don't deliver duplicates sent to multiple lists :0 Wh: .msgid.lock | formail -D 8192 .msgid.cache If your mailbox is not in `~/Mail`, then you should adjust the above accordingly. ## Configuring grokmirror Create a `~/.config/lore.conf` with the following contents. We'll use three lists as examples: git, linux-hardening, and linux-doc, but you'll obviously want to use the lists you care about. You can see which lists are available from https://lore.kernel.org/lists, or the exact git repositories on https://erol.kernel.org/. [core] toplevel = ~/.local/share/grokmirror/lore log = ${toplevel}/grokmirror.log [remote] site = https://lore.kernel.org manifest = https://lore.kernel.org/manifest.js.gz [pull] post_update_hook = ~/.local/bin/grok-pi-piper -c ~/.config/pi-piper.conf refresh = 300 include = /git/* /linux-hardening/* /linux-doc/* The above assumes that you installed grokmirror with `pip install --user`. Now make the toplevel directory: $ mkdir -p ~/.local/share/grokmirror/lore ## Configuring pi-piper The last step is to create `~/.config/pi-piper.conf`: [DEFAULT] pipe = /usr/bin/procmail shallow = yes The important bit here is `shallow = yes`. Public-inbox stores every mail message as a separate commit, so once a message is piped to procmail and delivered, we usually don't care about keeping a copy of that commit any more. If you set `shallow = yes`, pi-piper will prune all but the last successfully processed commit out of your local git copy by turning those repos into shallow git repositories. This helps to greatly save disk space, especially for large archives. If you *do* want to keep full archives, then don't set `shallow`. You can change your mind at any time by running `git fetch _grokmirror master --unshallow` in each underlying git repository (you can find them in `~/.local/share/grokmirror/lore/`). You can also specify this option per list: [DEFAULT] pipe = /usr/bin/procmail [linux-hardening] shallow = yes ## Running grok-pull You can now run grok-pull to get the initial repo copies. Note, that during the first run grokmirror will perform full clones even if you specified `shallow = yes` in the pi-piper config, so it may take some time for large archives like those for the git list. However, once the pi-piper hook runs, they will be repacked to almost nothing. Future versions of grokmirror may become smarter about this and perform shallow clones from the beginning. During the initial pi-piper run, there will be no mail delivered, as it will just perform initial setup and make a note where the HEAD is pointing. If you run grok-pull again, two things may happen: 1. There will be no changes and grok-pull will exit right away 2. If there are changes, they will be fetched and the hook will deliver them to procmail (and to your inbox) ## Running in the background You can run grok-pull in the background, where it will check for updates as frequently as the `refresh` setting says (300 seconds in the example above). You can either background it "the old way": grok-pull -o -c ~/.config/lore.conf & Or the new way, using a systemd user service: $ cat .config/systemd/user/grok-pull@.service [Unit] Description=Grok-pull service for %I ConditionPathExists=%h/.config/%i.conf [Service] ExecStart=%h/.local/bin/grok-pull -o -c %h/.config/%i.conf Type=simple Restart=on-failure [Install] WantedBy=default.target $ systemctl --user enable grok-pull@lore $ systemctl --user start grok-pull@lore If you make changes to `~/.config/lore.conf`, for example to add new lists, you will need to restart the service: $ systemctl --user restart grok-pull@lore ## Combining with mbsync You can totally combine this with mbsync and deliver into the same local inbox. As a perk, any messages injected from grokmirror will be uploaded to your remote imap mailbox. See this post from mcgrof about configuring mbsync: - https://people.kernel.org/mcgrof/replacing-offlineimap-with-mbsync ## Troubles Email tools@linux.kernel.org if you have any trouble getting the above to work. The grok-pi-piper utility is fairly new, so it's entirely possible that it's full of bugs.