Subscribing to public-inbox lists using grokmirror + procmail

Subscribing to public-inbox lists using grokmirror + procmail

[Konstantin Ryabitsev]

Hi, all:

I needed a way to pipe public-inbox straight into patchwork.kernel.org 
without having to manage yet another list subscription with postfix pipe 
integration. Then I realized that it's generally useful as a way to 
deliver straight from public-inbox archives into local inboxes -- 
similar to how l2md does it, but with some extra perks:

- we use manifest.js.gz provided by public-inbox to poll the remote 
  manifest instead of running "git remote update" in each mirrored list
- we automatically recognize when remote epochs roll over
- if configured with `shallow=yes`, we can automatically re-shallow 
  local git copies after each successful delivery, so they don't take up 
  any space

You can read the details here:
https://people.kernel.org/monsieuricon/subscribing-to-lore-lists-with-grokmirror

it's also available below.

Hope this is useful to others!

-K

----
# Subscribing to lore lists with grokmirror

You can have lore.kernel.org mailing lists delivered right into your inbox
straight from the git archive (in fact, this will work for any public-inbox
server, not just for lore.kernel.org). It's efficient and (optionally)
preserves a full copy of entire list archives on your system -- should you
wish to keep them.

Note: this requires grokmirror-2.0.2+, as earlier versions do not come with
the grok-pi-piper utility.

## Installing grokmirror-2.0

Easiest is to install from pip:

    pip install --user grokmirror~=2.0.2

You may have grokmirror available from your distro packages, too, but make
sure it's version 2.0.2 or above.

## Installing procmail

Procmail should be available with your distribution, so install it like any
other package.

## Configuring procmail

Procmail configuration can be a book in itself, but if you just want to have
messages delivered into your inbox, all you have to do is create a
`~/.procmailrc` with the following contents:

    DEFAULT=$HOME/Mail/

    # Don't deliver duplicates sent to multiple lists
    :0 Wh: .msgid.lock
    | formail -D 8192 .msgid.cache

If your mailbox is not in `~/Mail`, then you should adjust the above
accordingly.

## Configuring grokmirror

Create a `~/.config/lore.conf` with the following contents. We'll use three
lists as examples: git, linux-hardening, and linux-doc, but you'll obviously
want to use the lists you care about. You can see which lists are available
from https://lore.kernel.org/lists, or the exact git repositories on
https://erol.kernel.org/.

    [core]
    toplevel = ~/.local/share/grokmirror/lore
    log = ${toplevel}/grokmirror.log

    [remote]
    site = https://lore.kernel.org
    manifest = https://lore.kernel.org/manifest.js.gz

    [pull]
    post_update_hook = ~/.local/bin/grok-pi-piper -c ~/.config/pi-piper.conf
    refresh = 300
    include = /git/*
              /linux-hardening/*
              /linux-doc/*

The above assumes that you installed grokmirror with `pip install --user`. Now
make the toplevel directory:

    $ mkdir -p ~/.local/share/grokmirror/lore

## Configuring pi-piper

The last step is to create `~/.config/pi-piper.conf`:

    [DEFAULT]
    pipe = /usr/bin/procmail
    shallow = yes

The important bit here is `shallow = yes`. Public-inbox stores every mail
message as a separate commit, so once a message is piped to procmail and
delivered, we usually don't care about keeping a copy of that commit any more.
If you set `shallow = yes`, pi-piper will prune all but the last successfully
processed commit out of your local git copy by turning those repos into
shallow git repositories. This helps to greatly save disk space, especially
for large archives.

If you *do* want to keep full archives, then don't set `shallow`. You can
change your mind at any time by running `git fetch _grokmirror master
--unshallow` in each underlying git repository (you can find them in
`~/.local/share/grokmirror/lore/`).

You can also specify this option per list:

    [DEFAULT]
    pipe = /usr/bin/procmail

    [linux-hardening]
    shallow = yes

## Running grok-pull

You can now run grok-pull to get the initial repo copies. Note, that during
the first run grokmirror will perform full clones even if you specified
`shallow = yes` in the pi-piper config, so it may take some time for large
archives like those for the git list. However, once the pi-piper hook runs,
they will be repacked to almost nothing. Future versions of grokmirror may
become smarter about this and perform shallow clones from the beginning.

During the initial pi-piper run, there will be no mail delivered, as it will
just perform initial setup and make a note where the HEAD is pointing. If you
run grok-pull again, two things may happen:

1. There will be no changes and grok-pull will exit right away
2. If there are changes, they will be fetched and the hook will deliver them
   to procmail (and to your inbox)

## Running in the background

You can run grok-pull in the background, where it will check for updates as
frequently as the `refresh` setting says (300 seconds in the example above).
You can either background it "the old way":

    grok-pull -o -c ~/.config/lore.conf &

Or the new way, using a systemd user service:

    $ cat .config/systemd/user/grok-pull@.service

    [Unit]
    Description=Grok-pull service for %I
    ConditionPathExists=%h/.config/%i.conf

    [Service]
    ExecStart=%h/.local/bin/grok-pull -o -c %h/.config/%i.conf
    Type=simple
    Restart=on-failure

    [Install]
    WantedBy=default.target

    $ systemctl --user enable grok-pull@lore
    $ systemctl --user start grok-pull@lore

If you make changes to `~/.config/lore.conf`, for example to add new lists,
you will need to restart the service:

    $ systemctl --user restart grok-pull@lore

## Combining with mbsync

You can totally combine this with mbsync and deliver into the same local
inbox. As a perk, any messages injected from grokmirror will be uploaded to
your remote imap mailbox. See this post from mcgrof about configuring mbsync:

- https://people.kernel.org/mcgrof/replacing-offlineimap-with-mbsync

## Troubles

Email tools@linux.kernel.org if you have any trouble getting the above to
work. The grok-pi-piper utility is fairly new, so it's entirely possible that
it's full of bugs.