From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 7AFBE20248; Sat, 23 Feb 2019 22:07:38 +0000 (UTC) Date: Sat, 23 Feb 2019 22:07:38 +0000 From: Eric Wong To: Mateusz Loskot Cc: meta@public-inbox.org Subject: Re: Mirroring mailing lists directly Message-ID: <20190223220738.s5cij5pkqx7ke7ij@dcvr> References: <20190223031048.GA7084@dcvr> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: List-Id: Mateusz Loskot wrote: > All the details you gave about configs will be useful, I'm sure. > It's just that I'm not there yet. I'm still missing some > aspects of the bigger picture of mirroring a mailing list. > > I've realised, I'm missing an outline of the overall procedure: > > 0. Install public-inbox and its dependencies > 1. Find mailing list to mirror e.g. public@list.org > 2. Get a new e-mail address e.g. mirror@user.org > 3. Subscribe to list@host.org with mirror@user.org I typically use my normal address and not a list-specific one. > 4. Set up (to run manually or daemon) offlineimap to > sync from mirror@user.org to local Maildir You need to initialize the inbox repo: public-inbox-init -V2 NAME /path/to/inbox HTTP_URL list@host.org (I still need to write the manpage for -init :x) > 5. Set up public-inbox-watch > 6. Set up public-inbox-httpd to publish via HTTP Correct. > 7. Set up public-inbox-index to enable search Running -index separately is not necessary if you used "-V2" for public-inbox-init right now. V2 repos are significantly more scalable when you have hundreds of thousands of messages, but require SQLite and Xapian. (*) Xapian dependency can be removed, though For future updates to public-inbox code itself, you may need to run public-inbox-index if the Xapian schema changes incompatibly. > 8. Set up git daemon to allow `git clone` access to mirrored archives. public-inbox-httpd already supports smart HTTP clone. git-daemon is only necessary for git://, which seems to be falling out of favor given the popularity of HTTP/HTTPS. > Is this plan correct, complete or am I missing anything? Looks close to me. I also suggest running public-inbox-nntpd in addition to -httpd for NNTP users. It shares a common core with -nntpd and I have plans for a combined server to minimize memory use. > Do I need to bother with public-inbox-watch's bidirectional sync? There is no bidirectional sync in -watch. Perhaps you mean offlineimap? You may also want to prevent your Maildir and IMAP folder from growing too large. You can setup a cronjob to remove old mails from the Maildir; which causes offlineimap bidirectional sync to remove the old messages from IMAP, too. The following example removes mails older than 7 days: cd /path/to/Maildir && find new cur -ctime +7 -type f -print0 | xargs -0 rm -f > The docs of the public-inbox toolset are clear and they seem complete. > As explained above, I am missing a basic "where do I start" to > create mirror, especially for someone who is not a sysadmin, > like myself :) Yes, public-inbox-overview.pod probably needs to be updated. > Once I get to the 4. and later points of the plan above, > I will get back to your other suggestions on the implementation > details. Alright, please do :)