From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 6AF521F8C6 for ; Sun, 12 Sep 2021 07:47:16 +0000 (UTC) From: Eric Wong To: meta@public-inbox.org Subject: [PATCH 0/7] new public-inbox-{clone,fetch} commands Date: Sun, 12 Sep 2021 07:47:09 +0000 Message-Id: <20210912074716.4713-1-e@80x24.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit List-Id: Hopefully, these new commands make maintaining mirrors of a single (or handful of) multi-epoch v2 inboxes easier and less error-prone. Unlike grokmirror: * these commands do not require extra config files of any kind * they only allow cloning/fetching a single inbox per-invocation "description" files also default to something more meaningful for both public-inbox-init and -clone. PATCH 7/7 also begins laying the groundwork for a v1 => v2 migration path which doesn't involve existing mirrors having to redownload everything. Eric Wong (7): lei_mirror: simplify error reporting lei_mirror: fix error message new public-inbox-{clone,fetch} commands clone|lei_mirror: write description in mirrors import: do not write a "description" file init: set a useful description fetch: use manifest.js.gz for v1 Documentation/public-inbox-clone.pod | 71 +++++++++++ Documentation/public-inbox-fetch.pod | 63 ++++++++++ MANIFEST | 5 + lib/PublicInbox/Admin.pm | 8 ++ lib/PublicInbox/Fetch.pm | 172 +++++++++++++++++++++++++++ lib/PublicInbox/Import.pm | 3 - lib/PublicInbox/LEI.pm | 6 +- lib/PublicInbox/LeiMirror.pm | 167 ++++++++++++++++---------- lib/PublicInbox/TestCommon.pm | 3 +- script/public-inbox-clone | 58 +++++++++ script/public-inbox-fetch | 35 ++++++ script/public-inbox-init | 6 + t/init.t | 3 + t/lei-mirror.t | 52 ++++++++ t/www_listing.t | 1 - 15 files changed, 585 insertions(+), 68 deletions(-) create mode 100644 Documentation/public-inbox-clone.pod create mode 100644 Documentation/public-inbox-fetch.pod create mode 100644 lib/PublicInbox/Fetch.pm create mode 100755 script/public-inbox-clone create mode 100755 script/public-inbox-fetch