[bug#43637] [PATCH] gnu: packages: Add grokmirror.

[bug#43637] [PATCH] gnu: packages: Add grokmirror.

[Kyle Meyer]

* gnu/packages/version-control.scm (grokmirror): New variable.
---
 gnu/packages/version-control.scm | 39 +++++++++++++++++++++++++++++++-
 1 file changed, 38 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/version-control.scm b/gnu/packages/version-control.scm
index 2ee8247071..68971c60f9 100644
--- a/gnu/packages/version-control.scm
+++ b/gnu/packages/version-control.scm
@@ -7,7 +7,7 @@
 ;;; Copyright © 2014, 2015, 2016 Mark H Weaver <mhw@netris.org>
 ;;; Copyright © 2014, 2016, 2019 Eric Bavier <bavier@member.fsf.org>
 ;;; Copyright © 2015, 2016, 2017, 2018, 2019, 2020 Efraim Flashner <efraim@flashner.co.il>
-;;; Copyright © 2015, 2018 Kyle Meyer <kyle@kyleam.com>
+;;; Copyright © 2015, 2018, 2020 Kyle Meyer <kyle@kyleam.com>
 ;;; Copyright © 2015, 2017, 2018, 2020 Ricardo Wurmus <rekado@elephly.net>
 ;;; Copyright © 2016, 2017 Leo Famulari <leo@famulari.name>
 ;;; Copyright © 2016, 2017, 2018 Nikita <nikita@n0.is>
@@ -2269,6 +2269,43 @@ (define-public myrepos
 Mercurial, Bazaar, Darcs, CVS, Fossil, and Veracity.")
     (license license:gpl2+)))
 
+(define-public grokmirror
+  (package
+    (name "grokmirror")
+    (version "2.0.0")
+    (source
+     (origin
+       (method git-fetch)
+       (uri (git-reference
+             (url (string-append "https://git.kernel.org/pub/scm/"
+                                 "utils/grokmirror/grokmirror.git"))
+             (commit (string-append "v" version))))
+       (file-name (string-append name "-" version "-checkout"))
+       (sha256
+        (base32 "1cs43vf87x8x5k5ncgiwiclc92a1dvxpg2z6lh6psaiip808gylp"))))
+    (build-system python-build-system)
+    (arguments
+     `(#:tests? #f                      ; no test suite
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'install 'install-manpages
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let* ((man (string-append (assoc-ref outputs "out")
+                                        "/man/man1/")))
+               (mkdir-p man)
+               (for-each (lambda (file) (install-file file man))
+                         (find-files "." "\\.1$")))
+             #t)))))
+    (propagated-inputs
+     `(("python-requests" ,python-requests)))
+    (home-page
+     "https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git")
+    (synopsis "Framework to smartly mirror git repositories")
+    (description "Grokmirror enables replicating large git repository
+collections efficiently.  Mirrors decide to clone and update repositories
+based on a manifest file published by servers.")
+    (license license:gpl3+)))
+
 (define-public git-annex-remote-rclone
   (package
     (name "git-annex-remote-rclone")

base-commit: 3876ed32f25e160c47ba046d987ad9097b2bf0cb
-- 
2.28.0.849.g665beee440

[bug#43637] [PATCH] gnu: packages: Add grokmirror.

[Kyle Meyer]

A recent message on bug-guix [1] mentioned cloning the archives at
<https://yhetil.org>.  public-inbox archives, including those at
yhetil.org, publish a grokmirror-compatible manifest.  So, if you're
interested in mirroring the Guix-related archives at yhetil.org for any
reason [2], you can use grokmirror to do so in a way that's nicer than
periodically hitting yhetil.org with a git-fetch call for each
Guix-related inbox.

You can clone (and then later update) Guix-related inboxes with

    $ grok-pull -vc /path/to/config

and a config file that looks something like this:

--8<---------------cut here---------------start------------->8---
[core]
toplevel = /path/to/inboxes
manifest = ${toplevel}/manifest.js.gz
log = ${toplevel}/grok.log
loglevel = info
objstore = ${toplevel}/objstore

[pull]
pull_threads = 2
include = /guix-*

[remote]
site = https://yhetil.org
manifest = ${site}/manifest.js.gz
--8<---------------cut here---------------end--------------->8---

[1] https://issues.guix.gnu.org/43528#4
    CAJ3okZ0g25+-wsL2OepE+xMR5QWShmS73hSHsjEmEgYBiOWwAg@mail.gmail.com

[2] to have a fast local copy, to publish your own mirror, to feed the
    archives into some other tool...

bug#43637: [PATCH] gnu: packages: Add grokmirror.

[Ludovic Courtès]

Hi,

Kyle Meyer <kyle@kyleam.com> skribis:

> * gnu/packages/version-control.scm (grokmirror): New variable.

Applied, thanks!

> A recent message on bug-guix [1] mentioned cloning the archives at
> <https://yhetil.org>.  public-inbox archives, including those at
> yhetil.org, publish a grokmirror-compatible manifest.  So, if you're
> interested in mirroring the Guix-related archives at yhetil.org for any
> reason [2], you can use grokmirror to do so in a way that's nicer than
> periodically hitting yhetil.org with a git-fetch call for each
> Guix-related inbox.

So yhetil.org stores mailing list archives as a Git repo?

Sounds interesting.

Thanks,
Ludo’.

[bug#43637] [PATCH] gnu: packages: Add grokmirror.

[Kyle Meyer]

Ludovic Courtès writes:

> Hi,
>
> Kyle Meyer <kyle@kyleam.com> skribis:
>
>> * gnu/packages/version-control.scm (grokmirror): New variable.
>
> Applied, thanks!

Thanks for reviewing/applying!

>> A recent message on bug-guix [1] mentioned cloning the archives at
>> <https://yhetil.org>.  public-inbox archives, including those at
>> yhetil.org, publish a grokmirror-compatible manifest.  So, if you're
>> interested in mirroring the Guix-related archives at yhetil.org for any
>> reason [2], you can use grokmirror to do so in a way that's nicer than
>> periodically hitting yhetil.org with a git-fetch call for each
>> Guix-related inbox.
>
> So yhetil.org stores mailing list archives as a Git repo?
>
> Sounds interesting.

Yep, messages are stored in a Git repo or, for archives greater than
around one gigabyte, a collection of repos [*].  (A freshly gc'd
guix-devel weighs in at ~140M.)

  [*] documentation on public-inbox's v2 format:
      https://public-inbox.org/public-inbox-v2-format.html