[bug#53144] [PATCH 0/13] Make more git-using packages auto-updatable

[bug#53144] [PATCH 0/13] Make more git-using packages auto-updatable

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1231 bytes --]

X-Debbugs-CC: liliana.prikler@gmail.com ludo@gnu.org iskarian@mgsn.dev public@yoctocell.xyz

Hi,

This patch series makes more packages that use 'git-fetch'
auto-updatable.  The first 9 patches are specifically for Minetest packages.
The following three patches make packages written in the folowing style
auto-updatable:

(define-public foo
  ;; Upstream never makes any releases, so use the latest commit
  ;; from the 'stable' branch instead.
  (let ((commit "deadbeef...")
        (revision "9"))
    (package
      (name "foo")
      (version (git-version "0" revision commit))
      (source [git-fetch etc.])
      [etc.]
      (properties '((with-latest-git-commit . "refs/heads/stable"))))))

To do so, make sure with-latest-git-commit is set, and run
"./pre-inst-env guix refresh -t -u foo".  It will update
the commit and revision.

The last patch sets the property 'with-latest-git-commit'.

tests/channels.scm ("channel-news, one entry") fails, TBI.
"make check" is running, I'll send a message about what fails
and succeeds.

You can also grab a copy from https://notabug.org/maximed/guix-gnunet
(branch: wip-git-fetch2, 358c8f64966c423927f22f1c0ee29e85e70b1af8)

Greetings,
Maxime

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#53144] [PATCH 01/13] doc: Give some tips on Minetest packaging.

[Maxime Devos]

* doc/contributing.texi (Minetest Packages): New section.
* doc/guix.texi: Copyright update.
---
 doc/contributing.texi | 42 ++++++++++++++++++++++++++++++++++++++++++
 doc/guix.texi         |  2 +-
 2 files changed, 43 insertions(+), 1 deletion(-)

diff --git a/doc/contributing.texi b/doc/contributing.texi
index 72f5ce1e0e..5b91fc7867 100644
--- a/doc/contributing.texi
+++ b/doc/contributing.texi
@@ -394,6 +394,7 @@ needed is to review and apply the patch.
 * Synopses and Descriptions::   Helping users find the right package.
 * Snippets versus Phases::      Whether to use a snippet, or a build phase.
 * Emacs Packages::              Your Elisp fix.
+* Minetest Packages::           Building blocks.
 * Python Modules::              A touch of British comedy.
 * Perl Modules::                Little pearls.
 * Java Packages::               Coffee break.
@@ -703,6 +704,47 @@ When encountering problems, it is wise to check for the presence of the
 file, and whether any dependencies and their versions listed therein are
 satisfied.
 
+@node Minetest Packages
+@subsection Minetest Packages
+@cindex minetest, packaging
+
+A Minetest mod @code{foo} is named @code{minetest-foo} -- the author
+name from ContentDB is not included, unless required to resolve a name
+collision.
+
+Sometimes, it might be unclear what the version of a Minetest mod is.
+For example, ContentDB and the importer reports 2020-01-01, but
+according to the forums the version is 2.1.  Usually, in these cases the
+version on ContentDB is the newest and intended for distribution. As
+such, you can use the version from ContentDB without any special
+comments.
+
+@c Currently it's always checked out from git, but in principle
+@c tarballs could be used.
+
+Even though the source code is often checked out from version control,
+it is not necessary to use @code{git-version} or @code{hg-version}: the
+releases on ContentDB are formal releases; in fact they are upstream's
+official source of Minetest packages and they are not mutated in-place.
+
+@c Example (zip): mods by TenPlus1
+@c Example (git): basic_materials, ethereal
+While ContentDB provides the source code of packages in zip form, it is
+recommended not to use these, because users can and do delete old
+versions.  Likewise, sometimes the maintainer initially did tag versions
+but later stops doing so, breaking @command{guix refresh -u}.  As such,
+it is recommended not to use git tags in @code{origin} records and
+instead refer to the commit directly.
+
+The dependency information from ContentDB is usually correct.  However,
+sometimes to actually use a Minetest mod, an additional non-declared
+Minetest mod is required.  For example, to craft an item from a mod
+@code{minetest-basic-materials} in non-creative mode, an item from the
+mod @code{minetest-moreores} is required.  In such situations, it is
+recommended to add @code{minetest-moreores} to the inputs of
+@code{minetest-basic-materials}.  The dependencies go into
+@code{propagated-inputs}.
+
 @node Python Modules
 @subsection Python Modules
 
diff --git a/doc/guix.texi b/doc/guix.texi
index 08e5bfa111..0e19285c1c 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -88,7 +88,7 @@ Copyright @copyright{} 2020 Daniel Brooks@*
 Copyright @copyright{} 2020 John Soo@*
 Copyright @copyright{} 2020 Jonathan Brielmaier@*
 Copyright @copyright{} 2020 Edgar Vincent@*
-Copyright @copyright{} 2021 Maxime Devos@*
+Copyright @copyright{} 2021, 2022 Maxime Devos@*
 Copyright @copyright{} 2021 B. Wilson@*
 Copyright @copyright{} 2021 Xinglu Chen@*
 Copyright @copyright{} 2021 Raghav Gururajan@*

base-commit: 9fd4f4b09cc0495d6b1418f171ff738a1086cc00
-- 
2.34.0

[bug#53144] [PATCH 02/13] gnu: minetest-ethereal: Follow new versioning conventions.

[Maxime Devos]

* gnu/packages/minetest.scm (minetest-ethereal): Use version from ContentDB.
---
 gnu/packages/minetest.scm | 45 +++++++++++++++++----------------------
 1 file changed, 20 insertions(+), 25 deletions(-)

diff --git a/gnu/packages/minetest.scm b/gnu/packages/minetest.scm
index c431942473..56892a5aff 100644
--- a/gnu/packages/minetest.scm
+++ b/gnu/packages/minetest.scm
@@ -266,33 +266,28 @@ (define-public minetest-coloredwood
     (properties `((upstream-name . "VanessaE/coloredwood")))))
 
 (define-public minetest-ethereal
-  ;; ContentDB release 2021-07-28 is slightly ahead of the
-  ;; initial version 1.29 -- i.e., some released changes have been
-  ;; made to version 1.29 without a corresponding version bump.
-  (let ((commit "7670c1da9274901f57f6682384af2b3bae005a86")
-        (revision "0"))
-    (package
-      (name "minetest-ethereal")
-      (version (git-version "1.29" revision commit))
-      (source
-       (origin
-         (method git-fetch)
-         (uri (git-reference
-               (url "https://notabug.org/TenPlus1/ethereal")
-               (commit commit)))
-         (sha256
-          (base32 "1hal8bq4fydsip7s8rqz4vlaaqy9rhzxmryd0j2qnqm9286yjgkk"))
-         (file-name (git-file-name name version))))
-      (build-system minetest-mod-build-system)
-      (home-page (minetest-topic 14638))
-      (synopsis "The Ethereal mod adds many new biomes to Minetest")
-      (description
-       "The Ethereal Minetest mod uses the v7 map generator to add many new
+  (package
+    (name "minetest-ethereal")
+    (version "2021-07-28")
+    (source
+     (origin
+       (method git-fetch)
+       (uri (git-reference
+             (url "https://notabug.org/TenPlus1/ethereal")
+             (commit "7670c1da9274901f57f6682384af2b3bae005a86")))
+       (sha256
+        (base32 "1hal8bq4fydsip7s8rqz4vlaaqy9rhzxmryd0j2qnqm9286yjgkk"))
+       (file-name (git-file-name name version))))
+    (build-system minetest-mod-build-system)
+    (home-page (minetest-topic 14638))
+    (synopsis "The Ethereal mod adds many new biomes to Minetest")
+    (description
+     "The Ethereal Minetest mod uses the v7 map generator to add many new
 biomes to the world.  It adds new trees, plants, food items, tweaks and some
 special items, intending to make an interesting adventure.")
-      ;; CC0: some textures
-      (license (list license:cc0 license:expat))
-      (properties `((upstream-name . "TenPlus1/ethereal"))))))
+    ;; CC0: some textures
+    (license (list license:cc0 license:expat))
+    (properties `((upstream-name . "TenPlus1/ethereal")))))
 
 (define-public minetest-homedecor-modpack
   (package
-- 
2.34.0

[bug#53144] [PATCH 03/13] gnu: minetest-mesecons: Follow new versioning conventions.

[Maxime Devos]

* gnu/packages/minetest.scm (minetest-mesecons): Use version from ContentDB.
---
 gnu/packages/minetest.scm | 53 ++++++++++++++++++---------------------
 1 file changed, 25 insertions(+), 28 deletions(-)

diff --git a/gnu/packages/minetest.scm b/gnu/packages/minetest.scm
index 56892a5aff..fd68877330 100644
--- a/gnu/packages/minetest.scm
+++ b/gnu/packages/minetest.scm
@@ -320,40 +320,37 @@ (define-public minetest-homedecor-modpack
     (properties `((upstream-name . "VanessaE/homedecor_modpack")))))
 
 (define-public minetest-mesecons
-  ;; The release on ContentDB does not have its own version number.
-  (let ((commit "db5879706d04d3480bc4863ce0c03fa73e5f10c7")
-        (revision "0"))
-    (package
-      (name "minetest-mesecons")
-      (version (git-version "1.2.1" revision commit))
-      (source
-       (origin
-         (method git-fetch)
-         (uri (git-reference
-               (url "https://github.com/minetest-mods/mesecons")
-               (commit commit)))
-         (sha256
-          (base32 "04m9s9l3frw1lgki41hgvjsw2zkrvfv0sy750b6j12arzb3lv645"))
-         (file-name (git-file-name name version))))
-      (build-system minetest-mod-build-system)
-      (home-page "https://mesecons.net")
-      (synopsis
-       "Digital circuitry for Minetest, including wires, buttons and lights")
-      (description
-       "Mesecons is a mod for Minetest implementing various items related
+  (package
+    (name "minetest-mesecons")
+    (version "2021-07-25")
+    (source
+     (origin
+       (method git-fetch)
+       (uri (git-reference
+             (url "https://github.com/minetest-mods/mesecons")
+             (commit "db5879706d04d3480bc4863ce0c03fa73e5f10c7")))
+       (sha256
+        (base32 "04m9s9l3frw1lgki41hgvjsw2zkrvfv0sy750b6j12arzb3lv645"))
+       (file-name (git-file-name name version))))
+    (build-system minetest-mod-build-system)
+    (home-page "https://mesecons.net")
+    (synopsis
+     "Digital circuitry for Minetest, including wires, buttons and lights")
+    (description
+     "Mesecons is a mod for Minetest implementing various items related
 to digital circuitry, such as wires, buttons, lights and programmable
 controllers.  Among other things, there are also pistons, solar panels,
 pressure plates and note blocks.
 
 Mesecons has a similar goal to Redstone in Minecraft, but works in its own way,
 with different rules and mechanics.")
-      ;; LGPL for code, CC-BY-SA for textures.
-      ;; The README.md and COPYING.txt disagree about the "+" in license:lgpl3+.
-      ;; For now, assume README.md is correct.  Upstream has been asked to
-      ;; correct the inconsistency:
-      ;; <https://github.com/minetest-mods/mesecons/issues/575>.
-      (license (list license:lgpl3+ license:cc-by-sa3.0))
-      (properties `((upstream-name . "Jeija/mesecons"))))))
+    ;; LGPL for code, CC-BY-SA for textures.
+    ;; The README.md and COPYING.txt disagree about the "+" in license:lgpl3+.
+    ;; For now, assume README.md is correct.  Upstream has been asked to
+    ;; correct the inconsistency:
+    ;; <https://github.com/minetest-mods/mesecons/issues/575>.
+    (license (list license:lgpl3+ license:cc-by-sa3.0))
+    (properties `((upstream-name . "Jeija/mesecons")))))
 
 (define-public minetest-mineclone
   (package
-- 
2.34.0

[bug#53144] [PATCH 04/13] gnu: minetest-throwing: Follow new versioning conventions.

[Maxime Devos]

* gnu/packages/minetest.scm (minetest-throwing): Use version from ContentDB.
---
 gnu/packages/minetest.scm | 42 ++++++++++++++++++---------------------
 1 file changed, 19 insertions(+), 23 deletions(-)

diff --git a/gnu/packages/minetest.scm b/gnu/packages/minetest.scm
index fd68877330..cdf862e189 100644
--- a/gnu/packages/minetest.scm
+++ b/gnu/packages/minetest.scm
@@ -498,31 +498,27 @@ (define-public minetest-technic
     (properties `((upstream-name . "RealBadAngel/technic")))))
 
 (define-public minetest-throwing
-  ;; The latest release on ContentDB is ahead of the latet
-  ;; tagged commit.
-  (let ((commit "31f0cf5f868673dc82f24ddc432b45c9cd282d27")
-        (revision "0"))
-    (package
-      (name "minetest-throwing")
-      (version (git-version "1.1" revision commit))
-      (source
-       (origin
-         (method git-fetch)
-         (uri (git-reference
-               (url "https://github.com/minetest-mods/throwing")
-               (commit commit)))
-         (sha256
-          (base32 "1s5kkr6rxxv2dhbbjzv62gw1s617hnpjavw1v9fv11v3mgigdfjb"))
-         (file-name (git-file-name name version))))
-      (build-system minetest-mod-build-system)
-      (home-page (minetest-topic 16365))
-      (synopsis "API for throwing things in Minetest")
-      (description
-       "This Minetest mod provides an API for registering throwable things and
+  (package
+    (name "minetest-throwing")
+    (version "2020-08-14")
+    (source
+     (origin
+       (method git-fetch)
+       (uri (git-reference
+             (url "https://github.com/minetest-mods/throwing")
+             (commit "31f0cf5f868673dc82f24ddc432b45c9cd282d27")))
+       (sha256
+        (base32 "1s5kkr6rxxv2dhbbjzv62gw1s617hnpjavw1v9fv11v3mgigdfjb"))
+       (file-name (git-file-name name version))))
+    (build-system minetest-mod-build-system)
+    (home-page (minetest-topic 16365))
+    (synopsis "API for throwing things in Minetest")
+    (description
+     "This Minetest mod provides an API for registering throwable things and
 throwing things like arrows.  However, this mod does not provide an actual
 arrow and bow, but @code{minetest-throwing-arrows} does.")
-      (license license:mpl2.0)
-      (properties `((upstream-name . "Palige/throwing"))))))
+    (license license:mpl2.0)
+    (properties `((upstream-name . "Palige/throwing")))))
 
 (define-public minetest-throwing-arrows
   ;; There is only one tagged commit (version 1.1),
-- 
2.34.0

[bug#53144] [PATCH 05/13] gnu: minetest: Remove obsolete comments about version numbers.

[Maxime Devos]

According to the new contributing documentation about Minetest,
the release titles from ContentDB are acceptable.

* gnu/packages/minetest.scm
  (minetest-basic-materials,minetest-coloredwood,minetest-homedecor-modpack)
  (minetest-mobs,minetest-mobs-animal,minetest-pipeworks,minetest-technic):
  Remove comments about version numbers / dates.
---
 gnu/packages/minetest.scm | 13 -------------
 1 file changed, 13 deletions(-)

diff --git a/gnu/packages/minetest.scm b/gnu/packages/minetest.scm
index cdf862e189..8ade7bb966 100644
--- a/gnu/packages/minetest.scm
+++ b/gnu/packages/minetest.scm
@@ -211,7 +211,6 @@ (define-public minetest-moreores
 (define-public minetest-basic-materials
   (package
     (name "minetest-basic-materials")
-    ;; Upstream uses dates as version numbers.
     (version "2021-12-26")
     (source
      (origin
@@ -240,7 +239,6 @@ (define-public minetest-basic-materials
 (define-public minetest-coloredwood
   (package
     (name "minetest-coloredwood")
-    ;; Upstream uses dates as version numbers.
     (version "2021-04-14-1")
     (source
      (origin
@@ -292,8 +290,6 @@ (define-public minetest-ethereal
 (define-public minetest-homedecor-modpack
   (package
     (name "minetest-homedecor-modpack")
-    ;; Upstream doesn't tag releases, so use the release title from
-    ;; ContentDB as version.
     (version "2021-12-26")
     (source
      (origin
@@ -379,8 +375,6 @@ (define-public minetest-mineclone
 (define-public minetest-mobs
   (package
     (name "minetest-mobs")
-    ;; Upstream does not tag release, so use the ContentDB release
-    ;; title instead.
     (version "2021-12-12")
     (source
      (origin
@@ -408,8 +402,6 @@ (define-public minetest-mobs
 (define-public minetest-mobs-animal
   (package
     (name "minetest-mobs-animal")
-    ;; Upstream does not use version numbers, so use the release title
-    ;; from ContentDB instead;
     (version "2021-11-14")
     (source
      (origin
@@ -435,7 +427,6 @@ (define-public minetest-mobs-animal
 (define-public minetest-pipeworks
   (package
     (name "minetest-pipeworks")
-    ;; Upstream uses dates as version numbers.
     (version "2021-04-14-1")
     (source
      (origin
@@ -466,8 +457,6 @@ (define-public minetest-pipeworks
 (define-public minetest-technic
   (package
     (name "minetest-technic")
-    ;; Upstream doesn't keep version numbers, so use the release
-    ;; date on ContentDB instead.
     (version "2021-09-11")
     (source
      (origin
@@ -604,8 +593,6 @@ (define-public minetest-unifieddyes
 (define-public minetest-unified-inventory
   (package
     (name "minetest-unified-inventory")
-    ;; Upstream doesn't keep version numbers, so use the release title
-    ;; on ContentDB instead.
     (version "2021-12-26")
     (source
      (origin
-- 
2.34.0

[bug#53144] [PATCH 06/13] gnu: minetest-ethereal: Update to 2022-01-05.

[Maxime Devos]

* gnu/packages/minetest.scm (minetest-ethereal): Update to 2022-01-05.
---
 gnu/packages/minetest.scm | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/gnu/packages/minetest.scm b/gnu/packages/minetest.scm
index 8ade7bb966..40c658fdb3 100644
--- a/gnu/packages/minetest.scm
+++ b/gnu/packages/minetest.scm
@@ -266,15 +266,15 @@ (define-public minetest-coloredwood
 (define-public minetest-ethereal
   (package
     (name "minetest-ethereal")
-    (version "2021-07-28")
+    (version "2022-01-05")
     (source
      (origin
        (method git-fetch)
        (uri (git-reference
              (url "https://notabug.org/TenPlus1/ethereal")
-             (commit "7670c1da9274901f57f6682384af2b3bae005a86")))
+             (commit "fefa1b76fb04331dbde59dd745d120c1c728e9f7")))
        (sha256
-        (base32 "1hal8bq4fydsip7s8rqz4vlaaqy9rhzxmryd0j2qnqm9286yjgkk"))
+        (base32 "1hs1xrxb99kcrq03yl1bkbr0hnql9n0wblqn5sirs6wb3qshx9zd"))
        (file-name (git-file-name name version))))
     (build-system minetest-mod-build-system)
     (home-page (minetest-topic 14638))
-- 
2.34.0

[bug#53144] [PATCH 07/13] gnu: minetest-mesecons: Update to 2021-11-28.

[Maxime Devos]

* gnu/packages/minetest.scm (minetest-mesecons): Update to 2021-11-28.
---
 gnu/packages/minetest.scm | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/gnu/packages/minetest.scm b/gnu/packages/minetest.scm
index 40c658fdb3..8940672e33 100644
--- a/gnu/packages/minetest.scm
+++ b/gnu/packages/minetest.scm
@@ -318,15 +318,15 @@ (define-public minetest-homedecor-modpack
 (define-public minetest-mesecons
   (package
     (name "minetest-mesecons")
-    (version "2021-07-25")
+    (version "2021-11-28")
     (source
      (origin
        (method git-fetch)
        (uri (git-reference
              (url "https://github.com/minetest-mods/mesecons")
-             (commit "db5879706d04d3480bc4863ce0c03fa73e5f10c7")))
+             (commit "ecea0a2896d488f82f34505eb608ab5f527f50ec")))
        (sha256
-        (base32 "04m9s9l3frw1lgki41hgvjsw2zkrvfv0sy750b6j12arzb3lv645"))
+        (base32 "14v4vfwlk7dg5lk93sjdjg7n8amw0qlcxdn3d0gdfwvx8305vn5r"))
        (file-name (git-file-name name version))))
     (build-system minetest-mod-build-system)
     (home-page "https://mesecons.net")
-- 
2.34.0

[bug#53144] [PATCH 08/13] gnu: minetest-mobs: Update to 2022-01-03.

[Maxime Devos]

* gnu/packages/minetest.scm (minetest-mobs): Update to 2022-01-03.
---
 gnu/packages/minetest.scm | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/gnu/packages/minetest.scm b/gnu/packages/minetest.scm
index 8940672e33..9122e01463 100644
--- a/gnu/packages/minetest.scm
+++ b/gnu/packages/minetest.scm
@@ -375,15 +375,15 @@ (define-public minetest-mineclone
 (define-public minetest-mobs
   (package
     (name "minetest-mobs")
-    (version "2021-12-12")
+    (version "2022-01-03")
     (source
      (origin
        (method git-fetch)
        (uri (git-reference
              (url "https://notabug.org/TenPlus1/mobs_redo")
-             (commit "6a4a02f3fbf1038c69e72aaafa52a1e7d6106da8")))
+             (commit "e5d1958e16fa9df1b0b8cfcd425fb82981a29370")))
        (sha256
-        (base32 "0vgv7jpm9v3dwq4l9jxdd5z14yq164w8kin1d05jfv3ck4hwlwvr"))
+        (base32 "1pj08750m9123z0r5yxwcp5ncdml9c9rfv4sv45rvhq8npk5hvcw"))
        (file-name (git-file-name name version))))
     (build-system minetest-mod-build-system)
     (home-page (minetest-topic 9917))
-- 
2.34.0

[bug#53144] [PATCH 09/13] doc: Document the Minetest importer.

[Maxime Devos]

* doc/guix.texi (Invoking guix refresh): Include the Minetest importer in the
  list.
---
 doc/guix.texi | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/doc/guix.texi b/doc/guix.texi
index 0e19285c1c..5c1b9adb87 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -12575,6 +12575,8 @@ the updater for @uref{https://rubygems.org, RubyGems} packages.
 the updater for @uref{https://github.com, GitHub} packages.
 @item hackage
 the updater for @uref{https://hackage.haskell.org, Hackage} packages.
+@item minetest
+the updater for @uref{https://content.minetest.net, ContentDB} packages.
 @item stackage
 the updater for @uref{https://www.stackage.org, Stackage} packages.
 @item crate
-- 
2.34.0

[bug#53144] [PATCH 10/13] upstream: Support incrementing the revision of 'git-version'.

[Maxime Devos]

This is currently pointless, because no updater returns such versions.
A future patch will introduce an updater returning such versions.

* guix/upstream.scm
  (git-version-regexp): New variable.
  (maybe-git-version, maybe-git-version->revision)
  (maybe-git-versions->revision-replacements): New procedures.
  (update-package-source): Use 'package-definition-location' instead of
  'package-location'.  Also replace the revision.
---
 guix/upstream.scm  | 61 +++++++++++++++++++++++++++++++++++++++++++++-
 tests/upstream.scm | 37 ++++++++++++++++++++++++++++
 2 files changed, 97 insertions(+), 1 deletion(-)

diff --git a/guix/upstream.scm b/guix/upstream.scm
index 6666803a92..6b65147356 100644
--- a/guix/upstream.scm
+++ b/guix/upstream.scm
@@ -61,6 +61,10 @@ (define-module (guix upstream)
             url-prefix-predicate
             coalesce-sources
 
+            decompose-git-version
+            maybe-git-version->revision
+            maybe-git-versions->revision-replacements ; for tests
+
             upstream-updater
             upstream-updater?
             upstream-updater-name
@@ -230,6 +234,55 @@ (define (release>? r1 r2)
         (sort sources release>?)))
 
 \f
+
+
+;;;
+;;; Manipulating results of 'git-version'.
+;;; TODO: also supporting Mercurial ('hg-version') would be nice.
+;;;
+
+;; A regexp matching versions constructed by 'git-version'.
+(define git-version-regexp
+  (delay (make-regexp "^(.+)-([0123456789]+).([0123456789abcdefg]{7})$")))
+
+(define (decompose-git-version git-version-string)
+  "Split the version string GIT-VERSION-STRING constructed by 'git-version'
+in three parts: the version it was based on, the revision (as a string)
+and the abbreviated commit.  If GIT-VERSION-STRING does not correspond
+to a result of 'git-version', return #false (three times) instead."
+  (define m (regexp-exec (force git-version-regexp) git-version-string))
+  (if m
+      (values (match:substring m 1)
+              (match:substring m 2)
+              (match:substring m 3))
+      (values #false #false #false)))
+
+(define (maybe-git-version->revision maybe-git-version) ; string | #false
+  "If the string MAYBE-GIT-VERSION appears to be the result of a call to
+'git-version', return the revision (as a string).  Otherwise, return #false."
+  (let-values (((version-base revision abbreviated-commit)
+                (decompose-git-version maybe-git-version)))
+    revision))
+
+(define (maybe-git-versions->revision-replacements old new)
+  "If the two strings OLD and NEW appear to be the result of a call
+to 'git-version', return a list of replacements as expected by
+'update-expression' in 'update-package-source' for updating the revision.
+Otherwise, return the empty list."
+  (let* ((old-revision (maybe-git-version->revision old))
+         (new-revision (maybe-git-version->revision new)))
+    (if (and old-revision new-revision)
+        ;; Simply returning ((old-revision . new-revision)) would work.
+        ;; However, revision numbers are usually quite small,
+        ;; e.g. "0" or "1", so that would have a high risk of replacing
+        ;; something unrelated.  Instead, target the (revision ...) form
+        ;; in (let ((commit ...) (revision ...)) (package ...)).
+        `((,(object->string `(revision ,old-revision))
+           . ,(object->string `(revision ,new-revision))))
+        '())))
+
+\f
+
 ;;;
 ;;; Auto-update.
 ;;;
@@ -535,7 +588,11 @@ (define (update-expression expr replacements)
         (version     (upstream-source-version source))
         (version-loc (package-field-location package 'version)))
     (if version-loc
-        (let* ((loc         (package-location package))
+        ;; Use 'package-definition-location' instead of 'package-location'
+        ;; such that the commit and revision in
+        ;; (let ((commit ...) (revision ...)) (package ...)) forms can
+        ;; be updated.
+        (let* ((loc         (package-definition-location package))
                (old-version (package-version package))
                (old-hash    (content-hash-value
                              (origin-hash (package-source package))))
@@ -570,6 +627,8 @@ (define (update-expression expr replacements)
                                            'filename file))
                     (replacements `((,old-version . ,version)
                                     (,old-hash . ,hash)
+                                    ,@(maybe-git-versions->revision-replacements
+                                       old-version version)
                                     ,@(if (and old-commit new-commit)
                                           `((,old-commit . ,new-commit))
                                           '())
diff --git a/tests/upstream.scm b/tests/upstream.scm
index 9aacb77229..0b14b9867f 100644
--- a/tests/upstream.scm
+++ b/tests/upstream.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2016 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2022 Ricardo Wurmus <rekado@elephly.net>
+;;; Copyright © 2022 Maxime Devos <maximedevos@telenet.be>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -22,6 +23,7 @@ (define-module (test-upstream)
   #:use-module (guix download)
   #:use-module (guix packages)
   #:use-module (guix build-system gnu)
+  #:use-module (guix git-download)
   #:use-module (guix import print)
   #:use-module ((guix licenses) #:prefix license:)
   #:use-module (guix upstream)
@@ -210,4 +212,39 @@ (define test-new-package-sexp
                     '("hello" "sed" "tar" "grep"))))
       (else (pk else #false)))))
 
+(define (decompose-git-version* str)
+  (call-with-values (lambda () (decompose-git-version str)) list))
+
+(test-equal "decompose-git-version returns arguments if commit is short"
+  '("1.2.3" "900" "cabba9e")
+  (decompose-git-version* "1.2.3-900.cabba9e"))
+
+(test-equal "decompose-git-version handles - in versions"
+  '("1.2.3-rc0" "123" "ba99a9e")
+  (decompose-git-version* "1.2.3-rc0-123.ba99a9e"))
+
+(test-equal "decompose-git-version returns #false if not a git-version result"
+  '(#false #false #false)
+  (decompose-git-version* "1.2.3-rc0.ba99a9e"))
+
+(test-equal "maybe-git-version->revision returns the revision"
+  "12"
+  (maybe-git-version->revision "1.2.3-12.ba99a9e"))
+
+(test-equal "maybe-git-version->revision returns #false if not a git-version"
+  #false
+  (maybe-git-version->revision "1.2.3-12.nope"))
+
+(test-equal "maybe-git-version->revision-replacement can return ()"
+  '(() () ())
+  (map maybe-git-versions->revision-replacements
+       '("1.2.3" "1.2.3" "1.2.3-21.cabba9e")
+       '("1.2.3" "1.2.3-21.cabba9e" "1.2.3")))
+
+(test-equal "maybe-git-version->revision-replacement with git-version"
+  '(("(revision \"0\")" . "(revision \"1\")"))
+  (maybe-git-versions->revision-replacements
+   (git-version "1.2.3" "0" "cabba9e")
+   (git-version "1.2.3" "1" "ba99age")))
+
 (test-end)
-- 
2.34.0

[bug#53144] [PATCH 0/13] Make more git-using packages auto-updatable

[Ludovic Courtès]

Maxime Devos <maximedevos@telenet.be> skribis:

> This is currently pointless, because no updater returns such versions.

s/pointless/unused/  :-)

> A future patch will introduce an updater returning such versions.
>
> * guix/upstream.scm
>   (git-version-regexp): New variable.
>   (maybe-git-version, maybe-git-version->revision)
>   (maybe-git-versions->revision-replacements): New procedures.
>   (update-package-source): Use 'package-definition-location' instead of
>   'package-location'.  Also replace the revision.

Please mention tests/upstream.scm too.

Otherwise LGTM.

Ludo’.

[bug#53144] [PATCH 11/13] git: Support resolving references without cloning.

[Maxime Devos]

* guix/git.scm (remote-refs): Split off some logic to ...
  (call-with-detached-remote): ... this new procedure.
  (lookup-reference): New procedure.
* tests/git.scm ("lookup-reference: branch and HEAD"): New test.
---
 guix/git.scm  | 57 +++++++++++++++++++++++++++++++++++++++------------
 tests/git.scm | 22 +++++++++++++++++++-
 2 files changed, 65 insertions(+), 14 deletions(-)

diff --git a/guix/git.scm b/guix/git.scm
index 43e85a5026..1c07eba584 100644
--- a/guix/git.scm
+++ b/guix/git.scm
@@ -62,6 +62,7 @@ (define-module (guix git)
             commit-relation
 
             remote-refs
+            lookup-reference
 
             git-checkout
             git-checkout?
@@ -628,6 +629,22 @@ (define (commit-relation old new)
 ;;; Remote operations.
 ;;;
 
+;; TODO: it would be nice to use 'remote-create-detached' here,
+;; but that procedure isn't in any released version of guile-git yet.
+(define (call-with-detached-remote url proc)
+  "Call PROC with a remote for URL.  The remote is closed after PROC returns."
+  (call-with-temporary-directory
+   (lambda (cache-directory)
+     (let* ((repository (repository-init cache-directory))
+            ;; Create an in-memory remote so we don't touch disk.
+            (remote (remote-create-anonymous repository url)))
+       (remote-connect remote)
+       (let-values ((result (proc remote)))
+         ;; Wait until we're finished with the repository before closing it.
+         (remote-disconnect remote)
+         (repository-close! repository)
+         (apply values result))))))
+
 (define* (remote-refs url #:key tags?)
   "Return the list of references advertised at Git repository URL.  If TAGS?
 is true, limit to only refs/tags."
@@ -649,19 +666,33 @@ (define (remote-head->ref remote)
            name)))
 
   (with-libgit2
-   (call-with-temporary-directory
-    (lambda (cache-directory)
-      (let* ((repository (repository-init cache-directory))
-             ;; Create an in-memory remote so we don't touch disk.
-             (remote (remote-create-anonymous repository url)))
-        (remote-connect remote)
-
-        (let* ((remote-heads (remote-ls remote))
-               (refs (filter-map remote-head->ref remote-heads)))
-          ;; Wait until we're finished with the repository before closing it.
-          (remote-disconnect remote)
-          (repository-close! repository)
-          refs))))))
+   (call-with-detached-remote
+    url
+    (lambda (remote)
+      (define remote-heads (remote-ls remote))
+      (filter-map remote-head->ref remote-heads)))))
+
+(define* (lookup-reference url reference-name)
+  "Lookup the reference named REFERENCE-NAME advertised at the Git repository
+at URL and return the commit string.  If the reference was not found, return
+#false instead."
+  (define (oid->commit oid)
+    (define str (oid->string oid))
+    ;; FIXME: why is the result of oid->string prefixed by 8 zeroes
+    ;; when remote-ls is used?  To make hash collisions harder, it would
+    ;; be nice if the commit was not abbreviated.
+    (if (string-prefix? "00000000" str)
+        (string-drop str 8)
+        str))
+  (define (match? remote-head)
+    (string=? reference-name (remote-head-name remote-head)))
+  (with-libgit2
+   (call-with-detached-remote
+    url
+    (lambda (remote)
+      (define remote-heads (remote-ls remote))
+      (define head (find match? remote-heads))
+      (and=> head (compose oid->commit remote-head-oid))))))
 
 \f
 ;;;
diff --git a/tests/git.scm b/tests/git.scm
index d0646bbc85..f8eaf9e93b 100644
--- a/tests/git.scm
+++ b/tests/git.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2019, 2020 Ludovic Courtès <ludo@gnu.org>
-;;; Copyright © 2021 Xinglu Chen <public@yoctocell.xyz
+;;; Copyright © 2021 Xinglu Chen <public@yoctocell.xyz>
+;;; Copyright © 2022 Maxime Devos <maximedevos@telenet.be>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -189,4 +190,23 @@ (define-module (test-git)
         (tag "v1.1" "Release 1.1"))
     (remote-refs directory #:tags? #t)))
 
+(test-equal "lookup-reference: branch and HEAD"
+  '(#true #true)
+  (with-temporary-git-repository directory
+      '((add "a.txt" "A")
+        (commit "First commit")
+        (branch "a-branch")
+        (add "b.txt" "B")
+        (commit "Second commit"))
+    (with-repository directory repository
+      ;; See 'oid->commit' in (guix git) for why not string=?.
+      (list (string-prefix?
+             (lookup-reference directory "refs/heads/a-branch")
+             (oid->string
+              (commit-id (find-commit repository "First commit"))))
+            (string-prefix?
+             (lookup-reference directory "HEAD")
+             (oid->string
+              (commit-id (find-commit repository "Second commit"))))))))
+
 (test-end "git")
-- 
2.34.0

[bug#53144] [PATCH 0/13] Make more git-using packages auto-updatable

[Ludovic Courtès]

Maxime Devos <maximedevos@telenet.be> skribis:

> * guix/git.scm (remote-refs): Split off some logic to ...
>   (call-with-detached-remote): ... this new procedure.
>   (lookup-reference): New procedure.
> * tests/git.scm ("lookup-reference: branch and HEAD"): New test.

[...]

> +;; TODO: it would be nice to use 'remote-create-detached' here,
> +;; but that procedure isn't in any released version of guile-git yet.
> +(define (call-with-detached-remote url proc)

Let’s prepare a new Guile-Git release then (Erik Edrosa has been MIA
lately, but Marius, Mathieu, or myself can tag a release when needed).

> +  "Call PROC with a remote for URL.  The remote is closed after PROC returns."
> +  (call-with-temporary-directory
> +   (lambda (cache-directory)
> +     (let* ((repository (repository-init cache-directory))
> +            ;; Create an in-memory remote so we don't touch disk.
> +            (remote (remote-create-anonymous repository url)))
> +       (remote-connect remote)
> +       (let-values ((result (proc remote)))

s/let-values/let/

> +         ;; Wait until we're finished with the repository before closing it.
> +         (remote-disconnect remote)
> +         (repository-close! repository)
> +         (apply values result))))))

We can use this code until Guile-Git provides ‘remote-create-detached’
though.

Otherwise LGTM!

Ludo’.

[bug#53144] [PATCH 0/13] Make more git-using packages auto-updatable

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 723 bytes --]

Ludovic Courtès schreef op di 18-01-2022 om 18:39 [+0100]:
> > +;; TODO: it would be nice to use 'remote-create-detached' here,
> > +;; but that procedure isn't in any released version of guile-git
> > yet.
> > +(define (call-with-detached-remote url proc)
> 
> Let’s prepare a new Guile-Git release then (Erik Edrosa has been MIA
> lately, but Marius, Mathieu, or myself can tag a release when
> needed).

Looks like 'remote-create-detached' doesn't work yet (remote-create-
detached+remote-connect segfaults:
<https://gitlab.com/guile-git/guile-git/-/issues/24>).  To investigate
the issue, I have written a patch updating libgit2 to 1.4.2:
<https://issues.guix.gnu.org/54611#2>.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#53144] [PATCH 12/13] import: Add 'latest-git' updater.

[Maxime Devos]

* Makefile.am (MODULES, SCM_TESTS): Register new files.
* doc/guix.texi (Invoking guix refresh): Document it.
* guix/import/latest-git.scm: New importer file.
* guix/upstream.scm (increment-git-version): New procedure.
* tests/import-latest-git.scm: New test file.
---
 Makefile.am                 |   2 +
 doc/guix.texi               |  17 +++
 guix/import/latest-git.scm  | 104 ++++++++++++++++++
 guix/upstream.scm           |   9 ++
 tests/import-latest-git.scm | 204 ++++++++++++++++++++++++++++++++++++
 5 files changed, 336 insertions(+)
 create mode 100644 guix/import/latest-git.scm
 create mode 100644 tests/import-latest-git.scm

diff --git a/Makefile.am b/Makefile.am
index d6aabac261..e380c7c83d 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -267,6 +267,7 @@ MODULES =					\
   guix/import/json.scm				\
   guix/import/kde.scm				\
   guix/import/launchpad.scm   			\
+  guix/import/latest-git.scm			\
   guix/import/minetest.scm   			\
   guix/import/opam.scm				\
   guix/import/print.scm				\
@@ -482,6 +483,7 @@ SCM_TESTS =					\
   tests/hackage.scm				\
   tests/home-import.scm				\
   tests/import-git.scm				\
+  tests/import-latest-git.scm			\
   tests/import-utils.scm			\
   tests/inferior.scm				\
   tests/lint.scm				\
diff --git a/doc/guix.texi b/doc/guix.texi
index 5c1b9adb87..58ccc75ccf 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -12619,6 +12619,23 @@ property to @code{#t}.
       (release-tag-version-delimiter . ":"))))
 @end lisp
 
+@item latest-git
+@cindex latest-git
+@cindex with-latest-git-commit
+another updater for packages hosted on Git repositories.  The difference
+with @code{generic-git} is that it always choses the latest commit, even
+when it does not have a version tag.  As this practice should remain
+exceptional (@pxref{Version Numbers}), packages have to opt-in this
+updater, by using @code{git-version} to construct the version number and
+setting the @code{with-latest-git-commit} package property.
+
+Usually, it can be simply be set to @code{#true} to use the latest Git
+commit on the default branch---i.e., HEAD in Git parlance.  If this is
+not desired, for example if upstream has a branch that is considered
+‘stable’, it can be set to the name of a reference to take commits from.
+For example, to take commits from a branch named @code{stable}, the
+property @code{with-latest-git-commit} needs to be set to
+@code{refs/heads/stable}.
 
 @end table
 
diff --git a/guix/import/latest-git.scm b/guix/import/latest-git.scm
new file mode 100644
index 0000000000..208f112153
--- /dev/null
+++ b/guix/import/latest-git.scm
@@ -0,0 +1,104 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Maxime Devos <maximedevos@telenet.be>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import latest-git)
+  #:use-module (guix packages)
+  #:use-module (guix upstream)
+  #:use-module (guix ui)
+  #:use-module (guix git)
+  #:use-module (guix git-download)
+  #:use-module (ice-9 match)
+  #:use-module (srfi srfi-26)
+  #:export (%latest-git-updater))
+
+(define (check-valid-with-latest-git-commit? package value)
+  "Verify that VALUE is a valid value for the 'with-latest-git-commit'
+package property of PACKAGE.  If so, return #true.  Otherwise, emit a
+warning and return #false.  It is assumed VALUE is not false."
+  (or (string? value)
+      (eq? #true value)
+      (begin
+        (warning (or (package-field-location package 'properties)
+                     (package-location package))
+                 (G_ "Package ~a has an invalid 'with-latest-git-commit' \
+property.~%")
+                 (package-name package))
+        #false)))
+
+(define (with-latest-git-commit? package)
+  "Return true if PACKAGE is hosted on a Git repository and it is requested
+that the latest Git commit is used even when not formally released."
+  (match (package-source package)
+    ((? origin? origin)
+     (and (decompose-git-version (package-version package))
+          (eq? (origin-method origin) git-fetch)
+          (git-reference? (origin-uri origin))
+          (and=> (assq-ref (package-properties package)
+                           'with-latest-git-commit)
+                 (cut check-valid-with-latest-git-commit? package <>))))
+    (_ #f)))
+
+(define (latest-commit-reference-name package)
+  "Return the name of the reference that is expected to hold the latest Git
+commit to use as source code."
+  (match (assq-ref (package-properties package) 'with-latest-git-commit)
+    ('#true "HEAD")
+    ((? string? reference) reference)))
+
+(define (latest-git-upstream package)
+  "Return an <upstream-source> for the latest git commit of PACKAGE.
+If the reference pointing to the latest git commit has been deleted,
+return #false instead."
+  (let* ((name (package-name package))
+         (old-version (package-version package))
+         (old-reference (origin-uri (package-source package)))
+         (reference-name (latest-commit-reference-name package))
+         (commit (lookup-reference (git-reference-url old-reference)
+                                   reference-name)))
+    (if commit
+        (upstream-source
+         (package name)
+         (version
+          ;; See 'oid->commit' in (guix git) for why not string=?.
+          ;; Don't increment the revision if the commit remains the same.
+          (if (string-prefix? commit (git-reference-commit old-reference))
+              old-version
+              (increment-git-version old-version commit)))
+         (urls (git-reference
+                (inherit old-reference)
+                (commit commit))))
+        (begin
+          (warning (package-location package)
+                   (G_ "Cannot update ~a because the reference ~a of ~a has \
+disappeared.~%")
+                   (package-name package)
+                   reference-name
+                   (let ((maybe-hyperlink
+                          (if (supports-hyperlinks? (guix-warning-port))
+                              hyperlink
+                              (lambda (x y) x)))
+                         (url (git-reference-url old-reference)))
+                     (maybe-hyperlink url url)))
+          #false))))
+
+(define %latest-git-updater
+  (upstream-updater
+   (name 'latest-git)
+   (description "Updater for packages using latest Git commit")
+   (pred with-latest-git-commit?)
+   (latest latest-git-upstream)))
diff --git a/guix/upstream.scm b/guix/upstream.scm
index 6b65147356..a9211fe45b 100644
--- a/guix/upstream.scm
+++ b/guix/upstream.scm
@@ -64,6 +64,7 @@ (define-module (guix upstream)
             decompose-git-version
             maybe-git-version->revision
             maybe-git-versions->revision-replacements ; for tests
+            increment-git-version
 
             upstream-updater
             upstream-updater?
@@ -281,6 +282,14 @@ (define (maybe-git-versions->revision-replacements old new)
            . ,(object->string `(revision ,new-revision))))
         '())))
 
+(define (increment-git-version old-git-version commit)
+  "Increment the revision in OLD-GIT-VERSION by one, replacing the commit
+by COMMIT.  It is assumed OLD-GIT-VERSION is a result of 'git-version'."
+  (let-values (((old-base-version revision old-commit)
+                (decompose-git-version old-git-version)))
+    (git-version old-base-version
+                 (number->string (+ 1 (string->number revision))) commit)))
+
 \f
 
 ;;;
diff --git a/tests/import-latest-git.scm b/tests/import-latest-git.scm
new file mode 100644
index 0000000000..d0dc149ff8
--- /dev/null
+++ b/tests/import-latest-git.scm
@@ -0,0 +1,204 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Maxime Devos <maximedevos@telenet.be>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (test-import-latest-git)
+  #:use-module (git)
+  #:use-module (guix git)
+  #:use-module (guix tests)
+  #:use-module (guix packages)
+  #:use-module (guix import latest-git)
+  #:use-module (guix upstream)
+  #:use-module (guix git-download)
+  #:use-module (guix hg-download)
+  #:use-module (guix tests git)
+  #:use-module (guix build utils)
+  #:use-module (srfi srfi-64))
+
+(test-begin "git")
+
+(define latest-git-upstream
+  (upstream-updater-latest %latest-git-updater))
+
+(define with-latest-git-commit?
+  (upstream-updater-predicate %latest-git-updater))
+
+(define* (make-package directory base-version revision commit
+                       #:optional (properties
+                                   '((with-latest-git-commit . #true))))
+  (dummy-package "test-package"
+                 (version (git-version base-version revision commit))
+                 (source
+                  (origin
+                    (method git-fetch)
+                    (uri (git-reference
+                          (url (string-append "file://" directory))
+                          (commit commit)))
+                    (sha256 #f)))
+                 (properties properties)))
+
+(define (find-commit-as-string repository query)
+  (oid->string (commit-id (find-commit repository query))))
+
+(unless (which (git-command)) (test-skip 1))
+(test-equal "latest-git: an update"
+  '(#true #true #true)
+  (with-temporary-git-repository directory
+      '((add "a.txt" "A")
+        (commit "First commit")
+        (add "b.txt" "B")
+        (commit "Second commit"))
+    (with-repository directory repository
+      (let* ((old-commit
+              (find-commit-as-string repository "First commit"))
+             (new-commit
+              (find-commit-as-string repository "Second commit"))
+             (package (make-package directory "1.0" "0" old-commit))
+             (update (latest-git-upstream package)))
+        (list (with-latest-git-commit? package)
+              (string=? (upstream-source-version update)
+                        (git-version "1.0" "1" new-commit))
+              ;; See 'oid->commit in (guix git) for why not string=?.
+              (string-prefix?
+               (git-reference-commit (upstream-source-urls update))
+               new-commit))))))
+
+(unless (which (git-command)) (test-skip 1))
+(test-equal "latest-git: no new commit, no new revision"
+  '(#true #true #true)
+  (with-temporary-git-repository directory
+      '((add "a.txt" "A")
+        (commit "First commit"))
+    (with-repository directory repository
+      (let* ((commit
+              (find-commit-as-string repository "First commit"))
+             (package (make-package directory "1.0" "0" commit))
+             (update (latest-git-upstream package)))
+        ;; 'update' being #false would work as well.
+        (list (with-latest-git-commit? package)
+              (string=? (upstream-source-version update)
+                        (package-version package))
+              (string-prefix?
+               (git-reference-commit (upstream-source-urls update))
+               commit))))))
+
+(unless (which (git-command)) (test-skip 1))
+(test-equal "latest-git: non-HEAD commits ignored"
+  '(#true #true #true)
+  (with-temporary-git-repository directory
+      '((add "a.txt" "A")
+        (commit "First commit")
+        (tag "let-me-be-head")
+        (branch "dev")
+        (checkout "dev")
+        (add "b.txt" "B")
+        (commit "Not ready for distribution!")
+        (checkout "let-me-be-head"))
+    (with-repository directory repository
+      (let* ((commit
+              (find-commit-as-string repository "First commit"))
+             (package (make-package directory "1.0" "0" commit))
+             (update (latest-git-upstream package)))
+        (list (with-latest-git-commit? package)
+              (string=? (upstream-source-version update)
+                        (package-version package))
+              (string-prefix?
+               (git-reference-commit (upstream-source-urls update))
+               commit))))))
+
+(unless (which (git-command)) (test-skip 1))
+(test-equal "latest-git: non-HEAD branches can be chosen"
+  '(#true #true #true)
+  (with-temporary-git-repository directory
+      '((checkout "stable-for-distros" orphan)
+        (add "a.txt" "A")
+        (commit "First commit")
+        (add "b.txt" "B")
+        (commit "Here's a bugfix.")
+        (branch "unstable")
+        (checkout "unstable")
+        (add "c.txt" "C")
+        ;; This commit may not be chosen.
+        (commit "New feature, needs more work before distributing."))
+    (with-repository directory repository
+      (let* ((old-commit
+              (find-commit-as-string repository "First commit"))
+             (new-commit
+              (find-commit-as-string repository "Here's a bugfix"))
+             (properties
+              '((with-latest-git-commit . "refs/heads/stable-for-distros")))
+             (package (make-package directory "1.0" "0" old-commit properties))
+             (update (latest-git-upstream package)))
+        (list (with-latest-git-commit? package)
+              (string=? (upstream-source-version update)
+                        (git-version "1.0" "1" new-commit))
+              (string-prefix?
+               (git-reference-commit (upstream-source-urls update))
+               new-commit))))))
+
+(unless (which (git-command)) (test-skip 1))
+(test-equal "latest-git: deleted references handled gracefully"
+  #false
+  (with-temporary-git-repository directory
+      '((add "a.txt" "A")
+        (commit "First commit"))
+    (with-repository directory repository
+      (let* ((properties
+              '((with-latest-git-commit . "refs/heads/I-do-not-exist")))
+             (package (make-package directory "1.0" "0" "cabba9e" properties)))
+        (latest-git-upstream package)))))
+
+(test-equal "with-latest-git-commit?"
+  '(#true #false #true #true #false #false)
+  (map (lambda (properties)
+         (with-latest-git-commit?
+          (make-package "/dev/null" "1.0" "0" "cabba9e" properties)))
+       (list '((with-latest-git-commit . #true)) ; defaults to HEAD
+             '() ; packages have to opt-in, so #false
+             '((with-latest-git-commit . "HEAD")) ; explicit HEAD is ok
+             '((with-latest-git-commit . "refs/heads/main")) ; another branch
+             '((with-latest-git-commit . #xf00ba3)) ; bogus
+             '((irrelevant . #true)))))
+
+(test-equal "with-latest-git-commit?: not for other VCS"
+  #false
+  (with-latest-git-commit?
+   (package
+     (inherit (make-package "/dev/null" "1.0.0" "0" "cabba9e"))
+     (source
+      (origin
+        (method hg-fetch)
+        (uri (hg-reference
+              (url "https://foo")
+              (changeset "foo")))
+        (sha256 #false))))))
+
+(test-equal "with-latest-git-commit?: only if there's source code"
+  #false
+  (with-latest-git-commit?
+   (package
+     (inherit (make-package "/dev/null" "1.0.0" "0" "cabba9e"))
+     (source #false))))
+
+(test-equal "with-latest-git-commit?: only for git-version"
+  #false
+  (with-latest-git-commit?
+   (package
+     (inherit (make-package "/dev/null" "1.0.0" "0" "cabba9e"))
+     (version "1.0.0"))))
+
+(test-end "git")
-- 
2.34.0

[bug#53144] [PATCH 0/13] Make more git-using packages auto-updatable

[Ludovic Courtès]

Maxime Devos <maximedevos@telenet.be> skribis:

> * Makefile.am (MODULES, SCM_TESTS): Register new files.
> * doc/guix.texi (Invoking guix refresh): Document it.
> * guix/import/latest-git.scm: New importer file.
> * guix/upstream.scm (increment-git-version): New procedure.
> * tests/import-latest-git.scm: New test file.

[...]

> +@item latest-git
> +@cindex latest-git
> +@cindex with-latest-git-commit
> +another updater for packages hosted on Git repositories.  The difference
> +with @code{generic-git} is that it always choses the latest commit, even

“chooses”

> +when it does not have a version tag.  As this practice should remain
> +exceptional (@pxref{Version Numbers}), packages have to opt-in this
> +updater, by using @code{git-version} to construct the version number and
> +setting the @code{with-latest-git-commit} package property.

Instead of a new updater, should it be handled by ‘generic-git’?  It
could honor the property just as well, no?

Now in terms of code it does look nice to have it separate.  WDYT?

I’d call the property ‘release-from-git-reference’ or something, to
avoid confusion with the ‘--with-commit’ transformation option.

Thanks,
Ludo’.

[bug#53144] [PATCH 13/13] gnu: minetest-throwing-arrows: Use 'latest-git' updater.

[Maxime Devos]

* gnu/packages/minetest.scm
  (minetest-throwing-arrows)[properties]: New field.
---
 gnu/packages/minetest.scm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/minetest.scm b/gnu/packages/minetest.scm
index 9122e01463..9270acc6e0 100644
--- a/gnu/packages/minetest.scm
+++ b/gnu/packages/minetest.scm
@@ -537,7 +537,8 @@ (define-public minetest-throwing-arrows
        ;; not be translated.
        "This mod adds arrows and bows to Minetest.  It is a compatible
 replacement for the throwing mod by PilzAdam that uses the throwing API.")
-      (license license:mpl2.0))))
+      (license license:mpl2.0)
+      (properties '((with-latest-git-commit . #true))))))
 
 (define-public minetest-worldedit
   (package
-- 
2.34.0

[bug#53144] [PATCH 01/13] doc: Give some tips on Minetest packaging.

[Liliana Marie Prikler]

Am Sonntag, dem 09.01.2022 um 19:10 +0000 schrieb Maxime Devos:
> * doc/contributing.texi (Minetest Packages): New section.
> * doc/guix.texi: Copyright update.
> ---
>  doc/contributing.texi | 42
> ++++++++++++++++++++++++++++++++++++++++++
>  doc/guix.texi         |  2 +-
>  2 files changed, 43 insertions(+), 1 deletion(-)
> 
> diff --git a/doc/contributing.texi b/doc/contributing.texi
> index 72f5ce1e0e..5b91fc7867 100644
> --- a/doc/contributing.texi
> +++ b/doc/contributing.texi
> @@ -394,6 +394,7 @@ needed is to review and apply the patch.
>  * Synopses and Descriptions::   Helping users find the right
> package.
>  * Snippets versus Phases::      Whether to use a snippet, or a build
> phase.
>  * Emacs Packages::              Your Elisp fix.
> +* Minetest Packages::           Building blocks.
>  * Python Modules::              A touch of British comedy.
>  * Perl Modules::                Little pearls.
>  * Java Packages::               Coffee break.
> @@ -703,6 +704,47 @@ When encountering problems, it is wise to check
> for the presence of the
>  file, and whether any dependencies and their versions listed therein
> are
>  satisfied.
>  
> +@node Minetest Packages
> +@subsection Minetest Packages
> +@cindex minetest, packaging
> +
> +A Minetest mod @code{foo} is named @code{minetest-foo} -- the author
> +name from ContentDB is not included, unless required to resolve a
> name
> +collision.
> +
> +Sometimes, it might be unclear what the version of a Minetest mod
> is.
> +For example, ContentDB and the importer reports 2020-01-01, but
> +according to the forums the version is 2.1.  Usually, in these cases
> the
> +version on ContentDB is the newest and intended for distribution. As
> +such, you can use the version from ContentDB without any special
> +comments.
We might want to quote an authoritative resource on that, perhaps in
the footnote?

> +@c Currently it's always checked out from git, but in principle
> +@c tarballs could be used.
> +
> +Even though the source code is often checked out from version
> control,
> +it is not necessary to use @code{git-version} or @code{hg-version}:
> the
> +releases on ContentDB are formal releases; in fact they are
> upstream's
> +official source of Minetest packages and they are not mutated in-
> place.
> +
> +@c Example (zip): mods by TenPlus1
> +@c Example (git): basic_materials, ethereal
> +While ContentDB provides the source code of packages in zip form, it
> is
> +recommended not to use these, because users can and do delete old
> +versions.  Likewise, sometimes the maintainer initially did tag
> versions
> +but later stops doing so, breaking @command{guix refresh -u}.  As
> such,
> +it is recommended not to use git tags in @code{origin} records and
> +instead refer to the commit directly.
This combination of version+commit is something I'd generally
discourage (my reasoning for doing so already explained elsewhere), so
to me it might make sense to still explicitly point attention to it. 
Perhaps setting a package-property such as (upstream . contentdb),
which would also make it clear why we don't e.g. want the latest-git
updater to apply?


Otherwise LGTM.

[bug#53144] [PATCH 01/13] doc: Give some tips on Minetest packaging.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 6380 bytes --]

Liliana Marie Prikler schreef op zo 09-01-2022 om 22:15 [+0100]:
> [...]
> > +
> > +Sometimes, it might be unclear what the version of a Minetest mod
> > is.
> > +For example, ContentDB and the importer reports 2020-01-01, but
> > +according to the forums the version is 2.1.  Usually, in these cases
> > the
> > +version on ContentDB is the newest and intended for distribution. As
> > +such, you can use the version from ContentDB without any special
> > +comments.
> We might want to quote an authoritative resource on that, perhaps in
> the footnote?

Yes, quoting sources seems good.

About newest version: I don't have an authoritive source at hand; this
is more speaking of my own experience of what seems to be the case.  I
could compute some statistics w.r.t. how often is the latest version on
ContentDB >= latest tagged version in the repository (where > is the
is-(indirect)-child-commit-of relation, as the versions strings aren't
always directly comparable -- see discussion about CalVer vs. SemVer
and ContentDB sometimes using CalVer and sometimes SemVer).

This also seems a consequence of ContentDB being the official source
of mods and used by Minetest's built-in installer.

About ‘intended for distribution’: I'll look for a forum post or wiki
page targetting new mod authors that want to distribute their mod.

What do you think of me gathering this information and (if not
falsified) presenting it to upstream in some public location,
asking if it seems about right, and (presuming they answer ‘yes’),
quoting their response?

> > +@c Currently it's always checked out from git, but in principle
> > +@c tarballs could be used.
> > +
> > +Even though the source code is often checked out from version
> > control,
> > +it is not necessary to use @code{git-version} or @code{hg-version}:
> > the
> > +releases on ContentDB are formal releases; in fact they are
> > upstream's
> > +official source of Minetest packages and they are not mutated in-
> > place.
> > +
> > +@c Example (zip): mods by TenPlus1
> > +@c Example (git): basic_materials, ethereal
> > +While ContentDB provides the source code of packages in zip form, it
> > is
> > +recommended not to use these, because users can and do delete old
> > +versions.  Likewise, sometimes the maintainer initially did tag
> > versions
> > +but later stops doing so, breaking @command{guix refresh -u}.  As
> > such,
> > +it is recommended not to use git tags in @code{origin} records and
> > +instead refer to the commit directly.
> This combination of version+commit is something I'd generally
> discourage (my reasoning for doing so already explained elsewhere), so
> to me it might make sense to still explicitly point attention to it. 

That discussion I had in mind while writing this documentation.

I happen to disagree though, being mostly neutral about
version+commit with a slight preference towards including the commit
itself in the commit field of 'git-reference' (and not a tag pointing
to the commit), because it is more explicit and it fits slightly
better in some nebulous plans for decentralising source code
downloading/storage.  There are ‘tricking peer review’ issues
with this though, hence neutral.

(See <https://notabug.org/maximed/rehash> if interested)

Hence, adding something like ‘This is specific to Minetest packages,
for other packages it is advised to use git tags, see [...]’ doesn't
make much sense to me, though I understand why it would make sense
to you.  Furthermore, the reference [...] currently doesn't exist, so I
cannot point the reader to an explanation why for other packages we
want git tags and not commits.

If there was consensus (one way or the other) and some section in the
manual explaining what should usually be used (tags or commits)
(preferably also explaining the reasons and not simply stating things),
I could link to it though.  Or if there's no consensus, and the section
said something like

‘There are two options: tags or commits.  Currently there's no
consensus about what's best.  Here are some pros and cons of each: ...
Due to a lack of consensus, the patch submitter can make the choice.
When in doubt, throw a dice.’ (*) (to be reworded!),

then I could work with that as well.

Perhaps I could write out (*) a bit more, as a separate documentation
patch?  I'd have to ask on guix-devel@ if I'm
understanding/misunderstanding the lack of consensus, and see if
someone has already summarised things, maybe see if Nix people
have thought about this, etc.

> Perhaps setting a package-property such as (upstream . contentdb),
> which would also make it clear why we don't e.g. want the latest-git
> updater to apply?

More generally, being able to explicitely choose the updater
(minetest/github/elpa/...) seems useful!

However, in the context of this documentation section and the changes
to the ContentDB importer, I don't think latest-git is relevant here
(except for the infrequent edge cases like minetest-throwing-arrows and
emacs-next): we almost never want the latest-git updater to apply
(because formal releases etc.).  And when we do want it to apply, we
set

  (with-latest-git-commit . #true/"refs/heads/master"/...)

otherwise latest-git doesn't run.  Well, we don't do that yet except
for minetest-throwing-arrows, but that's the idea.

To summarise, I don't see the value that adding (upstream . contentdb)
would bring, it seems to me that it would only make the package
definitions longer.  Would this package property be pure documentation,
or would it be interpreted by the updater code in some way?


> Otherwise LGTM.

I'm not complaining, but from
<https://lists.gnu.org/archive/html/guix-devel/2021-12/msg00259.html>
and resulting discussion I was kind of expecting that you'd
want me to do

(let ((commit ...)
      (revision ...))
  (package ...
    (version (git-version "contentdb-version" revision commit)) ...)),

though you did write ‘I'd want a comment like the ones I find in
minetest.scm to [...]’ I suppose, and this new documentation is
explaining the reasons for using commits (raw commits in your
terminology) in some detail and for all Minetest packages at once.

Unrelated: git mirrors seem a cool idea!

Greetings,
Maxime

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#53144] [PATCH 01/13] doc: Give some tips on Minetest packaging.

[Liliana Marie Prikler]

Hi,

Am Montag, dem 10.01.2022 um 00:29 +0100 schrieb Maxime Devos:
> Liliana Marie Prikler schreef op zo 09-01-2022 om 22:15 [+0100]:
> > [...]
> > > +
> > > +Sometimes, it might be unclear what the version of a Minetest
> > > mod
> > > is.
> > > +For example, ContentDB and the importer reports 2020-01-01, but
> > > +according to the forums the version is 2.1.  Usually, in these
> > > cases
> > > the
> > > +version on ContentDB is the newest and intended for
> > > distribution. As
> > > +such, you can use the version from ContentDB without any special
> > > +comments.
> > We might want to quote an authoritative resource on that, perhaps
> > in the footnote?
> 
> Yes, quoting sources seems good.
> 
> About newest version: I don't have an authoritive source at hand;
> this is more speaking of my own experience of what seems to be the
> case.  I could compute some statistics w.r.t. how often is the latest
> version on ContentDB >= latest tagged version in the repository
> (where > is the is-(indirect)-child-commit-of relation, as the
> versions strings aren't always directly comparable -- see discussion
> about CalVer vs. SemVer and ContentDB sometimes using CalVer and
> sometimes SemVer).
> 
> This also seems a consequence of ContentDB being the official source
> of mods and used by Minetest's built-in installer.
> 
> About ‘intended for distribution’: I'll look for a forum post or wiki
> page targetting new mod authors that want to distribute their mod.
> 
> What do you think of me gathering this information and (if not
> falsified) presenting it to upstream in some public location,
> asking if it seems about right, and (presuming they answer ‘yes’),
> quoting their response?
In my opinion the fact that Minetest's built-in installer serves as
this ‘M-x package-install’-like gateway would already be authoritative
enough, but it still deserves being spelled out in a footnote that this
is the basis of our assumptions :)

Communicating with upstreams is also important, but a poll or similar
would be significantly harder to quote (perhaps we could insert it in a
comment), and that's not even talking about what we'd have to do to
account for input selection.

> > 
> I happen to disagree though, being mostly neutral about
> version+commit with a slight preference towards including the commit
> itself in the commit field of 'git-reference' (and not a tag pointing
> to the commit), because it is more explicit and it fits slightly
> better in some nebulous plans for decentralising source code
> downloading/storage.  There are ‘tricking peer review’ issues
> with this though, hence neutral.
> 
> (See <https://notabug.org/maximed/rehash> if interested)
> 
> Hence, adding something like ‘This is specific to Minetest packages,
> for other packages it is advised to use git tags, see [...]’ doesn't
> make much sense to me, though I understand why it would make sense
> to you.  Furthermore, the reference [...] currently doesn't exist, so
> I cannot point the reader to an explanation why for other packages we
> want git tags and not commits.
Again note that you're not only competing against version+tag, but also
against git-version+hash, with git-version embedding the hash.  Among
Guix packages, Minetest *is* rather unique in having a separate source
(ContentDB) pointing to the bespoke commits that serve as release
basis.

> If there was consensus (one way or the other) and some section in the
> manual explaining what should usually be used (tags or commits)
> (preferably also explaining the reasons and not simply stating
> things), I could link to it though.  Or if there's no consensus, and
> the section said something like
> 
> ‘There are two options: tags or commits.  Currently there's no
> consensus about what's best.  Here are some pros and cons of each:
> ...
> Due to a lack of consensus, the patch submitter can make the choice.
> When in doubt, throw a dice.’ (*) (to be reworded!),
> 
> then I could work with that as well.
> 
> Perhaps I could write out (*) a bit more, as a separate documentation
> patch?  I'd have to ask on guix-devel@ if I'm
> understanding/misunderstanding the lack of consensus, and see if
> someone has already summarised things, maybe see if Nix people
> have thought about this, etc.
One question whose answer is not quite clear to me is whether you want
to generally follow the commit vs. tag guidelines of the rest of Guix
(and therefore seek to formalize them in a similar way, though perhaps
with different reasoning from what I do), or if you want ContentDB
packages to follow their own style largely independent from the rest,
at least for the time being.

> > Perhaps setting a package-property such as (upstream . contentdb),
> > which would also make it clear why we don't e.g. want the latest-
> > git updater to apply?
> 
> More generally, being able to explicitely choose the updater
> (minetest/github/elpa/...) seems useful!
> 
> However, in the context of this documentation section and the changes
> to the ContentDB importer, I don't think latest-git is relevant here
> (except for the infrequent edge cases like minetest-throwing-arrows
> and emacs-next): we almost never want the latest-git updater to apply
> (because formal releases etc.).  And when we do want it to apply, we
> set
> 
>   (with-latest-git-commit . #true/"refs/heads/master"/...)
> 
> otherwise latest-git doesn't run.  Well, we don't do that yet except
> for minetest-throwing-arrows, but that's the idea.
I think the generic git updater would still be relevant here, hence why
I'm asking.  Yes, latest-git is restricted in that it needs opting in,
but the property combination

  ((upstream . latest-git) (latest-git-head "master"))

would also work, for example.

> To summarise, I don't see the value that adding (upstream .
> contentdb) would bring, it seems to me that it would only make the
> package definitions longer.  Would this package property be pure
> documentation, or would it be interpreted by the updater code in some
> way?
Ideally it'd serve as a better way of making the minetest-updater say
"Now this looks like a job for me" than relying on the minetest-
package? procedure.  Alternatively, we could make it so that the
updater has to have a contentdb URL as homepage.  You are correct in
that there's no immediate benefit, but given that we would be
shortening a four-line comment to a single-line property while
essentially keeping all the information is still a win in my opinion.

> > Otherwise LGTM.
> 
> I'm not complaining, but from
> <https://lists.gnu.org/archive/html/guix-devel/2021-12/msg00259.html>
> and resulting discussion I was kind of expecting that you'd
> want me to do
> 
> (let ((commit ...)
>       (revision ...))
>   (package ...
>     (version (git-version "contentdb-version" revision commit))
> ...)),
> 
> though you did write ‘I'd want a comment like the ones I find in
> minetest.scm to [...]’ I suppose, and this new documentation is
> explaining the reasons for using commits (raw commits in your
> terminology) in some detail and for all Minetest packages at once.
It does, but in a different location that requires a little bit of work
to do the mapping.  I think a comment line 
  ;; version from contentdb
or similar would still be appropriate even with said documentation
changes.  Alternatively, as pointed out above, we could have home-page
point to ContentDB, which IIRC should also present that information to
the user.

I think "Can we use ContentDB as a home-page for minetest-blub?" also
serves as a good leading question on what source is the most
authoritative one.  If the ContentDB page looks ill-maintained, but the
forum is active, the forum should probably be preferred (while
"generally, use ContentDB" still applies otherwise).

The distinction between "raw commits" and just commits is also an
important one that I want to keep.  ContentDB provides both release
titles and commits in a usable format, so that's why raw commits are in
this context permissible in my opinion.  There's no ContentDB for
guile-aiscm or node-something-that-loses-its-git-tag.

Cheers

[bug#53144] [PATCH 0/13] Make more git-using packages auto-updatable

[Ludovic Courtès]

Hi!

Maxime Devos <maximedevos@telenet.be> skribis:

> * doc/contributing.texi (Minetest Packages): New section.
> * doc/guix.texi: Copyright update.

[...]

> +@node Minetest Packages
> +@subsection Minetest Packages
> +@cindex minetest, packaging
> +
> +A Minetest mod @code{foo} is named @code{minetest-foo} -- the author
> +name from ContentDB is not included, unless required to resolve a name
> +collision.

The section looks great to me, but I’d suggest starting it with a couple
of sentences giving context, like “The @uref{…, Minetest} game can be
extended with so-called @dfn{mods}, which are …”

It might also be worth defining ContentDB or giving a @uref.

Last, parenthetical phrases should be enclosed in em dashes, represented
by three hyphens in Texinfo (two hyphens is for en dashes, used to
represent ranges.)

Ludo’.

[bug#53144] Acknowledgement ([PATCH 0/13] Make more git-using packages auto-updatable)

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 121 bytes --]

The failing tests are:

 * tests/channels.scm: "channel-news, one entry"
 * tests/pack.scm, due to networking issues

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#53144] [PATCH 0/13] Make more git-using packages auto-updatable

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 109 bytes --]

The remote-create-detached issue is fixed by
<https://gitlab.com/guile-git/guile-git/-/merge_requests/35>.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]