[bug#50359] [PATCH] import: Add 'generic-git' updater.

[Xinglu Chen <public@yoctocell.xyz>]

[-- Attachment #1.1: Type: text/plain, Size: 4713 bytes --]

On Tue, Sep 07 2021, Xinglu Chen wrote:

> On Mon, Sep 06 2021, Sarah Morgensen wrote:
>
>> Hi,
>>
>> Xinglu Chen <public@yoctocell.xyz> writes:
>>
>>> Any trick you used to find all of there weird version numbers?  :-)
>>
>> This monstrosity:
>>
>>   rg -U -B4 --pcre2 '(?!\(let.*(\n.*){0,1})\(version "([^\n"]*[^0-9\.][^"\n]*)".*(\n.*){0,10}commit.*version' gnu/packages
>>
>> and to show just the versions:
>>
>>   rg -Uor '$2' --pcre2 --no-filename --no-line-number 
>
> Wow!  I will try that and see for myself!  :-)
>
>>>> IMO, just get rid of the delimiter.  If we wanted to be *that* flexible,
>>>> we could make it so they provide a tag->version proc instead of (prefix,
>>>> suffix, delimiter).
>>>
>>> a ‘tag->version’ procedure would probably make things a bit too
>>> complicated for the people writing package definitions.  For example,
>>> having a delimiter would make it easy to match a tag like
>>> “2021-01-01-release”
>>>
>>> Delimiter is “.” (sorry if this hurts your eyes ;-))
>>>
>>> scheme@(guile-user)> (match:substring (string-match "^[^0-9]*([^\\.[:punct:]]+(\\.[^\\.[:punct:]]+)*).*$" "2021-01-01-release") 1)
>>> $28 = "2021"
>>>
>>> Delimiter is “-”
>>>
>>> scheme@(guile-user)> (match:substring (string-match "^[^0-9]*([^-[:punct:]]+(-[^-[:punct:]]+)*).*$" "2021-01-01-release") 1)
>>> $29 = "2021-01-01-release"
>>>
>>> And then, setting the suffix to “-release” would match just the version
>>> part.
>>
>> Right.  I missed that.
>>
>> In that vein, should we keep the dashes in "2021-01-01" or convert them
>> to periods?
>
> Having periods would be more consistent, then could have a
> ‘date->version’ procedure that replaces the hyphens with dots and have
>
>   (git-reference
>     (url "https://git.example.org")
>     (commit (date->version version)))
>
>> What about when a tag has underscores?
>
> Hmm, not sure about that, below is a list of packages I could find which had
> underscores as delimiters
>
> gnu/packages/graphics.scm
> 239:          (commit "DIRECTFB_1_7_7")))
> gnu/packages/gstreamer.scm
> 326:          (commit "ESOUND_0_2_41")))
> gnu/packages/java.scm
> 13925:                      (commit "jboss-transaction-api_1.2_spec-1.1.1.Final")))
>
> They all seem to use periods in the ‘version’ field, though, so I would
> say that the underscroes, should also be converted to periods.
>
>> What if a repo has tags in both formats?  Then "3.0.1" would be
>> considered older than "2011-01-01".
>
> That’s tricky, there isn’t really a way to know how old “3.0.1” is,
> without looking at the metadata of the tag.  Maybe this is one of those
> corner cases which can’t really automatically determine the latest
> release.  Should we have a ‘no-refresh?’ property to tell the refresh to
> not try to update the package?
>
>> Maybe we should just add an extra bit to detect a date format and only
>> consider it when there's no "proper versions"?
>
> That could be a good idea!
>
>> Aaaand I fell down a rabbit hole after that :) I've attached a patch
>> with what I've done.  It still has lots of issues--it requires the tag
>> to contain at least one version delimiter, it requires the first
>> character of the version to be a number... it might not even be better
>> than before I touched it, and even so the added complexity might not be
>> worth it.  But if you'd like to take it for a spin, I've attached it (it
>> applies straight on master).
>
> Great! I will try it out and see how it compares to my current WIP
> version.
>
> Not having characters in the first version number probably isn’t such a
> big deal, most version that contain characters end with a character.
> E.g., “1.2.3a” is not to uncommon, but “a1.2.3” is rarely seen.

I made some changes, to the patch and ran ‘guix refresh -t generic-git |
grep "^gnu/packages.*no valid’; I also modified the ‘github’ updater so
that it wouldn’t be used.
  
  (define %github-updater
    (upstream-updater
     (name 'github)
     (description "Updater for GitHub packages")
     (pred (const #f)) ;this right here
     (latest latest-release)))

Attached is a file with all the packages that didn’t have any valid
tags, and with a short description of perharps why no valid tags were
found.

Something I noticed was the a lot of Julia package use a version scheme
like this:

  (version "1.2.3+0")

The "+0" is included in the version field and acts like a “revision”; I
am not familiar with the Julia ecosystem, and I am not sure how we
should handle this situation.

The updated patch is also attached.


[-- Attachment #1.2: no-valid-tags --]
[-- Type: application/octet-stream, Size: 17569 bytes --]

;;; Custom prefix or suffix (should be solvable by us)
;; kmscon- prefix
gnu/packages/terminals.scm:281:14: warning: no valid tags found for kmscon

;; v1.2-<num>
gnu/packages/statistics.scm:5701:5: warning: no valid tags found for r-colorout

;; <CHAR><num>
gnu/packages/sdl.scm:662:14: warning: no valid tags found for sdl2-cs

;; test- prefix
gnu/packages/qt.scm:118:7: warning: no valid tags found for qite

;; "v" prefix, "-stable" suffix
gnu/packages/opencog.scm:76:14: warning: no valid tags found for atomspace
gnu/packages/opencog.scm:44:14: warning: no valid tags found for cogutil

;; +<num> suffix
gnu/packages/ocaml.scm:603:12: warning: no valid tags found for ocaml-mccs

;; "version-" prefix
gnu/packages/minetest.scm:600:5: warning: no valid tags found for minetest-unified-inventory

;; "maven-resources-plugin-" prefix
gnu/packages/maven.scm:2980:12: warning: no valid tags found for maven-resources-plugin

;; "txr-" prefix
gnu/packages/lisp.scm:912:5: warning: no valid tags found for txr

;; release- prefix
gnu/packages/algebra.scm:1420:12: warning: no valid tags found for m4ri
gnu/packages/algebra.scm:1510:12: warning: no valid tags found for m4rie

;; lrcalc- prefix
gnu/packages/algebra.scm:1566:12: warning: no valid tags found for lrcalc

;; 3.3.06-1, -<num> suffix
gnu/packages/admin.scm:4046:2: warning: no valid tags found for inxi
gnu/packages/admin.scm:3977:7: warning: no valid tags found for inxi-minimal

;; -release-DATE+SOME-RANDOM-NUMBERS, refs/tags/2.0.180-release-20210315231707
gnu/packages/bioinformatics.scm:11189:12: warning: no valid tags found for biobambam2
gnu/packages/bioinformatics.scm:11149:12: warning: no valid tags found for libmaus2

;; "+git<date>" suffix
gnu/packages/gnome.scm:11878:12: warning: no valid tags found for feedbackd

;; "fest-test-" prefix
gnu/packages/java.scm:10900:12: warning: no valid tags found for java-fest-test

;; "-stable" suffix
gnu/packages/game-development.scm:1722:12: warning: no valid tags found for godot

;; "v" prefix, "-debian" suffix
gnu/packages/gstreamer.scm:95:5: warning: no valid tags found for openni2


;; +<num> suffix, but this is part of the `version' field as well, not
;; sure what to do about this
;;
;; (version "1.5.0+0")
gnu/packages/julia-jll.scm:366:6: warning: no valid tags found for julia-fribidi-jll
gnu/packages/julia-jll.scm:2520:6: warning: no valid tags found for julia-xorg-xtrans-jll
gnu/packages/julia-jll.scm:1812:6: warning: no valid tags found for julia-xorg-libx11-jll
gnu/packages/julia-jll.scm:1149:6: warning: no valid tags found for julia-libvorbis-jll
gnu/packages/julia-jll.scm:1491:6: warning: no valid tags found for julia-qt5base-jll
gnu/packages/julia-jll.scm:1415:6: warning: no valid tags found for julia-pcre-jll
gnu/packages/julia-jll.scm:1964:6: warning: no valid tags found for julia-xorg-libxdmcp-jll
gnu/packages/julia-jll.scm:1581:6: warning: no valid tags found for julia-wayland-protocols-jll
gnu/packages/julia-jll.scm:694:6: warning: no valid tags found for julia-lame-jll
gnu/packages/julia-jll.scm:2261:6: warning: no valid tags found for julia-xorg-xcb-util-jll
gnu/packages/julia-jll.scm:1618:6: warning: no valid tags found for julia-x264-jll
gnu/packages/julia-jll.scm:1926:6: warning: no valid tags found for julia-xorg-libxcursor-jll
gnu/packages/julia-jll.scm:920:6: warning: no valid tags found for julia-libgpg-error-jll
gnu/packages/julia-jll.scm:882:6: warning: no valid tags found for julia-libglvnd-jll
gnu/packages/julia-jll.scm:486:6: warning: no valid tags found for julia-glib-jll
gnu/packages/julia-jll.scm:1850:6: warning: no valid tags found for julia-xorg-libxau-jll
gnu/packages/julia-jll.scm:106:6: warning: no valid tags found for julia-cairo-jll
gnu/packages/julia-jll.scm:2556:6: warning: no valid tags found for julia-xslt-jll
gnu/packages/julia-jll.scm:2298:6: warning: no valid tags found for julia-xorg-xcb-util-image-jll
gnu/packages/julia-jll.scm:2409:6: warning: no valid tags found for julia-xorg-xcb-util-wm-jll
gnu/packages/julia-jll.scm:657:6: warning: no valid tags found for julia-jpegturbo-jll
gnu/packages/julia-jll.scm:2074:6: warning: no valid tags found for julia-xorg-libxi-jll
gnu/packages/julia-jll.scm:1187:6: warning: no valid tags found for julia-lzo-jll
gnu/packages/julia-jll.scm:2000:6: warning: no valid tags found for julia-xorg-libxext-jll
gnu/packages/julia-jll.scm:843:6: warning: no valid tags found for julia-libgcrypt-jll
gnu/packages/julia-jll.scm:730:6: warning: no valid tags found for julia-libass-jll
gnu/packages/julia-jll.scm:1542:6: warning: no valid tags found for julia-wayland-jll
gnu/packages/julia-jll.scm:1111:6: warning: no valid tags found for julia-libuuid-jll
gnu/packages/julia-jll.scm:1072:6: warning: no valid tags found for julia-libtiff-jll
gnu/packages/julia-jll.scm:2224:6: warning: no valid tags found for julia-xorg-libxrender-jll
gnu/packages/julia-jll.scm:958:6: warning: no valid tags found for julia-libiconv-jll
gnu/packages/julia-jll.scm:996:6: warning: no valid tags found for julia-libmount-jll
gnu/packages/julia-jll.scm:1035:6: warning: no valid tags found for julia-libpng-jll
gnu/packages/julia-jll.scm:770:6: warning: no valid tags found for julia-libfdk-aac-jll
gnu/packages/julia-jll.scm:1735:6: warning: no valid tags found for julia-xml2-jll
gnu/packages/julia-jll.scm:806:6: warning: no valid tags found for julia-libffi-jll
gnu/packages/julia-jll.scm:1886:6: warning: no valid tags found for julia-xorg-libxcb-jll
gnu/packages/julia-jll.scm:2446:6: warning: no valid tags found for julia-xorg-xkbcomp-jll
gnu/packages/julia-jll.scm:1659:6: warning: no valid tags found for julia-x265-jll
gnu/packages/julia-jll.scm:2483:6: warning: no valid tags found for julia-xorg-xkeyboard-config-jll
gnu/packages/julia-jll.scm:443:6: warning: no valid tags found for julia-glfw-jll
gnu/packages/julia-jll.scm:2112:6: warning: no valid tags found for julia-xorg-libxinerama-jll
gnu/packages/julia-jll.scm:66:5: warning: no valid tags found for julia-bzip2-jll
gnu/packages/julia-jll.scm:569:6: warning: no valid tags found for julia-imagemagick-jll
gnu/packages/julia-jll.scm:1263:6: warning: no valid tags found for julia-ogg-jll
gnu/packages/julia-jll.scm:232:6: warning: no valid tags found for julia-ffmpeg-jll
gnu/packages/julia-jll.scm:1695:6: warning: no valid tags found for julia-xkbcommon-jll
gnu/packages/julia-jll.scm:2599:6: warning: no valid tags found for julia-zlib-jll
gnu/packages/julia-jll.scm:2335:6: warning: no valid tags found for julia-xorg-xcb-util-keysyms-jll
gnu/packages/julia-jll.scm:530:5: warning: no valid tags found for julia-gumbo-jll
gnu/packages/julia-jll.scm:2186:6: warning: no valid tags found for julia-xorg-libxrandr-jll
gnu/packages/julia-jll.scm:194:6: warning: no valid tags found for julia-expat-jll
gnu/packages/julia-jll.scm:1300:7: warning: no valid tags found for julia-openspecfun-jll
gnu/packages/julia-jll.scm:324:5: warning: no valid tags found for julia-freetype2-jll
gnu/packages/julia-jll.scm:1343:6: warning: no valid tags found for julia-openssl-jll
gnu/packages/julia-jll.scm:1225:5: warning: no valid tags found for julia-mbedtls-jll
gnu/packages/julia-jll.scm:152:5: warning: no valid tags found for julia-compilersupportlibraries-jll
gnu/packages/julia-jll.scm:2037:6: warning: no valid tags found for julia-xorg-libxfixes-jll
gnu/packages/julia-jll.scm:281:6: warning: no valid tags found for julia-fontconfig-jll
gnu/packages/julia-jll.scm:1775:6: warning: no valid tags found for julia-xorg-libpthread-stubs-jll
gnu/packages/julia-jll.scm:1379:6: warning: no valid tags found for julia-opus-jll
gnu/packages/julia-jll.scm:402:6: warning: no valid tags found for julia-gettext-jll
gnu/packages/julia-jll.scm:1453:6: warning: no valid tags found for julia-pixman-jll
gnu/packages/julia-jll.scm:2372:6: warning: no valid tags found for julia-xorg-xcb-util-renderutil-jll
gnu/packages/julia-jll.scm:2637:6: warning: no valid tags found for julia-zstd-jll
gnu/packages/julia-jll.scm:2149:6: warning: no valid tags found for julia-xorg-libxkbfile-jll

\f
;;; only pre-releases
gnu/packages/linux.scm:1306:7: warning: no valid tags found for rtl8812au-aircrack-ng-linux-module
gnu/packages/vim.scm:596:7: warning: no valid tags found for vim-solarized
gnu/packages/music.scm:2133:12: warning: no valid tags found for powertabeditor
gnu/packages/music.scm:4463:14: warning: no valid tags found for rkrlv2
gnu/packages/lua.scm:248:12: warning: no valid tags found for lua5.1-socket
gnu/packages/lua.scm:248:12: warning: no valid tags found for lua5.2-socket
gnu/packages/guile-xyz.scm:3863:12: warning: no valid tags found for guile-ac-d-bus
gnu/packages/golang.scm:4136:7: warning: no valid tags found for go-github-com-btcsuite-btcd-btcec
gnu/packages/games.scm:7274:5: warning: no valid tags found for colobot

;; _beta<num> suffix, refs/tags/v0.1_beta2
gnu/packages/compton.scm:50:7: warning: no valid tags found for compton

;; almost all tags are pre-releases
;; refs/tags/0.0.4-152-alpha
;; refs/tags/0.0.4-152-alpha^{}
;; refs/tags/0.0.4-172-alpha
;; refs/tags/0.0.4-172-alpha^{}
;; refs/tags/0.1.0-alpha
;; refs/tags/0.1.0-alpha^{}
;; refs/tags/0.1.1-alpha
;; refs/tags/0.1.1-alpha^{}
;; refs/tags/0.1.2-alpha
;; refs/tags/0.1.2-alpha^{}
;; refs/tags/0.2.0-alpha
;; refs/tags/0.2.0-alpha^{}
;; refs/tags/pre-extension
;; refs/tags/v0.0.2-alpha
;; refs/tags/v0.0.3-alpha
;; refs/tags/v0.0.4-alpha
;; refs/tags/v0.0.4-alpha^{}
;; refs/tags/v0.1.0-alpha
;; refs/tags/v0.1.0-alpha^{}
;; refs/tags/v0.1.2-candidate
;; refs/tags/v0.1.2-candidate^{}
gnu/packages/guile-xyz.scm:3488:12: warning: no valid tags found for nomad

\f
;;; no delimiter or just one version number
gnu/packages/mail.scm:587:5: warning: no valid tags found for neomutt
gnu/packages/sync.scm:518:12: warning: no valid tags found for casync
gnu/packages/pulseaudio.scm:315:12: warning: no valid tags found for ponymix
gnu/packages/networking.scm:1266:12: warning: no valid tags found for iputils
gnu/packages/music.scm:6884:12: warning: no valid tags found for a2jmidid
gnu/packages/emulators.scm:567:5: warning: no valid tags found for higan
gnu/packages/emacs-xyz.scm:11489:5: warning: no valid tags found for emacs-lua-mode
gnu/packages/diffoscope.scm:76:12: warning: no valid tags found for diffoscope
gnu/packages/cups.scm:67:14: warning: no valid tags found for brlaser
gnu/packages/code.scm:575:12: warning: no valid tags found for kcov
gnu/packages/gnome-xyz.scm:185:5: warning: no valid tags found for papirus-icon-theme
gnu/packages/gnome-xyz.scm:367:12: warning: no valid tags found for gnome-shell-extension-gsconnect
gnu/packages/gnome-xyz.scm:289:5: warning: no valid tags found for gnome-shell-extension-topicons-redux
gnu/packages/golang.scm:1116:5: warning: no valid tags found for go-github-com-op-go-logging
gnu/packages/java.scm:10579:12: warning: no valid tags found for java-javax-inject
gnu/packages/mail.scm:1415:12: warning: no valid tags found for notmuch-addrlookup-c






\f
;;; (Most likely) not solvable by us
;; only one tag "initial-commit"
gnu/packages/build-tools.scm:208:14: warning: no valid tags found for gn

;; ??? the weirdest version scheme???
;; refs/tags/bzr-revno-7894
;; refs/tags/moses-3.2.13-15-Nov-2012
;; refs/tags/moved-to-ure-based-r2l
;; refs/tags/obsolete-C++-PLN
;; refs/tags/obsolete-Python-PLN
;; refs/tags/old-embodiment_8-nov-2015
;; refs/tags/old-openpsi-Jinhua-Chua-2011
;; refs/tags/old-pattern-miner-11-Jun-2019
;; refs/tags/v0.1.4-stable
gnu/packages/opencog.scm:209:14: warning: no valid tags found for opencog

;; "quicklisp-" prefix, date instead of version number
gnu/packages/lisp-xyz.scm:3703:7: warning: no valid tags found for sbcl-portable-threads

;; no version number in tag
gnu/packages/guile-xyz.scm:3283:7: warning: no valid tags found for guile-srfi-180
gnu/packages/golang.scm:6193:14: warning: no valid tags found for go-github-com-golang-freetype
gnu/packages/messaging.scm:1495:14: warning: no valid tags found for libtoxcore
gnu/packages/networking.scm:3814:5: warning: no valid tags found for vde2
gnu/packages/lisp-xyz.scm:15892:6: warning: no valid tags found for cl-bodge-blobs-support
gnu/packages/lisp-xyz.scm:3120:7: warning: no valid tags found for cl-cffi-c-ref
gnu/packages/lisp-xyz.scm:5979:7: warning: no valid tags found for sbcl-ieee-floats
gnu/packages/engineering.scm:814:14: warning: no valid tags found for libfive
gnu/packages/cpp.scm:221:14: warning: no valid tags found for rct

;; "go.weekly." prefix, date instead of version
gnu/packages/golang.scm:3905:7: warning: no valid tags found for go-github-com-michiwend-golang-pretty

;; weird version scheme
;; refs/tags/V3
;; refs/tags/version_1
;; refs/tags/wyhash_alpha
;; refs/tags/wyhash_final
;; refs/tags/wyhash_v4
;; refs/tags/wyhash_v5
gnu/packages/digest.scm:36:12: warning: no valid tags found for wyhash




;;; couldn't be bothered to check...
gnu/packages/lisp-xyz.scm:6508:2: warning: no valid tags found for cl-trivia
gnu/packages/lisp-xyz.scm:12214:7: warning: no valid tags found for cl-mcclim
gnu/packages/lisp-xyz.scm:13633:7: warning: no valid tags found for cl-claw-utils
gnu/packages/lisp-xyz.scm:3195:7: warning: no valid tags found for sbcl-parenscript
gnu/packages/lisp-xyz.scm:6466:7: warning: no valid tags found for cl-trivia.trivial
gnu/packages/lisp-xyz.scm:6508:2: warning: no valid tags found for ecl-trivia
gnu/packages/lisp-xyz.scm:13566:5: warning: no valid tags found for cl-claw-support
gnu/packages/lisp-xyz.scm:9326:7: warning: no valid tags found for cl-jpeg
gnu/packages/lisp-xyz.scm:13566:5: warning: no valid tags found for sbcl-claw-support
gnu/packages/lisp-xyz.scm:18394:7: warning: no valid tags found for cl-tailrec
gnu/packages/lisp-xyz.scm:139:6: warning: no valid tags found for sbcl-bodge-utilities
gnu/packages/lisp-xyz.scm:15860:6: warning: no valid tags found for ecl-bodge-math
gnu/packages/lisp-xyz.scm:13566:5: warning: no valid tags found for ecl-claw-support
gnu/packages/lisp-xyz.scm:178:7: warning: no valid tags found for cl-bodge-queue
gnu/packages/lisp-xyz.scm:6508:2: warning: no valid tags found for sbcl-trivia
gnu/packages/lisp-xyz.scm:6807:7: warning: no valid tags found for sbcl-s-sysdeps
gnu/packages/lisp-xyz.scm:6466:7: warning: no valid tags found for ecl-trivia.trivial
gnu/packages/lisp-xyz.scm:519:7: warning: no valid tags found for cl-fiasco
gnu/packages/lisp-xyz.scm:6807:7: warning: no valid tags found for ecl-s-sysdeps
gnu/packages/lisp-xyz.scm:8929:7: warning: no valid tags found for ecl-cl-async
gnu/packages/lisp-xyz.scm:3195:7: warning: no valid tags found for cl-parenscript
gnu/packages/lisp-xyz.scm:9326:7: warning: no valid tags found for ecl-cl-jpeg
gnu/packages/lisp-xyz.scm:519:7: warning: no valid tags found for ecl-fiasco
gnu/packages/lisp-xyz.scm:3703:7: warning: no valid tags found for ecl-portable-threads
gnu/packages/lisp-xyz.scm:12214:7: warning: no valid tags found for sbcl-mcclim
gnu/packages/lisp-xyz.scm:8628:7: warning: no valid tags found for cl-misc-extensions
gnu/packages/lisp-xyz.scm:13633:7: warning: no valid tags found for ecl-claw-utils
gnu/packages/lisp-xyz.scm:5979:7: warning: no valid tags found for cl-ieee-floats
gnu/packages/lisp-xyz.scm:15892:6: warning: no valid tags found for ecl-bodge-blobs-support
gnu/packages/lisp-xyz.scm:3120:7: warning: no valid tags found for ecl-cffi-c-ref
gnu/packages/lisp-xyz.scm:5979:7: warning: no valid tags found for ecl-ieee-floats
gnu/packages/lisp-xyz.scm:139:6: warning: no valid tags found for cl-bodge-utilities
gnu/packages/lisp-xyz.scm:178:7: warning: no valid tags found for ecl-bodge-queue
gnu/packages/lisp-xyz.scm:9326:7: warning: no valid tags found for sbcl-cl-jpeg
gnu/packages/lisp-xyz.scm:519:7: warning: no valid tags found for sbcl-fiasco
gnu/packages/lisp-xyz.scm:15892:6: warning: no valid tags found for sbcl-bodge-blobs-support
gnu/packages/lisp-xyz.scm:139:6: warning: no valid tags found for ecl-bodge-utilities
gnu/packages/lisp-xyz.scm:3195:7: warning: no valid tags found for ecl-parenscript
gnu/packages/lisp-xyz.scm:3120:7: warning: no valid tags found for sbcl-cffi-c-ref
gnu/packages/lisp-xyz.scm:178:7: warning: no valid tags found for sbcl-bodge-queue
gnu/packages/lisp-xyz.scm:8628:7: warning: no valid tags found for ecl-misc-extensions
gnu/packages/lisp-xyz.scm:15860:6: warning: no valid tags found for sbcl-bodge-math
gnu/packages/lisp-xyz.scm:8628:7: warning: no valid tags found for sbcl-misc-extensions
gnu/packages/lisp-xyz.scm:15860:6: warning: no valid tags found for cl-bodge-math
gnu/packages/lisp-xyz.scm:13633:7: warning: no valid tags found for sbcl-claw-utils
gnu/packages/lisp-xyz.scm:6466:7: warning: no valid tags found for sbcl-trivia.trivial
gnu/packages/lisp-xyz.scm:18394:7: warning: no valid tags found for sbcl-tailrec
gnu/packages/lisp-xyz.scm:18394:7: warning: no valid tags found for ecl-tailrec
gnu/packages/lisp-xyz.scm:12214:7: warning: no valid tags found for ecl-mcclim
gnu/packages/lisp-xyz.scm:8929:7: warning: no valid tags found for sbcl-cl-async
gnu/packages/lisp-xyz.scm:6807:7: warning: no valid tags found for cl-s-sysdeps
gnu/packages/lisp-xyz.scm:8929:7: warning: no valid tags found for cl-async
gnu/packages/lisp-xyz.scm:3703:7: warning: no valid tags found for cl-portable-threads
gnu/packages/linux.scm:1306:7: warning: no valid tags found for rtl8812au-aircrack-ng-linux-module
gnu/packages/linux.scm:6284:5: warning: no valid tags found for mcelog
gnu/packages/kodi.scm:470:2: warning: no valid tags found for kodi-wayland
gnu/packages/kodi.scm:275:12: warning: no valid tags found for kodi

[-- Attachment #1.3: 0001-import-Add-generic-git-updater.patch --]
[-- Type: text/x-patch, Size: 12920 bytes --]

From f924dbb835425f6b9a5796918125592870391405 Mon Sep 17 00:00:00 2001
Message-Id: <f924dbb835425f6b9a5796918125592870391405.1631125652.git.public@yoctocell.xyz>
From: Xinglu Chen <public@yoctocell.xyz>
Date: Fri, 3 Sep 2021 17:50:56 +0200
Subject: [PATCH] import: Add 'generic-git' updater.

* guix/import/git.scm: New file.
* doc/guix.texi (Invoking guix refresh): Document it.
* Makefile.am (MODULES): Register it.
* guix/git.scm (ls-remote-refs): New procedure.

Co-authored-by: Sarah Morgensen <iskarian@mgsn.dev>
---
 Makefile.am         |   1 +
 doc/guix.texi       |  27 ++++++
 guix/git.scm        |  33 +++++++
 guix/import/git.scm | 217 ++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 278 insertions(+)
 create mode 100644 guix/import/git.scm

diff --git a/Makefile.am b/Makefile.am
index 3c79760734..c4d3a456b1 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -254,6 +254,7 @@ MODULES =					\
   guix/import/egg.scm   			\
   guix/import/elpa.scm   			\
   guix/import/gem.scm				\
+  guix/import/git.scm                           \
   guix/import/github.scm   			\
   guix/import/gnome.scm				\
   guix/import/gnu.scm				\
diff --git a/doc/guix.texi b/doc/guix.texi
index 36a0c7f5ec..26afb1607a 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -11920,6 +11920,33 @@ the updater for @uref{https://launchpad.net, Launchpad} packages.
 @item generic-html
 a generic updater that crawls the HTML page where the source tarball of
 the package is hosted, when applicable.
+@item generic-git
+a generic updater for packages hosted on Git repositories.  It tries to
+be smart about parsing Git tag names, but if it is not able to parse the
+tag name and compare tags correctly, users can define the following
+properties for a package.
+
+@itemize
+@item @code{tag-prefix}: a regular expression for matching a prefix of
+the tag name.
+
+@item @code{tag-suffix}: a regular expression for matching a suffix of
+the tag name.
+
+@item @code{tag-version-delimiter}: a string used as the delimiter in
+the tag name for separating the numbers of the version.
+@end itemize
+
+@lisp
+(package
+  (name "foo")
+  ;; ...
+  (properties
+    '((tag-prefix . "^release0-")
+      (tag-suffix . "[a-z]?$")
+      (tag-version-delimiter . ":"))))
+@end lisp      
+
 @end table
 
 For instance, the following command only checks for updates of Emacs
diff --git a/guix/git.scm b/guix/git.scm
index 9c6f326c36..c5d0d2da8e 100644
--- a/guix/git.scm
+++ b/guix/git.scm
@@ -56,6 +56,8 @@
             commit-difference
             commit-relation
 
+            ls-remote-refs
+
             git-checkout
             git-checkout?
             git-checkout-url
@@ -556,6 +558,37 @@ objects: 'ancestor (meaning that OLD is an ancestor of NEW), 'descendant, or
               (if (set-contains? oldest new)
                   'descendant
                   'unrelated))))))
+\f
+;;
+;;; Remote operations.
+;;;
+
+(define* (ls-remote-refs url #:key tags?)
+  "Return the list of references advertised at Git repository URL.  If TAGS?
+is true, limit to only refs/tags."
+  (define (ref? ref)
+    ;; Like `git ls-remote --refs', only show actual references.
+    (and (string-prefix? "refs/" ref)
+         (not (string-suffix? "^{}" ref))))
+
+  (define (tag? ref)
+    (string-prefix? "refs/tags/" ref))
+
+  (define (include? ref)
+    (and (ref? ref)
+         (or (not tags?) (tag? ref))))
+
+  (with-libgit2
+   (call-with-temporary-directory
+    (lambda (cache-directory)
+      (let* ((repository (repository-init cache-directory))
+             ;; Create an in-memory remote so we don't touch disk.
+             (remote (remote-create-anonymous repository url)))
+        (remote-connect remote)
+        (remote-disconnect remote)
+        (repository-close! repository)
+
+        (filter include? (map remote-head-name (remote-ls remote))))))))
 
 \f
 ;;;
diff --git a/guix/import/git.scm b/guix/import/git.scm
new file mode 100644
index 0000000000..52c98de197
--- /dev/null
+++ b/guix/import/git.scm
@@ -0,0 +1,217 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2021 Xinglu Chen <public@yoctocell.xyz>
+;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import git)
+  #:use-module (guix build utils)
+  #:use-module (guix diagnostics)
+  #:use-module (guix git)
+  #:use-module (guix git-download)
+  #:use-module (guix i18n)
+  #:use-module (guix packages)
+  #:use-module (guix upstream)
+  #:use-module (guix utils)
+  #:use-module (ice-9 format)
+  #:use-module (ice-9 match)
+  #:use-module (ice-9 rdelim)
+  #:use-module (ice-9 regex)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-26)
+  #:use-module (srfi srfi-34)
+  #:use-module (srfi srfi-35)
+  #:export (%generic-git-updater))
+
+;;; Commentary:
+;;;
+;;; This module provides a generic package updater for packages hosted on Git
+;;; repositories.
+;;;
+;;; It tries to be smart about tag names, but if it is not automatically able
+;;; to parse the tag names correctly, users can set the `tag-prefix',
+;;; `tag-suffix' and `tag-version-delimiter' properties of the package to make
+;;; the updater parse the Git tag name correctly.
+;;;
+;;; Code:
+
+;;; Errors & warnings
+
+(define-condition-type &git-no-valid-tags-error &error
+  git-no-valid-tags-error?)
+
+(define (git-no-valid-tags-error)
+  (raise (condition (&message (message "no valid tags found"))
+                    (&git-no-valid-tags-error))))
+
+(define-condition-type &git-no-tags-error &error
+  git-no-tags-error?)
+
+(define (git-no-tags-error)
+  (raise (condition (&message (message "no tags were found"))
+                    (&git-no-tags-error))))
+
+\f
+;;; Updater
+
+(define %pre-release-words
+  '("alpha" "beta" "rc" "dev" "test"))
+
+(define* (get-version-mapping tags #:key prefix suffix delim pre-releases?)
+  (define (guess-delim)
+    (let ((total (length tags))
+          (dots (reduce + 0 (map (cut string-count <> #\.) tags)))
+          (dashes (reduce + 0 (map (cut string-count <> #\-) tags)))
+          (underscores (reduce + 0 (map (cut string-count <> #\_) tags))))
+      (display (format #t "total: ~d, dots: ~d, dashes ~d, underscores ~d~%"
+              total dots dashes underscores))
+      (cond
+       ((>= dots (* total 0.35)) ".")
+       ((>= dashes (* total 0.8)) "-")
+       ((>= underscores (* total 0.8)) "_")
+       (else ""))))
+
+  (define delim-rx (regexp-quote (or delim (guess-delim))))
+  (define suffix-rx
+    (string-append
+     (or suffix
+         (if pre-releases?
+             (string-append ".*(" (string-join %pre-release-words "|") ").*")
+             ""))
+     "$"))
+  
+  (define prefix-rx (string-append "^" (or prefix "[^[:digit:]]*")))
+  (pk delim-rx)
+
+  (define tag-rx
+     (string-append "([[:digit:]][^" delim-rx "[:punct:]]*"
+                    "(" delim-rx "[^[:punct:]" delim-rx "]+)"
+                    ;; If there is are no delimiters, it could mean that the
+                    ;; version just contains one number (e.g., "2"), thus, use
+                    ;; "*" instead of "+" to match zero or more numbers.
+                    (if (string=? delim-rx "") "*" "+")
+                    ")" suffix-rx))
+
+  (define (get-version tag)
+    (let ((tag-match (regexp-exec (make-regexp tag-rx) tag)))
+      (pk tag)
+      (pk tag-match)
+      (pk tag-rx)
+      (and tag-match
+           (regexp-exec (make-regexp prefix-rx) (match:prefix tag-match))
+           (regexp-substitute/global
+            #f delim-rx (match:substring tag-match)
+            ;; Don't insert "." if there aren't any delimiters in the first
+            ;; place.
+            'pre (if (string=? delim-rx "") "" ".") 'post))))
+
+  (define (entry<? a b)
+    (eq? (version-compare (car a) (car b)) '<))
+
+  (let ((mapping (fold alist-cons '() (map get-version tags) tags)))
+    (stable-sort! (filter car mapping) entry<?)))
+
+(define* (get-latest-tag url #:key prefix suffix delim pre-releases?)
+  "Return the latest tag available from the Git repository at URL."
+  (define (pre-release? tag)
+    (any (lambda (rx) (regexp-exec (make-regexp rx regexp/icase) tag))
+              %pre-release-words))
+
+  (let* ((tags (map (cut string-drop <> (string-length "refs/tags/"))
+                    (ls-remote-refs url #:tags? #t)))
+         (versions->tags
+          (get-version-mapping (if pre-releases?
+                                   tags
+                                   (filter (negate pre-release?) tags))
+                               #:prefix prefix
+                               #:suffix suffix
+                               #:delim delim
+                               #:pre-releases? pre-releases?)))
+    (display versions->tags)
+    (newline)
+    (cond
+     ((null? tags)
+      (git-no-tags-error))
+     ((null? versions->tags)
+      (git-no-valid-tags-error))
+     (else
+      (match (last versions->tags)
+        ((version . tag)
+         (values version tag)))))))
+
+(define (latest-git-tag-version package tag-prefix tag-suffix
+                                tag-version-delimiter refresh-pre-releases?)
+  "Given a PACKAGE, the TAG-PREFIX, TAG-SUFFIX, TAG-VERSION-DELIMITER, and
+REFRESH-PRE-RELEASES?  properties of PACKAGE, returns the latest version of
+PACKAGE."
+  (guard (c ((or (git-no-tags-error? c) (git-no-valid-tags-error? c))
+             (warning (or (package-field-location package 'source)
+                          (package-location package))
+                      (G_ "~a for ~a~%")
+                      (condition-message c)
+                      (package-name package))
+             #f)
+            ((eq? (exception-kind c) 'git-error)
+             (warning (or (package-field-location package 'source)
+                          (package-location package))
+                      (G_ "failed to fetch Git repository for ~a~%")
+                      (package-name package))
+             #f))
+    (let* ((source (package-source package))
+           (url (git-reference-url (origin-uri source))))
+      ;;(format #t "~a~%" (package-name package))
+      (get-latest-tag url #:prefix tag-prefix #:suffix tag-suffix
+                      #:delim tag-version-delimiter
+                      #:pre-releases? refresh-pre-releases?))))
+
+(define (git-package? package)
+  "Whether the origin of PACKAGE is a Git repostiory."
+  (match (package-source package)
+    ((? origin? origin)
+     (and (eq? (origin-method origin) git-fetch)
+          (git-reference? (origin-uri origin))))
+    (_ #f)))
+
+(define (latest-git-release package)
+  "Return the latest release of PACKAGE."
+  (let* ((name (package-name package))
+         (properties (package-properties package))
+         (tag-prefix (assq-ref properties 'tag-prefix))
+         (tag-suffix (assq-ref properties 'tag-suffix))
+         (tag-version-delimiter (assq-ref properties 'tag-version-delimiter))
+         (refresh-pre-releases? (assq-ref properties 'refresh-pre-releases?))
+         (old-version (package-version package))
+         (url (git-reference-url (origin-uri (package-source package))))
+         (new-version (latest-git-tag-version package
+                                              tag-prefix
+                                              tag-suffix
+                                              tag-version-delimiter
+                                              refresh-pre-releases?)))
+
+    (if new-version
+        (upstream-source
+         (package name)
+         (version new-version)
+         (urls (list url)))
+        ;; No new release or no tags available.
+        #f)))
+
+(define %generic-git-updater
+  (upstream-updater
+   (name 'generic-git)
+   (description "Updater for packages hosted on Git repositories")
+   (pred git-package?)
+   (latest latest-git-release)))
-- 
2.33.0


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]