[bug#53818] [PATCH 0/3] Add Repology updater

[bug#53818] [PATCH 0/3] Add Repology updater

[Xinglu Chen]

[-- Attachment #1: Type: text/plain, Size: 1762 bytes --]

Hi,

This patchset adds a new updater, which scans Repology[1] for updates.
It should technically support all packages in Guix!  :-)

The data on Repology isn’t as detailed as the one on language-specific
repos, e.g., PyPI, so the updater doesn’t support things like ‘input
changes’.  If the source URL doesn’t contain the version verbatim[2], it
won’t be able reconstruct the URL of the updated version, meaning that
‘guix refresh -u’ won’t work.

Because of the way ‘%updaters’ in (guix upstream) works, the Repology
updater is the first or second updater that is used (since it
technically works on ever package), but because of the limitations I
mentioned above, the result might not always be the best.  The Repology
updater is mostly useful for things that don’t already have an updater,
e.g., ‘maven-dependency-tree’.  Would it make sense to hard-code the
‘%updaters’ variable and put the Repology last in the list?

[1]: <https://repology.org>
[2]: e.g., the version is “1.0.0” but the URL is
     “https://example.org/1_0_0.tar.gz”

Xinglu Chen (3):
  git-download: Export <git-reference>.
  import: Add 'repology' updater.
  gnu: xorg-server-xwayland: Set 'repology-name' property.

 Makefile.am               |   3 +
 doc/guix.texi             |   7 ++
 gnu/packages/xorg.scm     |   2 +
 guix/git-download.scm     |   3 +-
 guix/import/repology.scm  | 226 ++++++++++++++++++++++++++++++++++++++
 tests/import-repology.scm | 145 ++++++++++++++++++++++++
 6 files changed, 385 insertions(+), 1 deletion(-)
 create mode 100644 guix/import/repology.scm
 create mode 100644 tests/import-repology.scm


base-commit: 7c9ad54b0616112c7eea6dd02379616aef206490
-- 
2.34.1




[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

[bug#53818] [PATCH 0/3] Add Repology updater

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1293 bytes --]

Xinglu Chen schreef op zo 06-02-2022 om 12:50 [+0100]:
> Because of the way ‘%updaters’ in (guix upstream) works, the Repology
> updater is the first or second updater that is used (since it
> technically works on ever package), but because of the limitations I
> mentioned above, the result might not always be the best.  The Repology
> updater is mostly useful for things that don’t already have an updater,
> e.g., ‘maven-dependency-tree’.  Would it make sense to hard-code the
> ‘%updaters’ variable and put the Repology last in the list?

I would prefer not to hardcode %updaters and keep the current
discovery mechanism, such that people can experiment with updaters
outside a git checkout of guix and in channels.

FWIW it would be useful to have the same mechanism for importers.

However, it might be a good idea to do some _postprocessing_ on
the discovered list of updaters, e.g. they could be sorted on
'genericity' with 'stable-sort' (*):

(define (genericity x)
  (cond ((it is "generic-SOMETHING") 1)
        ((it is repology) 2)
        (#true 0)))

(define (less x y)
  (<= (genericity x) (genericity y)))

(*) stable-sort and not sort, to preserve alphabetical ordering
for updaters with the same genericity.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#53818] [PATCH 0/3] Add Repology updater

[Xinglu Chen]

[-- Attachment #1: Type: text/plain, Size: 1424 bytes --]

Maxime schrieb am Sonntag der 06. Februar 2022 um 13:41 +01:

> Xinglu Chen schreef op zo 06-02-2022 om 12:50 [+0100]:
>> Because of the way ‘%updaters’ in (guix upstream) works, the Repology
>> updater is the first or second updater that is used (since it
>> technically works on ever package), but because of the limitations I
>> mentioned above, the result might not always be the best.  The Repology
>> updater is mostly useful for things that don’t already have an updater,
>> e.g., ‘maven-dependency-tree’.  Would it make sense to hard-code the
>> ‘%updaters’ variable and put the Repology last in the list?
>
> I would prefer not to hardcode %updaters and keep the current
> discovery mechanism, such that people can experiment with updaters
> outside a git checkout of guix and in channels.

Good point.

> FWIW it would be useful to have the same mechanism for importers.
>
> However, it might be a good idea to do some _postprocessing_ on
> the discovered list of updaters, e.g. they could be sorted on
> 'genericity' with 'stable-sort' (*):
>
> (define (genericity x)
>   (cond ((it is "generic-SOMETHING") 1)
>         ((it is repology) 2)
>         (#true 0)))
>
> (define (less x y)
>   (<= (genericity x) (genericity y)))
>
> (*) stable-sort and not sort, to preserve alphabetical ordering
> for updaters with the same genericity.

That looks like a good idea

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

[bug#53818] [PATCH 1/3] git-download: Export <git-reference>.

[Xinglu Chen]

* guix/git-download.scm: Export <git-reference> record.
---
 guix/git-download.scm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/guix/git-download.scm b/guix/git-download.scm
index 5e624b9ae9..43424f2f19 100644
--- a/guix/git-download.scm
+++ b/guix/git-download.scm
@@ -44,7 +44,8 @@ (define-module (guix git-download)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-34)
   #:use-module (srfi srfi-35)
-  #:export (git-reference
+  #:export (<git-reference>
+            git-reference
             git-reference?
             git-reference-url
             git-reference-commit
-- 
2.34.1

[bug#53818] [PATCH 2/3] import: Add 'repology' updater.

[Xinglu Chen]

* guix/import/repology.scm
* tests/import-repology.scm: New files.
* Makefile.am (MODULES): Register them.
* doc/guix.texi (Invoking guix refresh): Document it.
---
 Makefile.am               |   3 +
 doc/guix.texi             |   7 ++
 guix/import/repology.scm  | 226 ++++++++++++++++++++++++++++++++++++++
 tests/import-repology.scm | 145 ++++++++++++++++++++++++
 4 files changed, 381 insertions(+)
 create mode 100644 guix/import/repology.scm
 create mode 100644 tests/import-repology.scm

diff --git a/Makefile.am b/Makefile.am
index 7463606d20..6792917b59 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -16,6 +16,7 @@
 # Copyright © 2019 Efraim Flashner <efraim@flashner.co.il>
 # Copyright © 2021 Chris Marusich <cmmarusich@gmail.com>
 # Copyright © 2021 Andrew Tropin <andrew@trop.in>
+# Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
 #
 # This file is part of GNU Guix.
 #
@@ -271,6 +272,7 @@ MODULES =					\
   guix/import/opam.scm				\
   guix/import/print.scm				\
   guix/import/pypi.scm				\
+  guix/import/repology.scm			\
   guix/import/stackage.scm			\
   guix/import/texlive.scm   			\
   guix/import/utils.scm				\
@@ -488,6 +490,7 @@ SCM_TESTS =					\
   tests/home-import.scm				\
   tests/import-git.scm				\
   tests/import-github.scm			\
+  tests/import-repology.scm			\
   tests/import-utils.scm			\
   tests/inferior.scm				\
   tests/lint.scm				\
diff --git a/doc/guix.texi b/doc/guix.texi
index 0cf865a672..15d215dd48 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -12932,6 +12932,13 @@
       (release-tag-version-delimiter . ":"))))
 @end lisp
 
+@item repology
+an updater that scans @uref{https://repology.org, Repology}, a website
+that tracks packages on various package repositories, for updates.
+
+The name of a package in Guix is not always that same as the name on
+Repology; users can set the @code{repology-name} package property to
+make the updater use the correct name.
 
 @end table
 
diff --git a/guix/import/repology.scm b/guix/import/repology.scm
new file mode 100644
index 0000000000..1d88022c1d
--- /dev/null
+++ b/guix/import/repology.scm
@@ -0,0 +1,226 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import repology)
+  #:use-module (guix diagnostics)
+  #:use-module (guix diagnostics)
+  #:use-module (guix git-download)
+  #:use-module (guix i18n)
+  #:use-module (guix import json)
+  #:use-module (guix import utils)
+  #:use-module (guix memoization)
+  #:use-module (guix packages)
+  #:use-module (guix upstream)
+  #:use-module (guix utils)
+  #:use-module (ice-9 match)
+  #:use-module (json)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-2)
+  #:use-module (srfi srfi-26)
+  #:use-module (srfi srfi-43)
+  #:export (repology-latest-release
+            %repology-updater))
+
+;;; Commentary:
+;;;
+;;; This module provides an updater which scans Repology, a site that monitors
+;;; several package repolsitories, for updates.  This means that if any other
+;;; package repository has a version of a package that is newer than the
+;;; version in Guix, the package should be able to be updated.  The updater
+;;; should in theory work for all packages in Guix, but the names of some
+;;; packages on Repology don't match the name in Guix.  The 'repology-name'
+;;; package property can be used to fix this.
+;;;
+;;; Guix already has many different updaters for language-specific packages,
+;;; and these typically provide more accurate data, e.g., input changes,
+;;; signature URLs.  The Repology updater should really be treated as a last
+;;; resort for those packages that don't have any other updater to rely on.
+;;;
+;;; See <https://repology.org/api/v1> for the API.
+;;;
+;;; Code:
+
+(define %repology-url
+   "https://repology.org/api/v1/project")
+
+(define* (package-name->repology-name name #:key (attempt 1))
+  "Convert NAME, the name of a Guix package, to the name of the package on
+Repology.  It doesn't always guess the correct name on the first attempt, so
+on the second attempt it will try to guess another name."
+  (match attempt
+    (1 (cond
+        ((string-prefix? "ghc-" name)
+         (string-append "haskell:"
+                        (string-drop name (string-length "ghc-"))))
+        ((string-prefix? "ocaml-" name)
+         (string-append "ocaml:"
+                        (string-drop name (string-length "ocaml-"))))
+        ((string-prefix? "perl-" name)
+         (string-append "perl:"
+                        (string-drop name (string-length "perl-"))))
+        ((string-prefix? "emacs-" name)
+         (string-append "emacs:"
+                        (string-drop name (string-length "emacs-"))))
+        ((string-prefix? "go-" name)
+         (string-append "go:"
+                        (string-drop name (string-length "go-"))))
+        ((string-prefix? "rust-" name)
+         (string-append "rust:"
+                        (string-drop name (string-length "rust-"))))
+        ((string-prefix? "lua-" name)
+         (string-append "lua:"
+                        (string-drop name (string-length "lua-"))))
+        ((string-prefix? "node-" name)
+         (string-append "node:"
+                        (string-drop name (string-length "node-"))))
+        ((string-prefix? "python-" name)
+         (string-append "python:"
+                        (string-drop name (string-length "python-"))))
+        ((string-prefix? "java-" name)
+         (string-append "java:"
+                        (string-drop name (string-length "java-"))))
+        ((string-prefix? "r-" name)
+         (string-append "r:"
+                        (string-drop name (string-length "r-"))))
+        ((string-prefix? "ruby-" name)
+         (string-append "ruby:"
+                        (string-drop name (string-length "ruby-"))))
+        ((string-prefix? "xf86-" name)
+         (string-append "xdrv:"
+                        (string-drop name (string-length "xf86-"))))
+        ((string-prefix? "font-" name)
+         (string-append "fonts:"
+                        (string-drop name (string-length "font-"))))
+        (else name)))
+    (2 (cond
+        ((string-prefix? "xf86-video" name)
+         (string-append "xdrv:"
+                        (string-drop name (string-length "xf86-video-"))))
+        ((string-prefix? "xf86-input" name)
+         (string-append "xdrv:"
+                        (string-drop name (string-length "xf86-input-"))))
+        (else name)))))  
+
+\f
+;;; JSON mappings.
+
+(define-json-mapping <repology-package> make-repology-package
+  repology-package?
+  json->repology-package
+  (repository repology-package-repository "repo")
+  (src-name repology-package-src-name "srcname")
+  (binary-name repology-package-binary-name "binname")
+  (visible-name repology-package-visible-name "visiblename")
+  (version repology-package-version)
+  (original-version repology-package-original-version "origversion")
+  (status repology-package-status)
+  (summary repology-package-summary)
+  (categories repology-package-categories)
+  (licenses repology-package-licenses)
+  (maintainers repology-package-maintainers))
+
+\f
+;;; Updater.
+
+(define repology-fetch-info
+  (memoize
+   (lambda (package)
+     "Fetch information about PACKAGE using the Repology API."
+     (define (name->info name )
+       (let ((url (string-append %repology-url "/" name)))
+         (and=> (json-fetch url)
+                (lambda (url)
+                  (vector-map (lambda (a b)
+                                (json->repology-package b))
+                              url)))))
+     
+     (let* ((name (or (assoc-ref (package-properties package)
+                                 'repology-name)
+                      (package-name->repology-name (package-name package))))
+            (info (name->info name)))
+       (if (and info (not (vector-empty? info)))
+           info
+           (let ((info (name->info (package-name->repology-name
+                                    (package-name package)
+                                    #:attempt 2))))
+             (if (and info (not (vector-empty? info)))
+                 info
+                 (begin
+                   (warning (G_ "package not found on Repology: ~a\n")
+                            (package-name package))
+                   #f))))))))
+
+(define (update-url url old-version new-version)
+  "Replace OLD-VERSION in URL with NEW-VERSION."
+  (match (factorize-uri url old-version)
+    ((? string? uri) #f)
+    ((factorized ...)
+     (apply string-append
+            (map (lambda (component)
+                   (match component
+                     ('version new-version)
+                     ((? string?) component)))
+                 factorized)))))
+
+(define (package-source-urls package version)
+  "Return a list of URLs for PACKAGE at VERSION.  If no URL was successfully constructed, return #f."
+  (let ((old-version (package-version package)))
+    ;; XXX: (guix upstream) only supports tarballs and Git repos for now.
+    (match (origin-uri (package-source package))
+      (($ <git-reference> url commit recursive?)
+       (and-let* ((updated-commit (if (string=? old-version commit)
+                                      version
+                                      (update-url commit old-version version))))
+         (list (git-reference
+                (url url)
+                (commit updated-commit)
+                (recursive? recursive?)))))
+      ((? string? url)
+       (list (update-url url old-version version)))
+      ((? list? urls)
+       (map (cut update-url <> old-version version) urls))
+      (_ #f))))
+
+(define (stable-version? repology-package)
+  (and (or (equal? "newest" (repology-package-status repology-package))
+           (equal? "unique" (repology-package-status repology-package)))
+       (repology-package-version repology-package)))
+
+;; XXX: 'package' will clash with the 'package' field of 'upstream-source'.
+(define (repology-latest-release pkg)
+  "Return the latest release of the PKG on Repology named NAME."
+  (and-let* ((packages (repology-fetch-info pkg))
+             (versions (filter-map stable-version?
+                                          (vector->list packages)))
+             (latest-version (if (null? versions)
+                                 #f
+                                 (car versions))))
+    ;; TODO: set 'signature-urls'.
+    (upstream-source
+     (package (package-name pkg))
+     (version latest-version)
+     (urls (package-source-urls pkg latest-version)))))
+
+(define %repology-updater
+(upstream-updater
+ (name 'repology)
+ (description "Updater for packages on Repology")
+ (pred (const #t))
+ (latest repology-latest-release)))
+
+;;; repology.scm ends here
diff --git a/tests/import-repology.scm b/tests/import-repology.scm
new file mode 100644
index 0000000000..f95071ce07
--- /dev/null
+++ b/tests/import-repology.scm
@@ -0,0 +1,145 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (test-import-repology)
+  #:use-module (guix download)
+  #:use-module (guix git-download)
+  #:use-module (guix import repology)
+  #:use-module (guix packages)
+  #:use-module (guix tests)
+  #:use-module (guix upstream)
+  #:use-module (json)
+  #:use-module (srfi srfi-64))
+
+(test-begin "repology")
+
+(define package-using-git-repository
+  (dummy-package
+   "foo"
+   (version "1.0")
+   (source
+    (origin
+      (method git-fetch)
+      (uri (git-reference
+            (url "https://git.example.org/foo")
+            (commit "1.0")))
+      (sha256 #f)))))
+
+(define package-using-tarball
+  (dummy-package
+   "foo"
+   (version "1.0")
+   (source
+    (origin
+      (method git-fetch)
+      (uri (string-append "https://example.org/foo-" version ".tar.gz"))
+      (sha256 #f)))))
+
+(define package-using-tarball-multiple-urls
+  (dummy-package
+   "foo"
+   (version "1.0")
+   (source
+    (origin
+      (method git-fetch)
+      (uri (list (string-append "https://example.org/foo-"
+                                version ".tar.gz")
+                 (string-append "https://mirror.example.org/foo-"
+                                version ".tar.gz")))
+      (sha256 #f)))))
+
+(define %test-json
+"[
+  {
+    \"repo\": \"aur\",
+    \"srcname\": \"foo\",
+    \"binname\": \"foo\",
+    \"visiblename\": \"foo\",
+    \"version\": \"1.0.r25.gb86405a\",
+    \"maintainers\": [
+      \"bob@aur\"
+    ],
+    \"licenses\": [
+      \"LGPL3+\"
+    ],
+    \"summary\": \"foo bar\"
+    \"status\": \"rolling\",
+    \"origversion\": \"1.0.r25.gb86405a-1\"
+  },
+  {
+    \"repo\": \"gnuguix\",
+    \"srcname\": \"foo\",
+    \"binname\": \"foo\",
+    \"visiblename\": \"foo\",
+    \"version\": \"1.0\",
+    \"summary\": \"foo bar\",
+    \"status\": \"outdated\",
+    \"origversion\": null
+  },
+  {
+    \"repo\": \"nix_unstable\",
+    \"name\": \"foo\",
+    \"visiblename\": \"foo\",
+    \"version\": \"2.0\",
+    \"maintainers\": [
+      \"bob@example.org\"
+    ],
+    \"licenses\": [
+      \"LGPL-3.0-or-later\"
+    ],
+    \"summary\": \"foo bar\",
+    \"status\": \"newest\",
+    \"origversion\": null
+  }
+]")
+
+(define (latest-release package)
+  (mock ((guix import json) json-fetch
+         (lambda (url)
+           (json-string->scm %test-json)))
+        (repology-latest-release package)))
+
+(test-equal "package using Git repo: version"
+  "2.0"
+  (upstream-source-version
+   (latest-release package-using-git-repository)))
+
+(test-equal "package using Git repo: git-reference"
+  (list (git-reference
+         (url "https://git.example.org/foo")
+         (commit "2.0")))
+  (upstream-source-urls
+   (latest-release package-using-git-repository)))
+
+(test-equal "package using tarball: version"
+  "2.0"
+  (upstream-source-version
+   (latest-release package-using-tarball)))
+
+(test-equal "package using tarball: URL"
+  (list "https://example.org/foo-2.0.tar.gz")
+  (upstream-source-urls
+   (latest-release package-using-tarball)))
+
+(test-equal "package using tarball: multiple URLs"
+  (list "https://example.org/foo-2.0.tar.gz"
+        "https://mirror.example.org/foo-2.0.tar.gz")
+  (upstream-source-urls
+   (latest-release package-using-tarball-multiple-urls)))
+
+(test-end "repology")
-- 
2.34.1

[bug#53818] [PATCH 2/3] import: Add 'repology' updater.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 553 bytes --]

Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
> +(define* (package-name->repology-name name #:key (attempt 1))
> +  "Convert NAME, the name of a Guix package, to the name of the package on
> +Repology.  It doesn't always guess the correct name on the first attempt, so
> +on the second attempt it will try to guess another name."
> +  (match attempt
> +    (1 (cond

Could you add 'minetest' to the list?  Sometimes they are named
'minetest-mod-foo' (Debian), sometimes 'minetest-foo' (Guix,
Archlinux).

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#53818] [PATCH 2/3] import: Add 'repology' updater.

[Xinglu Chen]

[-- Attachment #1: Type: text/plain, Size: 618 bytes --]

Maxime schrieb am Sonntag der 06. Februar 2022 um 14:11 +01:

> Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
>> +(define* (package-name->repology-name name #:key (attempt 1))
>> +  "Convert NAME, the name of a Guix package, to the name of the package on
>> +Repology.  It doesn't always guess the correct name on the first attempt, so
>> +on the second attempt it will try to guess another name."
>> +  (match attempt
>> +    (1 (cond
>
> Could you add 'minetest' to the list?  Sometimes they are named
> 'minetest-mod-foo' (Debian), sometimes 'minetest-foo' (Guix,
> Archlinux).

Sure!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

[bug#53818] [PATCH 2/3] import: Add 'repology' updater.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 545 bytes --]

Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
> +             (if (and info (not (vector-empty? info)))
> +                 info
> +                 (begin
> +                   (warning (G_ "package not found on Repology: ~a\n")
> +                            (package-name package))
> +                   #f))))))))

Why a warning here?  For other refreshers, we just return '#f' and
don't print a warning.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#53818] [PATCH 2/3] import: Add 'repology' updater.

[Xinglu Chen]

[-- Attachment #1: Type: text/plain, Size: 780 bytes --]

Maxime schrieb am Sonntag der 06. Februar 2022 um 14:13 +01:

> Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
>> +             (if (and info (not (vector-empty? info)))
>> +                 info
>> +                 (begin
>> +                   (warning (G_ "package not found on Repology: ~a\n")
>> +                            (package-name package))
>> +                   #f))))))))
>
> Why a warning here?  For other refreshers, we just return '#f' and
> don't print a warning.

Well, some actually do print a warning, e.g., opam.  The warning makes
it clear that a package wasn’t found when running ‘guix refresh’
(without arguments).

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

[bug#53818] [PATCH 2/3] import: Add 'repology' updater.

[Maxime Devos]

Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
> +     (define (name->info name )

Superfluous space.

[bug#53818] [PATCH 2/3] import: Add 'repology' updater.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 956 bytes --]

Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
> +      (($ <git-reference> url commit recursive?)
> +       (and-let* ((updated-commit (if (string=? old-version commit)
> +                                      version
> +                                      (update-url commit old-version version))))

When does the URL of a git-reference include a version?

> +         (list (git-reference
> +                (url url)
> +                (commit updated-commit)
> +                (recursive? recursive?)))))

(guix upstream) expects a single git-reference, not a list of git-
references.

Also, depending on the order of fields here doesn't seem great, I' do
something like

((? git-reference? reference)
 (git-reference
    (inherit reference)
    (commit updated-commit)))

instead.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#53818] [PATCH 2/3] import: Add 'repology' updater.

[Xinglu Chen]

[-- Attachment #1: Type: text/plain, Size: 1281 bytes --]

Maxime schrieb am Sonntag der 06. Februar 2022 um 14:17 +01:

> Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
>> +      (($ <git-reference> url commit recursive?)
>> +       (and-let* ((updated-commit (if (string=? old-version commit)
>> +                                      version
>> +                                      (update-url commit old-version version))))
>
> When does the URL of a git-reference include a version?

It doesn’t, but the tag usually does; I should probably rename the
‘update-url’ procedure to something else, maybe ‘update-version’?

>> +         (list (git-reference
>> +                (url url)
>> +                (commit updated-commit)
>> +                (recursive? recursive?)))))
>
> (guix upstream) expects a single git-reference, not a list of git-
> references.

Oops, noted.

> Also, depending on the order of fields here doesn't seem great, I' do
> something like
>
> ((? git-reference? reference)
>  (git-reference
>     (inherit reference)
>     (commit updated-commit)))
>
> instead.

Then I can drop the first patch of the series too.  :-)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

[bug#53818] [PATCH 2/3] import: Add 'repology' updater.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 469 bytes --]

Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
> +(define (stable-version? repology-package)
> +  (and (or (equal? "newest" (repology-package-status repology-package))
> +           (equal? "unique" (repology-package-status repology-package)))
> +       (repology-package-version repology-package)))

What does 'stable' mean here?  We usually just use the latest version
of a package, unlike, say, Debian stable.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#53818] [PATCH 2/3] import: Add 'repology' updater.

[Xinglu Chen]

[-- Attachment #1: Type: text/plain, Size: 698 bytes --]

Maxime schrieb am Sonntag der 06. Februar 2022 um 14:18 +01:

> Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
>> +(define (stable-version? repology-package)
>> +  (and (or (equal? "newest" (repology-package-status repology-package))
>> +           (equal? "unique" (repology-package-status repology-package)))
>> +       (repology-package-version repology-package)))
>
> What does 'stable' mean here?  We usually just use the latest version
> of a package, unlike, say, Debian stable.

“Stable” refers to a stable release, so things like pre-releases will be
ignored.  Maybe it should be renamed to ‘latest-release?’ or
‘latest-stable-release?’?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

[bug#53818] [PATCH 2/3] import: Add 'repology' updater.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 422 bytes --]

Xinglu Chen schreef op zo 06-02-2022 om 16:34 [+0100]:
> > of a package, unlike, say, Debian stable.
> 
> “Stable” refers to a stable release, so things like pre-releases will
> be
> ignored.  Maybe it should be renamed to ‘latest-release?’ or
> ‘latest-stable-release?’?

I would go with 'latest-release?' and a comment or docstring explaining
that pre-releases are ignored.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#53818] [PATCH 2/3] import: Add 'repology' updater.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 317 bytes --]

Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
> +(test-equal "package using Git repo: git-reference"
> +  (list (git-reference
> +         (url "https://git.example.org/foo")
> +         (commit "2.0")))

Drop 'list', (guix upstream) doesn't expect a list here.

Greetings,
Maxime

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#53818] [PATCH 2/3] import: Add 'repology' updater.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 492 bytes --]

Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
> +         (and=> (json-fetch url)

Given that we'll be contacting 'repology' for _every_ package
when doing "guix refresh -l", some caching seems in order.
Could 'http-fetch/cached' be used here?  json-fetch may
need to be modified to allow overriding the http-fetch procedure
used.

FWIW this could be useful for some other updaters (e.g. the
Minetest/ContentDB updater), to a lesser degree.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#53818] [PATCH 2/3] import: Add 'repology' updater.

[Xinglu Chen]

[-- Attachment #1: Type: text/plain, Size: 579 bytes --]

Maxime schrieb am Sonntag der 06. Februar 2022 um 14:23 +01:

> Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
>> +         (and=> (json-fetch url)
>
> Given that we'll be contacting 'repology' for _every_ package
> when doing "guix refresh -l", some caching seems in order.
> Could 'http-fetch/cached' be used here?  json-fetch may
> need to be modified to allow overriding the http-fetch procedure
> used.

That would be a good idea.  It seems like ‘http-fetch/cached’ doesn’t
take a #:headers argument, so it will have to be modified first.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

[bug#53818] [PATCH 2/3] import: Add 'repology' updater.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 201 bytes --]

Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
> +             (latest-version (if (null? versions)

Perhaps (and (pair? versions) (car versions)), YMMV.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#53818] [PATCH 2/3] import: Add 'repology' updater.

[Xinglu Chen]

[-- Attachment #1: Type: text/plain, Size: 288 bytes --]

Maxime schrieb am Sonntag der 06. Februar 2022 um 14:23 +01:

> Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
>> +             (latest-version (if (null? versions)
>
> Perhaps (and (pair? versions) (car versions)), YMMV.

That looks a little cleaner, thanks!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

[bug#53818] [PATCH 3/3] gnu: xorg-server-xwayland: Set 'repology-name' property.

[Xinglu Chen]

* gnu/packages/xorg.scm (xorg-server-xwayland):[properties]: Set
'repology-name'.
---
 gnu/packages/xorg.scm | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/gnu/packages/xorg.scm b/gnu/packages/xorg.scm
index 14e35d19ae..ee45630596 100644
--- a/gnu/packages/xorg.scm
+++ b/gnu/packages/xorg.scm
@@ -5520,6 +5520,8 @@ (define-public xorg-server-xwayland
                     (lambda _
                       (substitute* (find-files "." "\\.c$")
                         (("/bin/sh") (which "sh"))))))))
+    (properties
+     '((repology-name . "xwayland")))
     (synopsis "Xorg server with Wayland backend")
     (description "Xwayland is an X server for running X clients under
 Wayland.")
-- 
2.34.1

[bug#53818] [PATCH 3/3] gnu: xorg-server-xwayland: Set 'repology-name' property.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 559 bytes --]

Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
>                      (lambda _
>                        (substitute* (find-files "." "\\.c$")
>                          (("/bin/sh") (which "sh"))))))))

FWIW this is wrong when cross-compiling,
this needs to use (search-input-file inputs "bin/sh") instead
such that a (cross-compiled) shell from 'inputs' is used instead of a
(native) bash from 'native-inputs'.

That's a bug in the old package definition though.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#53818] [PATCH v2 0/7] Add Repology updater

[Xinglu Chen]

[-- Attachment #1: Type: text/plain, Size: 1339 bytes --]

Hi,

The following changes have been made since v1:

* Sort the list of updaters to make generic ones come last.

* Make ‘json-fetch’ use ‘http-fetch/cached’ if the #:cached? argument is
  #t.

* Added some additional rules for converting Guix package names to
  Repology package names.

* Some typo fixes and more docstrings.

Xinglu Chen (7):
  upstream: Sort list of updaters.
  http-client: Make 'http-fetch/cached' take '#:headers' argument.
  http-client: 'http-fetch/cached' accepts a string or a <uri>.
  import: json: Make 'json-fetch' take '#:cached?' argument.
  import: Add 'repology' updater.
  gnu: xorg-server-xwayland: Set 'repology-name' property.
  gnu: xorg-server-xwayland: Prepare for cross-compilation.

 Makefile.am               |   3 +
 doc/guix.texi             |   7 ++
 gnu/packages/xorg.scm     |   8 +-
 guix/http-client.scm      |  18 ++-
 guix/import/json.scm      |  14 ++-
 guix/import/repology.scm  | 235 ++++++++++++++++++++++++++++++++++++++
 guix/upstream.scm         |  29 ++++-
 tests/import-repology.scm | 145 +++++++++++++++++++++++
 8 files changed, 441 insertions(+), 18 deletions(-)
 create mode 100644 guix/import/repology.scm
 create mode 100644 tests/import-repology.scm


base-commit: 7c9ad54b0616112c7eea6dd02379616aef206490
-- 
2.34.1




[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

[bug#53818] [PATCH v2 1/7] upstream: Sort list of updaters.

[Xinglu Chen]

The order determines which updater gets used first when running ‘guix refresh’
on a package.  We want the most generic updaters to be last since they usually
don’t provide as much or accurate information as language-specific updaters.

* guix/upstream.scm (sort-updaters): New procedure.
(%updaters): Use it.
---
 guix/upstream.scm | 29 +++++++++++++++++++++++------
 1 file changed, 23 insertions(+), 6 deletions(-)

diff --git a/guix/upstream.scm b/guix/upstream.scm
index 6666803a92..d5faf9d3ee 100644
--- a/guix/upstream.scm
+++ b/guix/upstream.scm
@@ -250,14 +250,31 @@ (define (importer-modules)
                           %load-path)
                      #:warn warn-about-load-error)))
 
+(define (sort-updaters updaters)
+  "Sort UPDATERS by putting the more generic ones last."
+  (define (genericity updater)
+    (cond
+     ((equal? 'repology (upstream-updater-name updater))
+      2)
+     ((string-prefix? "generic-"
+                      (symbol->string (upstream-updater-name updater)))
+      1)     
+     (else 0)))
+
+  (define (less a b)
+    (<= (genericity a) (genericity b)))
+
+  (stable-sort updaters less))
+
 (define %updaters
   ;; The list of publically-known updaters.
-  (delay (fold-module-public-variables (lambda (obj result)
-                                         (if (upstream-updater? obj)
-                                             (cons obj result)
-                                             result))
-                                       '()
-                                       (importer-modules))))
+  (delay (sort-updaters
+          (fold-module-public-variables (lambda (obj result)
+                                          (if (upstream-updater? obj)
+                                              (cons obj result)
+                                              result))
+                                        '()
+                                        (importer-modules)))))
 
 ;; Tests need to mock this variable so mark it as "non-declarative".
 (set! %updaters %updaters)
-- 
2.34.1

[bug#53818] [PATCH v2 2/7] http-client: Make 'http-fetch/cached' take '#:headers' argument.

[Xinglu Chen]

* guix/http-client.scm (http-fetch/cached): Take ‘#:headers’ keyword argument.
---
 guix/http-client.scm | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/guix/http-client.scm b/guix/http-client.scm
index 10bc278023..058f09852f 100644
--- a/guix/http-client.scm
+++ b/guix/http-client.scm
@@ -3,6 +3,7 @@
 ;;; Copyright © 2015 Mark H Weaver <mhw@netris.org>
 ;;; Copyright © 2012, 2015 Free Software Foundation, Inc.
 ;;; Copyright © 2017 Tobias Geerinckx-Rice <me@tobias.gr>
+;;; Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -283,9 +284,11 @@ (define* (http-fetch/cached uri #:key (ttl (%http-cache-ttl)) text?
                             (write-cache dump-port)
                             (cache-miss (const #t))
                             (log-port (current-error-port))
-                            (timeout 10))
+                            (timeout 10)
+                            (headers '((user-agent . "GNU Guile"))))
   "Like 'http-fetch', return an input port, but cache its contents in
-~/.cache/guix.  The cache remains valid for TTL seconds.
+~/.cache/guix.  The cache remains valid for TTL seconds.  HEADERS is an alist
+of extra HTTP headers.  The cache time will automatically be added to HEADERS, so there is no need to specify it.
 
 Call WRITE-CACHE with the HTTP input port and the cache output port to write
 the data to cache.  Call CACHE-MISS with URI just before fetching data from
@@ -300,8 +303,8 @@ (define cache-time
         (and cache-port
              (stat:mtime (stat cache-port))))
 
-      (define headers
-        `((user-agent . "GNU Guile")
+      (define headers*
+        `(,@headers
           ,@(if cache-time
                 `((if-modified-since
                    . ,(time-utc->date (make-time time-utc 0 cache-time))))
@@ -316,7 +319,7 @@ (define headers
                      (raise c))))
         (let ((port (http-fetch uri #:text? text?
                                 #:log-port log-port
-                                #:headers headers #:timeout timeout)))
+                                #:headers headers* #:timeout timeout)))
           (cache-miss uri)
           (mkdir-p (dirname file))
           (when cache-port
-- 
2.34.1

[bug#53818] [PATCH v2 3/7] http-client: 'http-fetch/cached' accepts a string or a <uri>.

[Xinglu Chen]

This is consistent with the 'http-fetch' procedure.

* guix/http-client.scm (http-fetch/cached): The 'uri' argument can be a string
  or a <uri> record.
---
 guix/http-client.scm | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/guix/http-client.scm b/guix/http-client.scm
index 058f09852f..08efdd5e47 100644
--- a/guix/http-client.scm
+++ b/guix/http-client.scm
@@ -297,7 +297,10 @@ (define* (http-fetch/cached uri #:key (ttl (%http-cache-ttl)) text?
 TIMEOUT specifies the timeout in seconds for connection establishment.
 
 Write information about redirects to LOG-PORT."
-  (let ((file (cache-file-for-uri uri)))
+  (let* ((uri (if (string? uri)
+                  (string->uri uri)
+                  uri))
+         (file (cache-file-for-uri uri)))
     (define (update-cache cache-port)
       (define cache-time
         (and cache-port
-- 
2.34.1

[bug#53818] [PATCH v2 4/7] import: json: Make 'json-fetch' take '#:cached?' argument.

[Xinglu Chen]

* json.scm (json-fetch): Add ‘#:cached?’ keyword argument.
---
 guix/import/json.scm | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/guix/import/json.scm b/guix/import/json.scm
index 0c98bb25b8..6f88353659 100644
--- a/guix/import/json.scm
+++ b/guix/import/json.scm
@@ -3,6 +3,7 @@
 ;;; Copyright © 2015, 2016 Eric Bavier <bavier@member.fsf.org>
 ;;; Copyright © 2018, 2019 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2020 Ricardo Wurmus <rekado@elephly.net>
+;;; Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -37,17 +38,22 @@ (define-module (guix import json)
 (define* (json-fetch url
                      ;; Note: many websites returns 403 if we omit a
                      ;; 'User-Agent' header.
-                     #:key (headers `((user-agent . "GNU Guile")
-                                      (Accept . "application/json"))))
+                     #:key
+                     (headers `((user-agent . "GNU Guile")
+                                (Accept . "application/json")))
+                     cached?)
   "Return a representation of the JSON resource URL (a list or hash table), or
 #f if URL returns 403 or 404.  HEADERS is a list of HTTP headers to pass in
-the query."
+the query.  If CACHED? is #t, 'http-fetch/cached' will be used to fetch URL."
   (guard (c ((and (http-get-error? c)
                   (let ((error (http-get-error-code c)))
                     (or (= 403 error)
                         (= 404 error))))
              #f))
-    (let* ((port   (http-fetch url #:headers headers))
+    (let* ((port   ((if cached?
+                        http-fetch/cached
+                        http-fetch)
+                    url #:headers headers))
            (result (json->scm port)))
       (close-port port)
       result)))
-- 
2.34.1

[bug#53818] [PATCH v2 4/7] import: json: Make 'json-fetch' take '#:cached?' argument.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 648 bytes --]

Xinglu Chen schreef op ma 07-02-2022 om 10:07 [+0100]:
> +    (let* ((port   ((if cached?
> +                        http-fetch/cached
> +                        http-fetch)

Personally, instead of introdicing a cached? argument,
I would introduce a 'http-fetch' argument defaulting to 'http-fetch',
such that users of 'json-fetch' can set all the options of
'http-fetch/cached' (ttl, cache-miss ...).

(json-fetch #:http-fetch
            (lambda (uri . rest)
              (apply http-fetch/cached uri #:ttl foo #:cache-miss bar
                     rest)))

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#53818] [PATCH v2 5/7] import: Add 'repology' updater.

[Xinglu Chen]

* guix/import/repology.scm
* tests/import-repology.scm: New files.
* Makefile.am (MODULES): Register them.
* doc/guix.texi (Invoking guix refresh): Document it.
---
 Makefile.am               |   3 +
 doc/guix.texi             |   7 ++
 guix/import/repology.scm  | 235 ++++++++++++++++++++++++++++++++++++++
 tests/import-repology.scm | 145 +++++++++++++++++++++++
 4 files changed, 390 insertions(+)
 create mode 100644 guix/import/repology.scm
 create mode 100644 tests/import-repology.scm

diff --git a/Makefile.am b/Makefile.am
index 7463606d20..6792917b59 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -16,6 +16,7 @@
 # Copyright © 2019 Efraim Flashner <efraim@flashner.co.il>
 # Copyright © 2021 Chris Marusich <cmmarusich@gmail.com>
 # Copyright © 2021 Andrew Tropin <andrew@trop.in>
+# Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
 #
 # This file is part of GNU Guix.
 #
@@ -271,6 +272,7 @@ MODULES =					\
   guix/import/opam.scm				\
   guix/import/print.scm				\
   guix/import/pypi.scm				\
+  guix/import/repology.scm			\
   guix/import/stackage.scm			\
   guix/import/texlive.scm   			\
   guix/import/utils.scm				\
@@ -488,6 +490,7 @@ SCM_TESTS =					\
   tests/home-import.scm				\
   tests/import-git.scm				\
   tests/import-github.scm			\
+  tests/import-repology.scm			\
   tests/import-utils.scm			\
   tests/inferior.scm				\
   tests/lint.scm				\
diff --git a/doc/guix.texi b/doc/guix.texi
index 0cf865a672..15d215dd48 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -12932,6 +12932,13 @@
       (release-tag-version-delimiter . ":"))))
 @end lisp
 
+@item repology
+an updater that scans @uref{https://repology.org, Repology}, a website
+that tracks packages on various package repositories, for updates.
+
+The name of a package in Guix is not always that same as the name on
+Repology; users can set the @code{repology-name} package property to
+make the updater use the correct name.
 
 @end table
 
diff --git a/guix/import/repology.scm b/guix/import/repology.scm
new file mode 100644
index 0000000000..28f3a3af5f
--- /dev/null
+++ b/guix/import/repology.scm
@@ -0,0 +1,235 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import repology)
+  #:use-module (guix diagnostics)
+  #:use-module (guix diagnostics)
+  #:use-module (guix git-download)
+  #:use-module (guix i18n)
+  #:use-module (guix import json)
+  #:use-module (guix import utils)
+  #:use-module (guix memoization)
+  #:use-module (guix packages)
+  #:use-module (guix upstream)
+  #:use-module (guix utils)
+  #:use-module (ice-9 match)
+  #:use-module (json)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-2)
+  #:use-module (srfi srfi-26)
+  #:use-module (srfi srfi-43)
+  #:export (repology-latest-release
+            %repology-updater))
+
+;;; Commentary:
+;;;
+;;; This module provides an updater which scans Repology, a site that monitors
+;;; several package repolsitories, for updates.  This means that if any other
+;;; package repository has a version of a package that is newer than the
+;;; version in Guix, the package should be able to be updated.  The updater
+;;; should in theory work for all packages in Guix, but the names of some
+;;; packages on Repology don't match the name in Guix.  The 'repology-name'
+;;; package property can be used to fix this.
+;;;
+;;; Guix already has many different updaters for language-specific packages,
+;;; and these typically provide more accurate data, e.g., input changes,
+;;; signature URLs.  The Repology updater should really be treated as a last
+;;; resort for those packages that don't have any other updater to rely on.
+;;;
+;;; See <https://repology.org/api/v1> for the API.
+;;;
+;;; Code:
+
+(define %repology-url
+   "https://repology.org/api/v1/project")
+
+(define* (package-name->repology-name name #:key (attempt 1))
+  "Convert NAME, the name of a Guix package, to the name of the package on
+Repology.  It doesn't always guess the correct name on the first attempt, so
+on the second attempt it will try to guess another name."
+  (match attempt
+    (1 (cond
+        ((string-prefix? "ghc-" name)
+         (string-append "haskell:"
+                        (string-drop name (string-length "ghc-"))))
+        ((string-prefix? "ocaml-" name)
+         (string-append "ocaml:"
+                        (string-drop name (string-length "ocaml-"))))
+        ((string-prefix? "perl-" name)
+         (string-append "perl:"
+                        (string-drop name (string-length "perl-"))))
+        ((string-prefix? "emacs-" name)
+         (string-append "emacs:"
+                        (string-drop name (string-length "emacs-"))))
+        ((string-prefix? "go-" name)
+         (string-append "go:"
+                        (string-drop name (string-length "go-"))))
+        ((string-prefix? "rust-" name)
+         (string-append "rust:"
+                        (string-drop name (string-length "rust-"))))
+        ((string-prefix? "lua-" name)
+         (string-append "lua:"
+                        (string-drop name (string-length "lua-"))))
+        ((string-prefix? "node-" name)
+         (string-append "node:"
+                        (string-drop name (string-length "node-"))))
+        ((string-prefix? "python-" name)
+         (string-append "python:"
+                        (string-drop name (string-length "python-"))))
+        ((string-prefix? "java-" name)
+         (string-append "java:"
+                        (string-drop name (string-length "java-"))))
+        ((string-prefix? "r-" name)
+         (string-append "r:"
+                        (string-drop name (string-length "r-"))))
+        ((string-prefix? "ruby-" name)
+         (string-append "ruby:"
+                        (string-drop name (string-length "ruby-"))))
+        ((string-prefix? "xf86-" name)
+         (string-append "xdrv:"
+                        (string-drop name (string-length "xf86-"))))
+        ((string-prefix? "font-" name)
+         (string-append "fonts:"
+                        (string-drop name (string-length "font-"))))
+        ((string-suffix? "-minimal" name)
+         (string-drop-right name (string-length "-minimal")))
+        (else name)))
+    (2 (cond
+        ((string-prefix? "xf86-video" name)
+         (string-append "xdrv:"
+                        (string-drop name (string-length "xf86-video-"))))
+        ((string-prefix? "xf86-input" name)
+         (string-append "xdrv:"
+                        (string-drop name (string-length "xf86-input-"))))
+        ((string-prefix? "minetest-" name)
+         (string-append "minetest-mod-"
+                        (string-drop name (string-length "minetest-"))))
+        (else name)))))  
+
+\f
+;;; JSON mappings.
+
+(define-json-mapping <repology-package> make-repology-package
+  repology-package?
+  json->repology-package
+  (repository repology-package-repository "repo")
+  (src-name repology-package-src-name "srcname")
+  (binary-name repology-package-binary-name "binname")
+  (visible-name repology-package-visible-name "visiblename")
+  (version repology-package-version)
+  (original-version repology-package-original-version "origversion")
+  (status repology-package-status)
+  (summary repology-package-summary)
+  (categories repology-package-categories)
+  (licenses repology-package-licenses)
+  (maintainers repology-package-maintainers))
+
+\f
+;;; Updater.
+
+(define repology-fetch-info
+  (memoize
+   (lambda (package)
+     "Fetch information about PACKAGE using the Repology API."
+     (define (name->info name)
+       (let ((url (string-append %repology-url "/" name)))
+         (and=> (json-fetch url #:cached? #t)
+                (lambda (url)
+                  (vector-map (lambda (a b)
+                                (json->repology-package b))
+                              url)))))
+     
+     (let* ((name (or (assoc-ref (package-properties package)
+                                 'repology-name)
+                      (package-name->repology-name (package-name package))))
+            (info (name->info name)))
+       (if (and info (not (vector-empty? info)))
+           info
+           (let ((info (name->info (package-name->repology-name
+                                    (package-name package)
+                                    #:attempt 2))))
+             (if (and info (not (vector-empty? info)))
+                 info
+                 (begin
+                   (warning (G_ "package not found on Repology: ~a\n")
+                            (package-name package))
+                   #f))))))))
+
+(define (update-version string old-version new-version)
+  "Replace OLD-VERSION in STRING with NEW-VERSION.  This assumes that STRING
+contains OLD-VERSION verbatim; if it doesn't, #f is returned."
+  (match (factorize-uri string old-version)
+    ((? string?) #f)
+    ((factorized ...)
+     (apply string-append
+            (map (lambda (component)
+                   (match component
+                     ('version new-version)
+                     ((? string?) component)))
+                 factorized)))))
+
+(define (package-source-urls package version)
+  "Return a list of URLs for PACKAGE at VERSION.  If no URL was successfully constructed, return #f."
+  (let ((old-version (package-version package)))
+    ;; XXX: (guix upstream) only supports tarballs and Git repos for now.
+    (match (origin-uri (package-source package))
+      ((? git-reference? reference)
+       (and-let* ((old-commit (git-reference-commit reference))
+                  (new-commit (if (string=? old-version old-commit)
+                                      version
+                                      (update-version old-commit
+                                                      old-version
+                                                      version))))
+         (git-reference
+          (inherit reference)
+          (commit new-commit))))
+      ((? string? url)
+       (list (update-version url old-version version)))
+      ((? list? urls)
+       (map (cut update-version <> old-version version) urls))
+      (_ #f))))
+
+(define (latest-version? repology-package)
+  "Return the latest released version of REPOLOGY-PACKAGE.  If none are found,
+return #f."
+  (and (or (equal? "newest" (repology-package-status repology-package))
+           (equal? "unique" (repology-package-status repology-package)))
+       (repology-package-version repology-package)))
+
+;; XXX: We use 'pkg' because 'package' will clash with the 'package' field of
+;; 'upstream-source'.
+(define (repology-latest-release pkg)
+  "Return the latest release of the PKG on Repology named NAME."
+  (and-let* ((packages (repology-fetch-info pkg))
+             (versions (filter-map latest-version?
+                                   (vector->list packages)))
+             (latest-version (and (pair? versions) (car versions))))
+    ;; TODO: set 'signature-urls'.
+    (upstream-source
+     (package (package-name pkg))
+     (version latest-version)
+     (urls (package-source-urls pkg latest-version)))))
+
+(define %repology-updater
+  (upstream-updater
+   (name 'repology)
+   (description "Updater for packages on Repology")
+   (pred (const #t))
+   (latest repology-latest-release)))
+
+;;; repology.scm ends here
diff --git a/tests/import-repology.scm b/tests/import-repology.scm
new file mode 100644
index 0000000000..2d366db283
--- /dev/null
+++ b/tests/import-repology.scm
@@ -0,0 +1,145 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (test-import-repology)
+  #:use-module (guix download)
+  #:use-module (guix git-download)
+  #:use-module (guix import repology)
+  #:use-module (guix packages)
+  #:use-module (guix tests)
+  #:use-module (guix upstream)
+  #:use-module (json)
+  #:use-module (srfi srfi-64))
+
+(test-begin "repology")
+
+(define package-using-git-repository
+  (dummy-package
+   "foo"
+   (version "1.0")
+   (source
+    (origin
+      (method git-fetch)
+      (uri (git-reference
+            (url "https://git.example.org/foo")
+            (commit "1.0")))
+      (sha256 #f)))))
+
+(define package-using-tarball
+  (dummy-package
+   "foo"
+   (version "1.0")
+   (source
+    (origin
+      (method git-fetch)
+      (uri (string-append "https://example.org/foo-" version ".tar.gz"))
+      (sha256 #f)))))
+
+(define package-using-tarball-multiple-urls
+  (dummy-package
+   "foo"
+   (version "1.0")
+   (source
+    (origin
+      (method git-fetch)
+      (uri (list (string-append "https://example.org/foo-"
+                                version ".tar.gz")
+                 (string-append "https://mirror.example.org/foo-"
+                                version ".tar.gz")))
+      (sha256 #f)))))
+
+(define %test-json
+"[
+  {
+    \"repo\": \"aur\",
+    \"srcname\": \"foo\",
+    \"binname\": \"foo\",
+    \"visiblename\": \"foo\",
+    \"version\": \"1.0.r25.gb86405a\",
+    \"maintainers\": [
+      \"bob@aur\"
+    ],
+    \"licenses\": [
+      \"LGPL3+\"
+    ],
+    \"summary\": \"foo bar\"
+    \"status\": \"rolling\",
+    \"origversion\": \"1.0.r25.gb86405a-1\"
+  },
+  {
+    \"repo\": \"gnuguix\",
+    \"srcname\": \"foo\",
+    \"binname\": \"foo\",
+    \"visiblename\": \"foo\",
+    \"version\": \"1.0\",
+    \"summary\": \"foo bar\",
+    \"status\": \"outdated\",
+    \"origversion\": null
+  },
+  {
+    \"repo\": \"nix_unstable\",
+    \"name\": \"foo\",
+    \"visiblename\": \"foo\",
+    \"version\": \"2.0\",
+    \"maintainers\": [
+      \"bob@example.org\"
+    ],
+    \"licenses\": [
+      \"LGPL-3.0-or-later\"
+    ],
+    \"summary\": \"foo bar\",
+    \"status\": \"newest\",
+    \"origversion\": null
+  }
+]")
+
+(define (latest-release package)
+  (mock ((guix import json) json-fetch
+         (lambda* (url #:key cached?)
+           (json-string->scm %test-json)))
+        (repology-latest-release package)))
+
+(test-equal "package using Git repo: version"
+  "2.0"
+  (upstream-source-version
+   (latest-release package-using-git-repository)))
+
+(test-equal "package using Git repo: git-reference"
+  (git-reference
+   (url "https://git.example.org/foo")
+   (commit "2.0"))
+  (upstream-source-urls
+   (latest-release package-using-git-repository)))
+
+(test-equal "package using tarball: version"
+  "2.0"
+  (upstream-source-version
+   (latest-release package-using-tarball)))
+
+(test-equal "package using tarball: URL"
+  (list "https://example.org/foo-2.0.tar.gz")
+  (upstream-source-urls
+   (latest-release package-using-tarball)))
+
+(test-equal "package using tarball: multiple URLs"
+  (list "https://example.org/foo-2.0.tar.gz"
+        "https://mirror.example.org/foo-2.0.tar.gz")
+  (upstream-source-urls
+   (latest-release package-using-tarball-multiple-urls)))
+
+(test-end "repology")
-- 
2.34.1

[bug#53818] [PATCH v2 5/7] import: Add 'repology' updater.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 470 bytes --]

Xinglu Chen schreef op ma 07-02-2022 om 10:07 [+0100]:
> +The name of a package in Guix is not always that same as the name on
> +Repology; users can set the @code{repology-name} package property to
> +make the updater use the correct name.

Perhaps note that the updater has some heuristics for determining
the Repology name, so setting 'repology-name' often isn't necessary
even when the name in Guix doesn't match the name in Repology?

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#53818] [PATCH v2 5/7] import: Add 'repology' updater.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 718 bytes --]

Xinglu Chen schreef op ma 07-02-2022 om 10:07 [+0100]:
> +(define (latest-release package)
> +  (mock ((guix import json) json-fetch
> +         (lambda* (url #:key cached?)
> +           (json-string->scm %test-json)))
> +        (repology-latest-release package)))

It would be nice to verify the URL.
FWIW, there's a 'with-http-server' macro.
although it might need to be extended a little, see e.g.
<https://issues.guix.gnu.org/53389>.

Also, you are using 'memoize' in repology-fetch-info,
so the memoisation cache needs to be invalidated
(see e.g. tests/minetest.scm), or not, depending
on how 'repology-fetch-info' is used and tests are written.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#53818] [PATCH v2 5/7] import: Add 'repology' updater.

[Xinglu Chen]

[-- Attachment #1: Type: text/plain, Size: 885 bytes --]

Maxime schrieb am Montag der 07. Februar 2022 um 10:50 +01:

> Xinglu Chen schreef op ma 07-02-2022 om 10:07 [+0100]:
>> +(define (latest-release package)
>> +  (mock ((guix import json) json-fetch
>> +         (lambda* (url #:key cached?)
>> +           (json-string->scm %test-json)))
>> +        (repology-latest-release package)))
>
> It would be nice to verify the URL.

What do you mean with “verify”?

> FWIW, there's a 'with-http-server' macro.
> although it might need to be extended a little, see e.g.
> <https://issues.guix.gnu.org/53389>.
>
> Also, you are using 'memoize' in repology-fetch-info,
> so the memoisation cache needs to be invalidated
> (see e.g. tests/minetest.scm), or not, depending
> on how 'repology-fetch-info' is used and tests are written.

Good catch!  Invalidating the cache would be a good idea.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

[bug#53818] [PATCH v2 5/7] import: Add 'repology' updater.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 717 bytes --]

Xinglu Chen schreef op di 08-02-2022 om 13:29 [+0100]:
> > Xinglu Chen schreef op ma 07-02-2022 om 10:07 [+0100]:
> > > +(define (latest-release package)
> > > +  (mock ((guix import json) json-fetch
> > > +         (lambda* (url #:key cached?)
> > > +           (json-string->scm %test-json)))
> > > +        (repology-latest-release package)))
> > 
> > It would be nice to verify the URL.
> 
> What do you mean with “verify”

Something along the lines of:

(lambda* (url #:key cached?)
  (if (equal? url
"http://[hostname]/release/information/of/PACKAGE.json")
    (json-string->scm ...)
    (error "the refresher contacted the wrong URL!")))

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#53818] [PATCH v2 5/7] import: Add 'repology' updater.

[Xinglu Chen]

[-- Attachment #1: Type: text/plain, Size: 819 bytes --]

Maxime schrieb am Dienstag der 08. Februar 2022 um 13:49 +01:

> Xinglu Chen schreef op di 08-02-2022 om 13:29 [+0100]:
>> > Xinglu Chen schreef op ma 07-02-2022 om 10:07 [+0100]:
>> > > +(define (latest-release package)
>> > > +  (mock ((guix import json) json-fetch
>> > > +         (lambda* (url #:key cached?)
>> > > +           (json-string->scm %test-json)))
>> > > +        (repology-latest-release package)))
>> > 
>> > It would be nice to verify the URL.
>> 
>> What do you mean with “verify”
>
> Something along the lines of:
>
> (lambda* (url #:key cached?)
>   (if (equal? url
> "http://[hostname]/release/information/of/PACKAGE.json")
>     (json-string->scm ...)
>     (error "the refresher contacted the wrong URL!")))

Ah, OK, thanks for clarifying!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

[bug#53818] [PATCH v2 6/7] gnu: xorg-server-xwayland: Set 'repology-name' property.

[Xinglu Chen]

* gnu/packages/xorg.scm (xorg-server-xwayland):[properties]: Set
'repology-name'.
---
 gnu/packages/xorg.scm | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/gnu/packages/xorg.scm b/gnu/packages/xorg.scm
index 14e35d19ae..8be7017da7 100644
--- a/gnu/packages/xorg.scm
+++ b/gnu/packages/xorg.scm
@@ -33,6 +33,7 @@
 ;;; Copyright © 2021 Maxime Devos <maximedevos@telenet.be>
 ;;; Copyright © 2021 qblade <qblade@protonmail.com>
 ;;; Copyright © 2021 Lu Hui <luhux76@gmail.com>
+;;; Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -5520,6 +5521,8 @@ (define-public xorg-server-xwayland
                     (lambda _
                       (substitute* (find-files "." "\\.c$")
                         (("/bin/sh") (which "sh"))))))))
+    (properties
+     '((repology-name . "xwayland")))
     (synopsis "Xorg server with Wayland backend")
     (description "Xwayland is an X server for running X clients under
 Wayland.")
-- 
2.34.1

[bug#53818] [PATCH v2 7/7] gnu: xorg-server-xwayland: Prepare for cross-compilation.

[Xinglu Chen]

* gnu/packages/xorg.scm (xorg-server-xwayland)[arguments]<#:phases>: Use
  ‘search-input-file’ instead of ‘which’.
---
 gnu/packages/xorg.scm | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/xorg.scm b/gnu/packages/xorg.scm
index 8be7017da7..dea7268a2e 100644
--- a/gnu/packages/xorg.scm
+++ b/gnu/packages/xorg.scm
@@ -5518,9 +5518,10 @@ (define-public xorg-server-xwayland
              "--localstatedir=/var")
        #:phases (modify-phases %standard-phases
                   (add-after 'unpack 'patch-/bin/sh
-                    (lambda _
+                    (lambda* (#:key inputs #:allow-other-keys)
                       (substitute* (find-files "." "\\.c$")
-                        (("/bin/sh") (which "sh"))))))))
+                        (("/bin/sh")
+                         (search-input-file inputs "bin/sh"))))))))
     (properties
      '((repology-name . "xwayland")))
     (synopsis "Xorg server with Wayland backend")
-- 
2.34.1

[bug#53818] [PATCH v3 0/7] Add Repology updater

[Xinglu Chen]

[-- Attachment #1: Type: text/plain, Size: 1281 bytes --]

Changes since v2:

* Change ‘#:cached?’ argument to ‘#:http-fetch’.

* Invalidate cache in tests and make sure the requested URL is the one
  we expect.

* Expound a bit on the documentation in the manual.

* Some more rules for guessing the Repology name of a package.

Xinglu Chen (7):
  upstream: Sort list of updaters.
  http-client: Make 'http-fetch/cached' take '#:headers' argument.
  http-client: 'http-fetch/cached' accepts a string or a <uri>.
  import: json: Make 'json-fetch' take '#:http-fetch' argument.
  import: Add 'repology' updater.
  gnu: xorg-server-xwayland: Set 'repology-name' property.
  gnu: xorg-server-xwayland: Prepare for cross-compilation.

 Makefile.am               |   3 +
 doc/guix.texi             |   8 ++
 gnu/packages/xorg.scm     |   8 +-
 guix/http-client.scm      |  18 ++-
 guix/import/json.scm      |   9 +-
 guix/import/repology.scm  | 249 ++++++++++++++++++++++++++++++++++++++
 guix/upstream.scm         |  29 ++++-
 tests/import-repology.scm | 150 +++++++++++++++++++++++
 8 files changed, 457 insertions(+), 17 deletions(-)
 create mode 100644 guix/import/repology.scm
 create mode 100644 tests/import-repology.scm


base-commit: ad9cd004c81a01e33a605221a102f4d20abe442d
-- 
2.34.1




[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

[bug#53818] [PATCH v3 1/7] upstream: Sort list of updaters.

[Xinglu Chen]

The order determines which updater gets used first when running ‘guix refresh’
on a package.  We want the most generic updaters to be last since they usually
don’t provide as much or accurate information as language-specific updaters.

* guix/upstream.scm (sort-updaters): New procedure.
(%updaters): Use it.
---
 guix/upstream.scm | 29 +++++++++++++++++++++++------
 1 file changed, 23 insertions(+), 6 deletions(-)

diff --git a/guix/upstream.scm b/guix/upstream.scm
index 6666803a92..d5faf9d3ee 100644
--- a/guix/upstream.scm
+++ b/guix/upstream.scm
@@ -250,14 +250,31 @@ (define (importer-modules)
                           %load-path)
                      #:warn warn-about-load-error)))
 
+(define (sort-updaters updaters)
+  "Sort UPDATERS by putting the more generic ones last."
+  (define (genericity updater)
+    (cond
+     ((equal? 'repology (upstream-updater-name updater))
+      2)
+     ((string-prefix? "generic-"
+                      (symbol->string (upstream-updater-name updater)))
+      1)     
+     (else 0)))
+
+  (define (less a b)
+    (<= (genericity a) (genericity b)))
+
+  (stable-sort updaters less))
+
 (define %updaters
   ;; The list of publically-known updaters.
-  (delay (fold-module-public-variables (lambda (obj result)
-                                         (if (upstream-updater? obj)
-                                             (cons obj result)
-                                             result))
-                                       '()
-                                       (importer-modules))))
+  (delay (sort-updaters
+          (fold-module-public-variables (lambda (obj result)
+                                          (if (upstream-updater? obj)
+                                              (cons obj result)
+                                              result))
+                                        '()
+                                        (importer-modules)))))
 
 ;; Tests need to mock this variable so mark it as "non-declarative".
 (set! %updaters %updaters)
-- 
2.34.1

[bug#53818] [PATCH v3 2/7] http-client: Make 'http-fetch/cached' take '#:headers' argument.

[Xinglu Chen]

* guix/http-client.scm (http-fetch/cached): Take ‘#:headers’ keyword argument.
---
 guix/http-client.scm | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/guix/http-client.scm b/guix/http-client.scm
index 10bc278023..058f09852f 100644
--- a/guix/http-client.scm
+++ b/guix/http-client.scm
@@ -3,6 +3,7 @@
 ;;; Copyright © 2015 Mark H Weaver <mhw@netris.org>
 ;;; Copyright © 2012, 2015 Free Software Foundation, Inc.
 ;;; Copyright © 2017 Tobias Geerinckx-Rice <me@tobias.gr>
+;;; Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -283,9 +284,11 @@ (define* (http-fetch/cached uri #:key (ttl (%http-cache-ttl)) text?
                             (write-cache dump-port)
                             (cache-miss (const #t))
                             (log-port (current-error-port))
-                            (timeout 10))
+                            (timeout 10)
+                            (headers '((user-agent . "GNU Guile"))))
   "Like 'http-fetch', return an input port, but cache its contents in
-~/.cache/guix.  The cache remains valid for TTL seconds.
+~/.cache/guix.  The cache remains valid for TTL seconds.  HEADERS is an alist
+of extra HTTP headers.  The cache time will automatically be added to HEADERS, so there is no need to specify it.
 
 Call WRITE-CACHE with the HTTP input port and the cache output port to write
 the data to cache.  Call CACHE-MISS with URI just before fetching data from
@@ -300,8 +303,8 @@ (define cache-time
         (and cache-port
              (stat:mtime (stat cache-port))))
 
-      (define headers
-        `((user-agent . "GNU Guile")
+      (define headers*
+        `(,@headers
           ,@(if cache-time
                 `((if-modified-since
                    . ,(time-utc->date (make-time time-utc 0 cache-time))))
@@ -316,7 +319,7 @@ (define headers
                      (raise c))))
         (let ((port (http-fetch uri #:text? text?
                                 #:log-port log-port
-                                #:headers headers #:timeout timeout)))
+                                #:headers headers* #:timeout timeout)))
           (cache-miss uri)
           (mkdir-p (dirname file))
           (when cache-port
-- 
2.34.1

[bug#53818] [PATCH v3 3/7] http-client: 'http-fetch/cached' accepts a string or a <uri>.

[Xinglu Chen]

This is consistent with the 'http-fetch' procedure.

* guix/http-client.scm (http-fetch/cached): The 'uri' argument can be a string
  or a <uri> record.
---
 guix/http-client.scm | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/guix/http-client.scm b/guix/http-client.scm
index 058f09852f..08efdd5e47 100644
--- a/guix/http-client.scm
+++ b/guix/http-client.scm
@@ -297,7 +297,10 @@ (define* (http-fetch/cached uri #:key (ttl (%http-cache-ttl)) text?
 TIMEOUT specifies the timeout in seconds for connection establishment.
 
 Write information about redirects to LOG-PORT."
-  (let ((file (cache-file-for-uri uri)))
+  (let* ((uri (if (string? uri)
+                  (string->uri uri)
+                  uri))
+         (file (cache-file-for-uri uri)))
     (define (update-cache cache-port)
       (define cache-time
         (and cache-port
-- 
2.34.1

[bug#53818] [PATCH v3 4/7] import: json: Make 'json-fetch' take '#:http-fetch' argument.

[Xinglu Chen]

This will let users of the 'json-fetch' procedure use a custom procedure for
fetching the URL, e.g., 'http-fetch/cached'.

* json.scm (json-fetch): Add ‘#:http-fetch’ keyword argument.

Suggested-by: Maxime Devos <maximedevos@telenet.be>
---
 guix/import/json.scm | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/guix/import/json.scm b/guix/import/json.scm
index 0c98bb25b8..de7ed60e8d 100644
--- a/guix/import/json.scm
+++ b/guix/import/json.scm
@@ -3,6 +3,7 @@
 ;;; Copyright © 2015, 2016 Eric Bavier <bavier@member.fsf.org>
 ;;; Copyright © 2018, 2019 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2020 Ricardo Wurmus <rekado@elephly.net>
+;;; Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -37,11 +38,13 @@ (define-module (guix import json)
 (define* (json-fetch url
                      ;; Note: many websites returns 403 if we omit a
                      ;; 'User-Agent' header.
-                     #:key (headers `((user-agent . "GNU Guile")
-                                      (Accept . "application/json"))))
+                     #:key
+                     (headers `((user-agent . "GNU Guile")
+                                (Accept . "application/json")))
+                     (http-fetch http-fetch))
   "Return a representation of the JSON resource URL (a list or hash table), or
 #f if URL returns 403 or 404.  HEADERS is a list of HTTP headers to pass in
-the query."
+the query.  HTTP-FETCH is the procedure that is used to fetch the JSON data from URL.  It should take the URL as an argument and HEADERS as a keyword argument, and return an input port for the JSON data."
   (guard (c ((and (http-get-error? c)
                   (let ((error (http-get-error-code c)))
                     (or (= 403 error)
-- 
2.34.1

[bug#53818] [PATCH v3 5/7] import: Add 'repology' updater.

[Xinglu Chen]

* guix/import/repology.scm
* tests/import-repology.scm: New files.
* Makefile.am (MODULES): Register them.
* doc/guix.texi (Invoking guix refresh): Document it.
---
 Makefile.am               |   3 +
 doc/guix.texi             |   8 ++
 guix/import/repology.scm  | 249 ++++++++++++++++++++++++++++++++++++++
 tests/import-repology.scm | 150 +++++++++++++++++++++++
 4 files changed, 410 insertions(+)
 create mode 100644 guix/import/repology.scm
 create mode 100644 tests/import-repology.scm

diff --git a/Makefile.am b/Makefile.am
index 7463606d20..6792917b59 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -16,6 +16,7 @@
 # Copyright © 2019 Efraim Flashner <efraim@flashner.co.il>
 # Copyright © 2021 Chris Marusich <cmmarusich@gmail.com>
 # Copyright © 2021 Andrew Tropin <andrew@trop.in>
+# Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
 #
 # This file is part of GNU Guix.
 #
@@ -271,6 +272,7 @@ MODULES =					\
   guix/import/opam.scm				\
   guix/import/print.scm				\
   guix/import/pypi.scm				\
+  guix/import/repology.scm			\
   guix/import/stackage.scm			\
   guix/import/texlive.scm   			\
   guix/import/utils.scm				\
@@ -488,6 +490,7 @@ SCM_TESTS =					\
   tests/home-import.scm				\
   tests/import-git.scm				\
   tests/import-github.scm			\
+  tests/import-repology.scm			\
   tests/import-utils.scm			\
   tests/inferior.scm				\
   tests/lint.scm				\
diff --git a/doc/guix.texi b/doc/guix.texi
index 583ba1c61d..2d7612b09a 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -12932,6 +12932,14 @@
       (release-tag-version-delimiter . ":"))))
 @end lisp
 
+@item repology
+an updater that scans @uref{https://repology.org, Repology}, a website
+that tracks packages on various package repositories, for updates.
+
+The name of a package in Guix is not always that same as the name on
+Repology.  In most cases, the updater will be able to guess the name
+correctly.  If it doesn’t, users can set the @code{repology-name}
+package property.
 
 @end table
 
diff --git a/guix/import/repology.scm b/guix/import/repology.scm
new file mode 100644
index 0000000000..87fbd2ee6f
--- /dev/null
+++ b/guix/import/repology.scm
@@ -0,0 +1,249 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import repology)
+  #:use-module (guix diagnostics)
+  #:use-module (guix git-download)
+  #:use-module (guix http-client)
+  #:use-module (guix i18n)
+  #:use-module (guix import json)
+  #:use-module (guix import utils)
+  #:use-module (guix memoization)
+  #:use-module (guix packages)
+  #:use-module (guix upstream)
+  #:use-module (guix utils)
+  #:use-module (ice-9 match)
+  #:use-module (json)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-2)
+  #:use-module (srfi srfi-26)
+  #:use-module (srfi srfi-43)
+  #:export (%repology-url
+            repology-fetch-info
+            repology-latest-release
+            %repology-updater))
+
+;;; Commentary:
+;;;
+;;; This module provides an updater which scans Repology, a site that monitors
+;;; several package repolsitories, for updates.  This means that if any other
+;;; package repository has a version of a package that is newer than the
+;;; version in Guix, the package should be able to be updated.  The updater
+;;; should in theory work for all packages in Guix, but the names of some
+;;; packages on Repology don't match the name in Guix.  The 'repology-name'
+;;; package property can be used to fix this.
+;;;
+;;; Guix already has many different updaters for language-specific packages,
+;;; and these typically provide more accurate data, e.g., input changes,
+;;; signature URLs.  The Repology updater should really be treated as a last
+;;; resort for those packages that don't have any other updater to rely on.
+;;;
+;;; See <https://repology.org/api/v1> for the API.
+;;;
+;;; Code:
+
+(define %repology-url
+   "https://repology.org/api/v1/project")
+
+(define* (package-name->repology-name name #:key (attempt 1))
+  "Convert NAME, the name of a Guix package, to the name of the package on
+Repology.  It doesn't always guess the correct name on the first attempt, so
+on the second attempt it will try to guess another name."
+  (match attempt
+    (1 (cond
+        ((string-prefix? "ghc-" name)
+         (string-append "haskell:"
+                        (string-drop name (string-length "ghc-"))))
+        ((string-prefix? "ocaml-" name)
+         (string-append "ocaml:"
+                        (string-drop name (string-length "ocaml-"))))
+        ((string-prefix? "perl-" name)
+         (string-append "perl:"
+                        (string-drop name (string-length "perl-"))))
+        ((string-prefix? "emacs-" name)
+         (string-append "emacs:"
+                        (string-drop name (string-length "emacs-"))))
+        ((string-prefix? "go-" name)
+         (string-append "go:"
+                        (string-drop name (string-length "go-"))))
+        ((string-prefix? "rust-" name)
+         (string-append "rust:"
+                        (string-drop name (string-length "rust-"))))
+        ((string-prefix? "lua-" name)
+         (string-append "lua:"
+                        (string-drop name (string-length "lua-"))))
+        ((string-prefix? "node-" name)
+         (string-append "node:"
+                        (string-drop name (string-length "node-"))))
+        ((string-prefix? "python-" name)
+         (string-append "python:"
+                        (string-drop name (string-length "python-"))))
+        ((string-prefix? "java-" name)
+         (string-append "java:"
+                        (string-drop name (string-length "java-"))))
+        ((string-prefix? "r-" name)
+         (string-append "r:"
+                        (string-drop name (string-length "r-"))))
+        ((string-prefix? "ruby-" name)
+         (string-append "ruby:"
+                        (string-drop name (string-length "ruby-"))))
+        ((string-prefix? "xf86-" name)
+         (string-append "xdrv:"
+                        (string-drop name (string-length "xf86-"))))
+        ((string-prefix? "font-" name)
+         (string-append "fonts:"
+                        (string-drop name (string-length "font-"))))
+        ((string-prefix? "trytond-" name)
+         (string-append "tryton:"
+                        (string-drop name (string-length "trytond-"))))
+        ((string-prefix? "python-trytond-" name)
+         (string-append "tryton:"
+                        (string-drop name (string-length "python-trytond-"))))
+        ((string-suffix? "-minimal" name)
+         (string-drop-right name (string-length "-minimal")))        
+        (else name)))
+    (2 (cond
+        ((string-prefix? "xf86-video" name)
+         (string-append "xdrv:"
+                        (string-drop name (string-length "xf86-video-"))))
+        ((string-prefix? "xf86-input" name)
+         (string-append "xdrv:"
+                        (string-drop name (string-length "xf86-input-"))))
+        ((string-prefix? "minetest-" name)
+         (string-append "minetest-mod-"
+                        (string-drop name (string-length "minetest-"))))
+        ((string-prefix? "lib" name)
+         (string-drop name (string-length "lib")))
+        ((string-prefix? "vim-" name)
+         (string-append "vim:"
+                        (string-drop name (string-length "vim-"))))
+        (else name)))))  
+
+\f
+;;; JSON mappings.
+
+(define-json-mapping <repology-package> make-repology-package
+  repology-package?
+  json->repology-package
+  (repository repology-package-repository "repo")
+  (src-name repology-package-src-name "srcname")
+  (binary-name repology-package-binary-name "binname")
+  (visible-name repology-package-visible-name "visiblename")
+  (version repology-package-version)
+  (original-version repology-package-original-version "origversion")
+  (status repology-package-status)
+  (summary repology-package-summary)
+  (categories repology-package-categories)
+  (licenses repology-package-licenses)
+  (maintainers repology-package-maintainers))
+
+\f
+;;; Updater.
+
+(define repology-fetch-info
+  (memoize
+   (lambda (package)
+     "Fetch information about PACKAGE using the Repology API."
+     (define (name->info name)
+       (let ((url (string-append %repology-url "/" name)))
+         (and=> (json-fetch url #:http-fetch http-fetch/cached)
+                (lambda (url)
+                  (vector-map (lambda (a b)
+                                (json->repology-package b))
+                              url)))))
+     
+     (let* ((name (or (assoc-ref (package-properties package)
+                                 'repology-name)
+                      (package-name->repology-name (package-name package))))
+            (info (name->info name)))
+       (if (and info (not (vector-empty? info)))
+           info
+           (let ((info (name->info (package-name->repology-name
+                                    (package-name package)
+                                    #:attempt 2))))
+             (if (and info (not (vector-empty? info)))
+                 info
+                 (begin
+                   (warning (G_ "package not found on Repology: ~a\n")
+                            (package-name package))
+                   #f))))))))
+
+(define (update-version string old-version new-version)
+  "Replace OLD-VERSION in STRING with NEW-VERSION.  This assumes that STRING
+contains OLD-VERSION verbatim; if it doesn't, #f is returned."
+  (match (factorize-uri string old-version)
+    ((? string?) #f)
+    ((factorized ...)
+     (apply string-append
+            (map (lambda (component)
+                   (match component
+                     ('version new-version)
+                     ((? string?) component)))
+                 factorized)))))
+
+(define (package-source-urls package version)
+  "Return a list of URLs for PACKAGE at VERSION.  If no URL was successfully constructed, return #f."
+  (and-let* ((old-version (package-version package))
+             (source (package-source package)))
+    ;; XXX: (guix upstream) only supports tarballs and Git repos for now.
+    (match (origin-uri source)
+      ((? git-reference? reference)
+       (and-let* ((old-commit (git-reference-commit reference))
+                  (new-commit (if (string=? old-version old-commit)
+                                      version
+                                      (update-version old-commit
+                                                      old-version
+                                                      version))))
+         (git-reference
+          (inherit reference)
+          (commit new-commit))))
+      ((? string? url)
+       (list (update-version url old-version version)))
+      ((? list? urls)
+       (map (cut update-version <> old-version version) urls))
+      (_ #f))))
+
+(define (latest-version? repology-package)
+  "Return the latest released version of REPOLOGY-PACKAGE.  If none are found,
+return #f."
+  (and (or (equal? "newest" (repology-package-status repology-package))
+           (equal? "unique" (repology-package-status repology-package)))
+       (repology-package-version repology-package)))
+
+;; XXX: We use 'pkg' because 'package' will clash with the 'package' field of
+;; 'upstream-source'.
+(define (repology-latest-release pkg)
+  "Return the latest release of the PKG on Repology named NAME."
+  (and-let* ((packages (repology-fetch-info pkg))
+             (versions (filter-map latest-version?
+                                   (vector->list packages)))
+             (latest-version (and (pair? versions) (car versions))))
+    ;; TODO: set 'signature-urls'.
+    (upstream-source
+     (package (package-name pkg))
+     (version latest-version)
+     (urls (package-source-urls pkg latest-version)))))
+
+(define %repology-updater
+  (upstream-updater
+   (name 'repology)
+   (description "Updater for packages on Repology")
+   (pred (const #t))
+   (latest repology-latest-release)))
+
+;;; repology.scm ends here
diff --git a/tests/import-repology.scm b/tests/import-repology.scm
new file mode 100644
index 0000000000..4da01a4106
--- /dev/null
+++ b/tests/import-repology.scm
@@ -0,0 +1,150 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (test-import-repology)
+  #:use-module (guix download)
+  #:use-module (guix git-download)
+  #:use-module (guix import repology)
+  #:use-module (guix memoization)
+  #:use-module (guix packages)
+  #:use-module (guix tests)
+  #:use-module (guix upstream)
+  #:use-module (json)
+  #:use-module (srfi srfi-64))
+
+(test-begin "repology")
+
+(define package-using-git-repository
+  (dummy-package
+   "foo"
+   (version "1.0")
+   (source
+    (origin
+      (method git-fetch)
+      (uri (git-reference
+            (url "https://git.example.org/foo")
+            (commit "1.0")))
+      (sha256 #f)))))
+
+(define package-using-tarball
+  (dummy-package
+   "foo"
+   (version "1.0")
+   (source
+    (origin
+      (method git-fetch)
+      (uri (string-append "https://example.org/foo-" version ".tar.gz"))
+      (sha256 #f)))))
+
+(define package-using-tarball-multiple-urls
+  (dummy-package
+   "foo"
+   (version "1.0")
+   (source
+    (origin
+      (method git-fetch)
+      (uri (list (string-append "https://example.org/foo-"
+                                version ".tar.gz")
+                 (string-append "https://mirror.example.org/foo-"
+                                version ".tar.gz")))
+      (sha256 #f)))))
+
+(define %test-json
+"[
+  {
+    \"repo\": \"aur\",
+    \"srcname\": \"foo\",
+    \"binname\": \"foo\",
+    \"visiblename\": \"foo\",
+    \"version\": \"1.0.r25.gb86405a\",
+    \"maintainers\": [
+      \"bob@aur\"
+    ],
+    \"licenses\": [
+      \"LGPL3+\"
+    ],
+    \"summary\": \"foo bar\"
+    \"status\": \"rolling\",
+    \"origversion\": \"1.0.r25.gb86405a-1\"
+  },
+  {
+    \"repo\": \"gnuguix\",
+    \"srcname\": \"foo\",
+    \"binname\": \"foo\",
+    \"visiblename\": \"foo\",
+    \"version\": \"1.0\",
+    \"summary\": \"foo bar\",
+    \"status\": \"outdated\",
+    \"origversion\": null
+  },
+  {
+    \"repo\": \"nix_unstable\",
+    \"name\": \"foo\",
+    \"visiblename\": \"foo\",
+    \"version\": \"2.0\",
+    \"maintainers\": [
+      \"bob@example.org\"
+    ],
+    \"licenses\": [
+      \"LGPL-3.0-or-later\"
+    ],
+    \"summary\": \"foo bar\",
+    \"status\": \"newest\",
+    \"origversion\": null
+  }
+]")
+
+(define (latest-release package)
+  (invalidate-memoization! repology-fetch-info)
+  (mock ((guix import json) json-fetch
+         (lambda* (url #:key http-fetch)
+           (if (string=? url
+                         (string-append %repology-url "/foo"))
+               (json-string->scm %test-json)
+               (error "the URL is not correct"))))
+        (repology-latest-release package)))
+
+(test-equal "package using Git repo: version"
+  "2.0"
+  (upstream-source-version
+   (latest-release package-using-git-repository)))
+
+(test-equal "package using Git repo: git-reference"
+  (git-reference
+   (url "https://git.example.org/foo")
+   (commit "2.0"))
+  (upstream-source-urls
+   (latest-release package-using-git-repository)))
+
+(test-equal "package using tarball: version"
+  "2.0"
+  (upstream-source-version
+   (latest-release package-using-tarball)))
+
+(test-equal "package using tarball: URL"
+  (list "https://example.org/foo-2.0.tar.gz")
+  (upstream-source-urls
+   (latest-release package-using-tarball)))
+
+(test-equal "package using tarball: multiple URLs"
+  (list "https://example.org/foo-2.0.tar.gz"
+        "https://mirror.example.org/foo-2.0.tar.gz")
+  (upstream-source-urls
+   (latest-release package-using-tarball-multiple-urls)))
+
+(test-end "repology")
-- 
2.34.1

[bug#53818] [PATCH v3 6/7] gnu: xorg-server-xwayland: Set 'repology-name' property.

[Xinglu Chen]

* gnu/packages/xorg.scm (xorg-server-xwayland):[properties]: Set
'repology-name'.
---
 gnu/packages/xorg.scm | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/gnu/packages/xorg.scm b/gnu/packages/xorg.scm
index 14e35d19ae..8be7017da7 100644
--- a/gnu/packages/xorg.scm
+++ b/gnu/packages/xorg.scm
@@ -33,6 +33,7 @@
 ;;; Copyright © 2021 Maxime Devos <maximedevos@telenet.be>
 ;;; Copyright © 2021 qblade <qblade@protonmail.com>
 ;;; Copyright © 2021 Lu Hui <luhux76@gmail.com>
+;;; Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -5520,6 +5521,8 @@ (define-public xorg-server-xwayland
                     (lambda _
                       (substitute* (find-files "." "\\.c$")
                         (("/bin/sh") (which "sh"))))))))
+    (properties
+     '((repology-name . "xwayland")))
     (synopsis "Xorg server with Wayland backend")
     (description "Xwayland is an X server for running X clients under
 Wayland.")
-- 
2.34.1

[bug#53818] [PATCH v3 7/7] gnu: xorg-server-xwayland: Prepare for cross-compilation.

[Xinglu Chen]

* gnu/packages/xorg.scm (xorg-server-xwayland)[arguments]<#:phases>: Use
  ‘search-input-file’ instead of ‘which’.
---
 gnu/packages/xorg.scm | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/xorg.scm b/gnu/packages/xorg.scm
index 8be7017da7..dea7268a2e 100644
--- a/gnu/packages/xorg.scm
+++ b/gnu/packages/xorg.scm
@@ -5518,9 +5518,10 @@ (define-public xorg-server-xwayland
              "--localstatedir=/var")
        #:phases (modify-phases %standard-phases
                   (add-after 'unpack 'patch-/bin/sh
-                    (lambda _
+                    (lambda* (#:key inputs #:allow-other-keys)
                       (substitute* (find-files "." "\\.c$")
-                        (("/bin/sh") (which "sh"))))))))
+                        (("/bin/sh")
+                         (search-input-file inputs "bin/sh"))))))))
     (properties
      '((repology-name . "xwayland")))
     (synopsis "Xorg server with Wayland backend")
-- 
2.34.1

[bug#53818] [PATCH 0/3] Add Repology updater

[Ludovic Courtès]

Hi!

Xinglu Chen <public@yoctocell.xyz> skribis:

> This patchset adds a new updater, which scans Repology[1] for updates.
> It should technically support all packages in Guix!  :-)

I wouldn’t want to spoil the party, but I’m mildly enthusiastic.

Repology implements the same functionality as our updaters, so
repology.org is effectively “service as a software substitute” (SaaSS).

My preference would be to keep our existing updaters rather than
effectively ditch them and delegate the work to Repology.  It’s tempting
to think we can have both, but I’m not sure this would last long.

WDYT?

Ludo’.

[bug#53818] [PATCH 0/3] Add Repology updater

[Xinglu Chen]

[-- Attachment #1: Type: text/plain, Size: 1157 bytes --]

Hi,

Ludovic schrieb am Dienstag der 08. Februar 2022 um 23:59 +01:

> Hi!
>
> Xinglu Chen <public@yoctocell.xyz> skribis:
>
>> This patchset adds a new updater, which scans Repology[1] for updates.
>> It should technically support all packages in Guix!  :-)
>
> I wouldn’t want to spoil the party, but I’m mildly enthusiastic.
>
> Repology implements the same functionality as our updaters, so
> repology.org is effectively “service as a software substitute”
> (SaaSS).

Right, but it tracks a lot more repositories than what our updaters do,
so why not take advantage of that.

> My preference would be to keep our existing updaters rather than
> effectively ditch them and delegate the work to Repology.  It’s tempting
> to think we can have both, but I’m not sure this would last long.

The point of the Repology updater is to act as a fallback if none of
the other updaters can update a package, e.g., ‘maven-dependency-tree’.
I already mentioned that language-specific updaters usually provide more
accurate and detailed information, so they should be used when possible;
we aren’t losing anything here.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

[bug#53818] [PATCH 0/3] Add Repology updater

[Nicolas Goaziou]

Hello,

Xinglu Chen <public@yoctocell.xyz> writes:

> The point of the Repology updater is to act as a fallback if none of
> the other updaters can update a package, e.g., ‘maven-dependency-tree’.
> I already mentioned that language-specific updaters usually provide more
> accurate and detailed information, so they should be used when possible;
> we aren’t losing anything here.

One issue is that such an updater will introduce frequent false
positives. It is common for Repology to get the latest release wrong,
because some distribution is doing fancy versioning, or because
different distributions disagree about what is upstream.

I don't think we can rely on Repology's "newest" status. The updater may
need to provide its own version comparison tool, because Repology's tool
and Guix versioning do not play nice, in particular when using
`git-version'.

Regards,
-- 
Nicolas Goaziou

[bug#53818] [PATCH 0/3] Add Repology updater

[Xinglu Chen]

[-- Attachment #1: Type: text/plain, Size: 2580 bytes --]

Nicolas schrieb am Mittwoch der 09. Februar 2022 um 15:29 +01:

> Hello,
>
> Xinglu Chen <public@yoctocell.xyz> writes:
>
>> The point of the Repology updater is to act as a fallback if none of
>> the other updaters can update a package, e.g., ‘maven-dependency-tree’.
>> I already mentioned that language-specific updaters usually provide more
>> accurate and detailed information, so they should be used when possible;
>> we aren’t losing anything here.
>
> One issue is that such an updater will introduce frequent false
> positives. It is common for Repology to get the latest release wrong,
> because some distribution is doing fancy versioning, or because
> different distributions disagree about what is upstream.

Yeah, I have noticed that it sometimes thinks that a version like
“20080323” is newer than something like “0.1.2-0.a1b2b3d”, even though
it might not necessarily be true.  This seems to be the case for a lot
of Common Lisp packages which usually don’t have any proper releases.

> I don't think we can rely on Repology's "newest" status. The updater may
> need to provide its own version comparison tool, because Repology's tool
> and Guix versioning do not play nice, in particular when using
> `git-version'.

In my testing, the “newest” status does a pretty good job (besides the
problem I mentioned above)

Some other “bad” updates I found[*] are listed below (excluding Common Lisp
packages).

--8<---------------cut here---------------start------------->8---
guile-ac-d-bus would be upgraded from 1.0.0-beta.0 to 1.0.0-beta0
sic would be upgraded from 1.2 to 1.2+20210506_058547e
tla2tools would be upgraded from 1.7.1-0.6932e19 to 20140313
quickjs would be upgraded from 2021-03-27 to 2021.03.27
stow would be upgraded from 2.3.1 to 2.3.1+5.32
cube would be upgraded from 4.3.5 to 2005.08.29
python-ratelimiter would be upgraded from 1.2.0 to 1.2.0.post0
gr-osmosdr would be upgraded from 0.2.3-0.a100eb0 to 0.2.3.20210128
countdown would be upgraded from 1.0.0 to 20150606
http-parser would be upgraded from 2.9.4-1.ec8b5ee to 2.9.4.20201223
xlsx2csv would be upgraded from 0.7.4 to 20200427211949
keynav would be upgraded from 0.20110708.0 to 20150730+4ae486d
--8<---------------cut here---------------end--------------->8---

It seems like most of these could be solved by checking if the version
scheme changed from semver to calver.  I think that’s a pretty good
result considering how many packages we have.

[*] Until I ran into <https://issues.guix.gnu.org/53923>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

[bug#53818] [PATCH 0/3] Add Repology updater

[Nicolas Goaziou]

Hello,

Xinglu Chen <public@yoctocell.xyz> writes:

> Yeah, I have noticed that it sometimes thinks that a version like
> “20080323” is newer than something like “0.1.2-0.a1b2b3d”, even though
> it might not necessarily be true.  This seems to be the case for a lot
> of Common Lisp packages which usually don’t have any proper releases.

[...]

> In my testing, the “newest” status does a pretty good job (besides the
> problem I mentioned above)
>
> Some other “bad” updates I found[*] are listed below (excluding Common Lisp
> packages).
>
> --8<---------------cut here---------------start------------->8---
> guile-ac-d-bus would be upgraded from 1.0.0-beta.0 to 1.0.0-beta0
> sic would be upgraded from 1.2 to 1.2+20210506_058547e
> tla2tools would be upgraded from 1.7.1-0.6932e19 to 20140313
> quickjs would be upgraded from 2021-03-27 to 2021.03.27
> stow would be upgraded from 2.3.1 to 2.3.1+5.32
> cube would be upgraded from 4.3.5 to 2005.08.29
> python-ratelimiter would be upgraded from 1.2.0 to 1.2.0.post0
> gr-osmosdr would be upgraded from 0.2.3-0.a100eb0 to 0.2.3.20210128
> countdown would be upgraded from 1.0.0 to 20150606
> http-parser would be upgraded from 2.9.4-1.ec8b5ee to 2.9.4.20201223
> xlsx2csv would be upgraded from 0.7.4 to 20200427211949
> keynav would be upgraded from 0.20110708.0 to 20150730+4ae486d
> --8<---------------cut here---------------end--------------->8---
>
> It seems like most of these could be solved by checking if the version
> scheme changed from semver to calver.  I think that’s a pretty good
> result considering how many packages we have.

I think this would not cut it.

As I wrote, almost any package using `git-version' is going to create
a version mismatch. This is because we consider

  (git-version "X.Y" revision commit)

to be greater than "X.Y" whereas Repology either ignore the version, or
consider it to be a pre-release before "X.Y". See, e.g., "emacs:circe"
project, or "joycond". This, I think, the most prominent category of
comparison failures.

Also, there are versions which are plain wrong, e.g., "emacs:csv-mode",
and disqualify correct and up-to-date version. There are also version
disagreement in, e.g., "colobot", or upstream disagreement, e.g.,
"emacs:scala-mode".

See also "emacs:geiser-racket", "python:folium" or "higan" for other
projects with versioning issues.

Regards,
-- 
Nicolas Goaziou

[bug#53818] [PATCH 0/3] Add Repology updater

[Ludovic Courtès]

Hi,

Xinglu Chen <public@yoctocell.xyz> skribis:

> Ludovic schrieb am Dienstag der 08. Februar 2022 um 23:59 +01:

[...]

>> Repology implements the same functionality as our updaters, so
>> repology.org is effectively “service as a software substitute”
>> (SaaSS).
>
> Right, but it tracks a lot more repositories than what our updaters do,
> so why not take advantage of that.

True, but this is kinda self-reinforcing: it’ll sure keep tracking more
if we stop maintaining our own code (IIRC, Repology was started after
‘guix refresh’ and I believe it’s maintained by one person.)

>> My preference would be to keep our existing updaters rather than
>> effectively ditch them and delegate the work to Repology.  It’s tempting
>> to think we can have both, but I’m not sure this would last long.
>
> The point of the Repology updater is to act as a fallback if none of
> the other updaters can update a package, e.g., ‘maven-dependency-tree’.
> I already mentioned that language-specific updaters usually provide more
> accurate and detailed information, so they should be used when possible;
> we aren’t losing anything here.

Hmm yes, could be.  OTOH, like Nicolas writes, we would probably need
some filtering or post-processing to reduce false-positives, right?

Do you have examples where our updaters perform poorly and where
Repology does a better job?  I wonder if there are lessons to be drawn
and bugs to be fixed.

Thanks,
Ludo’.

[bug#53818] [PATCH 0/3] Add Repology updater

[Nicolas Goaziou]

Hello,

Ludovic Courtès <ludo@gnu.org> writes:

> Do you have examples where our updaters perform poorly and where
> Repology does a better job?  I wonder if there are lessons to be drawn
> and bugs to be fixed.

As a data point, I'm sorry to say that our updaters are useless to me.

I watch over more than one thousand packages. I would have a hard time
expressing what are those packages to the updater, besides writing and
keeping up-to-date a huge manifest file. Assuming I could manage this,
fetching all version information would take considerable time, and,
since many packages are from GitHub, the party would stop early anyway
with GitHub refusing to proceed and requesting some token I don't have.

OTOH, using Repology API, I get the information I want in about ten
seconds. Sure, I need to eyeball through the results, filtering false
positives (around 4% in my case), but it still is a practical solution.

IMO, to be useful, updaters may need to rely on an external service,
which may, or may not, belong to the Guix ecosystem. They also need
a good UI.

I don't want to sound too negative, though. And current updaters are
certainly good enough when watching over a couple of packages, which
might be the most common use-case.

Cheers,
-- 
Nicolas Goaziou

[bug#53818] [PATCH 0/3] Add Repology updater

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1338 bytes --]

Nicolas Goaziou schreef op ma 14-02-2022 om 11:40 [+0100]:
> [...]. Assuming I could manage this,
> fetching all version information would take considerable time, and,
> since many packages are from GitHub, the party would stop early anyway
> with GitHub refusing to proceed and requesting some token I don't have.
> 
> OTOH, using Repology API, I get the information I want in about ten
> seconds. Sure, I need to eyeball through the results, filtering false
> positives (around 4% in my case), but it still is a practical solution.
> 
> IMO, to be useful, updaters may need to rely on an external service,
> which may, or may not, belong to the Guix ecosystem. They also need
> a good UI.

To avoid exceeding API limits and reduce network traffic, I suggest the
following change:

  Cache HTTP responses, using http-fetch/cached instead of
  http-fetch.  When something is in the cache and not expired,
  this avoids some network traffic and does not bring us closer
  to the API limits.

  When it is expired (and in the cache), then at least
  http-fetch/cached makes a conditional request with
  If-Modified-Since, which GitHub does not count against the rate
  limit, assuming a ‘304 Not Modified’ response!

That does not address all your concerns but it should help I think.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#53818] [PATCH 0/3] Add Repology updater

[Ludovic Courtès]

Hi Nicolas,

Nicolas Goaziou <mail@nicolasgoaziou.fr> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Do you have examples where our updaters perform poorly and where
>> Repology does a better job?  I wonder if there are lessons to be drawn
>> and bugs to be fixed.
>
> As a data point, I'm sorry to say that our updaters are useless to me.
>
> I watch over more than one thousand packages. I would have a hard time
> expressing what are those packages to the updater, besides writing and
> keeping up-to-date a huge manifest file. Assuming I could manage this,
> fetching all version information would take considerable time, and,
> since many packages are from GitHub, the party would stop early anyway
> with GitHub refusing to proceed and requesting some token I don't have.
>
> OTOH, using Repology API, I get the information I want in about ten
> seconds. Sure, I need to eyeball through the results, filtering false
> positives (around 4% in my case), but it still is a practical solution.

(I’m confused because my understanding of what you first wrote was that
Repology had too many false positives to be useful.)

You wrote about your feelings and that’s insightful, but can we focus on
specific examples where updaters are not helpful so we can better
understand and improve the situation?

> IMO, to be useful, updaters may need to rely on an external service,
> which may, or may not, belong to the Guix ecosystem.

All the updaters rely on an external service.  Relying on a centralized
SaaSS is different, though.

> They also need a good UI.

Do you have examples of what’s wrong on the UI side?

To me, the main shortcoming is that ‘guix refresh’ doesn’t tell you that
if you update X, you may also need to update Y and Z.  That info is not
always available, but it is available in repos such as PyPI and ELPA.

Thanks,
Ludo’.

[bug#53818] [PATCH 0/3] Add Repology updater

[Nicolas Goaziou]

Hello,

Ludovic Courtès <ludo@gnu.org> writes:

> (I’m confused because my understanding of what you first wrote was that
> Repology had too many false positives to be useful.)

Repology is okay for my use-case because I've gotten accustomed to its
quirks. I wouldn't recommend it as a fall-back solution for Guix in its
current form, tho, for the reason above. Does that make sense?

> You wrote about your feelings and that’s insightful, but can we focus on
> specific examples where updaters are not helpful so we can better
> understand and improve the situation?

I wrote about the following facts:
- it is difficult to specify a large number of packages,
- when you have specified a large number of packages, the processing is
  slow,
- checking GitHub fails for me.

I don't see any feelings in there.

>> IMO, to be useful, updaters may need to rely on an external service,
>> which may, or may not, belong to the Guix ecosystem.
>
> All the updaters rely on an external service.  Relying on a centralized
> SaaSS is different, though.

Fair enough. I meant an external centralized service.

> Do you have examples of what’s wrong on the UI side?

It has no Emacs interface. Nuff said. ;)

Again, I don't know how to specify efficiently many packages, e.g., all
Emacs packages, or all games. Also, reading through a massive output in
the terminal is not very user friendly, IMO.

> To me, the main shortcoming is that ‘guix refresh’ doesn’t tell you that
> if you update X, you may also need to update Y and Z.  That info is not
> always available, but it is available in repos such as PyPI and ELPA.

I don't think solving this is realistic. Dependencies are sometimes very
loose.

Regards,
-- 
Nicolas Goaziou

[bug#53818] Improving updaters and ‘guix refresh’

[Ludovic Courtès]

Hi!

Nicolas Goaziou <mail@nicolasgoaziou.fr> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> (I’m confused because my understanding of what you first wrote was that
>> Repology had too many false positives to be useful.)
>
> Repology is okay for my use-case because I've gotten accustomed to its
> quirks. I wouldn't recommend it as a fall-back solution for Guix in its
> current form, tho, for the reason above. Does that make sense?

It sure does, thanks for explaining.

> I wrote about the following facts:
> - it is difficult to specify a large number of packages,
> - when you have specified a large number of packages, the processing is
>   slow,
> - checking GitHub fails for me.

Alright, I had missed that.

Regarding “specifying many packages”, do examples like these work for
you:

  • guix refresh -t elpa

  • guix refresh $(guix package -A ^emacs- | cut -f1)

  • guix refresh -r emacs-emms

  • guix refresh -s non-core -t generic-git

  • guix refresh -m packages-i-care-about.scm

If not, what kind of selection mechanism could help?  ‘-s’ currently
accepts only two values, but we could augment it.

Regarding slow processing, it very much depends on the updater.  For
example, on a warm cache, ‘guix refresh -t gnu’ is relatively fast
thanks to caching:

--8<---------------cut here---------------start------------->8---
$ time guix refresh -t gnu
gnu/packages/wget.scm:48:13: wget would be upgraded from 1.21.1 to 1.21.2
gnu/packages/tls.scm:86:13: libtasn1 would be upgraded from 4.17.0 to 4.18.0

[...]


real	0m38.314s
user	0m38.981s
sys	0m0.164s
--8<---------------cut here---------------end--------------->8---

It could be that some updaters do many HTTP round trips without any
caching, which slows things down.

[...]

>> Do you have examples of what’s wrong on the UI side?
>
> It has no Emacs interface. Nuff said. ;)

True!  :-)

I realize this is going off-topic, but let’s see if we can improve the
existing infrastructure to make it more convenient.

Thanks,
Ludo’.

[bug#53818] Improving updaters and ‘guix refresh’

[Nicolas Goaziou]

Hello,

Ludovic Courtès <ludo@gnu.org> writes:

> Regarding “specifying many packages”, do examples like these work for
> you:
>
>   • guix refresh -t elpa

I don't find it very useful in practice. As a user, the packages I'm
interested in probably rely on more than one updater. I'm not even
supposed to know what updater relates to a given package.

I actually only use this when I know a GNU ELPA package is outdated
already, and I want it to compute the hash for me:

  ./pre-inst-env guix refresh -t elpa -u emacs-foo

>   • guix refresh $(guix package -A ^emacs- | cut -f1)

This one is interesting. This illustrates that the UI is, from my point
of view, a bit lacking. It would be a nice improvement to add a regexp
mechanism built-in, like in "guix search".

In any case, this fails after reporting status of around 50 packages,
with this time:

  real	0m41,881s
  user	0m12,155s
  sys	0m0,726s

Assuming I don't get the "rate limit exceeded" error, at this rate, it
would take more than 15 minutes to check all the packages in
"emacs-xyz.scm". This is a bit long.

I don't see how this could reasonably be made faster without relying on
an external centralized service doing the checks regularly (e.g., once
a day) before the user actually requests them.

>   • guix refresh -r emacs-emms

It also fails with the "rate limit exceeded". While this sounds
theoretically nice, I wouldn't know how to make use of it yet.

>   • guix refresh -s non-core -t generic-git

See above about "-t elpa".

>   • guix refresh -m packages-i-care-about.scm

Yes, obviously, this is a nice, too. However, it doesn't scale if you
need to specify 1000+ packages.

> If not, what kind of selection mechanism could help?  ‘-s’ currently
> accepts only two values, but we could augment it.

Besides regexp matching, it may be useful to filter packages per module,
or source file name. Package categories is a bit awkward, tho, and
probably not satisfying.

> I realize this is going off-topic, but let’s see if we can improve the
> existing infrastructure to make it more convenient.

Is it really off-topic?

Anyway, all of this is only one data point, and, as a reminder,
I certainly don't want to disparage either Xinglu Chen's work, or
current "guix refresh" functionality.

HTH,
-- 
Nicolas Goaziou

[bug#53818] Improving updaters and ‘guix refresh’

[Ludovic Courtès]

Hi,

Nicolas Goaziou <mail@nicolasgoaziou.fr> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Regarding “specifying many packages”, do examples like these work for
>> you:
>>
>>   • guix refresh -t elpa
>
> I don't find it very useful in practice. As a user, the packages I'm
> interested in probably rely on more than one updater. I'm not even
> supposed to know what updater relates to a given package.

Right, that’s more for packagers than for users.

As a user, what works better is:

  guix refresh -r $(guix package -I |cut -f1) -s non-core

… or simply ‘--with-latest’, if I’m not interested in updating package
definitions.

>>   • guix refresh $(guix package -A ^emacs- | cut -f1)
>
> This one is interesting. This illustrates that the UI is, from my point
> of view, a bit lacking. It would be a nice improvement to add a regexp
> mechanism built-in, like in "guix search".

Makes sense, we can do that.

> In any case, this fails after reporting status of around 50 packages,
> with this time:
>
>   real	0m41,881s
>   user	0m12,155s
>   sys	0m0,726s

How does it fail?  If it’s the GitHub rate limit, then there’s only one
answer: you have to provide a token.

> Assuming I don't get the "rate limit exceeded" error, at this rate, it
> would take more than 15 minutes to check all the packages in
> "emacs-xyz.scm". This is a bit long.

> I don't see how this could reasonably be made faster without relying on
> an external centralized service doing the checks regularly (e.g., once
> a day) before the user actually requests them.

Maybe you’re right, but before jumping to the conclusion, we have to
investigate a bit.  Like I wrote, the ‘gnu’ updater for instance fetches
a single file that remains in cache afterwards—the cost is constant.

We should identify updaters that have linear cost and check what can be
done.  ‘github’, ‘generic-html’, and ‘generic-git’ are of that kind.

Now, the command I gave above looks at 1,134 packages, so is it even
something you want to do as a packager?

>>   • guix refresh -r emacs-emms
>
> It also fails with the "rate limit exceeded". While this sounds
> theoretically nice, I wouldn't know how to make use of it yet.
>
>>   • guix refresh -s non-core -t generic-git
>
> See above about "-t elpa".
>
>>   • guix refresh -m packages-i-care-about.scm
>
> Yes, obviously, this is a nice, too. However, it doesn't scale if you
> need to specify 1000+ packages.

You can use ‘fold-packages’ and have three lines that return a manifest
of 10K packages if you want it.

Honestly, since I mostly rely on others these days :-), I’m no longer
sure what the packager’s workflow is.  Also, the level of coupling
varies greatly between, say, a C/C++ package and a set of
Python/Emacs/Rust packages.

I find that ‘guix refresh’ works fine for loosely-coupled C/C++ packages
where often you’d want to upgrade packages individually.

But for Python and Emacs packages, what do we want?  Do packagers always
want to check 1K+ packages at once?  Or are there other patterns?

>> If not, what kind of selection mechanism could help?  ‘-s’ currently
>> accepts only two values, but we could augment it.
>
> Besides regexp matching, it may be useful to filter packages per module,
> or source file name. Package categories is a bit awkward, tho, and
> probably not satisfying.

We can add options to make it more convenient, but it’s already
possible:

  guix refresh $(guix package -A | grep emacs-xyz.scm | cut -f1)

>> I realize this is going off-topic, but let’s see if we can improve the
>> existing infrastructure to make it more convenient.
>
> Is it really off-topic?
>
> Anyway, all of this is only one data point, and, as a reminder,
> I certainly don't want to disparage either Xinglu Chen's work, or
> current "guix refresh" functionality.

Yup, same here!

I think we have nice infrastructure but you raise important
shortcomings.  What Xinglu Chen did might in fact be one way to address
it, and there may also be purely UI issues that we could address.

Thanks,
Ludo’.

[bug#53818] Improving updaters and ‘guix refresh’

[zimoun]

Hi,

On Thu, 17 Feb 2022 at 11:35, Ludovic Courtès <ludo@gnu.org> wrote:

>>>   • guix refresh $(guix package -A ^emacs- | cut -f1)
>>
>> This one is interesting. This illustrates that the UI is, from my point
>> of view, a bit lacking. It would be a nice improvement to add a regexp
>> mechanism built-in, like in "guix search".
>
> Makes sense, we can do that.

I agree the UI is not nice.  Well, at the command line, I never read the
complete output of “guix package -A” and I always pipe it with “cut
-f1”.  Well, I think this complete display is only useful for
third-party; the only one I have in mind is emacs-guix.  Therefore, are
we maintaining this CLI for backward compatibility when we could change
both?

Something more useful as output would be:

   name version synopsis

Whatever. :-)

Even the internal etc/completion/bash/guix has to pipe:

--8<---------------cut here---------------start------------->8---
_guix_complete_available_package ()
{
    local prefix="$1"
    if [ -z "$_guix_available_packages" ]
    then
	# Cache the complete list because it rarely changes and makes
	# completion much faster.
	_guix_available_packages="$(${COMP_WORDS[0]} package -A 2> /dev/null \
                                    | cut -f1)"
    fi
    COMPREPLY+=($(compgen -W "$_guix_available_packages" -- "$prefix"))
}
--8<---------------cut here---------------end--------------->8---


Last, I am not convinced that “guix search” would be help here.
Because:

  1. the output requires to pipe with recsel,
  2. it is much slower than “package -A” [1].


1: <https://issues.guix.gnu.org/39258#119>


>>>   • guix refresh -m packages-i-care-about.scm
>>
>> Yes, obviously, this is a nice, too. However, it doesn't scale if you
>> need to specify 1000+ packages.

[...]

>> In any case, this fails after reporting status of around 50 packages,
>> with this time:
>>
>>   real	0m41,881s
>>   user	0m12,155s
>>   sys	0m0,726s
>
> How does it fail?  If it’s the GitHub rate limit, then there’s only one
> answer: you have to provide a token.

Let mimick a collection if 1000+ packages I care about.  Consider this
manifest for packages using r-build-system only…

--8<---------------cut here---------------start------------->8---
(use-modules (guix packages)
             (gnu packages)
             (guix build-system r))

(packages->manifest
 (fold-packages (lambda (package result)
                  (if (eq? (package-build-system package) r-build-system)
                      (cons package result)
                      result))
                '()))
--8<---------------cut here---------------end--------------->8---

…it hits the issue of Github token…

--8<---------------cut here---------------start------------->8---
gnu/packages/bioconductor.scm:6034:13: 1.66.0 is already the latest version of r-plgem
gnu/packages/bioconductor.scm:6011:13: 1.22.0 is already the latest version of r-rots
gnu/packages/bioconductor.scm:12614:2: warning: 'bioconductor' updater failed to determine available releases for r-fourcseq
Backtrace:
          13 (primitive-load "/home/simon/.config/guix/current/bin/guix")

[...]

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
Error downloading release information through the GitHub
API. This may be fixed by using an access token and setting the environment
variable GUIX_GITHUB_TOKEN, for instance one procured from
https://github.com/settings/tokens

real	10m27.306s
user	4m14.077s
sys	0m12.467s
--8<---------------cut here---------------end--------------->8---

…when most R packages come from CRAN or Bioconductor archives.


Basically, ~5000 packages come from Github which represents ~25% of
overall.  Therefore, one needs to be really lucky when updating many
package and not hit the Github rate limit.


Yes, large collection of packages cannot be updated easily.  Somehow, it
is an issue from upstream and it is hard to fix… except by
duplicating upstream or provide a token. :-)


Well, using the external centralized Repology service is a first step to
update at scale, no?  A second step could be to have this feature
included in the Data Service; but before we have other fishes to fry,
IMHO. :-)


>> Assuming I don't get the "rate limit exceeded" error, at this rate, it
>> would take more than 15 minutes to check all the packages in
>> "emacs-xyz.scm". This is a bit long.
>>
>> I don't see how this could reasonably be made faster without relying on
>> an external centralized service doing the checks regularly (e.g., once
>> a day) before the user actually requests them.
>
> Maybe you’re right, but before jumping to the conclusion, we have to
> investigate a bit.  Like I wrote, the ‘gnu’ updater for instance fetches
> a single file that remains in cache afterwards—the cost is constant.

Repology acts as this “external centralized service”, no?  On one hand,
it is a practical solution; especially by being fast enough.  On the
other hand, it serves few false positives (say 4% to fix the ideas).

Nicolas, considering the complexity of packages and their origins, do
you think it would be possible to do better (fast and accurate) than
Repology at scale?


>>>   • guix refresh -m packages-i-care-about.scm
>>
>> Yes, obviously, this is a nice, too. However, it doesn't scale if you
>> need to specify 1000+ packages.
>
> You can use ‘fold-packages’ and have three lines that return a manifest
> of 10K packages if you want it.

Yes, see example above.


>>> If not, what kind of selection mechanism could help?  ‘-s’ currently
>>> accepts only two values, but we could augment it.
>>
>> Besides regexp matching, it may be useful to filter packages per module,
>> or source file name. Package categories is a bit awkward, tho, and
>> probably not satisfying.
>
> We can add options to make it more convenient, but it’s already
> possible:

Since these features are advanced, why not keep the CLI simple and
instead on rely manifest files for complex filtering?


>>> I realize this is going off-topic, but let’s see if we can improve the
>>> existing infrastructure to make it more convenient.

[...]

> I think we have nice infrastructure but you raise important
> shortcomings.  What Xinglu Chen did might in fact be one way to address
> it, and there may also be purely UI issues that we could address.

All the points raised here are important but appears to me orthogonal
with the patch series. :-)


Cheers,
simon

[bug#53818] Improving updaters and ‘guix refresh’

[Nicolas Goaziou]

Hello,

zimoun <zimon.toutoune@gmail.com> writes:

> On Thu, 17 Feb 2022 at 11:35, Ludovic Courtès <ludo@gnu.org> wrote:


>> How does it fail?  If it’s the GitHub rate limit, then there’s only one
>> answer: you have to provide a token.

IIUC, I have to register on GitHub to create this token. This is a bit
sad as a prerequisite to use one core feature of Guix.

> Let mimick a collection if 1000+ packages I care about.  Consider this
> manifest for packages using r-build-system only…
>
> --8<---------------cut here---------------start------------->8---
> (use-modules (guix packages)
>              (gnu packages)
>              (guix build-system r))
>
> (packages->manifest
>  (fold-packages (lambda (package result)
>                   (if (eq? (package-build-system package) r-build-system)
>                       (cons package result)
>                       result))
>                 '()))
> --8<---------------cut here---------------end--------------->8---

I have to learn about fold-packages.

> Nicolas, considering the complexity of packages and their origins, do
> you think it would be possible to do better (fast and accurate) than
> Repology at scale?

It's not about doing better, but doing differently. We do not need all
of Repology's features. 

As far as the updater part is concerned, Repology pokes at various
package repositories, which are usually not upstream, extracts package
versions, applies some heuristics to normalize and compare them, then
decides what is the newest version and which repositories provide
outdated packages. This has a two obvious shortcomings:

1. the version number usually doesn't come from an official source, so
   it may be wrong—e.g., our emacs-csv-mode is "outdated" because Funtoo
   1.4 chose a non-existing higher version number for the same package.

2. version comparison does not understand every local versioning
   scheme—e.g., our emacs-fold-dwim packages is currently at
   "1.2-0.c46f4bb", which is, in Guix parlance, after "1.2", yet
   Repology thinks this is actually older than "1.2".

Therefore, I think a (theoretical) centralized Guix-centric version
checker could be fast: it would only poke at what our packages consider
to be upstream, and accurate, since it would know about our versioning
rules. Basically it could boil down to calling current "guix refresh" on
every package daily, and serializing the results.

>>>>   • guix refresh -m packages-i-care-about.scm
>>>
>>> Yes, obviously, this is a nice, too. However, it doesn't scale if you
>>> need to specify 1000+ packages.
>>
>> You can use ‘fold-packages’ and have three lines that return a manifest
>> of 10K packages if you want it.
>
> Yes, see example above.

Point taken.

Regards,
-- 
Nicolas Goaziou

[bug#53818] Improving updaters and ‘guix refresh’

[Ludovic Courtès]

Hi!

Nicolas Goaziou <mail@nicolasgoaziou.fr> skribis:

> zimoun <zimon.toutoune@gmail.com> writes:
>
>> On Thu, 17 Feb 2022 at 11:35, Ludovic Courtès <ludo@gnu.org> wrote:
>
>
>>> How does it fail?  If it’s the GitHub rate limit, then there’s only one
>>> answer: you have to provide a token.
>
> IIUC, I have to register on GitHub to create this token. This is a bit
> sad as a prerequisite to use one core feature of Guix.

I have some good news!

  https://issues.guix.gnu.org/54241

Granted, it’s not a revolution, but it should fix one of the main
annoyances of ‘guix refresh’.

Regarding the “sad prerequisite”, the ‘github’ updater predates the
‘generic-git’ updater by several years, during which it was the only way
to get data for matching packages.

Actually I wonder if it’s still useful to keep.  In theory it can
provide more accurate data than the ‘generic-git’ updater; not sure if
this is the case in practice.

Ludo’.