[bug#74542] [PATCH 00/11] Improved tooling for package updates

[bug#74542] [PATCH 00/11] Improved tooling for package updates

[Ludovic Courtès]

Hello Guix!

This patch series aims to improve tooling for package upgrades:

  • By adding a ‘--dependents’ option to ‘guix build’ to build
    dependent packages (those returned by ‘guix refresh -l’).  This
    was suggested by my colleague Romain Garbage, who also prototyped
    a variant of this.

  • By improving individual updaters and updating metadata for
    some packages.

  • By adding an “upgrade manifest”, to automatically build the latest
    version (using the ‘with-latest’ package transformation) of a few
    security-critical packages along with their dependents at distance 2,
    as well as some of the leaf packages.

The goal is to use this upgrade manifest for a new jobset on ci.guix.
The idea is that when someone tries to upgrade one of these things,
chances are it will have been built already.

I already toyed with this idea here:

  https://guix.bordeaux.inria.fr/jobset/guix-hpc-cutting-edge
  https://gitlab.inria.fr/guix-hpc/guix-hpc/-/blob/master/cutting-edge.manifest?ref_type=heads

It’s experimental; surely we’ll find that the manifest needs to be
tweaked, or that some of the updaters abort to early.  But I think
it can help, and it’s a good way to use available x86_64 CPU power
at ci.guix.

According to a recent discussion with Nix hackers on Mastodon¹, mere
substitute availability is not perceived as a key factor in what
enables them to keep Nixpkgs up-to-date; the primary factor appears
to be the bot that sends patches updating packages.

Food for thought!

Ludo’.

¹ https://toot.aquilenet.fr/@civodul/113527656544893427

Ludovic Courtès (11):
  transformations: Export ‘package-with-upstream-version’.
  gnu-maintenance: ‘import-html-release’ doesn’t abort upon HTTP 404.
  gnu-maintenance: Savannah/Xorg updaters no longer abort on network
    errors.
  build: Add ‘--development’ option.
  packages: Factorize ‘all-packages’.
  guix build: Add ‘--dependents’.
  import: gnome: Keep going upon HTTP errors.
  gnu-maintenance: ‘gnu-ftp’ updater excludes GnuPG-hosted packages.
  gnu: Update updater properties for GnuPG-related packages.
  guix build: Validate that the file passed to ‘-m’ returns a manifest.
  etc: Add upgrade manifest.

 Makefile.am              |   1 +
 doc/contributing.texi    |   4 +-
 doc/guix.texi            |  52 +++++++++++
 etc/source-manifest.scm  |  13 +--
 etc/upgrade-manifest.scm |  98 +++++++++++++++++++++
 gnu/packages.scm         |  20 ++++-
 gnu/packages/gnupg.scm   |  40 +++------
 guix/download.scm        |   3 +-
 guix/gnu-maintenance.scm |  49 +++++++----
 guix/import/gnome.scm    |  13 ++-
 guix/scripts/build.scm   | 185 ++++++++++++++++++++++++++++++---------
 guix/scripts/graph.scm   |  10 ---
 guix/scripts/refresh.scm |  10 ---
 guix/scripts/weather.scm |  15 ----
 guix/transformations.scm |   2 +
 tests/guix-build.sh      |  11 +++
 16 files changed, 386 insertions(+), 140 deletions(-)
 create mode 100644 etc/upgrade-manifest.scm


base-commit: 827ec92f44d1b1daedc5446146801183560e861b
-- 
2.46.0

[bug#74542] [PATCH 01/11] transformations: Export ‘package-with-upstream-version’.

[Ludovic Courtès]

* guix/transformations.scm (package-with-upstream-version): Export.

Change-Id: I8e09e69e9c32bffc329c97ef1c0258f750181b11
---
 guix/transformations.scm | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/guix/transformations.scm b/guix/transformations.scm
index ea8b7a0844..9dfc4402c5 100644
--- a/guix/transformations.scm
+++ b/guix/transformations.scm
@@ -61,6 +61,8 @@ (define-module (guix transformations)
             tunable-package?
             tuned-package
 
+            package-with-upstream-version
+
             show-transformation-options-help
             transformation-option-key?
             cacheable-transformation-option-key?
-- 
2.46.0

[bug#74542] [PATCH 02/11] gnu-maintenance: ‘import-html-release’ doesn’t abort upon HTTP 404.

[Ludovic Courtès]

Previously, updaters using ‘import-html-release’ would throw to 'quit
when the HTML page isn’t found, aborting the whole process:

  $ guix refresh coreutils libchop
  guix refresh: error: https://de.freedif.org/savannah//libchop/: HTTP download failed: 404 ("Not Found")

* guix/gnu-maintenance.scm (url->links): Guard against ‘http-get-error?’
and return the empty list when it is raised.

Change-Id: I419ff96ddb424aa8c74d6bab3d66a5e85355c3dc
---
 guix/gnu-maintenance.scm | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)

diff --git a/guix/gnu-maintenance.scm b/guix/gnu-maintenance.scm
index 881e941fbf..f34dcfc13f 100644
--- a/guix/gnu-maintenance.scm
+++ b/guix/gnu-maintenance.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2010-2023 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2010-2024 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2012, 2013 Nikita Karetnikov <nikita@karetnikov.org>
 ;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
 ;;; Copyright © 2022 Maxime Devos <maximedevos@telenet.be>
@@ -30,6 +30,7 @@ (define-module (guix gnu-maintenance)
   #:use-module (srfi srfi-2)
   #:use-module (srfi srfi-11)
   #:use-module (srfi srfi-26)
+  #:use-module (srfi srfi-34)
   #:use-module (rnrs io ports)
   #:use-module ((guix http-client) #:hide (open-socket-for-uri))
   ;; not required in many cases, so autoloaded to reduce start-up costs.
@@ -496,11 +497,16 @@ (define (html-links sxml)
 
 (define (url->links url)
   "Return the unique links on the HTML page accessible at URL."
-  (let* ((uri   (string->uri url))
-         (port  (http-fetch/cached uri #:ttl 3600))
-         (sxml  (html->sxml port)))
-    (close-port port)
-    (delete-duplicates (html-links sxml))))
+  (guard (c ((http-get-error? c)
+             (warning (G_ "failed to download '~a': ~a (~a)~%")
+                      url (http-get-error-code c)
+                      (http-get-error-reason c))
+             '()))
+    (let* ((uri   (string->uri url))
+           (port  (http-fetch/cached uri #:ttl 3600))
+           (sxml  (html->sxml port)))
+      (close-port port)
+      (delete-duplicates (html-links sxml)))))
 
 (define (canonicalize-url url base-url)
   "Make relative URL absolute, by appending URL to BASE-URL as required.  If
-- 
2.46.0

[bug#74542] [PATCH 03/11] gnu-maintenance: Savannah/Xorg updaters no longer abort on network errors.

[Ludovic Courtès]

Previously these updaters would fail upon networking error, causing the
whole process to abort:

  $ guix refresh coreutils libgcrypt
  guix refresh: error: connect*: Connection refused

* guix/gnu-maintenance.scm (import-release*, import-xorg-release): Wrap
in ‘false-if-networking-error’.

Change-Id: I8d309bf8662e2d46460585b4a1b7da0a4f2e8907
---
 guix/gnu-maintenance.scm | 25 ++++++++++++++-----------
 1 file changed, 14 insertions(+), 11 deletions(-)

diff --git a/guix/gnu-maintenance.scm b/guix/gnu-maintenance.scm
index f34dcfc13f..866b960a4a 100644
--- a/guix/gnu-maintenance.scm
+++ b/guix/gnu-maintenance.scm
@@ -469,10 +469,12 @@ (define* (import-release* package #:key (version #f))
 \"emacs-auctex\", for instance.)"
   (let-values (((server directory)
                 (ftp-server/directory package)))
-    (false-if-ftp-error (import-release (package-upstream-name package)
-                                        #:version version
-                                        #:server server
-                                        #:directory directory))))
+    (false-if-networking-error
+     (false-if-ftp-error
+      (import-release (package-upstream-name package)
+                      #:version version
+                      #:server server
+                      #:directory directory)))))
 
 \f
 ;;;
@@ -913,13 +915,14 @@ (define* (import-xorg-release package #:key (version #f))
   "Return the latest release of PACKAGE.  Optionally include a VERSION string
 to fetch a specific version."
   (let ((uri (string->uri (origin-uri (package-source package)))))
-    (false-if-ftp-error
-     (import-ftp-release
-      (package-name package)
-      #:version version
-      #:server "ftp.freedesktop.org"
-      #:directory
-      (string-append "/pub/xorg/" (dirname (uri-path uri)))))))
+    (false-if-networking-error
+     (false-if-ftp-error
+      (import-ftp-release
+       (package-name package)
+       #:version version
+       #:server "ftp.freedesktop.org"
+       #:directory
+       (string-append "/pub/xorg/" (dirname (uri-path uri))))))))
 
 (define* (import-kernel.org-release package #:key (version #f))
   "Return the latest release of PACKAGE, a Linux kernel package.
-- 
2.46.0

[bug#74542] [PATCH 04/11] build: Add ‘--development’ option.

[Ludovic Courtès]

* guix/scripts/build.scm (show-help, %options): Add ‘-D’.
(options->things-to-build): Change ‘append-map’ to a loop.  Honor ‘-D’.
* tests/guix-build.sh: Add test.
* doc/guix.texi (Additional Build Options): Document it.

Change-Id: I99227aadfe861e43c001a4872292bd687b37f5d4
---
 doc/guix.texi          |  25 +++++++++
 guix/scripts/build.scm | 119 ++++++++++++++++++++++++++++-------------
 tests/guix-build.sh    |   5 ++
 3 files changed, 112 insertions(+), 37 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 26488b41c8..a9d0d044ae 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -13614,6 +13614,31 @@ Additional Build Options
 (@pxref{The Store Monad}).  The procedure must return a derivation as a
 monadic value, which is then passed through @code{run-with-store}.
 
+@item --development
+@itemx -D
+Build the ``development environment'' (build dependencies) of the
+following package.
+
+For example, the following command builds the inputs of @code{hello},
+but @emph{not} @code{hello} itself, and also builds @code{guile}:
+
+@example
+guix build -D hello guile
+@end example
+
+Notice that @option{-D} (or @option{--development}) only applies to the
+immediately following package on the command line.  Under the hood, it
+uses @code{package->development-manifest}
+(@pxref{package-development-manifest,
+@code{package->development-manifest}}).
+
+@quotation Note
+The effect of combining @option{--development} with @option{--target}
+(for cross-compilation) may not be what you expect: it will
+cross-compile all the dependencies of the given package when it is built
+natively.
+@end quotation
+
 @item --source
 @itemx -S
 Build the source derivations of the packages, rather than the packages
diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm
index 9e05bb532b..326d04f1f8 100644
--- a/guix/scripts/build.scm
+++ b/guix/scripts/build.scm
@@ -438,6 +438,8 @@ (define (show-help)
   -m, --manifest=FILE    build the packages that the manifest given in FILE
                          evaluates to"))
   (display (G_ "
+  -D, --development      build the inputs of the following package"))
+  (display (G_ "
   -S, --source           build the packages' source derivations"))
   (display (G_ "
       --sources[=TYPE]   build source derivations; TYPE may optionally be one
@@ -522,6 +524,9 @@ (define %options
          (option '(#\m "manifest") #t #f
                  (lambda (opt name arg result)
                    (alist-cons 'manifest arg result)))
+         (option '(#\D "development") #f #f
+                 (lambda (opt name arg result)
+                   (alist-cons 'development? #t result)))
          (option '(#\n "dry-run") #f #f
                  (lambda (opt name arg result)
                    (alist-cons 'dry-run? #t result)))
@@ -581,43 +586,83 @@ (define-public my-package
       (for-each validate-type lst)
       lst))
 
-  (append-map (match-lambda
-                (('argument . (? string? spec))
-                 (cond ((derivation-path? spec)
-                        (catch 'system-error
-                          (lambda ()
-                            ;; Ask for absolute file names so that .drv file
-                            ;; names passed from the user to 'read-derivation'
-                            ;; are absolute when it returns.
-                            (let ((spec (canonicalize-path spec)))
-                              (list (read-derivation-from-file spec))))
-                          (lambda args
-                            ;; Non-existent .drv files can be substituted down
-                            ;; the road, so don't error out.
-                            (if (= ENOENT (system-error-errno args))
-                                '()
-                                (apply throw args)))))
-                       ((store-path? spec)
-                        ;; Nothing to do; maybe for --log-file.
-                        '())
-                       (else
-                        (list (specification->package spec)))))
-                (('file . file)
-                 (let ((file (or (and (string-suffix? ".json" file)
-                                      (json->scheme-file file))
-                                 file)))
-                   (ensure-list (load* file (make-user-module '())))))
-                (('manifest . manifest)
-                 (map manifest-entry-item
-                      (manifest-entries
-                       (load* manifest
-                              (make-user-module '((guix profiles) (gnu)))))))
-                (('expression . str)
-                 (ensure-list (read/eval str)))
-                (('argument . (? derivation? drv))
-                 drv)
-                (_ '()))
-              opts))
+  (define system
+    (or (assoc-ref opts 'system) (%current-system)))
+
+  ;; Process OPTS in "the right order", meaning that if the user typed
+  ;; "-D hello", arrange to see the 'development? option before the "hello"
+  ;; spec.
+  (let loop ((opts (reverse opts))
+             (type 'regular)
+             (result '()))
+    (define (for-type obj)
+      ;; Return a list of objects corresponding to OBJ adjusted for TYPE.
+      (match type
+        ('regular
+         (list obj))
+        ('development
+         (if (package? obj)
+             (map manifest-entry-item
+                  (manifest-entries
+                   (package->development-manifest obj system)))
+             obj))))
+
+    (match opts
+      (()
+       (reverse result))
+      ((head . tail)
+       (match head
+         (('argument . (? string? spec))
+          (cond ((derivation-path? spec)
+                 (catch 'system-error
+                   (lambda ()
+                     ;; Ask for absolute file names so that .drv file
+                     ;; names passed from the user to 'read-derivation'
+                     ;; are absolute when it returns.
+                     (let ((spec (canonicalize-path spec)))
+                       (loop tail 'regular
+                             (cons (read-derivation-from-file spec)
+                                   result))))
+                   (lambda args
+                     ;; Non-existent .drv files can be substituted down
+                     ;; the road, so don't error out.
+                     (if (= ENOENT (system-error-errno args))
+                         (loop tail 'regular result)
+                         (apply throw args)))))
+                ((store-path? spec)
+                 ;; Nothing to do; maybe for --log-file.
+                 (loop tail type result))
+                (else
+                 (loop tail 'regular
+                       (append (for-type (specification->package spec))
+                               result)))))
+         (('file . file)
+          (let ((file (or (and (string-suffix? ".json" file)
+                               (json->scheme-file file))
+                          file)))
+            (loop tail 'regular
+                  (append (append-map
+                           for-type
+                           (ensure-list (load* file (make-user-module '()))))
+                          result))))
+         (('manifest . manifest)
+          (loop tail 'regular
+                (append (map manifest-entry-item
+                             (manifest-entries
+                              (load* manifest
+                                     (make-user-module '((guix profiles)
+                                                         (gnu))))))
+                        result)))
+         (('expression . str)
+          (loop tail 'regular
+                (append (append-map for-type (ensure-list (read/eval str)))
+                        result)))
+         (('argument . (? derivation? drv))
+          (loop tail 'regular (cons drv result)))
+         (('development? . #t)
+          (loop tail 'development result))
+         (_
+          (loop tail type result)))))))
 
 (define (options->derivations store opts)
   "Given OPTS, the result of 'args-fold', return a list of derivations to
diff --git a/tests/guix-build.sh b/tests/guix-build.sh
index 36eac2b7e0..3637bcdeb3 100644
--- a/tests/guix-build.sh
+++ b/tests/guix-build.sh
@@ -190,6 +190,11 @@ test `guix build -d --sources=transitive foo \
       | grep -e 'foo\.tar\.gz' -e 'bar\.tar\.gz' -e 'bar\.dat' \
       | wc -l` -eq 3
 
+# Building the inputs.
+guix build -D hello -n
+test `guix build -D hello -d \
+      | grep -e 'glibc.*\.drv$' -e 'gcc.*\.drv$' -e 'binutils.*\.drv$' \
+      | wc -l` -ge 3
 
 # Unbound variable in thunked field.
 cat > "$module_dir/foo.scm" <<EOF
-- 
2.46.0

[bug#74542] [PATCH 05/11] packages: Factorize ‘all-packages’.

[Ludovic Courtès]

* gnu/packages.scm (all-packages): New procedure.
* etc/source-manifest.scm (all-packages): Remove.
* guix/scripts/graph.scm (all-packages): Remove.
* guix/scripts/refresh.scm (all-packages): Remove.
* guix/scripts/weather.scm (all-packages): Remove.

Change-Id: I6072952c4b877b541037ce86402cfb7744eeb0a0
---
 etc/source-manifest.scm  | 13 +------------
 gnu/packages.scm         | 20 +++++++++++++++++++-
 guix/scripts/graph.scm   | 10 ----------
 guix/scripts/refresh.scm | 10 ----------
 guix/scripts/weather.scm | 15 ---------------
 5 files changed, 20 insertions(+), 48 deletions(-)

diff --git a/etc/source-manifest.scm b/etc/source-manifest.scm
index f96a5da6f7..3e1ae07959 100644
--- a/etc/source-manifest.scm
+++ b/etc/source-manifest.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2021 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2021, 2024 Ludovic Courtès <ludo@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -25,17 +25,6 @@
              (guix packages) (guix profiles)
              (gnu packages))
 
-(define (all-packages)
-  "Return the list of all the packages, public or private, omitting only
-superseded packages."
-  (fold-packages (lambda (package lst)
-                   (match (package-replacement package)
-                     (#f (cons package lst))
-                     (replacement
-                      (append (list replacement package) lst))))
-                 '()
-                 #:select? (negate package-superseded)))
-
 (define (upstream-origin source)
   "Return SOURCE without any patches or snippet."
   (origin (inherit source)
diff --git a/gnu/packages.scm b/gnu/packages.scm
index 80c22d1d7f..1af3b8d440 100644
--- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2012-2020, 2022-2023 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2012-2020, 2022-2024 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2013 Mark H Weaver <mhw@netris.org>
 ;;; Copyright © 2014 Eric Bavier <bavier@member.fsf.org>
 ;;; Copyright © 2016, 2017 Alex Kost <alezost@gmail.com>
@@ -56,6 +56,7 @@ (define-module (gnu packages)
             cache-is-authoritative?
 
             fold-packages
+            all-packages
             fold-available-packages
 
             find-newest-available-packages
@@ -253,6 +254,23 @@ (define* (fold-packages proc init
                                 init
                                 modules))
 
+(define all-packages
+  (mlambda ()
+    "Return the list of all public packages, including replacements and hidden
+packages, excluding superseded packages."
+    (delete-duplicates
+     (fold-packages (lambda (package result)
+                      (match (package-replacement package)
+                        ((? package? replacement)
+                         (cons* replacement package result))
+                        (#f
+                         (cons package result))))
+                    '()
+
+                    ;; Dismiss deprecated packages but keep hidden packages.
+                    #:select? (negate package-superseded))
+     eq?)))
+
 (define %package-cache-file
   ;; Location of the package cache.
   "/lib/guix/package.cache")
diff --git a/guix/scripts/graph.scm b/guix/scripts/graph.scm
index 6740858d8b..935721edea 100644
--- a/guix/scripts/graph.scm
+++ b/guix/scripts/graph.scm
@@ -119,16 +119,6 @@ (define %package-node-type
 ;;; Reverse package DAG.
 ;;;
 
-(define (all-packages)            ;XXX: duplicated from (guix scripts refresh)
-  "Return the list of all the distro's packages."
-  (fold-packages (lambda (package result)
-                   ;; Ignore deprecated packages.
-                   (if (package-superseded package)
-                       result
-                       (cons package result)))
-                 '()
-                 #:select? (const #t)))           ;include hidden packages
-
 (define %reverse-package-node-type
   ;; For this node type we first need to compute the list of packages and the
   ;; list of back-edges.  Since we want to do it only once, we use the
diff --git a/guix/scripts/refresh.scm b/guix/scripts/refresh.scm
index ec7d38c22a..8c72d0c545 100644
--- a/guix/scripts/refresh.scm
+++ b/guix/scripts/refresh.scm
@@ -455,16 +455,6 @@ (define* (check-for-package-update update-spec updaters #:key warn?)
 ;;; Dependents.
 ;;;
 
-(define (all-packages)
-  "Return the list of all the distro's packages."
-  (fold-packages (lambda (package result)
-                   ;; Ignore deprecated packages.
-                   (if (package-superseded package)
-                       result
-                       (cons package result)))
-                 '()
-                 #:select? (const #t)))           ;include hidden packages
-
 (define (list-dependents packages)
   "List all the things that would need to be rebuilt if PACKAGES are changed."
   ;; Using %BAG-NODE-TYPE is more accurate than using %PACKAGE-NODE-TYPE
diff --git a/guix/scripts/weather.scm b/guix/scripts/weather.scm
index 08a1b22a74..29432fd923 100644
--- a/guix/scripts/weather.scm
+++ b/guix/scripts/weather.scm
@@ -55,21 +55,6 @@ (define-module (guix scripts weather)
   #:use-module (ice-9 vlist)
   #:export (guix-weather))
 
-(define (all-packages)
-  "Return the list of public packages we are going to query."
-  (delete-duplicates
-   (fold-packages (lambda (package result)
-                    (match (package-replacement package)
-                      ((? package? replacement)
-                       (cons* replacement package result))
-                      (#f
-                       (cons package result))))
-                  '()
-
-                  ;; Dismiss deprecated packages but keep hidden packages.
-                  #:select? (negate package-superseded))
-   eq?))
-
 (define (call-with-progress-reporter reporter proc)
   "This is a variant of 'call-with-progress-reporter' that works with monadic
 scope."
-- 
2.46.0

[bug#74542] [PATCH 06/11] guix build: Add ‘--dependents’.

[Ludovic Courtès]

* guix/scripts/build.scm (show-help, %options): Add ‘--dependents’.
(dependents): New procedure.
(options->things-to-build): Add ‘store’ parameter; honor ‘dependents’
option.
[for-type]: Handle ‘dependents’ type.
(options->derivations): Update call to ‘options->things-to-build’.
* tests/guix-build.sh: Add test.
* doc/guix.texi (Additional Build Options): Document ‘--dependents’.
(Invoking guix refresh): Cross-reference it.
* doc/contributing.texi (Submitting Patches): Mention it.

Change-Id: I00b6d5831e1f1d35dc8b84a82605391d5a8f417c
---
 doc/contributing.texi  |  4 ++-
 doc/guix.texi          | 27 ++++++++++++++++++++
 guix/scripts/build.scm | 56 ++++++++++++++++++++++++++++++++++++++++--
 tests/guix-build.sh    |  6 +++++
 4 files changed, 90 insertions(+), 3 deletions(-)

diff --git a/doc/contributing.texi b/doc/contributing.texi
index b063169189..5a778466d7 100644
--- a/doc/contributing.texi
+++ b/doc/contributing.texi
@@ -1914,7 +1914,9 @@ Submitting Patches
 
 @item
 Make sure the package builds on your platform, using @command{guix build
-@var{package}}.
+@var{package}}.  Also build at least its direct dependents with
+@command{guix build --dependents=1 @var{package}}
+(@pxref{build-dependents, @command{guix build}}).
 
 @item
 We recommend you also try building the package on other supported
diff --git a/doc/guix.texi b/doc/guix.texi
index a9d0d044ae..5734cf306b 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -13639,6 +13639,31 @@ Additional Build Options
 natively.
 @end quotation
 
+@cindex dependents of a package, building them
+@cindex building the dependents of a package
+@anchor{build-dependents}
+@item --dependents[=@var{depth}]
+@itemx -T [@var{depth}]
+Build the dependents of the following package.  By default, build all
+the direct and indirect dependents; when @var{depth} is provided, limit
+to dependents at that distance: 1 for direct dependents, 2 for
+dependents of dependents, and so on.
+
+For example, the command below builds @emph{all} the dependents of libgit2:
+
+@example
+guix build --dependents libgit2
+@end example
+
+To build all the packages that directly depend on NumPy, run:
+
+@example
+guix build -T1 python-numpy
+@end example
+
+The list of dependents is computed in the same way as with @command{guix
+refresh --list-dependent} (@pxref{Invoking guix refresh}).
+
 @item --source
 @itemx -S
 Build the source derivations of the packages, rather than the packages
@@ -15142,6 +15167,8 @@ Invoking guix refresh
 @command{guix graph}}, for information on how to visualize the list of
 dependents of a package.
 
+@xref{build-dependents, @command{guix build --dependents}}, for a
+convenient way to build all the dependents of a package.
 @end table
 
 Be aware that the @option{--list-dependent} option only
diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm
index 326d04f1f8..565bfd48e9 100644
--- a/guix/scripts/build.scm
+++ b/guix/scripts/build.scm
@@ -33,6 +33,9 @@ (define-module (guix scripts build)
   #:use-module (guix profiles)
   #:use-module (guix diagnostics)
   #:autoload   (guix http-client) (http-fetch http-get-error?)
+  #:autoload   (guix scripts graph) (%bag-node-type)
+  #:autoload   (guix graph) (node-back-edges)
+  #:autoload   (guix sets) (setq set-contains? set-insert)
   #:use-module (ice-9 format)
   #:use-module (ice-9 match)
   #:use-module (srfi srfi-1)
@@ -440,6 +443,9 @@ (define (show-help)
   (display (G_ "
   -D, --development      build the inputs of the following package"))
   (display (G_ "
+  -T, --dependents[=N]   build dependents of the following package, up to
+                         depth N"))
+  (display (G_ "
   -S, --source           build the packages' source derivations"))
   (display (G_ "
       --sources[=TYPE]   build source derivations; TYPE may optionally be one
@@ -527,6 +533,11 @@ (define %options
          (option '(#\D "development") #f #f
                  (lambda (opt name arg result)
                    (alist-cons 'development? #t result)))
+         (option '(#\T "dependents") #f #t
+                 (lambda (opt name arg result)
+                   (alist-cons 'dependents
+                               (or (and=> arg string->number*) +inf.0)
+                               result)))
          (option '(#\n "dry-run") #f #f
                  (lambda (opt name arg result)
                    (alist-cons 'dry-run? #t result)))
@@ -551,7 +562,39 @@ (define %options
                  %standard-cross-build-options
                  %standard-native-build-options)))
 
-(define (options->things-to-build opts)
+(define (dependents store packages max-depth)
+  "List all the things that would need to be rebuilt if PACKAGES are changed."
+  ;; Using %BAG-NODE-TYPE is more accurate than using %PACKAGE-NODE-TYPE
+  ;; because it includes implicit dependencies.
+  (define (get-dependents packages edges)
+    (let loop ((packages packages)
+               (result '())
+               (depth 0)
+               (visited (setq)))
+      (if (> depth max-depth)
+          (values result visited)
+          (match packages
+            (()
+             (values result visited))
+            ((head . tail)
+             (if (set-contains? visited head)
+                 (loop tail result depth visited)
+                 (let ((next (edges head)))
+                   (call-with-values
+                       (lambda ()
+                         (loop next
+                               (cons head result)
+                               (+ depth 1)
+                               (set-insert head visited)))
+                     (lambda (result visited)
+                       (loop tail result depth visited))))))))))
+
+  (with-store store
+    (run-with-store store
+      (mlet %store-monad ((edges (node-back-edges %bag-node-type (all-packages))))
+        (return (get-dependents packages edges))))))
+
+(define (options->things-to-build store opts)
   "Read the arguments from OPTS and return a list of high-level objects to
 build---packages, gexps, derivations, and so on."
   (define (validate-type x)
@@ -600,6 +643,13 @@ (define-public my-package
       (match type
         ('regular
          (list obj))
+        (('dependents . depth)
+         (if (package? obj)
+             (begin
+               (info (G_ "computing dependents of package ~a...~%")
+                     (package-full-name obj))
+               (dependents store (list obj) depth))
+             (list obj)))
         ('development
          (if (package? obj)
              (map manifest-entry-item
@@ -661,6 +711,8 @@ (define-public my-package
           (loop tail 'regular (cons drv result)))
          (('development? . #t)
           (loop tail 'development result))
+         (('dependents . depth)
+          (loop tail `(dependents . ,depth) result))
          (_
           (loop tail type result)))))))
 
@@ -687,7 +739,7 @@ (define (options->derivations store opts)
       (systems systems)))
 
   (define things-to-build
-    (map transform (options->things-to-build opts)))
+    (map transform (options->things-to-build store opts)))
 
   (define warn-if-unsupported
     (let ((target (assoc-ref opts 'target)))
diff --git a/tests/guix-build.sh b/tests/guix-build.sh
index 3637bcdeb3..42e2ecafb1 100644
--- a/tests/guix-build.sh
+++ b/tests/guix-build.sh
@@ -196,6 +196,12 @@ test `guix build -D hello -d \
       | grep -e 'glibc.*\.drv$' -e 'gcc.*\.drv$' -e 'binutils.*\.drv$' \
       | wc -l` -ge 3
 
+# Building the dependents.
+test `guix build -T1 libgit2 -T1 libssh -d \
+      | grep -e 'guile-git.*\.drv$' -e 'guile-ssh.*\.drv$' \
+             -e 'libgit2.*\.drv$' -e 'libssh.*\.drv$' \
+      | wc -l` -eq 4
+
 # Unbound variable in thunked field.
 cat > "$module_dir/foo.scm" <<EOF
 (define-module (foo)
-- 
2.46.0

[bug#74542] [PATCH 07/11] import: gnome: Keep going upon HTTP errors.

[Ludovic Courtès]

* guix/import/gnome.scm (import-gnome-release): Do not re-raise
condition C, emit a warning instead.

Change-Id: I9a7c337a17d1f7ed4677398e7b0f383befca7d74
---
 guix/import/gnome.scm | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/guix/import/gnome.scm b/guix/import/gnome.scm
index 054ae44f7a..3ba8ae02e5 100644
--- a/guix/import/gnome.scm
+++ b/guix/import/gnome.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2017, 2019, 2021 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2017, 2019, 2021, 2024 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2022 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;; Copyright © 2022 Hartmut Goebel <h.goebel@crazy-compilers.com>
 ;;;
@@ -23,6 +23,8 @@ (define-module (guix import gnome)
   #:use-module (guix utils)
   #:use-module (guix packages)
   #:use-module (guix http-client)
+  #:use-module (guix diagnostics)
+  #:use-module (guix i18n)
   #:use-module (json)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-34)
@@ -111,9 +113,12 @@ (define* (import-gnome-release package #:key (version #f))
           releases))
 
   (guard (c ((http-get-error? c)
-             (if (= 404 (http-get-error-code c))
-                 #f
-                 (raise c))))
+             (unless (= 404 (http-get-error-code c))
+               (warning (G_ "failed to download from '~a': ~a (~s)~%")
+                        (uri->string (http-get-error-uri c))
+                        (http-get-error-code c)
+                        (http-get-error-reason c)))
+             #f))
     (let* ((port (http-fetch/cached
                   (string->uri (string-append
                                 "https://ftp.gnome.org/pub/gnome/sources/"
-- 
2.46.0

[bug#74542] [PATCH 08/11] gnu-maintenance: ‘gnu-ftp’ updater excludes GnuPG-hosted packages.

[Ludovic Courtès]

Since ftp://ftp.gnupg.org no longer exists, another updater should be
used for packages with code fetched from mirror://gnupg.

* guix/gnu-maintenance.scm (gnupg-hosted?): New procedure.
(%gnu-ftp-updater): Exclude packages that match ‘gnupg-hosted?’.
* guix/download.scm (%mirrors): Remove ftp://ftp.gnupg.org.

Change-Id: I653877ad843da60281d3a234acc85743bee64856
---
 guix/download.scm        | 3 +--
 guix/gnu-maintenance.scm | 6 +++++-
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/guix/download.scm b/guix/download.scm
index e37c2195ee..d88ad0ee44 100644
--- a/guix/download.scm
+++ b/guix/download.scm
@@ -78,8 +78,7 @@ (define %mirrors
        "ftp://mirrors.dotsrc.org/gcrypt/"
        "ftp://ftp.heanet.ie/mirrors/ftp.gnupg.org/gcrypt/"
        "ftp://ftp.mirrorservice.org/sites/ftp.gnupg.org/gcrypt/"
-       "ftp://ftp.ring.gr.jp/pub/net/gnupg/"
-       "ftp://ftp.gnupg.org/gcrypt/")
+       "ftp://ftp.ring.gr.jp/pub/net/gnupg/")
       (gnome
        "https://download.gnome.org/"
        "http://ftp.gnome.org/pub/GNOME/")
diff --git a/guix/gnu-maintenance.scm b/guix/gnu-maintenance.scm
index 866b960a4a..b612b11c00 100644
--- a/guix/gnu-maintenance.scm
+++ b/guix/gnu-maintenance.scm
@@ -1025,15 +1025,19 @@ (define %gnu-updater
            (false-if-networking-error (gnu-hosted? package))))
    (import import-gnu-release)))
 
+(define gnupg-hosted?
+  (url-prefix-predicate "mirror://gnupg/"))
+
 (define %gnu-ftp-updater
   ;; This is for GNU packages taken from alternate locations, such as
-  ;; alpha.gnu.org, ftp.gnupg.org, etc.  It is obsolescent.
+  ;; alpha.gnu.org (ftp.gnupg.org is no longer available).  It is obsolescent.
   (upstream-updater
    (name 'gnu-ftp)
    (description "Updater for GNU packages only available via FTP")
    (pred (lambda (package)
            (false-if-networking-error
             (and (not (gnu-hosted? package))
+                 (not (gnupg-hosted? package))
                  (pure-gnu-package? package)))))
    (import import-release*)))
 
-- 
2.46.0

[bug#74542] [PATCH 09/11] gnu: Update updater properties for GnuPG-related packages.

[Ludovic Courtès]

* gnu/packages/gnupg.scm (libgpg-error)[properties]: Remove.
(libgcrypt)[properties]: Remove.
(libassuan)[properties]: Remove.
(libksba)[properties]: Remove.
(npth)[properties]: Remove.
(gnupg)[properties]: Remove.
(gpgme)[properties]: Remove.
(pinentry-tty)[properties]: Change for ‘generic-html’.
(gpa)[properties]: Remove.

Change-Id: I1eb9d6ad5bb3ecbfbef3d44dbae281a3de7ebac7
---
 gnu/packages/gnupg.scm | 40 ++++++++++++----------------------------
 1 file changed, 12 insertions(+), 28 deletions(-)

diff --git a/gnu/packages/gnupg.scm b/gnu/packages/gnupg.scm
index 2e97c2244a..3f55ccaf5b 100644
--- a/gnu/packages/gnupg.scm
+++ b/gnu/packages/gnupg.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2012-2021, 2023 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2012-2021, 2023, 2024 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2013, 2015, 2018 Andreas Enge <andreas@enge.fr>
 ;;; Copyright © 2014, 2018 Eric Bavier <bavier@member.fsf.org>
 ;;; Copyright © 2014, 2015, 2016, 2020 Mark H Weaver <mhw@netris.org>
@@ -160,9 +160,7 @@ (define-public libgpg-error
 for all GnuPG components.  Among these are GPG, GPGSM, GPGME,
 GPG-Agent, libgcrypt, Libksba, DirMngr, Pinentry, SmartCard
 Daemon and possibly more in the future.")
-    (license license:lgpl2.0+)
-    (properties '((ftp-server . "ftp.gnupg.org")
-                  (ftp-directory . "/gcrypt/libgpg-error")))))
+    (license license:lgpl2.0+)))
 
 (define-public libgcrypt
   (package
@@ -217,9 +215,7 @@ (define-public libgcrypt
 standard cryptographic building blocks such as symmetric ciphers, hash
 algorithms, public key algorithms, large integer functions and random number
 generation.")
-    (license license:lgpl2.0+)
-    (properties '((ftp-server . "ftp.gnupg.org")
-                  (ftp-directory . "/gcrypt/libgcrypt")))))
+    (license license:lgpl2.0+)))
 
 (define-public libassuan
   (package
@@ -250,9 +246,7 @@ (define-public libassuan
 protocol.  This protocol is used for IPC between most newer
 GnuPG components.  Both, server and client side functions are
 provided.")
-    (license license:lgpl2.0+)
-    (properties '((ftp-server . "ftp.gnupg.org")
-                  (ftp-directory . "/gcrypt/libassuan")))))
+    (license license:lgpl2.0+)))
 
 (define-public libksba
   (package
@@ -283,9 +277,7 @@ (define-public libksba
      "KSBA (pronounced Kasbah) is a library to make X.509 certificates
 as well as the CMS easily accessible by other applications.  Both
 specifications are building blocks of S/MIME and TLS.")
-    (license license:gpl3+)
-    (properties '((ftp-server . "ftp.gnupg.org")
-                  (ftp-directory . "/gcrypt/libksba")))))
+    (license license:gpl3+)))
 
 (define-public npth
   (package
@@ -307,9 +299,7 @@ (define-public npth
 In contrast to GNU Pth is is based on the system's standard threads
 implementation.  This allows the use of libraries which are not
 compatible to GNU Pth.")
-    (license (list license:lgpl3+ license:gpl2+)) ; dual license
-    (properties '((ftp-server . "ftp.gnupg.org")
-                  (ftp-directory . "/gcrypt/npth")))))
+    (license (list license:lgpl3+ license:gpl2+)))) ;dual license
 
 (define-public gnupg
   (package
@@ -404,9 +394,7 @@ (define-public gnupg
 servers.  It includes several libraries: libassuan (IPC between GnuPG
 components), libgpg-error (centralized GnuPG error values), and
 libskba (working with X.509 certificates and CMS data).")
-    (license license:gpl3+)
-    (properties '((ftp-server . "ftp.gnupg.org")
-                  (ftp-directory . "/gcrypt/gnupg")))))
+    (license license:gpl3+)))
 
 (define-public gnupg-1
   (package (inherit gnupg)
@@ -463,9 +451,7 @@ (define-public gpgme
 programming task, it is suggested that all software should try to use GPGME
 instead.  This way bug fixes or improvements can be done at a central place
 and every application benefits from this.")
-    (license license:lgpl2.1+)
-    (properties '((ftp-server . "ftp.gnupg.org")
-                  (ftp-directory . "/gcrypt/gpgme")))))
+    (license license:lgpl2.1+)))
 
 (define-public gpgme-1.23
   (hidden-package
@@ -886,9 +872,9 @@ (define-public pinentry-tty
      "Pinentry provides a console that allows users to enter a passphrase when
 @code{gpg} is run and needs it.")
     (license license:gpl2+)
-    (properties '((ftp-server . "ftp.gnupg.org")
-                  (ftp-directory . "/gcrypt/pinentry")
-                  (upstream-name . "pinentry")))))
+    (properties
+     '((release-monitoring-url . "https://gnupg.org/ftp/gcrypt/pinentry/")
+       (upstream-name . "pinentry")))))
 
 (define-public pinentry-emacs
   (package
@@ -1134,9 +1120,7 @@ (define-public gpa
      "GPA, the GNU Privacy Assistant, is a graphical user interface for
 @uref{https://gnupg.org, GnuPG}.  It can be used to encrypt, decrypt, and sign
 files, to verify signatures, and to manage the private and public keys.")
-    (license license:gpl3+)
-    (properties '((ftp-server . "ftp.gnupg.org")
-                  (ftp-directory . "/gcrypt/gpa")))))
+    (license license:gpl3+)))
 
 (define-public parcimonie
   (package
-- 
2.46.0

[bug#74542] [PATCH 10/11] guix build: Validate that the file passed to ‘-m’ returns a manifest.

[Ludovic Courtès]

* guix/scripts/build.scm (options->things-to-build)[ensure-manifest]:
New procedure.
Use it.

Change-Id: If64c483d7079f441a296d5bd4e06e67f44cbb7bf
---
 guix/scripts/build.scm | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm
index 565bfd48e9..1b0b006ad5 100644
--- a/guix/scripts/build.scm
+++ b/guix/scripts/build.scm
@@ -629,6 +629,12 @@ (define-public my-package
       (for-each validate-type lst)
       lst))
 
+  (define (ensure-manifest x file)
+    (unless (manifest? x)
+      (raise (formatted-message (G_ "file '~a' does not return a manifest")
+                                file)))
+    x)
+
   (define system
     (or (assoc-ref opts 'system) (%current-system)))
 
@@ -699,9 +705,11 @@ (define-public my-package
           (loop tail 'regular
                 (append (map manifest-entry-item
                              (manifest-entries
-                              (load* manifest
-                                     (make-user-module '((guix profiles)
-                                                         (gnu))))))
+                              (ensure-manifest
+                               (load* manifest
+                                      (make-user-module '((guix profiles)
+                                                          (gnu))))
+                               manifest)))
                         result)))
          (('expression . str)
           (loop tail 'regular
-- 
2.46.0

[bug#74542] [PATCH 11/11] etc: Add upgrade manifest.

[Ludovic Courtès]

* guix/scripts/build.scm (dependents): Export.
* etc/upgrade-manifest.scm: New file.
* Makefile.am (EXTRA_DIST): Add it.

Change-Id: I1b2a2ebd09e559c68da9f25772bf33caacb4c031
---
 Makefile.am              |  1 +
 etc/upgrade-manifest.scm | 98 ++++++++++++++++++++++++++++++++++++++++
 guix/scripts/build.scm   |  2 +
 3 files changed, 101 insertions(+)
 create mode 100644 etc/upgrade-manifest.scm

diff --git a/Makefile.am b/Makefile.am
index e94ba87797..0cff32c607 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -743,6 +743,7 @@ EXTRA_DIST +=						\
   etc/source-manifest.scm				\
   etc/system-tests.scm					\
   etc/time-travel-manifest.scm				\
+  etc/upgrade-manifest.scm				\
   scripts/guix.in					\
   tests/cve-sample.json					\
   tests/keys/civodul.pub				\
diff --git a/etc/upgrade-manifest.scm b/etc/upgrade-manifest.scm
new file mode 100644
index 0000000000..6dd605ef03
--- /dev/null
+++ b/etc/upgrade-manifest.scm
@@ -0,0 +1,98 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2024 Ludovic Courtès <ludo@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+;; This manifest computes upgrades of key packages using the 'with-latest'
+;; package transformation.
+
+(use-modules (guix monads)
+             (guix graph)
+             (guix packages)
+             (guix profiles)
+             (guix store)
+             (guix transformations)
+             ((guix scripts build) #:select (dependents))
+             ((guix scripts graph) #:select (%bag-node-type))
+             ((guix import github) #:select (%github-api))
+             (guix build-system gnu)
+             (guix build-system cmake)
+             ((gnu packages) #:select (all-packages))
+             (ice-9 match)
+             (srfi srfi-1))
+
+;; Bypass the GitHub updater: we'd need an API token or we would hit the rate
+;; limit.
+(%github-api "http://example.org")
+
+(define (leaf-packages)
+  (with-store store
+    (run-with-store store
+      (mlet %store-monad ((edges (node-back-edges %bag-node-type (all-packages))))
+        (return (filter (lambda (package)
+                          (null? (edges package)))
+                        (all-packages)))))))
+
+(define security-packages
+  '("git" "git-minimal"
+    "xorg-server"
+    "elogind"
+    "openssl"
+    "gnutls"
+    "libarchive"
+    "libgit2"
+    "libssh"
+
+    ;; GnuPG.
+    "libassuan"
+    "libgpg-error"
+    "libgcrypt"
+    "libksba"
+    "npth"
+    "gnupg"
+    "gpgme"
+    "pinentry"))
+
+(define security-upgrades
+  ;; Upgrades of individual packages with their dependents built against that
+  ;; upgrade.
+  (manifest
+   (with-store store
+     (append-map (match-lambda
+                   ((package . output)
+                    (let* ((name (package-name package))
+                           (latest (options->transformation
+                                    `((with-latest . ,name)))))
+                      (map (lambda (package)
+                             (manifest-entry
+                               (inherit (package->manifest-entry
+                                         (latest (pk 'latest package))))
+                               (name (string-append (package-name package)
+                                                    "-with-latest-" name))))
+                           (dependents store (list package) 2)))))
+                 (specifications->packages security-packages)))))
+
+(define leaf-package-updates
+  ;; Select a subset (~22%) of all the leaf packages, typically small C/C++
+  ;; packages not part of a bigger "collection" or repo (CRAN, PyPI, etc.).
+  (manifest
+   (filter-map (lambda (package)
+                 (and (memq (package-build-system package)
+                            (list gnu-build-system cmake-build-system))
+                      (package-with-upstream-version (pk 'up package))))
+               (leaf-packages))))
+
+(concatenate-manifest (list leaf-package-updates security-upgrades))
diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm
index 1b0b006ad5..ddebcaf743 100644
--- a/guix/scripts/build.scm
+++ b/guix/scripts/build.scm
@@ -63,6 +63,8 @@ (define-module (guix scripts build)
             show-cross-build-options-help
             show-native-build-options-help
 
+            dependents
+
             guix-build
             register-root
             register-root*))
-- 
2.46.0

[bug#74542] [PATCH 00/11] Improved tooling for package updates

[Ludovic Courtès]

Ludovic Courtès <ludo@gnu.org> skribis:

> I already toyed with this idea here:
>
>   https://guix.bordeaux.inria.fr/jobset/guix-hpc-cutting-edge
>   https://gitlab.inria.fr/guix-hpc/guix-hpc/-/blob/master/cutting-edge.manifest?ref_type=heads
>
> It’s experimental; surely we’ll find that the manifest needs to be
> tweaked, or that some of the updaters abort to early.  But I think
> it can help, and it’s a good way to use available x86_64 CPU power
> at ci.guix.

Another point: having a single “upgrade manifest” probably won’t scale,
first because it takes time to evaluate it (looking for the latest
upstream version of all these packages), second because the jobset will
be messy if it contains tons of unrelated upgrades.

So I’m thinking that probably we’ll want focused upgrade manifests: one
for selected security-critical packages, one for the astronomy packages
so Sharlatan can save a bit on energy at home ;-), one for R packages,
and so on.

Another one I’d like to have: automatic ungrafting so that we could
apply ungrafting packages “anytime” (ideally!).

The sky’s the limit!

Ludo’.

[bug#74542] [PATCH 00/11] Improved tooling for package updates

[Simon Tournier]

Hi,

On Tue, 26 Nov 2024 at 11:32, Ludovic Courtès <ludo@gnu.org> wrote:

> This patch series aims to improve tooling for package upgrades:
>
>   • By adding a ‘--dependents’ option to ‘guix build’ to build
>     dependent packages (those returned by ‘guix refresh -l’).  This
>     was suggested by my colleague Romain Garbage, who also prototyped
>     a variant of this.
>
>   • By improving individual updaters and updating metadata for
>     some packages.

Cool!

>
>   • By adding an “upgrade manifest”, to automatically build the latest
>     version (using the ‘with-latest’ package transformation) of a few
>     security-critical packages along with their dependents at distance 2,
>     as well as some of the leaf packages.

I am more reserved here.

--8<---------------cut here---------------start------------->8---
scheme@(guix-user)> (define all-leafs (leaf-packages))
scheme@(guix-user)> (length all-leafs)
$2 = 13680
scheme@(guix-user)> (define leaf-to-updates
   (filter-map (lambda (package)
                 (and (memq (package-build-system package)
                            (list gnu-build-system cmake-build-system))
                      package))
               all-leafs))
scheme@(guix-user)> (length leaf-to-updates)
$3 = 2843
--8<---------------cut here---------------end--------------->8---

This list contains multiversion packages, for instance:

    "clang"
    "clang"
    "clang"
    "gccgo"
    "gccgo"
    "gccgo"
    "gccgo"
    "gccgo"
    "glibc"
    "glibc"
    "glibc"
    "glibc"
    "glibc"
    "glibc-locales"
    "glibc-locales-2.29"
    "glibc-locales-2.30"
    "glibc-locales-2.31"
    "glibc-locales-2.32"
    "glibc-locales-2.33"
    "glibc-locales-2.35"
    "linux-libre"
    "linux-libre"
    "linux-libre"
    "linux-libre"
    "linux-libre"
    "linux-libre-arm-generic"
    "linux-libre-arm-generic"
    "linux-libre-arm-generic"
    "linux-libre-arm-generic"
    "linux-libre-arm-omap2plus"
    "linux-libre-arm-omap2plus"
    "linux-libre-arm64-generic"
    "linux-libre-arm64-generic"
    "linux-libre-arm64-generic"
    "linux-libre-arm64-generic"
    "linux-libre-arm64-honeycomb"
    "linux-libre-documentation"
    "linux-libre-headers"
    "linux-libre-headers"
    "linux-libre-headers"
    "linux-libre-headers"
    "linux-libre-headers"
    "linux-libre-headers"
 

And this list also contains PyPI, for instance:

    "pyconfigure"
    "pypy"
    "python-cffi-documentation"
    "python-clingo-dl"
    "python-debug"
    "python-graph-tool"
    "python-libfreenect"
    "python-libmpsse"
    "python-libsvm"
    "python-next"
    "python-on-guile"
    "python-openturns"
    "python-sepolgen"
    "python2-dbus"
    "python2-minimal"

Maybe, we could be a bit more stringent on the list and filter out more
packages than only the ones based on build-systems.

Well, it could be helpful to avoid to burn resources for nothing.  For
example, it could be better to exclude the GitHub importer instead of
bypassing it with a wrong URL.

> Ludovic Courtès (11):
>   transformations: Export ‘package-with-upstream-version’.
>   gnu-maintenance: ‘import-html-release’ doesn’t abort upon HTTP 404.
>   gnu-maintenance: Savannah/Xorg updaters no longer abort on network
>     errors.
>   build: Add ‘--development’ option.
>   packages: Factorize ‘all-packages’.
>   guix build: Add ‘--dependents’.
>   import: gnome: Keep going upon HTTP errors.
>   gnu-maintenance: ‘gnu-ftp’ updater excludes GnuPG-hosted packages.
>   gnu: Update updater properties for GnuPG-related packages.
>   guix build: Validate that the file passed to ‘-m’ returns a manifest.
>   etc: Add upgrade manifest.

Let review patch by patch. :-)

Cheers,
simon

[bug#74542] [PATCH 01/11] transformations: Export ‘package-with-upstream-version’.

[Simon Tournier]

On Tue, 26 Nov 2024 at 11:33, Ludovic Courtès <ludo@gnu.org> wrote:
> * guix/transformations.scm (package-with-upstream-version): Export.
>
> Change-Id: I8e09e69e9c32bffc329c97ef1c0258f750181b11
> ---
>  guix/transformations.scm | 2 ++
>  1 file changed, 2 insertions(+)

LGTM!  Good to start with the simplest. :-)

[bug#74542] [PATCH 02/11] gnu-maintenance: ‘import-html-release’ doesn’t abort upon HTTP 404.

[Simon Tournier]

Hi,

On Tue, 26 Nov 2024 at 11:33, Ludovic Courtès <ludo@gnu.org> wrote:

> +  #:use-module (srfi srfi-34)

[...]

> +  (guard (c ((http-get-error? c)
> +             (warning (G_ "failed to download '~a': ~a (~a)~%")
> +                      url (http-get-error-code c)
> +                      (http-get-error-reason c))
> +             '()))
> +    (let* ((uri   (string->uri url))
> +           (port  (http-fetch/cached uri #:ttl 3600))
> +           (sxml  (html->sxml port)))
> +      (close-port port)
> +      (delete-duplicates (html-links sxml)))))

I’m always confused by which framework for exception does the project
rely on?  There is old ’catch’ and ’throw’ then sometimes new
’with-exception-handler’ and ’raise’.  And also SRFI 34.

What’s the preferred convention?

Cheers,
simon

[bug#74542] [PATCH 03/11] gnu-maintenance: Savannah/Xorg updaters no longer abort on network errors.

[Simon Tournier]

Hi,

On Tue, 26 Nov 2024 at 11:33, Ludovic Courtès <ludo@gnu.org> wrote:

>  guix/gnu-maintenance.scm | 25 ++++++++++++++-----------
>  1 file changed, 14 insertions(+), 11 deletions(-)

LGTM.

Cheers,
simon

[bug#74542] [PATCH 04/11] build: Add ‘--development’ option.

[Simon Tournier]

Hi,

Here the oneline commit message is:

     build: Add ‘--development’ option.

when elsewhere in the same series, it reads:

     guix build: Add ‘--dependents’.

I suggest consistency; ’build:’ or ’guix build:’ for both.

On Tue, 26 Nov 2024 at 11:33, Ludovic Courtès <ludo@gnu.org> wrote:

> +  ;; Process OPTS in "the right order", meaning that if the user typed
> +  ;; "-D hello", arrange to see the 'development? option before the "hello"
> +  ;; spec.
> +  (let loop ((opts (reverse opts))
> +             (type 'regular)
> +             (result '()))
> +    (define (for-type obj)
> +      ;; Return a list of objects corresponding to OBJ adjusted for TYPE.
> +      (match type
> +        ('regular
> +         (list obj))
> +        ('development
> +         (if (package? obj)
> +             (map manifest-entry-item
> +                  (manifest-entries
> +                   (package->development-manifest obj system)))
> +             obj))))

Well, the loop seems overcomplicated but nothing better to propose. ;-)

> +    (match opts
> +      (()
> +       (reverse result))
> +      ((head . tail)
> +       (match head
> +         (('argument . (? string? spec))
> +          (cond ((derivation-path? spec)
> +                 (catch 'system-error
> +                   (lambda ()
> +                     ;; Ask for absolute file names so that .drv file
> +                     ;; names passed from the user to 'read-derivation'
> +                     ;; are absolute when it returns.
> +                     (let ((spec (canonicalize-path spec)))
> +                       (loop tail 'regular
> +                             (cons (read-derivation-from-file spec)
> +                                   result))))
> +                   (lambda args
> +                     ;; Non-existent .drv files can be substituted down
> +                     ;; the road, so don't error out.
> +                     (if (= ENOENT (system-error-errno args))
> +                         (loop tail 'regular result)
> +                         (apply throw args)))))
> +                ((store-path? spec)
> +                 ;; Nothing to do; maybe for --log-file.
> +                 (loop tail type result))
> +                (else
> +                 (loop tail 'regular
> +                       (append (for-type (specification->package spec))
> +                               result)))))
> +         (('file . file)
> +          (let ((file (or (and (string-suffix? ".json" file)
> +                               (json->scheme-file file))
> +                          file)))
> +            (loop tail 'regular
> +                  (append (append-map
> +                           for-type
> +                           (ensure-list (load* file (make-user-module '()))))
> +                          result))))
> +         (('manifest . manifest)
> +          (loop tail 'regular
> +                (append (map manifest-entry-item
> +                             (manifest-entries
> +                              (load* manifest
> +                                     (make-user-module '((guix profiles)
> +                                                         (gnu))))))
> +                        result)))
> +         (('expression . str)
> +          (loop tail 'regular
> +                (append (append-map for-type (ensure-list (read/eval str)))
> +                        result)))
> +         (('argument . (? derivation? drv))
> +          (loop tail 'regular (cons drv result)))

Since you are at it, why not move this pattern close to the other
pattern matching ’argument’ too?


Cheers,
simon

[bug#74542] [PATCH 07/11] import: gnome: Keep going upon HTTP errors.

[Simon Tournier]

Hi,

On Tue, 26 Nov 2024 at 11:33, Ludovic Courtès <ludo@gnu.org> wrote:
> * guix/import/gnome.scm (import-gnome-release): Do not re-raise
> condition C, emit a warning instead.
>
> Change-Id: I9a7c337a17d1f7ed4677398e7b0f383befca7d74
> ---
>  guix/import/gnome.scm | 13 +++++++++----
>  1 file changed, 9 insertions(+), 4 deletions(-)

LGTM.

Cheers,
simon

[bug#74542] [PATCH 08/11] gnu-maintenance: ‘gnu-ftp’ updater excludes GnuPG-hosted packages.

[Simon Tournier]

Hi,

On Tue, 26 Nov 2024 at 11:33, Ludovic Courtès <ludo@gnu.org> wrote:
> Since ftp://ftp.gnupg.org no longer exists, another updater should be
> used for packages with code fetched from mirror://gnupg.
>
> * guix/gnu-maintenance.scm (gnupg-hosted?): New procedure.
> (%gnu-ftp-updater): Exclude packages that match ‘gnupg-hosted?’.
> * guix/download.scm (%mirrors): Remove ftp://ftp.gnupg.org.
>
> Change-Id: I653877ad843da60281d3a234acc85743bee64856
> ---
>  guix/download.scm        | 3 +--
>  guix/gnu-maintenance.scm | 6 +++++-
>  2 files changed, 6 insertions(+), 3 deletions(-)

LGTM.

Cheers,
simon

[bug#74542] [PATCH 09/11] gnu: Update updater properties for GnuPG-related packages.

[Simon Tournier]

Hi,

On Tue, 26 Nov 2024 at 11:33, Ludovic Courtès <ludo@gnu.org> wrote:
> * gnu/packages/gnupg.scm (libgpg-error)[properties]: Remove.
> (libgcrypt)[properties]: Remove.
> (libassuan)[properties]: Remove.
> (libksba)[properties]: Remove.
> (npth)[properties]: Remove.
> (gnupg)[properties]: Remove.
> (gpgme)[properties]: Remove.
> (pinentry-tty)[properties]: Change for ‘generic-html’.
> (gpa)[properties]: Remove.
>
> Change-Id: I1eb9d6ad5bb3ecbfbef3d44dbae281a3de7ebac7
> ---
>  gnu/packages/gnupg.scm | 40 ++++++++++++----------------------------
>  1 file changed, 12 insertions(+), 28 deletions(-)

LGTM.

Cool!  Nice clean up!

Cheers,
simon

[bug#74542] [PATCH 10/11] guix build: Validate that the file passed to ‘-m’ returns a manifest.

[Simon Tournier]

Hi,

On Tue, 26 Nov 2024 at 11:33, Ludovic Courtès <ludo@gnu.org> wrote:
> * guix/scripts/build.scm (options->things-to-build)[ensure-manifest]:
> New procedure.
> Use it.
>
> Change-Id: If64c483d7079f441a296d5bd4e06e67f44cbb7bf
> ---
>  guix/scripts/build.scm | 14 +++++++++++---
>  1 file changed, 11 insertions(+), 3 deletions(-)

LGTM.

Cheers,
simon

[bug#74542] [PATCH 11/11] etc: Add upgrade manifest.

[Simon Tournier]

Hi,

On Tue, 26 Nov 2024 at 11:33, Ludovic Courtès <ludo@gnu.org> wrote:

> +(define leaf-package-updates
> +  ;; Select a subset (~22%) of all the leaf packages, typically small C/C++
> +  ;; packages not part of a bigger "collection" or repo (CRAN, PyPI, etc.).
> +  (manifest
> +   (filter-map (lambda (package)
> +                 (and (memq (package-build-system package)
> +                            (list gnu-build-system cmake-build-system))
> +                      (package-with-upstream-version (pk 'up package))))
> +               (leaf-packages))))

I would be finer here.  For instance, the manifest contains the same
package at several versions, i.e., ’package-with-upstream-version’
applies to the same package at different version. Slurm is one example
among many others:

--8<---------------cut here---------------start------------->8---
scheme@(guix-user)> ,pp (sort
                         (map
                          (lambda (p) `(,(package-name p) . ,(package-version p)))
                          (filter-map
                            (lambda (package)
                               (and (memq (package-build-system package)
                                           (list gnu-build-system cmake-build-system))
                                       package))
                                  (leaf-packages)))
                         (lambda (p1 p2)
                           (match p1 ((n1 . _) (match p2 ((n2 . _) (string< n1 n2)))))))
[...]

 ("slurm" . "20.11.9")
 ("slurm" . "20.02.6-1")
 ("slurm" . "23.02.6")
 ("slurm" . "21.08.8")
 ("slurm" . "22.05.1")
 ("slurm" . "19.05.8")
 ("slurm" . "18.08.9")
 ("slurm-drmaa" . "1.1.3")
 ("slurm-minimal" . "20.02.6-1")
 ("slurm-minimal" . "22.05.1")
 ("slurm-minimal" . "23.02.6")
 ("slurm-minimal" . "21.08.8")
 ("slurm-minimal" . "18.08.9")
 ("slurm-minimal" . "20.11.9")
 ("slurm-minimal" . "23.11.10")
 ("slurm-minimal" . "19.05.8")

[...]
--8<---------------cut here---------------end--------------->8---

Cheers,
simon

[bug#74542] [PATCH 00/11] Improved tooling for package updates

[Simon Tournier]

Hi,

On Tue, 26 Nov 2024 at 15:42, Ludovic Courtès <ludo@gnu.org> wrote:

> Another point: having a single “upgrade manifest” probably won’t scale,
> first because it takes time to evaluate it (looking for the latest
> upstream version of all these packages), second because the jobset will
> be messy if it contains tons of unrelated upgrades.

Yes, that’d be my suggestion: having a better filtering strategy for
’leaf-packages’…

> So I’m thinking that probably we’ll want focused upgrade manifests: one
> for selected security-critical packages, one for the astronomy packages
> so Sharlatan can save a bit on energy at home ;-), one for R packages,
> and so on.

…so yes, it appears to me better to have security for one, then maybe
one per team?  Or per updater?

> Another one I’d like to have: automatic ungrafting so that we could
> apply ungrafting packages “anytime” (ideally!).

You read in my dreams! ;-)

> The sky’s the limit!

Let touch the void…

Cheers,
simon

[bug#74542] [PATCH 00/11] Improved tooling for package updates

[Suhail Singh]

Ludovic Courtès <ludo@gnu.org> writes:

> The goal is to use this upgrade manifest for a new jobset on ci.guix.
> The idea is that when someone tries to upgrade one of these things,
> chances are it will have been built already.
>
> ...
>
> According to a recent discussion with Nix hackers on Mastodon¹, mere
> substitute availability is not perceived as a key factor in what
> enables them to keep Nixpkgs up-to-date; the primary factor appears
> to be the bot that sends patches updating packages.
>
> Food for thought!
>
> Ludo’.
>
> ¹ https://toot.aquilenet.fr/@civodul/113527656544893427

If the goal is to improve how up-to-date a distro Guix is, then the
below would help.  In no particular order (the numbers are only to
provide named references):

1. Ensure a periodic process (let's say alice-bot) that runs on the CI
   server that submits patches for updates.
2. alice-bot could, in fact, be part of the "new jobset on ci.guix" that
   would be updating the substitutes.  If the substitute building is
   deemed successful it could result in the corresponding upgrade patch
   being submitted.
3. Having a trusted (i.e., trusted by guix authenticate), review bot
   (let's say bob-bot), whose job is to review alice-bot submissions and
   determine whether desirable invariants are being upheld.  If they
   are, bob-bot merges the patch.  If not, it requests human
   intervention.  It would help for bob-bot to add/remove usertags on
   the debbugs issue (and for those state-changes to be visible on
   <https://issues.guix.gnu.org/>) to ensure the attention of human
   reviewers is not wasted.

Having bob-bot be distinct from alice-bot would help keep the process
transparent and reduce bob-bot's code size (thus aiding auditability).
Additionally, it would also allow for bob-bot to be used to provide
automated lint/style checks for patch submissions by people.

The patch series in #74542 provides some necessary tooling to help keep
the distro up-to-date.  However, we still need alice-bot and bob-bot as
described above to get to sufficiency.

--
Suhail

[bug#74542] [PATCH 02/11] gnu-maintenance: ‘import-html-release’ doesn’t abort upon HTTP 404.

[Ludovic Courtès]

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> On Tue, 26 Nov 2024 at 11:33, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> +  #:use-module (srfi srfi-34)
>
> [...]
>
>> +  (guard (c ((http-get-error? c)
>> +             (warning (G_ "failed to download '~a': ~a (~a)~%")
>> +                      url (http-get-error-code c)
>> +                      (http-get-error-reason c))
>> +             '()))
>> +    (let* ((uri   (string->uri url))
>> +           (port  (http-fetch/cached uri #:ttl 3600))
>> +           (sxml  (html->sxml port)))
>> +      (close-port port)
>> +      (delete-duplicates (html-links sxml)))))
>
> I’m always confused by which framework for exception does the project
> rely on?  There is old ’catch’ and ’throw’ then sometimes new
> ’with-exception-handler’ and ’raise’.  And also SRFI 34.
>
> What’s the preferred convention?

My take: SRFI-34.  Before 3.0 we had to use ‘catch’ for key+args
exceptions, which is why it’s still used a lot.

Hopefully it’ll get more homogeneous over time.  :-)

Ludo’.

[bug#74542] [PATCH 11/11] etc: Add upgrade manifest.

[Ludovic Courtès]

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> On Tue, 26 Nov 2024 at 11:33, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> +(define leaf-package-updates
>> +  ;; Select a subset (~22%) of all the leaf packages, typically small C/C++
>> +  ;; packages not part of a bigger "collection" or repo (CRAN, PyPI, etc.).
>> +  (manifest
>> +   (filter-map (lambda (package)
>> +                 (and (memq (package-build-system package)
>> +                            (list gnu-build-system cmake-build-system))
>> +                      (package-with-upstream-version (pk 'up package))))
>> +               (leaf-packages))))
>
> I would be finer here.  For instance, the manifest contains the same
> package at several versions, i.e., ’package-with-upstream-version’
> applies to the same package at different version. Slurm is one example
> among many others:

Right, so there will be duplicates.  Is that what you mean?

(I think it’s okay to have duplicates at this stage, no big deal.)

Ludo’.

[bug#74542] [PATCH 00/11] Improved tooling for package updates

[Ludovic Courtès]

Hi,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> Maybe, we could be a bit more stringent on the list and filter out more
> packages than only the ones based on build-systems.

Yes, I agree.  I’m not too concerned about multi-versioned package
(clang, slurm, etc.): worst that can happen is that we’ll in fact build
just a single package, even though it appears several times in the
resulting manifest.

But things like PyPI, the kernel, etc., should probably be avoided.

Not sure what filtering predicate to use.  Thoughts?

Ludo’.

[bug#74542] [PATCH 00/11] Improved tooling for package updates

[Ludovic Courtès]

Suhail Singh <suhailsingh247@gmail.com> skribis:

> If the goal is to improve how up-to-date a distro Guix is, then the
> below would help.  In no particular order (the numbers are only to
> provide named references):

That’s the goal, and what you propose makes a lot of sense to me.  This
patch series is just first steps in that direction, I hope.

Ludo’.