[bug#74542] [PATCH 00/11] Improved tooling for package updates

[bug#74542] [PATCH 00/11] Improved tooling for package updates

[Ludovic Courtès]

Hello Guix!

This patch series aims to improve tooling for package upgrades:

  • By adding a ‘--dependents’ option to ‘guix build’ to build
    dependent packages (those returned by ‘guix refresh -l’).  This
    was suggested by my colleague Romain Garbage, who also prototyped
    a variant of this.

  • By improving individual updaters and updating metadata for
    some packages.

  • By adding an “upgrade manifest”, to automatically build the latest
    version (using the ‘with-latest’ package transformation) of a few
    security-critical packages along with their dependents at distance 2,
    as well as some of the leaf packages.

The goal is to use this upgrade manifest for a new jobset on ci.guix.
The idea is that when someone tries to upgrade one of these things,
chances are it will have been built already.

I already toyed with this idea here:

  https://guix.bordeaux.inria.fr/jobset/guix-hpc-cutting-edge
  https://gitlab.inria.fr/guix-hpc/guix-hpc/-/blob/master/cutting-edge.manifest?ref_type=heads

It’s experimental; surely we’ll find that the manifest needs to be
tweaked, or that some of the updaters abort to early.  But I think
it can help, and it’s a good way to use available x86_64 CPU power
at ci.guix.

According to a recent discussion with Nix hackers on Mastodon¹, mere
substitute availability is not perceived as a key factor in what
enables them to keep Nixpkgs up-to-date; the primary factor appears
to be the bot that sends patches updating packages.

Food for thought!

Ludo’.

¹ https://toot.aquilenet.fr/@civodul/113527656544893427

Ludovic Courtès (11):
  transformations: Export ‘package-with-upstream-version’.
  gnu-maintenance: ‘import-html-release’ doesn’t abort upon HTTP 404.
  gnu-maintenance: Savannah/Xorg updaters no longer abort on network
    errors.
  build: Add ‘--development’ option.
  packages: Factorize ‘all-packages’.
  guix build: Add ‘--dependents’.
  import: gnome: Keep going upon HTTP errors.
  gnu-maintenance: ‘gnu-ftp’ updater excludes GnuPG-hosted packages.
  gnu: Update updater properties for GnuPG-related packages.
  guix build: Validate that the file passed to ‘-m’ returns a manifest.
  etc: Add upgrade manifest.

 Makefile.am              |   1 +
 doc/contributing.texi    |   4 +-
 doc/guix.texi            |  52 +++++++++++
 etc/source-manifest.scm  |  13 +--
 etc/upgrade-manifest.scm |  98 +++++++++++++++++++++
 gnu/packages.scm         |  20 ++++-
 gnu/packages/gnupg.scm   |  40 +++------
 guix/download.scm        |   3 +-
 guix/gnu-maintenance.scm |  49 +++++++----
 guix/import/gnome.scm    |  13 ++-
 guix/scripts/build.scm   | 185 ++++++++++++++++++++++++++++++---------
 guix/scripts/graph.scm   |  10 ---
 guix/scripts/refresh.scm |  10 ---
 guix/scripts/weather.scm |  15 ----
 guix/transformations.scm |   2 +
 tests/guix-build.sh      |  11 +++
 16 files changed, 386 insertions(+), 140 deletions(-)
 create mode 100644 etc/upgrade-manifest.scm


base-commit: 827ec92f44d1b1daedc5446146801183560e861b
-- 
2.46.0

[bug#74542] [PATCH 01/11] transformations: Export ‘package-with-upstream-version’.

[Ludovic Courtès]

* guix/transformations.scm (package-with-upstream-version): Export.

Change-Id: I8e09e69e9c32bffc329c97ef1c0258f750181b11
---
 guix/transformations.scm | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/guix/transformations.scm b/guix/transformations.scm
index ea8b7a0844..9dfc4402c5 100644
--- a/guix/transformations.scm
+++ b/guix/transformations.scm
@@ -61,6 +61,8 @@ (define-module (guix transformations)
             tunable-package?
             tuned-package
 
+            package-with-upstream-version
+
             show-transformation-options-help
             transformation-option-key?
             cacheable-transformation-option-key?
-- 
2.46.0

[bug#74542] [PATCH 01/11] transformations: Export ‘package-with-upstream-version’.

[Simon Tournier]

On Tue, 26 Nov 2024 at 11:33, Ludovic Courtès <ludo@gnu.org> wrote:
> * guix/transformations.scm (package-with-upstream-version): Export.
>
> Change-Id: I8e09e69e9c32bffc329c97ef1c0258f750181b11
> ---
>  guix/transformations.scm | 2 ++
>  1 file changed, 2 insertions(+)

LGTM!  Good to start with the simplest. :-)

[bug#74542] [PATCH 02/11] gnu-maintenance: ‘import-html-release’ doesn’t abort upon HTTP 404.

[Ludovic Courtès]

Previously, updaters using ‘import-html-release’ would throw to 'quit
when the HTML page isn’t found, aborting the whole process:

  $ guix refresh coreutils libchop
  guix refresh: error: https://de.freedif.org/savannah//libchop/: HTTP download failed: 404 ("Not Found")

* guix/gnu-maintenance.scm (url->links): Guard against ‘http-get-error?’
and return the empty list when it is raised.

Change-Id: I419ff96ddb424aa8c74d6bab3d66a5e85355c3dc
---
 guix/gnu-maintenance.scm | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)

diff --git a/guix/gnu-maintenance.scm b/guix/gnu-maintenance.scm
index 881e941fbf..f34dcfc13f 100644
--- a/guix/gnu-maintenance.scm
+++ b/guix/gnu-maintenance.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2010-2023 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2010-2024 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2012, 2013 Nikita Karetnikov <nikita@karetnikov.org>
 ;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
 ;;; Copyright © 2022 Maxime Devos <maximedevos@telenet.be>
@@ -30,6 +30,7 @@ (define-module (guix gnu-maintenance)
   #:use-module (srfi srfi-2)
   #:use-module (srfi srfi-11)
   #:use-module (srfi srfi-26)
+  #:use-module (srfi srfi-34)
   #:use-module (rnrs io ports)
   #:use-module ((guix http-client) #:hide (open-socket-for-uri))
   ;; not required in many cases, so autoloaded to reduce start-up costs.
@@ -496,11 +497,16 @@ (define (html-links sxml)
 
 (define (url->links url)
   "Return the unique links on the HTML page accessible at URL."
-  (let* ((uri   (string->uri url))
-         (port  (http-fetch/cached uri #:ttl 3600))
-         (sxml  (html->sxml port)))
-    (close-port port)
-    (delete-duplicates (html-links sxml))))
+  (guard (c ((http-get-error? c)
+             (warning (G_ "failed to download '~a': ~a (~a)~%")
+                      url (http-get-error-code c)
+                      (http-get-error-reason c))
+             '()))
+    (let* ((uri   (string->uri url))
+           (port  (http-fetch/cached uri #:ttl 3600))
+           (sxml  (html->sxml port)))
+      (close-port port)
+      (delete-duplicates (html-links sxml)))))
 
 (define (canonicalize-url url base-url)
   "Make relative URL absolute, by appending URL to BASE-URL as required.  If
-- 
2.46.0

[bug#74542] [PATCH 02/11] gnu-maintenance: ‘import-html-release’ doesn’t abort upon HTTP 404.

[Simon Tournier]

Hi,

On Tue, 26 Nov 2024 at 11:33, Ludovic Courtès <ludo@gnu.org> wrote:

> +  #:use-module (srfi srfi-34)

[...]

> +  (guard (c ((http-get-error? c)
> +             (warning (G_ "failed to download '~a': ~a (~a)~%")
> +                      url (http-get-error-code c)
> +                      (http-get-error-reason c))
> +             '()))
> +    (let* ((uri   (string->uri url))
> +           (port  (http-fetch/cached uri #:ttl 3600))
> +           (sxml  (html->sxml port)))
> +      (close-port port)
> +      (delete-duplicates (html-links sxml)))))

I’m always confused by which framework for exception does the project
rely on?  There is old ’catch’ and ’throw’ then sometimes new
’with-exception-handler’ and ’raise’.  And also SRFI 34.

What’s the preferred convention?

Cheers,
simon

[bug#74542] [PATCH 02/11] gnu-maintenance: ‘import-html-release’ doesn’t abort upon HTTP 404.

[Ludovic Courtès]

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> On Tue, 26 Nov 2024 at 11:33, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> +  #:use-module (srfi srfi-34)
>
> [...]
>
>> +  (guard (c ((http-get-error? c)
>> +             (warning (G_ "failed to download '~a': ~a (~a)~%")
>> +                      url (http-get-error-code c)
>> +                      (http-get-error-reason c))
>> +             '()))
>> +    (let* ((uri   (string->uri url))
>> +           (port  (http-fetch/cached uri #:ttl 3600))
>> +           (sxml  (html->sxml port)))
>> +      (close-port port)
>> +      (delete-duplicates (html-links sxml)))))
>
> I’m always confused by which framework for exception does the project
> rely on?  There is old ’catch’ and ’throw’ then sometimes new
> ’with-exception-handler’ and ’raise’.  And also SRFI 34.
>
> What’s the preferred convention?

My take: SRFI-34.  Before 3.0 we had to use ‘catch’ for key+args
exceptions, which is why it’s still used a lot.

Hopefully it’ll get more homogeneous over time.  :-)

Ludo’.

[bug#74542] [PATCH 02/11] gnu-maintenance: ‘import-html-release’ doesn’t abort upon HTTP 404.

[Simon Tournier]

Hi Ludo,

On Tue, 26 Nov 2024 at 18:16, Ludovic Courtès <ludo@gnu.org> wrote:

> My take: SRFI-34.  Before 3.0 we had to use ‘catch’ for key+args
> exceptions, which is why it’s still used a lot.

Thanks for explaining.

> Hopefully it’ll get more homogeneous over time.  :-)

I recently introduced catch/throw to stay uniform with the rest of the
file.  From my point of view, it makes easier to read weeks or months
later.  That’s said, someone™ will homogenize the files, one day…
eventually. :-)

Cheers,
simon

[bug#74542] [PATCH 03/11] gnu-maintenance: Savannah/Xorg updaters no longer abort on network errors.

[Ludovic Courtès]

Previously these updaters would fail upon networking error, causing the
whole process to abort:

  $ guix refresh coreutils libgcrypt
  guix refresh: error: connect*: Connection refused

* guix/gnu-maintenance.scm (import-release*, import-xorg-release): Wrap
in ‘false-if-networking-error’.

Change-Id: I8d309bf8662e2d46460585b4a1b7da0a4f2e8907
---
 guix/gnu-maintenance.scm | 25 ++++++++++++++-----------
 1 file changed, 14 insertions(+), 11 deletions(-)

diff --git a/guix/gnu-maintenance.scm b/guix/gnu-maintenance.scm
index f34dcfc13f..866b960a4a 100644
--- a/guix/gnu-maintenance.scm
+++ b/guix/gnu-maintenance.scm
@@ -469,10 +469,12 @@ (define* (import-release* package #:key (version #f))
 \"emacs-auctex\", for instance.)"
   (let-values (((server directory)
                 (ftp-server/directory package)))
-    (false-if-ftp-error (import-release (package-upstream-name package)
-                                        #:version version
-                                        #:server server
-                                        #:directory directory))))
+    (false-if-networking-error
+     (false-if-ftp-error
+      (import-release (package-upstream-name package)
+                      #:version version
+                      #:server server
+                      #:directory directory)))))
 
 \f
 ;;;
@@ -913,13 +915,14 @@ (define* (import-xorg-release package #:key (version #f))
   "Return the latest release of PACKAGE.  Optionally include a VERSION string
 to fetch a specific version."
   (let ((uri (string->uri (origin-uri (package-source package)))))
-    (false-if-ftp-error
-     (import-ftp-release
-      (package-name package)
-      #:version version
-      #:server "ftp.freedesktop.org"
-      #:directory
-      (string-append "/pub/xorg/" (dirname (uri-path uri)))))))
+    (false-if-networking-error
+     (false-if-ftp-error
+      (import-ftp-release
+       (package-name package)
+       #:version version
+       #:server "ftp.freedesktop.org"
+       #:directory
+       (string-append "/pub/xorg/" (dirname (uri-path uri))))))))
 
 (define* (import-kernel.org-release package #:key (version #f))
   "Return the latest release of PACKAGE, a Linux kernel package.
-- 
2.46.0

[bug#74542] [PATCH 03/11] gnu-maintenance: Savannah/Xorg updaters no longer abort on network errors.

[Simon Tournier]

Hi,

On Tue, 26 Nov 2024 at 11:33, Ludovic Courtès <ludo@gnu.org> wrote:

>  guix/gnu-maintenance.scm | 25 ++++++++++++++-----------
>  1 file changed, 14 insertions(+), 11 deletions(-)

LGTM.

Cheers,
simon

[bug#74542] [PATCH 04/11] build: Add ‘--development’ option.

[Ludovic Courtès]

* guix/scripts/build.scm (show-help, %options): Add ‘-D’.
(options->things-to-build): Change ‘append-map’ to a loop.  Honor ‘-D’.
* tests/guix-build.sh: Add test.
* doc/guix.texi (Additional Build Options): Document it.

Change-Id: I99227aadfe861e43c001a4872292bd687b37f5d4
---
 doc/guix.texi          |  25 +++++++++
 guix/scripts/build.scm | 119 ++++++++++++++++++++++++++++-------------
 tests/guix-build.sh    |   5 ++
 3 files changed, 112 insertions(+), 37 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 26488b41c8..a9d0d044ae 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -13614,6 +13614,31 @@ Additional Build Options
 (@pxref{The Store Monad}).  The procedure must return a derivation as a
 monadic value, which is then passed through @code{run-with-store}.
 
+@item --development
+@itemx -D
+Build the ``development environment'' (build dependencies) of the
+following package.
+
+For example, the following command builds the inputs of @code{hello},
+but @emph{not} @code{hello} itself, and also builds @code{guile}:
+
+@example
+guix build -D hello guile
+@end example
+
+Notice that @option{-D} (or @option{--development}) only applies to the
+immediately following package on the command line.  Under the hood, it
+uses @code{package->development-manifest}
+(@pxref{package-development-manifest,
+@code{package->development-manifest}}).
+
+@quotation Note
+The effect of combining @option{--development} with @option{--target}
+(for cross-compilation) may not be what you expect: it will
+cross-compile all the dependencies of the given package when it is built
+natively.
+@end quotation
+
 @item --source
 @itemx -S
 Build the source derivations of the packages, rather than the packages
diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm
index 9e05bb532b..326d04f1f8 100644
--- a/guix/scripts/build.scm
+++ b/guix/scripts/build.scm
@@ -438,6 +438,8 @@ (define (show-help)
   -m, --manifest=FILE    build the packages that the manifest given in FILE
                          evaluates to"))
   (display (G_ "
+  -D, --development      build the inputs of the following package"))
+  (display (G_ "
   -S, --source           build the packages' source derivations"))
   (display (G_ "
       --sources[=TYPE]   build source derivations; TYPE may optionally be one
@@ -522,6 +524,9 @@ (define %options
          (option '(#\m "manifest") #t #f
                  (lambda (opt name arg result)
                    (alist-cons 'manifest arg result)))
+         (option '(#\D "development") #f #f
+                 (lambda (opt name arg result)
+                   (alist-cons 'development? #t result)))
          (option '(#\n "dry-run") #f #f
                  (lambda (opt name arg result)
                    (alist-cons 'dry-run? #t result)))
@@ -581,43 +586,83 @@ (define-public my-package
       (for-each validate-type lst)
       lst))
 
-  (append-map (match-lambda
-                (('argument . (? string? spec))
-                 (cond ((derivation-path? spec)
-                        (catch 'system-error
-                          (lambda ()
-                            ;; Ask for absolute file names so that .drv file
-                            ;; names passed from the user to 'read-derivation'
-                            ;; are absolute when it returns.
-                            (let ((spec (canonicalize-path spec)))
-                              (list (read-derivation-from-file spec))))
-                          (lambda args
-                            ;; Non-existent .drv files can be substituted down
-                            ;; the road, so don't error out.
-                            (if (= ENOENT (system-error-errno args))
-                                '()
-                                (apply throw args)))))
-                       ((store-path? spec)
-                        ;; Nothing to do; maybe for --log-file.
-                        '())
-                       (else
-                        (list (specification->package spec)))))
-                (('file . file)
-                 (let ((file (or (and (string-suffix? ".json" file)
-                                      (json->scheme-file file))
-                                 file)))
-                   (ensure-list (load* file (make-user-module '())))))
-                (('manifest . manifest)
-                 (map manifest-entry-item
-                      (manifest-entries
-                       (load* manifest
-                              (make-user-module '((guix profiles) (gnu)))))))
-                (('expression . str)
-                 (ensure-list (read/eval str)))
-                (('argument . (? derivation? drv))
-                 drv)
-                (_ '()))
-              opts))
+  (define system
+    (or (assoc-ref opts 'system) (%current-system)))
+
+  ;; Process OPTS in "the right order", meaning that if the user typed
+  ;; "-D hello", arrange to see the 'development? option before the "hello"
+  ;; spec.
+  (let loop ((opts (reverse opts))
+             (type 'regular)
+             (result '()))
+    (define (for-type obj)
+      ;; Return a list of objects corresponding to OBJ adjusted for TYPE.
+      (match type
+        ('regular
+         (list obj))
+        ('development
+         (if (package? obj)
+             (map manifest-entry-item
+                  (manifest-entries
+                   (package->development-manifest obj system)))
+             obj))))
+
+    (match opts
+      (()
+       (reverse result))
+      ((head . tail)
+       (match head
+         (('argument . (? string? spec))
+          (cond ((derivation-path? spec)
+                 (catch 'system-error
+                   (lambda ()
+                     ;; Ask for absolute file names so that .drv file
+                     ;; names passed from the user to 'read-derivation'
+                     ;; are absolute when it returns.
+                     (let ((spec (canonicalize-path spec)))
+                       (loop tail 'regular
+                             (cons (read-derivation-from-file spec)
+                                   result))))
+                   (lambda args
+                     ;; Non-existent .drv files can be substituted down
+                     ;; the road, so don't error out.
+                     (if (= ENOENT (system-error-errno args))
+                         (loop tail 'regular result)
+                         (apply throw args)))))
+                ((store-path? spec)
+                 ;; Nothing to do; maybe for --log-file.
+                 (loop tail type result))
+                (else
+                 (loop tail 'regular
+                       (append (for-type (specification->package spec))
+                               result)))))
+         (('file . file)
+          (let ((file (or (and (string-suffix? ".json" file)
+                               (json->scheme-file file))
+                          file)))
+            (loop tail 'regular
+                  (append (append-map
+                           for-type
+                           (ensure-list (load* file (make-user-module '()))))
+                          result))))
+         (('manifest . manifest)
+          (loop tail 'regular
+                (append (map manifest-entry-item
+                             (manifest-entries
+                              (load* manifest
+                                     (make-user-module '((guix profiles)
+                                                         (gnu))))))
+                        result)))
+         (('expression . str)
+          (loop tail 'regular
+                (append (append-map for-type (ensure-list (read/eval str)))
+                        result)))
+         (('argument . (? derivation? drv))
+          (loop tail 'regular (cons drv result)))
+         (('development? . #t)
+          (loop tail 'development result))
+         (_
+          (loop tail type result)))))))
 
 (define (options->derivations store opts)
   "Given OPTS, the result of 'args-fold', return a list of derivations to
diff --git a/tests/guix-build.sh b/tests/guix-build.sh
index 36eac2b7e0..3637bcdeb3 100644
--- a/tests/guix-build.sh
+++ b/tests/guix-build.sh
@@ -190,6 +190,11 @@ test `guix build -d --sources=transitive foo \
       | grep -e 'foo\.tar\.gz' -e 'bar\.tar\.gz' -e 'bar\.dat' \
       | wc -l` -eq 3
 
+# Building the inputs.
+guix build -D hello -n
+test `guix build -D hello -d \
+      | grep -e 'glibc.*\.drv$' -e 'gcc.*\.drv$' -e 'binutils.*\.drv$' \
+      | wc -l` -ge 3
 
 # Unbound variable in thunked field.
 cat > "$module_dir/foo.scm" <<EOF
-- 
2.46.0

[bug#74542] [PATCH 04/11] build: Add ‘--development’ option.

[Simon Tournier]

Hi,

Here the oneline commit message is:

     build: Add ‘--development’ option.

when elsewhere in the same series, it reads:

     guix build: Add ‘--dependents’.

I suggest consistency; ’build:’ or ’guix build:’ for both.

On Tue, 26 Nov 2024 at 11:33, Ludovic Courtès <ludo@gnu.org> wrote:

> +  ;; Process OPTS in "the right order", meaning that if the user typed
> +  ;; "-D hello", arrange to see the 'development? option before the "hello"
> +  ;; spec.
> +  (let loop ((opts (reverse opts))
> +             (type 'regular)
> +             (result '()))
> +    (define (for-type obj)
> +      ;; Return a list of objects corresponding to OBJ adjusted for TYPE.
> +      (match type
> +        ('regular
> +         (list obj))
> +        ('development
> +         (if (package? obj)
> +             (map manifest-entry-item
> +                  (manifest-entries
> +                   (package->development-manifest obj system)))
> +             obj))))

Well, the loop seems overcomplicated but nothing better to propose. ;-)

> +    (match opts
> +      (()
> +       (reverse result))
> +      ((head . tail)
> +       (match head
> +         (('argument . (? string? spec))
> +          (cond ((derivation-path? spec)
> +                 (catch 'system-error
> +                   (lambda ()
> +                     ;; Ask for absolute file names so that .drv file
> +                     ;; names passed from the user to 'read-derivation'
> +                     ;; are absolute when it returns.
> +                     (let ((spec (canonicalize-path spec)))
> +                       (loop tail 'regular
> +                             (cons (read-derivation-from-file spec)
> +                                   result))))
> +                   (lambda args
> +                     ;; Non-existent .drv files can be substituted down
> +                     ;; the road, so don't error out.
> +                     (if (= ENOENT (system-error-errno args))
> +                         (loop tail 'regular result)
> +                         (apply throw args)))))
> +                ((store-path? spec)
> +                 ;; Nothing to do; maybe for --log-file.
> +                 (loop tail type result))
> +                (else
> +                 (loop tail 'regular
> +                       (append (for-type (specification->package spec))
> +                               result)))))
> +         (('file . file)
> +          (let ((file (or (and (string-suffix? ".json" file)
> +                               (json->scheme-file file))
> +                          file)))
> +            (loop tail 'regular
> +                  (append (append-map
> +                           for-type
> +                           (ensure-list (load* file (make-user-module '()))))
> +                          result))))
> +         (('manifest . manifest)
> +          (loop tail 'regular
> +                (append (map manifest-entry-item
> +                             (manifest-entries
> +                              (load* manifest
> +                                     (make-user-module '((guix profiles)
> +                                                         (gnu))))))
> +                        result)))
> +         (('expression . str)
> +          (loop tail 'regular
> +                (append (append-map for-type (ensure-list (read/eval str)))
> +                        result)))
> +         (('argument . (? derivation? drv))
> +          (loop tail 'regular (cons drv result)))

Since you are at it, why not move this pattern close to the other
pattern matching ’argument’ too?


Cheers,
simon

[bug#74542] [PATCH 04/11] build: Add ‘--development’ option.

[Ludovic Courtès]

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> Here the oneline commit message is:
>
>      build: Add ‘--development’ option.
>
> when elsewhere in the same series, it reads:
>
>      guix build: Add ‘--dependents’.
>
> I suggest consistency; ’build:’ or ’guix build:’ for both.

Oops, fixed (will send v2 later).

[bug#74542] [PATCH 05/11] packages: Factorize ‘all-packages’.

[Ludovic Courtès]

* gnu/packages.scm (all-packages): New procedure.
* etc/source-manifest.scm (all-packages): Remove.
* guix/scripts/graph.scm (all-packages): Remove.
* guix/scripts/refresh.scm (all-packages): Remove.
* guix/scripts/weather.scm (all-packages): Remove.

Change-Id: I6072952c4b877b541037ce86402cfb7744eeb0a0
---
 etc/source-manifest.scm  | 13 +------------
 gnu/packages.scm         | 20 +++++++++++++++++++-
 guix/scripts/graph.scm   | 10 ----------
 guix/scripts/refresh.scm | 10 ----------
 guix/scripts/weather.scm | 15 ---------------
 5 files changed, 20 insertions(+), 48 deletions(-)

diff --git a/etc/source-manifest.scm b/etc/source-manifest.scm
index f96a5da6f7..3e1ae07959 100644
--- a/etc/source-manifest.scm
+++ b/etc/source-manifest.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2021 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2021, 2024 Ludovic Courtès <ludo@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -25,17 +25,6 @@
              (guix packages) (guix profiles)
              (gnu packages))
 
-(define (all-packages)
-  "Return the list of all the packages, public or private, omitting only
-superseded packages."
-  (fold-packages (lambda (package lst)
-                   (match (package-replacement package)
-                     (#f (cons package lst))
-                     (replacement
-                      (append (list replacement package) lst))))
-                 '()
-                 #:select? (negate package-superseded)))
-
 (define (upstream-origin source)
   "Return SOURCE without any patches or snippet."
   (origin (inherit source)
diff --git a/gnu/packages.scm b/gnu/packages.scm
index 80c22d1d7f..1af3b8d440 100644
--- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2012-2020, 2022-2023 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2012-2020, 2022-2024 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2013 Mark H Weaver <mhw@netris.org>
 ;;; Copyright © 2014 Eric Bavier <bavier@member.fsf.org>
 ;;; Copyright © 2016, 2017 Alex Kost <alezost@gmail.com>
@@ -56,6 +56,7 @@ (define-module (gnu packages)
             cache-is-authoritative?
 
             fold-packages
+            all-packages
             fold-available-packages
 
             find-newest-available-packages
@@ -253,6 +254,23 @@ (define* (fold-packages proc init
                                 init
                                 modules))
 
+(define all-packages
+  (mlambda ()
+    "Return the list of all public packages, including replacements and hidden
+packages, excluding superseded packages."
+    (delete-duplicates
+     (fold-packages (lambda (package result)
+                      (match (package-replacement package)
+                        ((? package? replacement)
+                         (cons* replacement package result))
+                        (#f
+                         (cons package result))))
+                    '()
+
+                    ;; Dismiss deprecated packages but keep hidden packages.
+                    #:select? (negate package-superseded))
+     eq?)))
+
 (define %package-cache-file
   ;; Location of the package cache.
   "/lib/guix/package.cache")
diff --git a/guix/scripts/graph.scm b/guix/scripts/graph.scm
index 6740858d8b..935721edea 100644
--- a/guix/scripts/graph.scm
+++ b/guix/scripts/graph.scm
@@ -119,16 +119,6 @@ (define %package-node-type
 ;;; Reverse package DAG.
 ;;;
 
-(define (all-packages)            ;XXX: duplicated from (guix scripts refresh)
-  "Return the list of all the distro's packages."
-  (fold-packages (lambda (package result)
-                   ;; Ignore deprecated packages.
-                   (if (package-superseded package)
-                       result
-                       (cons package result)))
-                 '()
-                 #:select? (const #t)))           ;include hidden packages
-
 (define %reverse-package-node-type
   ;; For this node type we first need to compute the list of packages and the
   ;; list of back-edges.  Since we want to do it only once, we use the
diff --git a/guix/scripts/refresh.scm b/guix/scripts/refresh.scm
index ec7d38c22a..8c72d0c545 100644
--- a/guix/scripts/refresh.scm
+++ b/guix/scripts/refresh.scm
@@ -455,16 +455,6 @@ (define* (check-for-package-update update-spec updaters #:key warn?)
 ;;; Dependents.
 ;;;
 
-(define (all-packages)
-  "Return the list of all the distro's packages."
-  (fold-packages (lambda (package result)
-                   ;; Ignore deprecated packages.
-                   (if (package-superseded package)
-                       result
-                       (cons package result)))
-                 '()
-                 #:select? (const #t)))           ;include hidden packages
-
 (define (list-dependents packages)
   "List all the things that would need to be rebuilt if PACKAGES are changed."
   ;; Using %BAG-NODE-TYPE is more accurate than using %PACKAGE-NODE-TYPE
diff --git a/guix/scripts/weather.scm b/guix/scripts/weather.scm
index 08a1b22a74..29432fd923 100644
--- a/guix/scripts/weather.scm
+++ b/guix/scripts/weather.scm
@@ -55,21 +55,6 @@ (define-module (guix scripts weather)
   #:use-module (ice-9 vlist)
   #:export (guix-weather))
 
-(define (all-packages)
-  "Return the list of public packages we are going to query."
-  (delete-duplicates
-   (fold-packages (lambda (package result)
-                    (match (package-replacement package)
-                      ((? package? replacement)
-                       (cons* replacement package result))
-                      (#f
-                       (cons package result))))
-                  '()
-
-                  ;; Dismiss deprecated packages but keep hidden packages.
-                  #:select? (negate package-superseded))
-   eq?))
-
 (define (call-with-progress-reporter reporter proc)
   "This is a variant of 'call-with-progress-reporter' that works with monadic
 scope."
-- 
2.46.0

[bug#74542] [PATCH 05/11] packages: Factorize ‘all-packages’.

[Simon Tournier]

Hi,

On Tue, 26 Nov 2024 at 11:33, Ludovic Courtès <ludo@gnu.org> wrote:

> Change-Id: I6072952c4b877b541037ce86402cfb7744eeb0a0
> ---
>  etc/source-manifest.scm  | 13 +------------
>  gnu/packages.scm         | 20 +++++++++++++++++++-
>  guix/scripts/graph.scm   | 10 ----------
>  guix/scripts/refresh.scm | 10 ----------
>  guix/scripts/weather.scm | 15 ---------------
>  5 files changed, 20 insertions(+), 48 deletions(-)

LGTM.

Cheers,
simon

[bug#74542] [PATCH 06/11] guix build: Add ‘--dependents’.

[Ludovic Courtès]

* guix/scripts/build.scm (show-help, %options): Add ‘--dependents’.
(dependents): New procedure.
(options->things-to-build): Add ‘store’ parameter; honor ‘dependents’
option.
[for-type]: Handle ‘dependents’ type.
(options->derivations): Update call to ‘options->things-to-build’.
* tests/guix-build.sh: Add test.
* doc/guix.texi (Additional Build Options): Document ‘--dependents’.
(Invoking guix refresh): Cross-reference it.
* doc/contributing.texi (Submitting Patches): Mention it.

Change-Id: I00b6d5831e1f1d35dc8b84a82605391d5a8f417c
---
 doc/contributing.texi  |  4 ++-
 doc/guix.texi          | 27 ++++++++++++++++++++
 guix/scripts/build.scm | 56 ++++++++++++++++++++++++++++++++++++++++--
 tests/guix-build.sh    |  6 +++++
 4 files changed, 90 insertions(+), 3 deletions(-)

diff --git a/doc/contributing.texi b/doc/contributing.texi
index b063169189..5a778466d7 100644
--- a/doc/contributing.texi
+++ b/doc/contributing.texi
@@ -1914,7 +1914,9 @@ Submitting Patches
 
 @item
 Make sure the package builds on your platform, using @command{guix build
-@var{package}}.
+@var{package}}.  Also build at least its direct dependents with
+@command{guix build --dependents=1 @var{package}}
+(@pxref{build-dependents, @command{guix build}}).
 
 @item
 We recommend you also try building the package on other supported
diff --git a/doc/guix.texi b/doc/guix.texi
index a9d0d044ae..5734cf306b 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -13639,6 +13639,31 @@ Additional Build Options
 natively.
 @end quotation
 
+@cindex dependents of a package, building them
+@cindex building the dependents of a package
+@anchor{build-dependents}
+@item --dependents[=@var{depth}]
+@itemx -T [@var{depth}]
+Build the dependents of the following package.  By default, build all
+the direct and indirect dependents; when @var{depth} is provided, limit
+to dependents at that distance: 1 for direct dependents, 2 for
+dependents of dependents, and so on.
+
+For example, the command below builds @emph{all} the dependents of libgit2:
+
+@example
+guix build --dependents libgit2
+@end example
+
+To build all the packages that directly depend on NumPy, run:
+
+@example
+guix build -T1 python-numpy
+@end example
+
+The list of dependents is computed in the same way as with @command{guix
+refresh --list-dependent} (@pxref{Invoking guix refresh}).
+
 @item --source
 @itemx -S
 Build the source derivations of the packages, rather than the packages
@@ -15142,6 +15167,8 @@ Invoking guix refresh
 @command{guix graph}}, for information on how to visualize the list of
 dependents of a package.
 
+@xref{build-dependents, @command{guix build --dependents}}, for a
+convenient way to build all the dependents of a package.
 @end table
 
 Be aware that the @option{--list-dependent} option only
diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm
index 326d04f1f8..565bfd48e9 100644
--- a/guix/scripts/build.scm
+++ b/guix/scripts/build.scm
@@ -33,6 +33,9 @@ (define-module (guix scripts build)
   #:use-module (guix profiles)
   #:use-module (guix diagnostics)
   #:autoload   (guix http-client) (http-fetch http-get-error?)
+  #:autoload   (guix scripts graph) (%bag-node-type)
+  #:autoload   (guix graph) (node-back-edges)
+  #:autoload   (guix sets) (setq set-contains? set-insert)
   #:use-module (ice-9 format)
   #:use-module (ice-9 match)
   #:use-module (srfi srfi-1)
@@ -440,6 +443,9 @@ (define (show-help)
   (display (G_ "
   -D, --development      build the inputs of the following package"))
   (display (G_ "
+  -T, --dependents[=N]   build dependents of the following package, up to
+                         depth N"))
+  (display (G_ "
   -S, --source           build the packages' source derivations"))
   (display (G_ "
       --sources[=TYPE]   build source derivations; TYPE may optionally be one
@@ -527,6 +533,11 @@ (define %options
          (option '(#\D "development") #f #f
                  (lambda (opt name arg result)
                    (alist-cons 'development? #t result)))
+         (option '(#\T "dependents") #f #t
+                 (lambda (opt name arg result)
+                   (alist-cons 'dependents
+                               (or (and=> arg string->number*) +inf.0)
+                               result)))
          (option '(#\n "dry-run") #f #f
                  (lambda (opt name arg result)
                    (alist-cons 'dry-run? #t result)))
@@ -551,7 +562,39 @@ (define %options
                  %standard-cross-build-options
                  %standard-native-build-options)))
 
-(define (options->things-to-build opts)
+(define (dependents store packages max-depth)
+  "List all the things that would need to be rebuilt if PACKAGES are changed."
+  ;; Using %BAG-NODE-TYPE is more accurate than using %PACKAGE-NODE-TYPE
+  ;; because it includes implicit dependencies.
+  (define (get-dependents packages edges)
+    (let loop ((packages packages)
+               (result '())
+               (depth 0)
+               (visited (setq)))
+      (if (> depth max-depth)
+          (values result visited)
+          (match packages
+            (()
+             (values result visited))
+            ((head . tail)
+             (if (set-contains? visited head)
+                 (loop tail result depth visited)
+                 (let ((next (edges head)))
+                   (call-with-values
+                       (lambda ()
+                         (loop next
+                               (cons head result)
+                               (+ depth 1)
+                               (set-insert head visited)))
+                     (lambda (result visited)
+                       (loop tail result depth visited))))))))))
+
+  (with-store store
+    (run-with-store store
+      (mlet %store-monad ((edges (node-back-edges %bag-node-type (all-packages))))
+        (return (get-dependents packages edges))))))
+
+(define (options->things-to-build store opts)
   "Read the arguments from OPTS and return a list of high-level objects to
 build---packages, gexps, derivations, and so on."
   (define (validate-type x)
@@ -600,6 +643,13 @@ (define-public my-package
       (match type
         ('regular
          (list obj))
+        (('dependents . depth)
+         (if (package? obj)
+             (begin
+               (info (G_ "computing dependents of package ~a...~%")
+                     (package-full-name obj))
+               (dependents store (list obj) depth))
+             (list obj)))
         ('development
          (if (package? obj)
              (map manifest-entry-item
@@ -661,6 +711,8 @@ (define-public my-package
           (loop tail 'regular (cons drv result)))
          (('development? . #t)
           (loop tail 'development result))
+         (('dependents . depth)
+          (loop tail `(dependents . ,depth) result))
          (_
           (loop tail type result)))))))
 
@@ -687,7 +739,7 @@ (define (options->derivations store opts)
       (systems systems)))
 
   (define things-to-build
-    (map transform (options->things-to-build opts)))
+    (map transform (options->things-to-build store opts)))
 
   (define warn-if-unsupported
     (let ((target (assoc-ref opts 'target)))
diff --git a/tests/guix-build.sh b/tests/guix-build.sh
index 3637bcdeb3..42e2ecafb1 100644
--- a/tests/guix-build.sh
+++ b/tests/guix-build.sh
@@ -196,6 +196,12 @@ test `guix build -D hello -d \
       | grep -e 'glibc.*\.drv$' -e 'gcc.*\.drv$' -e 'binutils.*\.drv$' \
       | wc -l` -ge 3
 
+# Building the dependents.
+test `guix build -T1 libgit2 -T1 libssh -d \
+      | grep -e 'guile-git.*\.drv$' -e 'guile-ssh.*\.drv$' \
+             -e 'libgit2.*\.drv$' -e 'libssh.*\.drv$' \
+      | wc -l` -eq 4
+
 # Unbound variable in thunked field.
 cat > "$module_dir/foo.scm" <<EOF
 (define-module (foo)
-- 
2.46.0

[bug#74542] [PATCH 06/11] guix build: Add ‘--dependents’.

[Simon Tournier]

Hi,

On Tue, 26 Nov 2024 at 11:33, Ludovic Courtès <ludo@gnu.org> wrote:

> +@item --dependents[=@var{depth}]
> +@itemx -T [@var{depth}]

Cosmetic: argh, ’-M’ is already taken under “guix build”! Because it
would be nice to have something similar as with “guix graph
--max-depth”.  Yes, both depths does not have the same meaning but
that’s still a ’max depth’ for both cases. ;-)

Well…

> +The list of dependents is computed in the same way as with @command{guix
> +refresh --list-dependent} (@pxref{Invoking guix refresh}).

… “guix refresh” already uses ’-T’ but for another meaning
(--list-transitive).  Therefore, I think something other than ’-T’
should be better.  Maybe ’-D’ since it cannot conflict with “guix
shell”..

> +  -T, --dependents[=N]   build dependents of the following package, up to
> +                         depth N"))

As said, ’-T’ might be confusing with “guix refresh -T” especially when
’dependents’ is more or less connected with “guix refresh -l”.

> +         (option '(#\T "dependents") #f #t
> +                 (lambda (opt name arg result)
> +                   (alist-cons 'dependents
> +                               (or (and=> arg string->number*) +inf.0)
> +                               result)))

Idem.

> +(define (options->things-to-build store opts)

[...]

> -    (map transform (options->things-to-build opts)))
> +    (map transform (options->things-to-build store opts)))

I thought ’store’ could have been hidden, but not.  Since it’s already
passed to ’options->derivations’. :-)


Well, aside the cosmetic, LGTM.

Cheers,
simon

[bug#74542] [PATCH 06/11] guix build: Add ‘--dependents’.

[Ludovic Courtès]

Hi,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

>> +@item --dependents[=@var{depth}]
>> +@itemx -T [@var{depth}]
>
> Cosmetic: argh, ’-M’ is already taken under “guix build”! Because it
> would be nice to have something similar as with “guix graph
> --max-depth”.  Yes, both depths does not have the same meaning but
> that’s still a ’max depth’ for both cases. ;-)
>
> Well…
>
>> +The list of dependents is computed in the same way as with @command{guix
>> +refresh --list-dependent} (@pxref{Invoking guix refresh}).
>
> … “guix refresh” already uses ’-T’ but for another meaning
> (--list-transitive).  Therefore, I think something other than ’-T’
> should be better.  Maybe ’-D’ since it cannot conflict with “guix
> shell”..

The previous patch adds ‘-D’ with the same meaning as in ‘guix shell’ so
it’s not available either.

I looked for other options and found ‘-P’, which is unused in ‘guix
build’ and ‘guix refresh’; it’s an alias for ‘--link-profile’ in ‘guix
shell’, an option that wouldn’t make sense in ‘guix build’.

Thoughts?

Ludo’.

[bug#74542] [PATCH 07/11] import: gnome: Keep going upon HTTP errors.

[Ludovic Courtès]

* guix/import/gnome.scm (import-gnome-release): Do not re-raise
condition C, emit a warning instead.

Change-Id: I9a7c337a17d1f7ed4677398e7b0f383befca7d74
---
 guix/import/gnome.scm | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/guix/import/gnome.scm b/guix/import/gnome.scm
index 054ae44f7a..3ba8ae02e5 100644
--- a/guix/import/gnome.scm
+++ b/guix/import/gnome.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2017, 2019, 2021 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2017, 2019, 2021, 2024 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2022 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;; Copyright © 2022 Hartmut Goebel <h.goebel@crazy-compilers.com>
 ;;;
@@ -23,6 +23,8 @@ (define-module (guix import gnome)
   #:use-module (guix utils)
   #:use-module (guix packages)
   #:use-module (guix http-client)
+  #:use-module (guix diagnostics)
+  #:use-module (guix i18n)
   #:use-module (json)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-34)
@@ -111,9 +113,12 @@ (define* (import-gnome-release package #:key (version #f))
           releases))
 
   (guard (c ((http-get-error? c)
-             (if (= 404 (http-get-error-code c))
-                 #f
-                 (raise c))))
+             (unless (= 404 (http-get-error-code c))
+               (warning (G_ "failed to download from '~a': ~a (~s)~%")
+                        (uri->string (http-get-error-uri c))
+                        (http-get-error-code c)
+                        (http-get-error-reason c)))
+             #f))
     (let* ((port (http-fetch/cached
                   (string->uri (string-append
                                 "https://ftp.gnome.org/pub/gnome/sources/"
-- 
2.46.0

[bug#74542] [PATCH 07/11] import: gnome: Keep going upon HTTP errors.

[Simon Tournier]

Hi,

On Tue, 26 Nov 2024 at 11:33, Ludovic Courtès <ludo@gnu.org> wrote:
> * guix/import/gnome.scm (import-gnome-release): Do not re-raise
> condition C, emit a warning instead.
>
> Change-Id: I9a7c337a17d1f7ed4677398e7b0f383befca7d74
> ---
>  guix/import/gnome.scm | 13 +++++++++----
>  1 file changed, 9 insertions(+), 4 deletions(-)

LGTM.

Cheers,
simon

[bug#74542] [PATCH 08/11] gnu-maintenance: ‘gnu-ftp’ updater excludes GnuPG-hosted packages.

[Ludovic Courtès]

Since ftp://ftp.gnupg.org no longer exists, another updater should be
used for packages with code fetched from mirror://gnupg.

* guix/gnu-maintenance.scm (gnupg-hosted?): New procedure.
(%gnu-ftp-updater): Exclude packages that match ‘gnupg-hosted?’.
* guix/download.scm (%mirrors): Remove ftp://ftp.gnupg.org.

Change-Id: I653877ad843da60281d3a234acc85743bee64856
---
 guix/download.scm        | 3 +--
 guix/gnu-maintenance.scm | 6 +++++-
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/guix/download.scm b/guix/download.scm
index e37c2195ee..d88ad0ee44 100644
--- a/guix/download.scm
+++ b/guix/download.scm
@@ -78,8 +78,7 @@ (define %mirrors
        "ftp://mirrors.dotsrc.org/gcrypt/"
        "ftp://ftp.heanet.ie/mirrors/ftp.gnupg.org/gcrypt/"
        "ftp://ftp.mirrorservice.org/sites/ftp.gnupg.org/gcrypt/"
-       "ftp://ftp.ring.gr.jp/pub/net/gnupg/"
-       "ftp://ftp.gnupg.org/gcrypt/")
+       "ftp://ftp.ring.gr.jp/pub/net/gnupg/")
       (gnome
        "https://download.gnome.org/"
        "http://ftp.gnome.org/pub/GNOME/")
diff --git a/guix/gnu-maintenance.scm b/guix/gnu-maintenance.scm
index 866b960a4a..b612b11c00 100644
--- a/guix/gnu-maintenance.scm
+++ b/guix/gnu-maintenance.scm
@@ -1025,15 +1025,19 @@ (define %gnu-updater
            (false-if-networking-error (gnu-hosted? package))))
    (import import-gnu-release)))
 
+(define gnupg-hosted?
+  (url-prefix-predicate "mirror://gnupg/"))
+
 (define %gnu-ftp-updater
   ;; This is for GNU packages taken from alternate locations, such as
-  ;; alpha.gnu.org, ftp.gnupg.org, etc.  It is obsolescent.
+  ;; alpha.gnu.org (ftp.gnupg.org is no longer available).  It is obsolescent.
   (upstream-updater
    (name 'gnu-ftp)
    (description "Updater for GNU packages only available via FTP")
    (pred (lambda (package)
            (false-if-networking-error
             (and (not (gnu-hosted? package))
+                 (not (gnupg-hosted? package))
                  (pure-gnu-package? package)))))
    (import import-release*)))
 
-- 
2.46.0

[bug#74542] [PATCH 08/11] gnu-maintenance: ‘gnu-ftp’ updater excludes GnuPG-hosted packages.

[Simon Tournier]

Hi,

On Tue, 26 Nov 2024 at 11:33, Ludovic Courtès <ludo@gnu.org> wrote:
> Since ftp://ftp.gnupg.org no longer exists, another updater should be
> used for packages with code fetched from mirror://gnupg.
>
> * guix/gnu-maintenance.scm (gnupg-hosted?): New procedure.
> (%gnu-ftp-updater): Exclude packages that match ‘gnupg-hosted?’.
> * guix/download.scm (%mirrors): Remove ftp://ftp.gnupg.org.
>
> Change-Id: I653877ad843da60281d3a234acc85743bee64856
> ---
>  guix/download.scm        | 3 +--
>  guix/gnu-maintenance.scm | 6 +++++-
>  2 files changed, 6 insertions(+), 3 deletions(-)

LGTM.

Cheers,
simon

[bug#74542] [PATCH 09/11] gnu: Update updater properties for GnuPG-related packages.

[Ludovic Courtès]

* gnu/packages/gnupg.scm (libgpg-error)[properties]: Remove.
(libgcrypt)[properties]: Remove.
(libassuan)[properties]: Remove.
(libksba)[properties]: Remove.
(npth)[properties]: Remove.
(gnupg)[properties]: Remove.
(gpgme)[properties]: Remove.
(pinentry-tty)[properties]: Change for ‘generic-html’.
(gpa)[properties]: Remove.

Change-Id: I1eb9d6ad5bb3ecbfbef3d44dbae281a3de7ebac7
---
 gnu/packages/gnupg.scm | 40 ++++++++++++----------------------------
 1 file changed, 12 insertions(+), 28 deletions(-)

diff --git a/gnu/packages/gnupg.scm b/gnu/packages/gnupg.scm
index 2e97c2244a..3f55ccaf5b 100644
--- a/gnu/packages/gnupg.scm
+++ b/gnu/packages/gnupg.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2012-2021, 2023 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2012-2021, 2023, 2024 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2013, 2015, 2018 Andreas Enge <andreas@enge.fr>
 ;;; Copyright © 2014, 2018 Eric Bavier <bavier@member.fsf.org>
 ;;; Copyright © 2014, 2015, 2016, 2020 Mark H Weaver <mhw@netris.org>
@@ -160,9 +160,7 @@ (define-public libgpg-error
 for all GnuPG components.  Among these are GPG, GPGSM, GPGME,
 GPG-Agent, libgcrypt, Libksba, DirMngr, Pinentry, SmartCard
 Daemon and possibly more in the future.")
-    (license license:lgpl2.0+)
-    (properties '((ftp-server . "ftp.gnupg.org")
-                  (ftp-directory . "/gcrypt/libgpg-error")))))
+    (license license:lgpl2.0+)))
 
 (define-public libgcrypt
   (package
@@ -217,9 +215,7 @@ (define-public libgcrypt
 standard cryptographic building blocks such as symmetric ciphers, hash
 algorithms, public key algorithms, large integer functions and random number
 generation.")
-    (license license:lgpl2.0+)
-    (properties '((ftp-server . "ftp.gnupg.org")
-                  (ftp-directory . "/gcrypt/libgcrypt")))))
+    (license license:lgpl2.0+)))
 
 (define-public libassuan
   (package
@@ -250,9 +246,7 @@ (define-public libassuan
 protocol.  This protocol is used for IPC between most newer
 GnuPG components.  Both, server and client side functions are
 provided.")
-    (license license:lgpl2.0+)
-    (properties '((ftp-server . "ftp.gnupg.org")
-                  (ftp-directory . "/gcrypt/libassuan")))))
+    (license license:lgpl2.0+)))
 
 (define-public libksba
   (package
@@ -283,9 +277,7 @@ (define-public libksba
      "KSBA (pronounced Kasbah) is a library to make X.509 certificates
 as well as the CMS easily accessible by other applications.  Both
 specifications are building blocks of S/MIME and TLS.")
-    (license license:gpl3+)
-    (properties '((ftp-server . "ftp.gnupg.org")
-                  (ftp-directory . "/gcrypt/libksba")))))
+    (license license:gpl3+)))
 
 (define-public npth
   (package
@@ -307,9 +299,7 @@ (define-public npth
 In contrast to GNU Pth is is based on the system's standard threads
 implementation.  This allows the use of libraries which are not
 compatible to GNU Pth.")
-    (license (list license:lgpl3+ license:gpl2+)) ; dual license
-    (properties '((ftp-server . "ftp.gnupg.org")
-                  (ftp-directory . "/gcrypt/npth")))))
+    (license (list license:lgpl3+ license:gpl2+)))) ;dual license
 
 (define-public gnupg
   (package
@@ -404,9 +394,7 @@ (define-public gnupg
 servers.  It includes several libraries: libassuan (IPC between GnuPG
 components), libgpg-error (centralized GnuPG error values), and
 libskba (working with X.509 certificates and CMS data).")
-    (license license:gpl3+)
-    (properties '((ftp-server . "ftp.gnupg.org")
-                  (ftp-directory . "/gcrypt/gnupg")))))
+    (license license:gpl3+)))
 
 (define-public gnupg-1
   (package (inherit gnupg)
@@ -463,9 +451,7 @@ (define-public gpgme
 programming task, it is suggested that all software should try to use GPGME
 instead.  This way bug fixes or improvements can be done at a central place
 and every application benefits from this.")
-    (license license:lgpl2.1+)
-    (properties '((ftp-server . "ftp.gnupg.org")
-                  (ftp-directory . "/gcrypt/gpgme")))))
+    (license license:lgpl2.1+)))
 
 (define-public gpgme-1.23
   (hidden-package
@@ -886,9 +872,9 @@ (define-public pinentry-tty
      "Pinentry provides a console that allows users to enter a passphrase when
 @code{gpg} is run and needs it.")
     (license license:gpl2+)
-    (properties '((ftp-server . "ftp.gnupg.org")
-                  (ftp-directory . "/gcrypt/pinentry")
-                  (upstream-name . "pinentry")))))
+    (properties
+     '((release-monitoring-url . "https://gnupg.org/ftp/gcrypt/pinentry/")
+       (upstream-name . "pinentry")))))
 
 (define-public pinentry-emacs
   (package
@@ -1134,9 +1120,7 @@ (define-public gpa
      "GPA, the GNU Privacy Assistant, is a graphical user interface for
 @uref{https://gnupg.org, GnuPG}.  It can be used to encrypt, decrypt, and sign
 files, to verify signatures, and to manage the private and public keys.")
-    (license license:gpl3+)
-    (properties '((ftp-server . "ftp.gnupg.org")
-                  (ftp-directory . "/gcrypt/gpa")))))
+    (license license:gpl3+)))
 
 (define-public parcimonie
   (package
-- 
2.46.0

[bug#74542] [PATCH 09/11] gnu: Update updater properties for GnuPG-related packages.

[Simon Tournier]

Hi,

On Tue, 26 Nov 2024 at 11:33, Ludovic Courtès <ludo@gnu.org> wrote:
> * gnu/packages/gnupg.scm (libgpg-error)[properties]: Remove.
> (libgcrypt)[properties]: Remove.
> (libassuan)[properties]: Remove.
> (libksba)[properties]: Remove.
> (npth)[properties]: Remove.
> (gnupg)[properties]: Remove.
> (gpgme)[properties]: Remove.
> (pinentry-tty)[properties]: Change for ‘generic-html’.
> (gpa)[properties]: Remove.
>
> Change-Id: I1eb9d6ad5bb3ecbfbef3d44dbae281a3de7ebac7
> ---
>  gnu/packages/gnupg.scm | 40 ++++++++++++----------------------------
>  1 file changed, 12 insertions(+), 28 deletions(-)

LGTM.

Cool!  Nice clean up!

Cheers,
simon

[bug#74542] [PATCH 10/11] guix build: Validate that the file passed to ‘-m’ returns a manifest.

[Ludovic Courtès]

* guix/scripts/build.scm (options->things-to-build)[ensure-manifest]:
New procedure.
Use it.

Change-Id: If64c483d7079f441a296d5bd4e06e67f44cbb7bf
---
 guix/scripts/build.scm | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm
index 565bfd48e9..1b0b006ad5 100644
--- a/guix/scripts/build.scm
+++ b/guix/scripts/build.scm
@@ -629,6 +629,12 @@ (define-public my-package
       (for-each validate-type lst)
       lst))
 
+  (define (ensure-manifest x file)
+    (unless (manifest? x)
+      (raise (formatted-message (G_ "file '~a' does not return a manifest")
+                                file)))
+    x)
+
   (define system
     (or (assoc-ref opts 'system) (%current-system)))
 
@@ -699,9 +705,11 @@ (define-public my-package
           (loop tail 'regular
                 (append (map manifest-entry-item
                              (manifest-entries
-                              (load* manifest
-                                     (make-user-module '((guix profiles)
-                                                         (gnu))))))
+                              (ensure-manifest
+                               (load* manifest
+                                      (make-user-module '((guix profiles)
+                                                          (gnu))))
+                               manifest)))
                         result)))
          (('expression . str)
           (loop tail 'regular
-- 
2.46.0

[bug#74542] [PATCH 10/11] guix build: Validate that the file passed to ‘-m’ returns a manifest.

[Simon Tournier]

Hi,

On Tue, 26 Nov 2024 at 11:33, Ludovic Courtès <ludo@gnu.org> wrote:
> * guix/scripts/build.scm (options->things-to-build)[ensure-manifest]:
> New procedure.
> Use it.
>
> Change-Id: If64c483d7079f441a296d5bd4e06e67f44cbb7bf
> ---
>  guix/scripts/build.scm | 14 +++++++++++---
>  1 file changed, 11 insertions(+), 3 deletions(-)

LGTM.

Cheers,
simon

[bug#74542] [PATCH 11/11] etc: Add upgrade manifest.

[Ludovic Courtès]

* guix/scripts/build.scm (dependents): Export.
* etc/upgrade-manifest.scm: New file.
* Makefile.am (EXTRA_DIST): Add it.

Change-Id: I1b2a2ebd09e559c68da9f25772bf33caacb4c031
---
 Makefile.am              |  1 +
 etc/upgrade-manifest.scm | 98 ++++++++++++++++++++++++++++++++++++++++
 guix/scripts/build.scm   |  2 +
 3 files changed, 101 insertions(+)
 create mode 100644 etc/upgrade-manifest.scm

diff --git a/Makefile.am b/Makefile.am
index e94ba87797..0cff32c607 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -743,6 +743,7 @@ EXTRA_DIST +=						\
   etc/source-manifest.scm				\
   etc/system-tests.scm					\
   etc/time-travel-manifest.scm				\
+  etc/upgrade-manifest.scm				\
   scripts/guix.in					\
   tests/cve-sample.json					\
   tests/keys/civodul.pub				\
diff --git a/etc/upgrade-manifest.scm b/etc/upgrade-manifest.scm
new file mode 100644
index 0000000000..6dd605ef03
--- /dev/null
+++ b/etc/upgrade-manifest.scm
@@ -0,0 +1,98 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2024 Ludovic Courtès <ludo@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+;; This manifest computes upgrades of key packages using the 'with-latest'
+;; package transformation.
+
+(use-modules (guix monads)
+             (guix graph)
+             (guix packages)
+             (guix profiles)
+             (guix store)
+             (guix transformations)
+             ((guix scripts build) #:select (dependents))
+             ((guix scripts graph) #:select (%bag-node-type))
+             ((guix import github) #:select (%github-api))
+             (guix build-system gnu)
+             (guix build-system cmake)
+             ((gnu packages) #:select (all-packages))
+             (ice-9 match)
+             (srfi srfi-1))
+
+;; Bypass the GitHub updater: we'd need an API token or we would hit the rate
+;; limit.
+(%github-api "http://example.org")
+
+(define (leaf-packages)
+  (with-store store
+    (run-with-store store
+      (mlet %store-monad ((edges (node-back-edges %bag-node-type (all-packages))))
+        (return (filter (lambda (package)
+                          (null? (edges package)))
+                        (all-packages)))))))
+
+(define security-packages
+  '("git" "git-minimal"
+    "xorg-server"
+    "elogind"
+    "openssl"
+    "gnutls"
+    "libarchive"
+    "libgit2"
+    "libssh"
+
+    ;; GnuPG.
+    "libassuan"
+    "libgpg-error"
+    "libgcrypt"
+    "libksba"
+    "npth"
+    "gnupg"
+    "gpgme"
+    "pinentry"))
+
+(define security-upgrades
+  ;; Upgrades of individual packages with their dependents built against that
+  ;; upgrade.
+  (manifest
+   (with-store store
+     (append-map (match-lambda
+                   ((package . output)
+                    (let* ((name (package-name package))
+                           (latest (options->transformation
+                                    `((with-latest . ,name)))))
+                      (map (lambda (package)
+                             (manifest-entry
+                               (inherit (package->manifest-entry
+                                         (latest (pk 'latest package))))
+                               (name (string-append (package-name package)
+                                                    "-with-latest-" name))))
+                           (dependents store (list package) 2)))))
+                 (specifications->packages security-packages)))))
+
+(define leaf-package-updates
+  ;; Select a subset (~22%) of all the leaf packages, typically small C/C++
+  ;; packages not part of a bigger "collection" or repo (CRAN, PyPI, etc.).
+  (manifest
+   (filter-map (lambda (package)
+                 (and (memq (package-build-system package)
+                            (list gnu-build-system cmake-build-system))
+                      (package-with-upstream-version (pk 'up package))))
+               (leaf-packages))))
+
+(concatenate-manifest (list leaf-package-updates security-upgrades))
diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm
index 1b0b006ad5..ddebcaf743 100644
--- a/guix/scripts/build.scm
+++ b/guix/scripts/build.scm
@@ -63,6 +63,8 @@ (define-module (guix scripts build)
             show-cross-build-options-help
             show-native-build-options-help
 
+            dependents
+
             guix-build
             register-root
             register-root*))
-- 
2.46.0

[bug#74542] [PATCH 11/11] etc: Add upgrade manifest.

[Simon Tournier]

Hi,

On Tue, 26 Nov 2024 at 11:33, Ludovic Courtès <ludo@gnu.org> wrote:

> +(define leaf-package-updates
> +  ;; Select a subset (~22%) of all the leaf packages, typically small C/C++
> +  ;; packages not part of a bigger "collection" or repo (CRAN, PyPI, etc.).
> +  (manifest
> +   (filter-map (lambda (package)
> +                 (and (memq (package-build-system package)
> +                            (list gnu-build-system cmake-build-system))
> +                      (package-with-upstream-version (pk 'up package))))
> +               (leaf-packages))))

I would be finer here.  For instance, the manifest contains the same
package at several versions, i.e., ’package-with-upstream-version’
applies to the same package at different version. Slurm is one example
among many others:

--8<---------------cut here---------------start------------->8---
scheme@(guix-user)> ,pp (sort
                         (map
                          (lambda (p) `(,(package-name p) . ,(package-version p)))
                          (filter-map
                            (lambda (package)
                               (and (memq (package-build-system package)
                                           (list gnu-build-system cmake-build-system))
                                       package))
                                  (leaf-packages)))
                         (lambda (p1 p2)
                           (match p1 ((n1 . _) (match p2 ((n2 . _) (string< n1 n2)))))))
[...]

 ("slurm" . "20.11.9")
 ("slurm" . "20.02.6-1")
 ("slurm" . "23.02.6")
 ("slurm" . "21.08.8")
 ("slurm" . "22.05.1")
 ("slurm" . "19.05.8")
 ("slurm" . "18.08.9")
 ("slurm-drmaa" . "1.1.3")
 ("slurm-minimal" . "20.02.6-1")
 ("slurm-minimal" . "22.05.1")
 ("slurm-minimal" . "23.02.6")
 ("slurm-minimal" . "21.08.8")
 ("slurm-minimal" . "18.08.9")
 ("slurm-minimal" . "20.11.9")
 ("slurm-minimal" . "23.11.10")
 ("slurm-minimal" . "19.05.8")

[...]
--8<---------------cut here---------------end--------------->8---

Cheers,
simon

[bug#74542] [PATCH 11/11] etc: Add upgrade manifest.

[Ludovic Courtès]

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> On Tue, 26 Nov 2024 at 11:33, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> +(define leaf-package-updates
>> +  ;; Select a subset (~22%) of all the leaf packages, typically small C/C++
>> +  ;; packages not part of a bigger "collection" or repo (CRAN, PyPI, etc.).
>> +  (manifest
>> +   (filter-map (lambda (package)
>> +                 (and (memq (package-build-system package)
>> +                            (list gnu-build-system cmake-build-system))
>> +                      (package-with-upstream-version (pk 'up package))))
>> +               (leaf-packages))))
>
> I would be finer here.  For instance, the manifest contains the same
> package at several versions, i.e., ’package-with-upstream-version’
> applies to the same package at different version. Slurm is one example
> among many others:

Right, so there will be duplicates.  Is that what you mean?

(I think it’s okay to have duplicates at this stage, no big deal.)

Ludo’.

[bug#74542] [PATCH 11/11] etc: Add upgrade manifest.

[Simon Tournier]

Hi,

On Tue, 26 Nov 2024 at 18:18, Ludovic Courtès <ludo@gnu.org> wrote:

>>> +(define leaf-package-updates
>>> +  ;; Select a subset (~22%) of all the leaf packages, typically small C/C++
>>> +  ;; packages not part of a bigger "collection" or repo (CRAN, PyPI, etc.).
>>> +  (manifest
>>> +   (filter-map (lambda (package)
>>> +                 (and (memq (package-build-system package)
>>> +                            (list gnu-build-system cmake-build-system))
>>> +                      (package-with-upstream-version (pk 'up package))))
>>> +               (leaf-packages))))
>>
>> I would be finer here.  For instance, the manifest contains the same
>> package at several versions, i.e., ’package-with-upstream-version’
>> applies to the same package at different version. Slurm is one example
>> among many others:
>
> Right, so there will be duplicates.  Is that what you mean?

Yes, duplicates.  And also Emacs or Python packages.  As discussed
elsewhere [1] in the thread, when looking more closely to the 2843
packages, the list could be shortened a bit.  In other words, I would be
fine here in the way to filter out. :-)

> (I think it’s okay to have duplicates at this stage, no big deal.)

It will trigger updaters for nothing because we know beforehand either
the updater will fail for whatever reasons, either the updater will
return the same package and thus the duplicates will be removed when
building the manifest.

Well, maybe it’s useless indeed because it saves nothing compared to
watching a online videos or else, but still it hurts me to waste
resource when it could be done a bit differently to avoid such waste.


Cheers,
simon


1: [bug#74542] [PATCH 00/11] Improved tooling for package updates
Simon Tournier <zimon.toutoune@gmail.com>
Tue, 26 Nov 2024 15:59:53 +0100
id:8734je3us6.fsf@gmail.com
https://issues.guix.gnu.org/74542
https://issues.guix.gnu.org/msgid/8734je3us6.fsf@gmail.com
https://yhetil.org/guix/8734je3us6.fsf@gmail.com

[bug#74542] [PATCH 00/11] Improved tooling for package updates

[Ludovic Courtès]

Ludovic Courtès <ludo@gnu.org> skribis:

> I already toyed with this idea here:
>
>   https://guix.bordeaux.inria.fr/jobset/guix-hpc-cutting-edge
>   https://gitlab.inria.fr/guix-hpc/guix-hpc/-/blob/master/cutting-edge.manifest?ref_type=heads
>
> It’s experimental; surely we’ll find that the manifest needs to be
> tweaked, or that some of the updaters abort to early.  But I think
> it can help, and it’s a good way to use available x86_64 CPU power
> at ci.guix.

Another point: having a single “upgrade manifest” probably won’t scale,
first because it takes time to evaluate it (looking for the latest
upstream version of all these packages), second because the jobset will
be messy if it contains tons of unrelated upgrades.

So I’m thinking that probably we’ll want focused upgrade manifests: one
for selected security-critical packages, one for the astronomy packages
so Sharlatan can save a bit on energy at home ;-), one for R packages,
and so on.

Another one I’d like to have: automatic ungrafting so that we could
apply ungrafting packages “anytime” (ideally!).

The sky’s the limit!

Ludo’.

[bug#74542] [PATCH 00/11] Improved tooling for package updates

[Simon Tournier]

Hi,

On Tue, 26 Nov 2024 at 15:42, Ludovic Courtès <ludo@gnu.org> wrote:

> Another point: having a single “upgrade manifest” probably won’t scale,
> first because it takes time to evaluate it (looking for the latest
> upstream version of all these packages), second because the jobset will
> be messy if it contains tons of unrelated upgrades.

Yes, that’d be my suggestion: having a better filtering strategy for
’leaf-packages’…

> So I’m thinking that probably we’ll want focused upgrade manifests: one
> for selected security-critical packages, one for the astronomy packages
> so Sharlatan can save a bit on energy at home ;-), one for R packages,
> and so on.

…so yes, it appears to me better to have security for one, then maybe
one per team?  Or per updater?

> Another one I’d like to have: automatic ungrafting so that we could
> apply ungrafting packages “anytime” (ideally!).

You read in my dreams! ;-)

> The sky’s the limit!

Let touch the void…

Cheers,
simon

[bug#74542] [PATCH 00/11] Improved tooling for package updates

[Simon Tournier]

Hi,

On Tue, 26 Nov 2024 at 11:32, Ludovic Courtès <ludo@gnu.org> wrote:

> This patch series aims to improve tooling for package upgrades:
>
>   • By adding a ‘--dependents’ option to ‘guix build’ to build
>     dependent packages (those returned by ‘guix refresh -l’).  This
>     was suggested by my colleague Romain Garbage, who also prototyped
>     a variant of this.
>
>   • By improving individual updaters and updating metadata for
>     some packages.

Cool!

>
>   • By adding an “upgrade manifest”, to automatically build the latest
>     version (using the ‘with-latest’ package transformation) of a few
>     security-critical packages along with their dependents at distance 2,
>     as well as some of the leaf packages.

I am more reserved here.

--8<---------------cut here---------------start------------->8---
scheme@(guix-user)> (define all-leafs (leaf-packages))
scheme@(guix-user)> (length all-leafs)
$2 = 13680
scheme@(guix-user)> (define leaf-to-updates
   (filter-map (lambda (package)
                 (and (memq (package-build-system package)
                            (list gnu-build-system cmake-build-system))
                      package))
               all-leafs))
scheme@(guix-user)> (length leaf-to-updates)
$3 = 2843
--8<---------------cut here---------------end--------------->8---

This list contains multiversion packages, for instance:

    "clang"
    "clang"
    "clang"
    "gccgo"
    "gccgo"
    "gccgo"
    "gccgo"
    "gccgo"
    "glibc"
    "glibc"
    "glibc"
    "glibc"
    "glibc"
    "glibc-locales"
    "glibc-locales-2.29"
    "glibc-locales-2.30"
    "glibc-locales-2.31"
    "glibc-locales-2.32"
    "glibc-locales-2.33"
    "glibc-locales-2.35"
    "linux-libre"
    "linux-libre"
    "linux-libre"
    "linux-libre"
    "linux-libre"
    "linux-libre-arm-generic"
    "linux-libre-arm-generic"
    "linux-libre-arm-generic"
    "linux-libre-arm-generic"
    "linux-libre-arm-omap2plus"
    "linux-libre-arm-omap2plus"
    "linux-libre-arm64-generic"
    "linux-libre-arm64-generic"
    "linux-libre-arm64-generic"
    "linux-libre-arm64-generic"
    "linux-libre-arm64-honeycomb"
    "linux-libre-documentation"
    "linux-libre-headers"
    "linux-libre-headers"
    "linux-libre-headers"
    "linux-libre-headers"
    "linux-libre-headers"
    "linux-libre-headers"
 

And this list also contains PyPI, for instance:

    "pyconfigure"
    "pypy"
    "python-cffi-documentation"
    "python-clingo-dl"
    "python-debug"
    "python-graph-tool"
    "python-libfreenect"
    "python-libmpsse"
    "python-libsvm"
    "python-next"
    "python-on-guile"
    "python-openturns"
    "python-sepolgen"
    "python2-dbus"
    "python2-minimal"

Maybe, we could be a bit more stringent on the list and filter out more
packages than only the ones based on build-systems.

Well, it could be helpful to avoid to burn resources for nothing.  For
example, it could be better to exclude the GitHub importer instead of
bypassing it with a wrong URL.

> Ludovic Courtès (11):
>   transformations: Export ‘package-with-upstream-version’.
>   gnu-maintenance: ‘import-html-release’ doesn’t abort upon HTTP 404.
>   gnu-maintenance: Savannah/Xorg updaters no longer abort on network
>     errors.
>   build: Add ‘--development’ option.
>   packages: Factorize ‘all-packages’.
>   guix build: Add ‘--dependents’.
>   import: gnome: Keep going upon HTTP errors.
>   gnu-maintenance: ‘gnu-ftp’ updater excludes GnuPG-hosted packages.
>   gnu: Update updater properties for GnuPG-related packages.
>   guix build: Validate that the file passed to ‘-m’ returns a manifest.
>   etc: Add upgrade manifest.

Let review patch by patch. :-)

Cheers,
simon

[bug#74542] [PATCH 00/11] Improved tooling for package updates

[Ludovic Courtès]

Hi,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> Maybe, we could be a bit more stringent on the list and filter out more
> packages than only the ones based on build-systems.

Yes, I agree.  I’m not too concerned about multi-versioned package
(clang, slurm, etc.): worst that can happen is that we’ll in fact build
just a single package, even though it appears several times in the
resulting manifest.

But things like PyPI, the kernel, etc., should probably be avoided.

Not sure what filtering predicate to use.  Thoughts?

Ludo’.

[bug#74542] [PATCH 00/11] Improved tooling for package updates

[Simon Tournier]

Hi,

On Tue, 26 Nov 2024 at 18:21, Ludovic Courtès <ludo@gnu.org> wrote:

> But things like PyPI, the kernel, etc., should probably be avoided.
>
> Not sure what filtering predicate to use.  Thoughts?

For now, I have nothing better to propose other than filtering on
package name.  For instance, exclude “emacs-”, “python-”, “r-” (there is
one ;-)), “ghc-”, “ocaml-” etc.  Maybe “linux-” too.

Cheers,
simon

[bug#74542] [PATCH 00/11] Improved tooling for package updates

[Suhail Singh]

Ludovic Courtès <ludo@gnu.org> writes:

> The goal is to use this upgrade manifest for a new jobset on ci.guix.
> The idea is that when someone tries to upgrade one of these things,
> chances are it will have been built already.
>
> ...
>
> According to a recent discussion with Nix hackers on Mastodon¹, mere
> substitute availability is not perceived as a key factor in what
> enables them to keep Nixpkgs up-to-date; the primary factor appears
> to be the bot that sends patches updating packages.
>
> Food for thought!
>
> Ludo’.
>
> ¹ https://toot.aquilenet.fr/@civodul/113527656544893427

If the goal is to improve how up-to-date a distro Guix is, then the
below would help.  In no particular order (the numbers are only to
provide named references):

1. Ensure a periodic process (let's say alice-bot) that runs on the CI
   server that submits patches for updates.
2. alice-bot could, in fact, be part of the "new jobset on ci.guix" that
   would be updating the substitutes.  If the substitute building is
   deemed successful it could result in the corresponding upgrade patch
   being submitted.
3. Having a trusted (i.e., trusted by guix authenticate), review bot
   (let's say bob-bot), whose job is to review alice-bot submissions and
   determine whether desirable invariants are being upheld.  If they
   are, bob-bot merges the patch.  If not, it requests human
   intervention.  It would help for bob-bot to add/remove usertags on
   the debbugs issue (and for those state-changes to be visible on
   <https://issues.guix.gnu.org/>) to ensure the attention of human
   reviewers is not wasted.

Having bob-bot be distinct from alice-bot would help keep the process
transparent and reduce bob-bot's code size (thus aiding auditability).
Additionally, it would also allow for bob-bot to be used to provide
automated lint/style checks for patch submissions by people.

The patch series in #74542 provides some necessary tooling to help keep
the distro up-to-date.  However, we still need alice-bot and bob-bot as
described above to get to sufficiency.

--
Suhail

[bug#74542] [PATCH 00/11] Improved tooling for package updates

[Ludovic Courtès]

Suhail Singh <suhailsingh247@gmail.com> skribis:

> If the goal is to improve how up-to-date a distro Guix is, then the
> below would help.  In no particular order (the numbers are only to
> provide named references):

That’s the goal, and what you propose makes a lot of sense to me.  This
patch series is just first steps in that direction, I hope.

Ludo’.

[bug#74542] [PATCH v2 00/16] Improved tooling for package updates

[Ludovic Courtès]

Hello!

This version aims to address comments by Simon and additional issues
I stumbled upon:

  • Short option for ‘--dependents’ is now ‘-P’ (instead of ‘-T’).

  • More package refresher metadata updates (GnuTLS, Git).

  • ‘generic-html’ updater correctly computes URLs for ‘curl’ and
    any other package that uses <base href="…"> in its release
    monitoring page.

  • ‘package-with-upstream-version’ can preserve patches.  This turns
    out to be a hard requirement if we want to make this useful.

    While ‘guix build openssl --with-latest=openssl’ dismisses patches,
    what we want here instead is to preserve patches, such that what’s
    being built is exactly the same as what we’d get with ‘guix
    refresh -u openssl’.

  • ‘package-with-upstream-version’ can disable authentication: it’s
    essentially required if we are to run this in a non-interactive
    way, given the fact that we cannot reliably fetch keys from OpenPGP
    key servers.

  • Manifest is now limited to a dozen security-critical packages (I
    left out leaf packages entirely; we can work on it separately
    later).

  • Manifest is more precise: it refers to the packages to update by
    their variable (rather than by package specs), which again makes
    sure we’re upgrading the exact variant we want to upgrade.

Thoughts?

I’d like to have the manifest built by CI soon so we can assess
its usefulness and tweak it as needed.

Ludo’.

Ludovic Courtès (16):
  transformations: Export ‘package-with-upstream-version’.
  gnu-maintenance: ‘import-html-release’ doesn’t abort upon HTTP 404.
  gnu-maintenance: Savannah/Xorg updaters no longer abort on network
    errors.
  guix build: Add ‘--development’ option.
  packages: Factorize ‘all-packages’.
  guix build: Add ‘--dependents’.
  import: gnome: Keep going upon HTTP errors.
  gnu-maintenance: ‘gnu-ftp’ updater excludes GnuPG-hosted packages.
  gnu: Update updater properties for GnuPG-related packages.
  gnu: gnutls: Change release monitoring URL.
  gnu: git-minimal: Add ‘upstream-name’ property.
  gnu-maintenance: ‘generic-html’ update honors <base href="…">.
  guix build: Validate that the file passed to ‘-m’ returns a manifest.
  transformations: ‘package-with-upstream-version’ can preserve patches.
  transformations: Add #:authenticate? to
    ‘package-with-upstream-version’.
  etc: Add upgrade manifest.

 Makefile.am                      |   1 +
 doc/contributing.texi            |   4 +-
 doc/guix.texi                    |  52 +++++++++
 etc/source-manifest.scm          |  13 +--
 etc/upgrade-manifest.scm         | 128 +++++++++++++++++++++
 gnu/packages.scm                 |  20 +++-
 gnu/packages/gnupg.scm           |  40 ++-----
 gnu/packages/tls.scm             |   6 +-
 gnu/packages/version-control.scm |   3 +-
 guix/download.scm                |   3 +-
 guix/gnu-maintenance.scm         |  90 +++++++++------
 guix/import/gnome.scm            |  13 ++-
 guix/scripts/build.scm           | 185 ++++++++++++++++++++++++-------
 guix/scripts/graph.scm           |  10 --
 guix/scripts/refresh.scm         |  10 --
 guix/scripts/weather.scm         |  15 ---
 guix/transformations.scm         |  43 ++++++-
 tests/guix-build.sh              |  11 ++
 tests/transformations.scm        |  31 +++++-
 19 files changed, 517 insertions(+), 161 deletions(-)
 create mode 100644 etc/upgrade-manifest.scm


base-commit: f8979b4bcc8772d02640f6f665b4195380d57df9
-- 
2.46.0

[bug#74542] [PATCH v2 00/16] Improved tooling for package updates

[Maxim Cournoyer]

Hi Ludovic!

Ludovic Courtès <ludo@gnu.org> writes:


[...]

>     While ‘guix build openssl --with-latest=openssl’ dismisses patches,
>     what we want here instead is to preserve patches, such that what’s
>     being built is exactly the same as what we’d get with ‘guix
>     refresh -u openssl’.

Could 'guix build openssl --with-latest' also have an option to preserve
patches?  This has often tripped me.

-- 
Thanks,
Maxim

[bug#74542] [PATCH v2 00/16] Improved tooling for package updates

[Suhail Singh]

Ludovic Courtès <ludo@gnu.org> writes:

>   • Short option for ‘--dependents’ is now ‘-P’ (instead of ‘-T’).

Thank you to both Simon and yourself for prioritizing some kind of
consistency between the various guix sub-commands.

>   • ‘package-with-upstream-version’ can preserve patches.  This turns
>     out to be a hard requirement if we want to make this useful.

I agree that this is essential.  At times, updates, can break existing
patches.  I am assuming it would be easy enough to spot such build
failures?  I.e., I am assuming that we will be preserving sufficient
details from the build log so core developers may be informed.

>   • Manifest is now limited to a dozen security-critical packages (I
>     left out leaf packages entirely; we can work on it separately
>     later).

Prioritizing the security-critical packages makes sense.  Especially
since we lack an automated "integration" QA system at the moment,
leaving the leaf packages for later is prudent.  The learnings from
trying this on the security-critical packages may also help inform the
approach we take for the other packages.

>   • Manifest is more precise: it refers to the packages to update by
>     their variable (rather than by package specs), which again makes
>     sure we’re upgrading the exact variant we want to upgrade.

I think this is an essential change as well.

> Thoughts?

Thank you for devoting time to this.  We need to improve our automation.

> I’d like to have the manifest built by CI soon so we can assess
> its usefulness and tweak it as needed.

Could we please add some minimal code to submit-patch-if-build-succeeds
to this series?  That would provide a working example of a CI job that
interacts with debbugs/sends-emails and would allow for greater eyeballs
on the working state of this jobset.

-- 
Suhail

[bug#74542] [PATCH v2 00/16] Improved tooling for package updates

[Simon Tournier]

Hi Ludo,

On Fri, 29 Nov 2024 at 10:40, Ludovic Courtès <ludo@gnu.org> wrote:

>   • ‘generic-html’ updater correctly computes URLs for ‘curl’ and
>     any other package that uses <base href="…"> in its release
>     monitoring page.

Oh! :-)

>   • Manifest is now limited to a dozen security-critical packages (I
>     left out leaf packages entirely; we can work on it separately
>     later).

I agree.

> I’d like to have the manifest built by CI soon so we can assess
> its usefulness and tweak it as needed.

I agree.


> Ludovic Courtès (16):
>   transformations: Export ‘package-with-upstream-version’.
>   gnu-maintenance: ‘import-html-release’ doesn’t abort upon HTTP 404.
>   gnu-maintenance: Savannah/Xorg updaters no longer abort on network
>     errors.
>   guix build: Add ‘--development’ option.
>   packages: Factorize ‘all-packages’.
>   guix build: Add ‘--dependents’.
>   import: gnome: Keep going upon HTTP errors.
>   gnu-maintenance: ‘gnu-ftp’ updater excludes GnuPG-hosted packages.
>   gnu: Update updater properties for GnuPG-related packages.
>   gnu: gnutls: Change release monitoring URL.
>   gnu: git-minimal: Add ‘upstream-name’ property.
>   gnu-maintenance: ‘generic-html’ update honors <base href="…">.
>   guix build: Validate that the file passed to ‘-m’ returns a manifest.
>   transformations: ‘package-with-upstream-version’ can preserve patches.
>   transformations: Add #:authenticate? to
>     ‘package-with-upstream-version’.
>   etc: Add upgrade manifest.
>
>  Makefile.am                      |   1 +
>  doc/contributing.texi            |   4 +-
>  doc/guix.texi                    |  52 +++++++++
>  etc/source-manifest.scm          |  13 +--
>  etc/upgrade-manifest.scm         | 128 +++++++++++++++++++++
>  gnu/packages.scm                 |  20 +++-
>  gnu/packages/gnupg.scm           |  40 ++-----
>  gnu/packages/tls.scm             |   6 +-
>  gnu/packages/version-control.scm |   3 +-
>  guix/download.scm                |   3 +-
>  guix/gnu-maintenance.scm         |  90 +++++++++------
>  guix/import/gnome.scm            |  13 ++-
>  guix/scripts/build.scm           | 185 ++++++++++++++++++++++++-------
>  guix/scripts/graph.scm           |  10 --
>  guix/scripts/refresh.scm         |  10 --
>  guix/scripts/weather.scm         |  15 ---
>  guix/transformations.scm         |  43 ++++++-
>  tests/guix-build.sh              |  11 ++
>  tests/transformations.scm        |  31 +++++-
>  19 files changed, 517 insertions(+), 161 deletions(-)
>  create mode 100644 etc/upgrade-manifest.scm

Well, I have not given another look to all this v2 patchset.  But from I
have checked, it’s LGTM.

Cheers,
simon

[bug#74542] [PATCH v2 01/16] transformations: Export ‘package-with-upstream-version’.

[Ludovic Courtès]

* guix/transformations.scm (package-with-upstream-version): Export.

Change-Id: I8e09e69e9c32bffc329c97ef1c0258f750181b11
---
 guix/transformations.scm | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/guix/transformations.scm b/guix/transformations.scm
index ea8b7a0844..9dfc4402c5 100644
--- a/guix/transformations.scm
+++ b/guix/transformations.scm
@@ -61,6 +61,8 @@ (define-module (guix transformations)
             tunable-package?
             tuned-package
 
+            package-with-upstream-version
+
             show-transformation-options-help
             transformation-option-key?
             cacheable-transformation-option-key?
-- 
2.46.0

[bug#74542] [PATCH v2 02/16] gnu-maintenance: ‘import-html-release’ doesn’t abort upon HTTP 404.

[Ludovic Courtès]

Previously, updaters using ‘import-html-release’ would throw to 'quit
when the HTML page isn’t found, aborting the whole process:

  $ guix refresh coreutils libchop
  guix refresh: error: https://de.freedif.org/savannah//libchop/: HTTP download failed: 404 ("Not Found")

* guix/gnu-maintenance.scm (url->links): Guard against ‘http-get-error?’
and return the empty list when it is raised.

Change-Id: I419ff96ddb424aa8c74d6bab3d66a5e85355c3dc
---
 guix/gnu-maintenance.scm | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)

diff --git a/guix/gnu-maintenance.scm b/guix/gnu-maintenance.scm
index 881e941fbf..f34dcfc13f 100644
--- a/guix/gnu-maintenance.scm
+++ b/guix/gnu-maintenance.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2010-2023 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2010-2024 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2012, 2013 Nikita Karetnikov <nikita@karetnikov.org>
 ;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
 ;;; Copyright © 2022 Maxime Devos <maximedevos@telenet.be>
@@ -30,6 +30,7 @@ (define-module (guix gnu-maintenance)
   #:use-module (srfi srfi-2)
   #:use-module (srfi srfi-11)
   #:use-module (srfi srfi-26)
+  #:use-module (srfi srfi-34)
   #:use-module (rnrs io ports)
   #:use-module ((guix http-client) #:hide (open-socket-for-uri))
   ;; not required in many cases, so autoloaded to reduce start-up costs.
@@ -496,11 +497,16 @@ (define (html-links sxml)
 
 (define (url->links url)
   "Return the unique links on the HTML page accessible at URL."
-  (let* ((uri   (string->uri url))
-         (port  (http-fetch/cached uri #:ttl 3600))
-         (sxml  (html->sxml port)))
-    (close-port port)
-    (delete-duplicates (html-links sxml))))
+  (guard (c ((http-get-error? c)
+             (warning (G_ "failed to download '~a': ~a (~a)~%")
+                      url (http-get-error-code c)
+                      (http-get-error-reason c))
+             '()))
+    (let* ((uri   (string->uri url))
+           (port  (http-fetch/cached uri #:ttl 3600))
+           (sxml  (html->sxml port)))
+      (close-port port)
+      (delete-duplicates (html-links sxml)))))
 
 (define (canonicalize-url url base-url)
   "Make relative URL absolute, by appending URL to BASE-URL as required.  If
-- 
2.46.0

[bug#74542] [PATCH v2 02/16] gnu-maintenance: ‘import-html-release’ doesn’t abort upon HTTP 404.

[Maxim Cournoyer]

Ludovic Courtès <ludo@gnu.org> writes:

> Previously, updaters using ‘import-html-release’ would throw to 'quit
> when the HTML page isn’t found, aborting the whole process:
>
>   $ guix refresh coreutils libchop
>   guix refresh: error: https://de.freedif.org/savannah//libchop/: HTTP download failed: 404 ("Not Found")
>
> * guix/gnu-maintenance.scm (url->links): Guard against ‘http-get-error?’
> and return the empty list when it is raised.
>
> Change-Id: I419ff96ddb424aa8c74d6bab3d66a5e85355c3dc

Reviewed-by: Maxim Cournoyer <maxim.cournoyer@gmail>

-- 
Thanks,
Maxim

[bug#74542] [PATCH v2 03/16] gnu-maintenance: Savannah/Xorg updaters no longer abort on network errors.

[Ludovic Courtès]

Previously these updaters would fail upon networking error, causing the
whole process to abort:

  $ guix refresh coreutils libgcrypt
  guix refresh: error: connect*: Connection refused

* guix/gnu-maintenance.scm (import-release*, import-xorg-release): Wrap
in ‘false-if-networking-error’.

Change-Id: I8d309bf8662e2d46460585b4a1b7da0a4f2e8907
---
 guix/gnu-maintenance.scm | 25 ++++++++++++++-----------
 1 file changed, 14 insertions(+), 11 deletions(-)

diff --git a/guix/gnu-maintenance.scm b/guix/gnu-maintenance.scm
index f34dcfc13f..866b960a4a 100644
--- a/guix/gnu-maintenance.scm
+++ b/guix/gnu-maintenance.scm
@@ -469,10 +469,12 @@ (define* (import-release* package #:key (version #f))
 \"emacs-auctex\", for instance.)"
   (let-values (((server directory)
                 (ftp-server/directory package)))
-    (false-if-ftp-error (import-release (package-upstream-name package)
-                                        #:version version
-                                        #:server server
-                                        #:directory directory))))
+    (false-if-networking-error
+     (false-if-ftp-error
+      (import-release (package-upstream-name package)
+                      #:version version
+                      #:server server
+                      #:directory directory)))))
 
 \f
 ;;;
@@ -913,13 +915,14 @@ (define* (import-xorg-release package #:key (version #f))
   "Return the latest release of PACKAGE.  Optionally include a VERSION string
 to fetch a specific version."
   (let ((uri (string->uri (origin-uri (package-source package)))))
-    (false-if-ftp-error
-     (import-ftp-release
-      (package-name package)
-      #:version version
-      #:server "ftp.freedesktop.org"
-      #:directory
-      (string-append "/pub/xorg/" (dirname (uri-path uri)))))))
+    (false-if-networking-error
+     (false-if-ftp-error
+      (import-ftp-release
+       (package-name package)
+       #:version version
+       #:server "ftp.freedesktop.org"
+       #:directory
+       (string-append "/pub/xorg/" (dirname (uri-path uri))))))))
 
 (define* (import-kernel.org-release package #:key (version #f))
   "Return the latest release of PACKAGE, a Linux kernel package.
-- 
2.46.0

[bug#74542] [PATCH v2 04/16] guix build: Add ‘--development’ option.

[Ludovic Courtès]

* guix/scripts/build.scm (show-help, %options): Add ‘-D’.
(options->things-to-build): Change ‘append-map’ to a loop.  Honor ‘-D’.
* tests/guix-build.sh: Add test.
* doc/guix.texi (Additional Build Options): Document it.

Change-Id: I99227aadfe861e43c001a4872292bd687b37f5d4
---
 doc/guix.texi          |  25 +++++++++
 guix/scripts/build.scm | 119 ++++++++++++++++++++++++++++-------------
 tests/guix-build.sh    |   5 ++
 3 files changed, 112 insertions(+), 37 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 26488b41c8..a9d0d044ae 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -13614,6 +13614,31 @@ Additional Build Options
 (@pxref{The Store Monad}).  The procedure must return a derivation as a
 monadic value, which is then passed through @code{run-with-store}.
 
+@item --development
+@itemx -D
+Build the ``development environment'' (build dependencies) of the
+following package.
+
+For example, the following command builds the inputs of @code{hello},
+but @emph{not} @code{hello} itself, and also builds @code{guile}:
+
+@example
+guix build -D hello guile
+@end example
+
+Notice that @option{-D} (or @option{--development}) only applies to the
+immediately following package on the command line.  Under the hood, it
+uses @code{package->development-manifest}
+(@pxref{package-development-manifest,
+@code{package->development-manifest}}).
+
+@quotation Note
+The effect of combining @option{--development} with @option{--target}
+(for cross-compilation) may not be what you expect: it will
+cross-compile all the dependencies of the given package when it is built
+natively.
+@end quotation
+
 @item --source
 @itemx -S
 Build the source derivations of the packages, rather than the packages
diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm
index 9e05bb532b..e8384c8d2d 100644
--- a/guix/scripts/build.scm
+++ b/guix/scripts/build.scm
@@ -438,6 +438,8 @@ (define (show-help)
   -m, --manifest=FILE    build the packages that the manifest given in FILE
                          evaluates to"))
   (display (G_ "
+  -D, --development      build the inputs of the following package"))
+  (display (G_ "
   -S, --source           build the packages' source derivations"))
   (display (G_ "
       --sources[=TYPE]   build source derivations; TYPE may optionally be one
@@ -522,6 +524,9 @@ (define %options
          (option '(#\m "manifest") #t #f
                  (lambda (opt name arg result)
                    (alist-cons 'manifest arg result)))
+         (option '(#\D "development") #f #f
+                 (lambda (opt name arg result)
+                   (alist-cons 'development? #t result)))
          (option '(#\n "dry-run") #f #f
                  (lambda (opt name arg result)
                    (alist-cons 'dry-run? #t result)))
@@ -581,43 +586,83 @@ (define-public my-package
       (for-each validate-type lst)
       lst))
 
-  (append-map (match-lambda
-                (('argument . (? string? spec))
-                 (cond ((derivation-path? spec)
-                        (catch 'system-error
-                          (lambda ()
-                            ;; Ask for absolute file names so that .drv file
-                            ;; names passed from the user to 'read-derivation'
-                            ;; are absolute when it returns.
-                            (let ((spec (canonicalize-path spec)))
-                              (list (read-derivation-from-file spec))))
-                          (lambda args
-                            ;; Non-existent .drv files can be substituted down
-                            ;; the road, so don't error out.
-                            (if (= ENOENT (system-error-errno args))
-                                '()
-                                (apply throw args)))))
-                       ((store-path? spec)
-                        ;; Nothing to do; maybe for --log-file.
-                        '())
-                       (else
-                        (list (specification->package spec)))))
-                (('file . file)
-                 (let ((file (or (and (string-suffix? ".json" file)
-                                      (json->scheme-file file))
-                                 file)))
-                   (ensure-list (load* file (make-user-module '())))))
-                (('manifest . manifest)
-                 (map manifest-entry-item
-                      (manifest-entries
-                       (load* manifest
-                              (make-user-module '((guix profiles) (gnu)))))))
-                (('expression . str)
-                 (ensure-list (read/eval str)))
-                (('argument . (? derivation? drv))
-                 drv)
-                (_ '()))
-              opts))
+  (define system
+    (or (assoc-ref opts 'system) (%current-system)))
+
+  ;; Process OPTS in "the right order", meaning that if the user typed
+  ;; "-D hello", arrange to see the 'development? option before the "hello"
+  ;; spec.
+  (let loop ((opts (reverse opts))
+             (type 'regular)
+             (result '()))
+    (define (for-type obj)
+      ;; Return a list of objects corresponding to OBJ adjusted for TYPE.
+      (match type
+        ('regular
+         (list obj))
+        ('development
+         (if (package? obj)
+             (map manifest-entry-item
+                  (manifest-entries
+                   (package->development-manifest obj system)))
+             obj))))
+
+    (match opts
+      (()
+       (reverse result))
+      ((head . tail)
+       (match head
+         (('argument . (? string? spec))
+          (cond ((derivation-path? spec)
+                 (catch 'system-error
+                   (lambda ()
+                     ;; Ask for absolute file names so that .drv file
+                     ;; names passed from the user to 'read-derivation'
+                     ;; are absolute when it returns.
+                     (let ((spec (canonicalize-path spec)))
+                       (loop tail 'regular
+                             (cons (read-derivation-from-file spec)
+                                   result))))
+                   (lambda args
+                     ;; Non-existent .drv files can be substituted down
+                     ;; the road, so don't error out.
+                     (if (= ENOENT (system-error-errno args))
+                         (loop tail 'regular result)
+                         (apply throw args)))))
+                ((store-path? spec)
+                 ;; Nothing to do; maybe for --log-file.
+                 (loop tail type result))
+                (else
+                 (loop tail 'regular
+                       (append (for-type (specification->package spec))
+                               result)))))
+         (('argument . (? derivation? drv))
+          (loop tail 'regular (cons drv result)))
+         (('file . file)
+          (let ((file (or (and (string-suffix? ".json" file)
+                               (json->scheme-file file))
+                          file)))
+            (loop tail 'regular
+                  (append (append-map
+                           for-type
+                           (ensure-list (load* file (make-user-module '()))))
+                          result))))
+         (('manifest . manifest)
+          (loop tail 'regular
+                (append (map manifest-entry-item
+                             (manifest-entries
+                              (load* manifest
+                                     (make-user-module '((guix profiles)
+                                                         (gnu))))))
+                        result)))
+         (('expression . str)
+          (loop tail 'regular
+                (append (append-map for-type (ensure-list (read/eval str)))
+                        result)))
+         (('development? . #t)
+          (loop tail 'development result))
+         (_
+          (loop tail type result)))))))
 
 (define (options->derivations store opts)
   "Given OPTS, the result of 'args-fold', return a list of derivations to
diff --git a/tests/guix-build.sh b/tests/guix-build.sh
index 36eac2b7e0..3637bcdeb3 100644
--- a/tests/guix-build.sh
+++ b/tests/guix-build.sh
@@ -190,6 +190,11 @@ test `guix build -d --sources=transitive foo \
       | grep -e 'foo\.tar\.gz' -e 'bar\.tar\.gz' -e 'bar\.dat' \
       | wc -l` -eq 3
 
+# Building the inputs.
+guix build -D hello -n
+test `guix build -D hello -d \
+      | grep -e 'glibc.*\.drv$' -e 'gcc.*\.drv$' -e 'binutils.*\.drv$' \
+      | wc -l` -ge 3
 
 # Unbound variable in thunked field.
 cat > "$module_dir/foo.scm" <<EOF
-- 
2.46.0

[bug#74542] [PATCH v2 04/16] guix build: Add ‘--development’ option.

[Maxim Cournoyer]

Ludovic Courtès <ludo@gnu.org> writes:

> * guix/scripts/build.scm (show-help, %options): Add ‘-D’.
> (options->things-to-build): Change ‘append-map’ to a loop.  Honor ‘-D’.
> * tests/guix-build.sh: Add test.
> * doc/guix.texi (Additional Build Options): Document it.
>
> Change-Id: I99227aadfe861e43c001a4872292bd687b37f5d4

Reviewed-by: Maxim Cournoyer <maxim.cournoyer@gmail>

-- 
Thanks,
Maxim

[bug#74542] [PATCH v2 05/16] packages: Factorize ‘all-packages’.

[Ludovic Courtès]

* gnu/packages.scm (all-packages): New procedure.
* etc/source-manifest.scm (all-packages): Remove.
* guix/scripts/graph.scm (all-packages): Remove.
* guix/scripts/refresh.scm (all-packages): Remove.
* guix/scripts/weather.scm (all-packages): Remove.

Change-Id: I6072952c4b877b541037ce86402cfb7744eeb0a0
---
 etc/source-manifest.scm  | 13 +------------
 gnu/packages.scm         | 20 +++++++++++++++++++-
 guix/scripts/graph.scm   | 10 ----------
 guix/scripts/refresh.scm | 10 ----------
 guix/scripts/weather.scm | 15 ---------------
 5 files changed, 20 insertions(+), 48 deletions(-)

diff --git a/etc/source-manifest.scm b/etc/source-manifest.scm
index f96a5da6f7..3e1ae07959 100644
--- a/etc/source-manifest.scm
+++ b/etc/source-manifest.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2021 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2021, 2024 Ludovic Courtès <ludo@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -25,17 +25,6 @@
              (guix packages) (guix profiles)
              (gnu packages))
 
-(define (all-packages)
-  "Return the list of all the packages, public or private, omitting only
-superseded packages."
-  (fold-packages (lambda (package lst)
-                   (match (package-replacement package)
-                     (#f (cons package lst))
-                     (replacement
-                      (append (list replacement package) lst))))
-                 '()
-                 #:select? (negate package-superseded)))
-
 (define (upstream-origin source)
   "Return SOURCE without any patches or snippet."
   (origin (inherit source)
diff --git a/gnu/packages.scm b/gnu/packages.scm
index 80c22d1d7f..1af3b8d440 100644
--- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2012-2020, 2022-2023 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2012-2020, 2022-2024 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2013 Mark H Weaver <mhw@netris.org>
 ;;; Copyright © 2014 Eric Bavier <bavier@member.fsf.org>
 ;;; Copyright © 2016, 2017 Alex Kost <alezost@gmail.com>
@@ -56,6 +56,7 @@ (define-module (gnu packages)
             cache-is-authoritative?
 
             fold-packages
+            all-packages
             fold-available-packages
 
             find-newest-available-packages
@@ -253,6 +254,23 @@ (define* (fold-packages proc init
                                 init
                                 modules))
 
+(define all-packages
+  (mlambda ()
+    "Return the list of all public packages, including replacements and hidden
+packages, excluding superseded packages."
+    (delete-duplicates
+     (fold-packages (lambda (package result)
+                      (match (package-replacement package)
+                        ((? package? replacement)
+                         (cons* replacement package result))
+                        (#f
+                         (cons package result))))
+                    '()
+
+                    ;; Dismiss deprecated packages but keep hidden packages.
+                    #:select? (negate package-superseded))
+     eq?)))
+
 (define %package-cache-file
   ;; Location of the package cache.
   "/lib/guix/package.cache")
diff --git a/guix/scripts/graph.scm b/guix/scripts/graph.scm
index 6740858d8b..935721edea 100644
--- a/guix/scripts/graph.scm
+++ b/guix/scripts/graph.scm
@@ -119,16 +119,6 @@ (define %package-node-type
 ;;; Reverse package DAG.
 ;;;
 
-(define (all-packages)            ;XXX: duplicated from (guix scripts refresh)
-  "Return the list of all the distro's packages."
-  (fold-packages (lambda (package result)
-                   ;; Ignore deprecated packages.
-                   (if (package-superseded package)
-                       result
-                       (cons package result)))
-                 '()
-                 #:select? (const #t)))           ;include hidden packages
-
 (define %reverse-package-node-type
   ;; For this node type we first need to compute the list of packages and the
   ;; list of back-edges.  Since we want to do it only once, we use the
diff --git a/guix/scripts/refresh.scm b/guix/scripts/refresh.scm
index ec7d38c22a..8c72d0c545 100644
--- a/guix/scripts/refresh.scm
+++ b/guix/scripts/refresh.scm
@@ -455,16 +455,6 @@ (define* (check-for-package-update update-spec updaters #:key warn?)
 ;;; Dependents.
 ;;;
 
-(define (all-packages)
-  "Return the list of all the distro's packages."
-  (fold-packages (lambda (package result)
-                   ;; Ignore deprecated packages.
-                   (if (package-superseded package)
-                       result
-                       (cons package result)))
-                 '()
-                 #:select? (const #t)))           ;include hidden packages
-
 (define (list-dependents packages)
   "List all the things that would need to be rebuilt if PACKAGES are changed."
   ;; Using %BAG-NODE-TYPE is more accurate than using %PACKAGE-NODE-TYPE
diff --git a/guix/scripts/weather.scm b/guix/scripts/weather.scm
index 08a1b22a74..29432fd923 100644
--- a/guix/scripts/weather.scm
+++ b/guix/scripts/weather.scm
@@ -55,21 +55,6 @@ (define-module (guix scripts weather)
   #:use-module (ice-9 vlist)
   #:export (guix-weather))
 
-(define (all-packages)
-  "Return the list of public packages we are going to query."
-  (delete-duplicates
-   (fold-packages (lambda (package result)
-                    (match (package-replacement package)
-                      ((? package? replacement)
-                       (cons* replacement package result))
-                      (#f
-                       (cons package result))))
-                  '()
-
-                  ;; Dismiss deprecated packages but keep hidden packages.
-                  #:select? (negate package-superseded))
-   eq?))
-
 (define (call-with-progress-reporter reporter proc)
   "This is a variant of 'call-with-progress-reporter' that works with monadic
 scope."
-- 
2.46.0

[bug#74542] [PATCH v2 05/16] packages: Factorize ‘all-packages’.

[Maxim Cournoyer]

Hi Ludovic,

Ludovic Courtès <ludo@gnu.org> writes:

> * gnu/packages.scm (all-packages): New procedure.
> * etc/source-manifest.scm (all-packages): Remove.
> * guix/scripts/graph.scm (all-packages): Remove.
> * guix/scripts/refresh.scm (all-packages): Remove.
> * guix/scripts/weather.scm (all-packages): Remove.

LGTM.

[...]

> @@ -253,6 +254,23 @@ (define* (fold-packages proc init
>                                  init
>                                  modules))
>  
> +(define all-packages
> +  (mlambda ()
> +    "Return the list of all public packages, including replacements and hidden
> +packages, excluding superseded packages."

Reading the above doc made me question; are replacements always supposed
to be made public?  I typically would leave them private to avoid
cluttering the CLI with duplicate packages.

-- 
Thanks,
Maxim

[bug#74542] [PATCH v2 06/16] guix build: Add ‘--dependents’.

[Ludovic Courtès]

* guix/scripts/build.scm (show-help, %options): Add ‘--dependents’.
(dependents): New procedure.
(options->things-to-build): Add ‘store’ parameter; honor ‘dependents’
option.
[for-type]: Handle ‘dependents’ type.
(options->derivations): Update call to ‘options->things-to-build’.
* tests/guix-build.sh: Add test.
* doc/guix.texi (Additional Build Options): Document ‘--dependents’.
(Invoking guix refresh): Cross-reference it.
* doc/contributing.texi (Submitting Patches): Mention it.

Change-Id: I00b6d5831e1f1d35dc8b84a82605391d5a8f417c
---
 doc/contributing.texi  |  4 ++-
 doc/guix.texi          | 27 ++++++++++++++++++++
 guix/scripts/build.scm | 56 ++++++++++++++++++++++++++++++++++++++++--
 tests/guix-build.sh    |  6 +++++
 4 files changed, 90 insertions(+), 3 deletions(-)

diff --git a/doc/contributing.texi b/doc/contributing.texi
index b063169189..5a778466d7 100644
--- a/doc/contributing.texi
+++ b/doc/contributing.texi
@@ -1914,7 +1914,9 @@ Submitting Patches
 
 @item
 Make sure the package builds on your platform, using @command{guix build
-@var{package}}.
+@var{package}}.  Also build at least its direct dependents with
+@command{guix build --dependents=1 @var{package}}
+(@pxref{build-dependents, @command{guix build}}).
 
 @item
 We recommend you also try building the package on other supported
diff --git a/doc/guix.texi b/doc/guix.texi
index a9d0d044ae..219f19f2d6 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -13639,6 +13639,31 @@ Additional Build Options
 natively.
 @end quotation
 
+@cindex dependents of a package, building them
+@cindex building the dependents of a package
+@anchor{build-dependents}
+@item --dependents[=@var{depth}]
+@itemx -P [@var{depth}]
+Build the dependents of the following package.  By default, build all
+the direct and indirect dependents; when @var{depth} is provided, limit
+to dependents at that distance: 1 for direct dependents, 2 for
+dependents of dependents, and so on.
+
+For example, the command below builds @emph{all} the dependents of libgit2:
+
+@example
+guix build --dependents libgit2
+@end example
+
+To build all the packages that directly depend on NumPy, run:
+
+@example
+guix build -P1 python-numpy
+@end example
+
+The list of dependents is computed in the same way as with @command{guix
+refresh --list-dependent} (@pxref{Invoking guix refresh}).
+
 @item --source
 @itemx -S
 Build the source derivations of the packages, rather than the packages
@@ -15142,6 +15167,8 @@ Invoking guix refresh
 @command{guix graph}}, for information on how to visualize the list of
 dependents of a package.
 
+@xref{build-dependents, @command{guix build --dependents}}, for a
+convenient way to build all the dependents of a package.
 @end table
 
 Be aware that the @option{--list-dependent} option only
diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm
index e8384c8d2d..5c93dc78f8 100644
--- a/guix/scripts/build.scm
+++ b/guix/scripts/build.scm
@@ -33,6 +33,9 @@ (define-module (guix scripts build)
   #:use-module (guix profiles)
   #:use-module (guix diagnostics)
   #:autoload   (guix http-client) (http-fetch http-get-error?)
+  #:autoload   (guix scripts graph) (%bag-node-type)
+  #:autoload   (guix graph) (node-back-edges)
+  #:autoload   (guix sets) (setq set-contains? set-insert)
   #:use-module (ice-9 format)
   #:use-module (ice-9 match)
   #:use-module (srfi srfi-1)
@@ -440,6 +443,9 @@ (define (show-help)
   (display (G_ "
   -D, --development      build the inputs of the following package"))
   (display (G_ "
+  -P, --dependents[=N]   build dependents of the following package, up to
+                         depth N"))
+  (display (G_ "
   -S, --source           build the packages' source derivations"))
   (display (G_ "
       --sources[=TYPE]   build source derivations; TYPE may optionally be one
@@ -527,6 +533,11 @@ (define %options
          (option '(#\D "development") #f #f
                  (lambda (opt name arg result)
                    (alist-cons 'development? #t result)))
+         (option '(#\P "dependents") #f #t
+                 (lambda (opt name arg result)
+                   (alist-cons 'dependents
+                               (or (and=> arg string->number*) +inf.0)
+                               result)))
          (option '(#\n "dry-run") #f #f
                  (lambda (opt name arg result)
                    (alist-cons 'dry-run? #t result)))
@@ -551,7 +562,39 @@ (define %options
                  %standard-cross-build-options
                  %standard-native-build-options)))
 
-(define (options->things-to-build opts)
+(define (dependents store packages max-depth)
+  "List all the things that would need to be rebuilt if PACKAGES are changed."
+  ;; Using %BAG-NODE-TYPE is more accurate than using %PACKAGE-NODE-TYPE
+  ;; because it includes implicit dependencies.
+  (define (get-dependents packages edges)
+    (let loop ((packages packages)
+               (result '())
+               (depth 0)
+               (visited (setq)))
+      (if (> depth max-depth)
+          (values result visited)
+          (match packages
+            (()
+             (values result visited))
+            ((head . tail)
+             (if (set-contains? visited head)
+                 (loop tail result depth visited)
+                 (let ((next (edges head)))
+                   (call-with-values
+                       (lambda ()
+                         (loop next
+                               (cons head result)
+                               (+ depth 1)
+                               (set-insert head visited)))
+                     (lambda (result visited)
+                       (loop tail result depth visited))))))))))
+
+  (with-store store
+    (run-with-store store
+      (mlet %store-monad ((edges (node-back-edges %bag-node-type (all-packages))))
+        (return (get-dependents packages edges))))))
+
+(define (options->things-to-build store opts)
   "Read the arguments from OPTS and return a list of high-level objects to
 build---packages, gexps, derivations, and so on."
   (define (validate-type x)
@@ -600,6 +643,13 @@ (define-public my-package
       (match type
         ('regular
          (list obj))
+        (('dependents . depth)
+         (if (package? obj)
+             (begin
+               (info (G_ "computing dependents of package ~a...~%")
+                     (package-full-name obj))
+               (dependents store (list obj) depth))
+             (list obj)))
         ('development
          (if (package? obj)
              (map manifest-entry-item
@@ -661,6 +711,8 @@ (define-public my-package
                         result)))
          (('development? . #t)
           (loop tail 'development result))
+         (('dependents . depth)
+          (loop tail `(dependents . ,depth) result))
          (_
           (loop tail type result)))))))
 
@@ -687,7 +739,7 @@ (define (options->derivations store opts)
       (systems systems)))
 
   (define things-to-build
-    (map transform (options->things-to-build opts)))
+    (map transform (options->things-to-build store opts)))
 
   (define warn-if-unsupported
     (let ((target (assoc-ref opts 'target)))
diff --git a/tests/guix-build.sh b/tests/guix-build.sh
index 3637bcdeb3..8707ed32c0 100644
--- a/tests/guix-build.sh
+++ b/tests/guix-build.sh
@@ -196,6 +196,12 @@ test `guix build -D hello -d \
       | grep -e 'glibc.*\.drv$' -e 'gcc.*\.drv$' -e 'binutils.*\.drv$' \
       | wc -l` -ge 3
 
+# Building the dependents.
+test `guix build -P1 libgit2 -P1 libssh -d \
+      | grep -e 'guile-git.*\.drv$' -e 'guile-ssh.*\.drv$' \
+             -e 'libgit2.*\.drv$' -e 'libssh.*\.drv$' \
+      | wc -l` -eq 4
+
 # Unbound variable in thunked field.
 cat > "$module_dir/foo.scm" <<EOF
 (define-module (foo)
-- 
2.46.0

[bug#74542] [PATCH v2 07/16] import: gnome: Keep going upon HTTP errors.

[Ludovic Courtès]

* guix/import/gnome.scm (import-gnome-release): Do not re-raise
condition C, emit a warning instead.

Change-Id: I9a7c337a17d1f7ed4677398e7b0f383befca7d74
---
 guix/import/gnome.scm | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/guix/import/gnome.scm b/guix/import/gnome.scm
index 054ae44f7a..3ba8ae02e5 100644
--- a/guix/import/gnome.scm
+++ b/guix/import/gnome.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2017, 2019, 2021 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2017, 2019, 2021, 2024 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2022 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;; Copyright © 2022 Hartmut Goebel <h.goebel@crazy-compilers.com>
 ;;;
@@ -23,6 +23,8 @@ (define-module (guix import gnome)
   #:use-module (guix utils)
   #:use-module (guix packages)
   #:use-module (guix http-client)
+  #:use-module (guix diagnostics)
+  #:use-module (guix i18n)
   #:use-module (json)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-34)
@@ -111,9 +113,12 @@ (define* (import-gnome-release package #:key (version #f))
           releases))
 
   (guard (c ((http-get-error? c)
-             (if (= 404 (http-get-error-code c))
-                 #f
-                 (raise c))))
+             (unless (= 404 (http-get-error-code c))
+               (warning (G_ "failed to download from '~a': ~a (~s)~%")
+                        (uri->string (http-get-error-uri c))
+                        (http-get-error-code c)
+                        (http-get-error-reason c)))
+             #f))
     (let* ((port (http-fetch/cached
                   (string->uri (string-append
                                 "https://ftp.gnome.org/pub/gnome/sources/"
-- 
2.46.0

[bug#74542] [PATCH v2 08/16] gnu-maintenance: ‘gnu-ftp’ updater excludes GnuPG-hosted packages.

[Ludovic Courtès]

Since ftp://ftp.gnupg.org no longer exists, another updater should be
used for packages with code fetched from mirror://gnupg.

* guix/gnu-maintenance.scm (gnupg-hosted?): New procedure.
(%gnu-ftp-updater): Exclude packages that match ‘gnupg-hosted?’.
* guix/download.scm (%mirrors): Remove ftp://ftp.gnupg.org.

Change-Id: I653877ad843da60281d3a234acc85743bee64856
---
 guix/download.scm        | 3 +--
 guix/gnu-maintenance.scm | 6 +++++-
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/guix/download.scm b/guix/download.scm
index e37c2195ee..d88ad0ee44 100644
--- a/guix/download.scm
+++ b/guix/download.scm
@@ -78,8 +78,7 @@ (define %mirrors
        "ftp://mirrors.dotsrc.org/gcrypt/"
        "ftp://ftp.heanet.ie/mirrors/ftp.gnupg.org/gcrypt/"
        "ftp://ftp.mirrorservice.org/sites/ftp.gnupg.org/gcrypt/"
-       "ftp://ftp.ring.gr.jp/pub/net/gnupg/"
-       "ftp://ftp.gnupg.org/gcrypt/")
+       "ftp://ftp.ring.gr.jp/pub/net/gnupg/")
       (gnome
        "https://download.gnome.org/"
        "http://ftp.gnome.org/pub/GNOME/")
diff --git a/guix/gnu-maintenance.scm b/guix/gnu-maintenance.scm
index 866b960a4a..b612b11c00 100644
--- a/guix/gnu-maintenance.scm
+++ b/guix/gnu-maintenance.scm
@@ -1025,15 +1025,19 @@ (define %gnu-updater
            (false-if-networking-error (gnu-hosted? package))))
    (import import-gnu-release)))
 
+(define gnupg-hosted?
+  (url-prefix-predicate "mirror://gnupg/"))
+
 (define %gnu-ftp-updater
   ;; This is for GNU packages taken from alternate locations, such as
-  ;; alpha.gnu.org, ftp.gnupg.org, etc.  It is obsolescent.
+  ;; alpha.gnu.org (ftp.gnupg.org is no longer available).  It is obsolescent.
   (upstream-updater
    (name 'gnu-ftp)
    (description "Updater for GNU packages only available via FTP")
    (pred (lambda (package)
            (false-if-networking-error
             (and (not (gnu-hosted? package))
+                 (not (gnupg-hosted? package))
                  (pure-gnu-package? package)))))
    (import import-release*)))
 
-- 
2.46.0

[bug#74542] [PATCH v2 09/16] gnu: Update updater properties for GnuPG-related packages.

[Ludovic Courtès]

* gnu/packages/gnupg.scm (libgpg-error)[properties]: Remove.
(libgcrypt)[properties]: Remove.
(libassuan)[properties]: Remove.
(libksba)[properties]: Remove.
(npth)[properties]: Remove.
(gnupg)[properties]: Remove.
(gpgme)[properties]: Remove.
(pinentry-tty)[properties]: Change for ‘generic-html’.
(gpa)[properties]: Remove.

Change-Id: I1eb9d6ad5bb3ecbfbef3d44dbae281a3de7ebac7
---
 gnu/packages/gnupg.scm | 40 ++++++++++++----------------------------
 1 file changed, 12 insertions(+), 28 deletions(-)

diff --git a/gnu/packages/gnupg.scm b/gnu/packages/gnupg.scm
index 2e97c2244a..3f55ccaf5b 100644
--- a/gnu/packages/gnupg.scm
+++ b/gnu/packages/gnupg.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2012-2021, 2023 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2012-2021, 2023, 2024 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2013, 2015, 2018 Andreas Enge <andreas@enge.fr>
 ;;; Copyright © 2014, 2018 Eric Bavier <bavier@member.fsf.org>
 ;;; Copyright © 2014, 2015, 2016, 2020 Mark H Weaver <mhw@netris.org>
@@ -160,9 +160,7 @@ (define-public libgpg-error
 for all GnuPG components.  Among these are GPG, GPGSM, GPGME,
 GPG-Agent, libgcrypt, Libksba, DirMngr, Pinentry, SmartCard
 Daemon and possibly more in the future.")
-    (license license:lgpl2.0+)
-    (properties '((ftp-server . "ftp.gnupg.org")
-                  (ftp-directory . "/gcrypt/libgpg-error")))))
+    (license license:lgpl2.0+)))
 
 (define-public libgcrypt
   (package
@@ -217,9 +215,7 @@ (define-public libgcrypt
 standard cryptographic building blocks such as symmetric ciphers, hash
 algorithms, public key algorithms, large integer functions and random number
 generation.")
-    (license license:lgpl2.0+)
-    (properties '((ftp-server . "ftp.gnupg.org")
-                  (ftp-directory . "/gcrypt/libgcrypt")))))
+    (license license:lgpl2.0+)))
 
 (define-public libassuan
   (package
@@ -250,9 +246,7 @@ (define-public libassuan
 protocol.  This protocol is used for IPC between most newer
 GnuPG components.  Both, server and client side functions are
 provided.")
-    (license license:lgpl2.0+)
-    (properties '((ftp-server . "ftp.gnupg.org")
-                  (ftp-directory . "/gcrypt/libassuan")))))
+    (license license:lgpl2.0+)))
 
 (define-public libksba
   (package
@@ -283,9 +277,7 @@ (define-public libksba
      "KSBA (pronounced Kasbah) is a library to make X.509 certificates
 as well as the CMS easily accessible by other applications.  Both
 specifications are building blocks of S/MIME and TLS.")
-    (license license:gpl3+)
-    (properties '((ftp-server . "ftp.gnupg.org")
-                  (ftp-directory . "/gcrypt/libksba")))))
+    (license license:gpl3+)))
 
 (define-public npth
   (package
@@ -307,9 +299,7 @@ (define-public npth
 In contrast to GNU Pth is is based on the system's standard threads
 implementation.  This allows the use of libraries which are not
 compatible to GNU Pth.")
-    (license (list license:lgpl3+ license:gpl2+)) ; dual license
-    (properties '((ftp-server . "ftp.gnupg.org")
-                  (ftp-directory . "/gcrypt/npth")))))
+    (license (list license:lgpl3+ license:gpl2+)))) ;dual license
 
 (define-public gnupg
   (package
@@ -404,9 +394,7 @@ (define-public gnupg
 servers.  It includes several libraries: libassuan (IPC between GnuPG
 components), libgpg-error (centralized GnuPG error values), and
 libskba (working with X.509 certificates and CMS data).")
-    (license license:gpl3+)
-    (properties '((ftp-server . "ftp.gnupg.org")
-                  (ftp-directory . "/gcrypt/gnupg")))))
+    (license license:gpl3+)))
 
 (define-public gnupg-1
   (package (inherit gnupg)
@@ -463,9 +451,7 @@ (define-public gpgme
 programming task, it is suggested that all software should try to use GPGME
 instead.  This way bug fixes or improvements can be done at a central place
 and every application benefits from this.")
-    (license license:lgpl2.1+)
-    (properties '((ftp-server . "ftp.gnupg.org")
-                  (ftp-directory . "/gcrypt/gpgme")))))
+    (license license:lgpl2.1+)))
 
 (define-public gpgme-1.23
   (hidden-package
@@ -886,9 +872,9 @@ (define-public pinentry-tty
      "Pinentry provides a console that allows users to enter a passphrase when
 @code{gpg} is run and needs it.")
     (license license:gpl2+)
-    (properties '((ftp-server . "ftp.gnupg.org")
-                  (ftp-directory . "/gcrypt/pinentry")
-                  (upstream-name . "pinentry")))))
+    (properties
+     '((release-monitoring-url . "https://gnupg.org/ftp/gcrypt/pinentry/")
+       (upstream-name . "pinentry")))))
 
 (define-public pinentry-emacs
   (package
@@ -1134,9 +1120,7 @@ (define-public gpa
      "GPA, the GNU Privacy Assistant, is a graphical user interface for
 @uref{https://gnupg.org, GnuPG}.  It can be used to encrypt, decrypt, and sign
 files, to verify signatures, and to manage the private and public keys.")
-    (license license:gpl3+)
-    (properties '((ftp-server . "ftp.gnupg.org")
-                  (ftp-directory . "/gcrypt/gpa")))))
+    (license license:gpl3+)))
 
 (define-public parcimonie
   (package
-- 
2.46.0

[bug#74542] [PATCH v2 10/16] gnu: gnutls: Change release monitoring URL.

[Ludovic Courtès]

* gnu/packages/tls.scm (gnutls)[properties]: Change release monitoring
URL.  Add ‘upstream-name’.

Change-Id: If6ee941246b130914ddf7ca11c074a9547b9d739
---
 gnu/packages/tls.scm | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/tls.scm b/gnu/packages/tls.scm
index 43806e24da..06f4098f5a 100644
--- a/gnu/packages/tls.scm
+++ b/gnu/packages/tls.scm
@@ -300,7 +300,11 @@ (define-public gnutls
 required structures.")
     (license license:lgpl2.1+)
     (properties
-     '((release-monitoring-url . "https://gnutls.org/download.html")))))
+     ;; Since gnutls.org doesn't have a page with a direct link to the
+     ;; tarball, defer to fellow LFS hackers.
+     '((release-monitoring-url
+        . "https://www.linuxfromscratch.org/blfs/view/svn/postlfs/gnutls.html")
+       (upstream-name . "gnutls")))))
 
 (define-deprecated/public-alias gnutls-latest gnutls)
 
-- 
2.46.0

[bug#74542] [PATCH v2 11/16] gnu: git-minimal: Add ‘upstream-name’ property.

[Ludovic Courtès]

* gnu/packages/version-control.scm (git-minimal)[properties]: Add
‘upstream-name’.

Change-Id: I2d2516003b7f686852a3531859d06ca7f4e4820d
---
 gnu/packages/version-control.scm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/version-control.scm b/gnu/packages/version-control.scm
index c892a83a98..936d511cb3 100644
--- a/gnu/packages/version-control.scm
+++ b/gnu/packages/version-control.scm
@@ -518,7 +518,8 @@ (define-public git-minimal
                                       "CVE-2022-30949"
                                       "CVE-2022-36882"
                                       "CVE-2022-36883"
-                                      "CVE-2022-36884"))))
+                                      "CVE-2022-36884"))
+                  (upstream-name . "git")))
     (license license:gpl2)
     (home-page "https://git-scm.com/")))
 
-- 
2.46.0

[bug#74542] [PATCH v2 12/16] gnu-maintenance: ‘generic-html’ update honors <base href="…">.

[Ludovic Courtès]

This fixes updates of ‘curl’: <https://curl.se/download/> includes
<base href="…"> in its head and ignoring it would lead to incorrect
download URLs.

* guix/gnu-maintenance.scm (html-links): Keep track of <base href="…">
in ‘loop’.  Rewrite relative links at the end.

Change-Id: I989da78df3431034c9a584f8e10cad87ae6dc920
---
 guix/gnu-maintenance.scm | 41 +++++++++++++++++++++++++++-------------
 1 file changed, 28 insertions(+), 13 deletions(-)

diff --git a/guix/gnu-maintenance.scm b/guix/gnu-maintenance.scm
index b612b11c00..ee4882326f 100644
--- a/guix/gnu-maintenance.scm
+++ b/guix/gnu-maintenance.scm
@@ -39,6 +39,7 @@ (define-module (guix gnu-maintenance)
   #:use-module (guix utils)
   #:use-module (guix diagnostics)
   #:use-module (guix i18n)
+  #:autoload   (guix combinators) (fold2)
   #:use-module (guix memoization)
   #:use-module (guix records)
   #:use-module (guix upstream)
@@ -483,19 +484,33 @@ (define* (import-release* package #:key (version #f))
 
 (define (html-links sxml)
   "Return the list of links found in SXML, the SXML tree of an HTML page."
-  (let loop ((sxml sxml)
-             (links '()))
-    (match sxml
-      (('a ('@ attributes ...) body ...)
-       (match (assq 'href attributes)
-         (#f          (fold loop links body))
-         (('href url) (fold loop (cons url links) body))))
-      ((tag ('@ _ ...) body ...)
-       (fold loop links body))
-      ((tag body ...)
-       (fold loop links body))
-      (_
-       links))))
+  (define-values (links base)
+    (let loop ((sxml sxml)
+               (links '())
+               (base #f))
+      (match sxml
+        (('a ('@ attributes ...) body ...)
+         (match (assq 'href attributes)
+           (#f          (fold2 loop links base body))
+           (('href url) (fold2 loop (cons url links) base body))))
+        (('base ('@ ('href new-base)))
+         ;; The base against which relative URL paths must be resolved.
+         (values links new-base))
+        ((tag ('@ _ ...) body ...)
+         (fold2 loop links base body))
+        ((tag body ...)
+         (fold2 loop links base body))
+        (_
+         (values links base)))))
+
+  (if base
+      (map (lambda (link)
+             (let ((uri (string->uri link)))
+               (if (or uri (string-prefix? "/" link))
+                   link
+                   (in-vicinity base link))))
+           links)
+      links))
 
 (define (url->links url)
   "Return the unique links on the HTML page accessible at URL."
-- 
2.46.0

[bug#74542] [PATCH v2 13/16] guix build: Validate that the file passed to ‘-m’ returns a manifest.

[Ludovic Courtès]

* guix/scripts/build.scm (options->things-to-build)[ensure-manifest]:
New procedure.
Use it.

Change-Id: If64c483d7079f441a296d5bd4e06e67f44cbb7bf
---
 guix/scripts/build.scm | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm
index 5c93dc78f8..d1368f7e48 100644
--- a/guix/scripts/build.scm
+++ b/guix/scripts/build.scm
@@ -629,6 +629,12 @@ (define-public my-package
       (for-each validate-type lst)
       lst))
 
+  (define (ensure-manifest x file)
+    (unless (manifest? x)
+      (raise (formatted-message (G_ "file '~a' does not return a manifest")
+                                file)))
+    x)
+
   (define system
     (or (assoc-ref opts 'system) (%current-system)))
 
@@ -701,9 +707,11 @@ (define-public my-package
           (loop tail 'regular
                 (append (map manifest-entry-item
                              (manifest-entries
-                              (load* manifest
-                                     (make-user-module '((guix profiles)
-                                                         (gnu))))))
+                              (ensure-manifest
+                               (load* manifest
+                                      (make-user-module '((guix profiles)
+                                                          (gnu))))
+                               manifest)))
                         result)))
          (('expression . str)
           (loop tail 'regular
-- 
2.46.0

[bug#74542] [PATCH v2 14/16] transformations: ‘package-with-upstream-version’ can preserve patches.

[Ludovic Courtès]

* guix/transformations.scm (upstream-fetch): New procedure.
(package-with-upstream-version): Add #:preserve-patches? and honor it.

Change-Id: Ib56b84957d8bdad2eebe2551e2a6e477506fc55e
---
 guix/transformations.scm  | 25 ++++++++++++++++++++++---
 tests/transformations.scm | 31 ++++++++++++++++++++++++++++++-
 2 files changed, 52 insertions(+), 4 deletions(-)

diff --git a/guix/transformations.scm b/guix/transformations.scm
index 9dfc4402c5..a32aad39f2 100644
--- a/guix/transformations.scm
+++ b/guix/transformations.scm
@@ -848,9 +848,20 @@ (define (transform-package-patches specs)
         (rewrite obj)
         obj)))
 
-(define* (package-with-upstream-version p #:optional version)
+(define* (upstream-fetch source hash-algo hash
+                         #:optional name
+                         #:key (system (%current-system))
+                         (guile (default-guile))
+                         executable?)
+  "This origin method simply downloads SOURCE, an <upstream-source> record."
+  (lower-object source system))
+
+(define* (package-with-upstream-version p #:optional version
+                                        #:key (preserve-patches? #f))
   "Return package P changed to use the given upstream VERSION or, if VERSION
-is #f, the latest known upstream version."
+is #f, the latest known upstream version.  When PRESERVE-PATCHES? is true,
+preserve patches and snippets found in the source of P, provided it's an
+origin."
   (let ((source (package-latest-release p #:version version)))
     (cond ((not source)
            (if version
@@ -885,7 +896,15 @@ (define* (package-with-upstream-version p #:optional version)
            (package
              (inherit p)
              (version (upstream-source-version source))
-             (source source))))))
+             (source (if (and preserve-patches?
+                              (origin? (package-source p)))
+                         ;; Inherit P's origin so snippets and patches are
+                         ;; applied as if we had run 'guix refresh -u'.
+                         (origin
+                           (inherit (package-source p))
+                           (method upstream-fetch)
+                           (uri source))
+                         source)))))))
 
 (define (transform-package-latest specs)
   "Return a procedure that rewrites package graphs such that those in SPECS
diff --git a/tests/transformations.scm b/tests/transformations.scm
index 755211d65d..5285d98f17 100644
--- a/tests/transformations.scm
+++ b/tests/transformations.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2016-2017, 2019-2023 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2016-2017, 2019-2024 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2021 Marius Bakke <marius@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
@@ -497,6 +497,35 @@ (define* (depends-on-toolchain? p #:optional (toolchain "gcc-toolchain"))
     (let ((new (t coreutils)))
       (assq-ref (package-properties new) 'transformations))))
 
+(test-equal "package-with-upstream-version"
+  '("42.0" "42.0"
+    ("http://example.org")
+    ("a" "b") (do something))
+  (mock ((guix upstream) %updaters
+         (delay (list (upstream-updater
+                       (name 'dummy)
+                       (pred (const #t))
+                       (description "")
+                       (import (const (upstream-source
+                                       (package "foo")
+                                       (version "42.0")
+                                       (urls '("http://example.org")))))))))
+        (let* ((old (dummy-package "foo" (version "1.0")
+                                   (source (dummy-origin
+                                            (patches '("a" "b"))
+                                            (snippet '(do something))))))
+               (new (package-with-upstream-version old))
+               (new+patches (package-with-upstream-version
+                             old #:preserve-patches? #t)))
+          (list (package-version new) (package-version new+patches)
+
+                ;; Source of NEW is directly an <upstream-source>.
+                (upstream-source-urls (package-source new))
+
+                ;; Check that #:preserve-patches? #t gave us an origin.
+                (origin-patches (package-source new+patches))
+                (origin-snippet (package-source new+patches))))))
+
 (test-equal "options->transformation, with-latest"
   "42.0"
   (mock ((guix upstream) %updaters
-- 
2.46.0

[bug#74542] [PATCH v2 15/16] transformations: Add #:authenticate? to ‘package-with-upstream-version’.

[Ludovic Courtès]

* guix/transformations.scm (upstream-source-without-signatures): New
procedures.
(package-with-upstream-version): Add #:authenticate? and honor it.

Change-Id: Ib86feb765f57bc1b129e813780abc605c43ed97b
---
 guix/transformations.scm | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/guix/transformations.scm b/guix/transformations.scm
index a32aad39f2..3f7d20beaa 100644
--- a/guix/transformations.scm
+++ b/guix/transformations.scm
@@ -31,7 +31,8 @@ (define-module (guix transformations)
   #:autoload   (guix download) (download-to-store)
   #:autoload   (guix git-download) (git-reference? git-reference-url)
   #:autoload   (guix git) (git-checkout git-checkout? git-checkout-url)
-  #:autoload   (guix upstream) (package-latest-release
+  #:autoload   (guix upstream) (upstream-source
+                                package-latest-release
                                 upstream-source-version
                                 upstream-source-signature-urls)
   #:autoload   (guix cpu) (current-cpu
@@ -856,13 +857,24 @@ (define* (upstream-fetch source hash-algo hash
   "This origin method simply downloads SOURCE, an <upstream-source> record."
   (lower-object source system))
 
+(define (upstream-source-without-signatures source)
+  "Return SOURCE with #f as its 'signature-urls' field."
+  (upstream-source (inherit source)
+                   (signature-urls #f)))
+
 (define* (package-with-upstream-version p #:optional version
-                                        #:key (preserve-patches? #f))
+                                        #:key
+                                        (preserve-patches? #f)
+                                        (authenticate? #t))
   "Return package P changed to use the given upstream VERSION or, if VERSION
 is #f, the latest known upstream version.  When PRESERVE-PATCHES? is true,
 preserve patches and snippets found in the source of P, provided it's an
-origin."
-  (let ((source (package-latest-release p #:version version)))
+origin.  When AUTHENTICATE? is false, disable OpenPGP signature verification
+of upstream source code."
+  (let ((source (and=> (package-latest-release p #:version version)
+                       (if authenticate?
+                           identity
+                           upstream-source-without-signatures))))
     (cond ((not source)
            (if version
                (warning
-- 
2.46.0

[bug#74542] [PATCH v2 16/16] etc: Add upgrade manifest.

[Ludovic Courtès]

* guix/scripts/build.scm (dependents): Export.
* etc/upgrade-manifest.scm: New file.
* Makefile.am (EXTRA_DIST): Add it.

Change-Id: I1b2a2ebd09e559c68da9f25772bf33caacb4c031
---
 Makefile.am              |   1 +
 etc/upgrade-manifest.scm | 128 +++++++++++++++++++++++++++++++++++++++
 guix/scripts/build.scm   |   2 +
 3 files changed, 131 insertions(+)
 create mode 100644 etc/upgrade-manifest.scm

diff --git a/Makefile.am b/Makefile.am
index e94ba87797..0cff32c607 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -743,6 +743,7 @@ EXTRA_DIST +=						\
   etc/source-manifest.scm				\
   etc/system-tests.scm					\
   etc/time-travel-manifest.scm				\
+  etc/upgrade-manifest.scm				\
   scripts/guix.in					\
   tests/cve-sample.json					\
   tests/keys/civodul.pub				\
diff --git a/etc/upgrade-manifest.scm b/etc/upgrade-manifest.scm
new file mode 100644
index 0000000000..5e6d7d85e4
--- /dev/null
+++ b/etc/upgrade-manifest.scm
@@ -0,0 +1,128 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2024 Ludovic Courtès <ludo@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+;; This manifest computes upgrades of key packages using updaters from (guix
+;; upstream) and supporting code for the 'with-latest' transformation.
+
+(use-modules (guix memoization)
+             (guix monads)
+             (guix graph)
+             (guix packages)
+             (guix profiles)
+             (guix store)
+             (guix transformations)
+             (guix upstream)
+             ((guix scripts build) #:select (dependents))
+             ((guix scripts graph) #:select (%bag-node-type))
+             ((guix import github) #:select (%github-api))
+             (guix build-system gnu)
+             (guix build-system cmake)
+             ((gnu packages) #:select (all-packages))
+
+             (gnu packages backup)
+             (gnu packages curl)
+             (gnu packages freedesktop)
+             (gnu packages gnupg)
+             (gnu packages ssh)
+             (gnu packages tls)
+             (gnu packages version-control)
+             (gnu packages xorg)
+
+             (ice-9 match)
+             (srfi srfi-1))
+
+;; Bypass the GitHub updater: we'd need an API token or we would hit the rate
+;; limit.
+(%github-api "http://example.org")
+
+(define security-packages
+  (list xorg-server
+        elogind
+
+        openssl
+        gnutls
+        curl
+        curl-ssh
+
+        libarchive
+        libgit2
+        libssh
+
+        ;; GnuPG.
+        libassuan
+        libgpg-error
+        libgcrypt
+        libksba
+        npth
+        gnupg
+        gpgme
+        pinentry))
+
+(define latest-version
+  (mlambdaq (package)
+    (package-with-upstream-version package
+                                   ;; Preserve patches and snippets to get
+                                   ;; exactly the same as what we'd have with
+                                   ;; 'guix refresh -u PACKAGE'.
+                                   #:preserve-patches? #t
+
+                                   ;; XXX: Disable source code authentication:
+                                   ;; this requires a local keyring, populated
+                                   ;; from key servers, but key servers may be
+                                   ;; unreliable or may lack the upstream
+                                   ;; keys.  Leave it up to packagers to
+                                   ;; actually authenticate code and make sure
+                                   ;; it matches what this manifest computed.
+                                   #:authenticate? #f)))
+
+(define individual-security-upgrades
+  ;; Upgrades of individual packages with their direct dependents built
+  ;; against that upgrade.
+  (manifest
+   (with-store store
+     (append-map (lambda (package)
+                   (let* ((name (package-name package))
+                          (newest (latest-version package))
+                          (update (package-input-rewriting
+                                   `((,package . ,newest)))))
+                     (map (lambda (package)
+                            (manifest-entry
+                              (inherit (package->manifest-entry
+                                        (update package)))
+                              (name (string-append (package-name package)
+                                                   "-with-latest-" name))))
+                          (dependents store (list package) 1))))
+                 security-packages))))
+
+(define joint-security-upgrades
+  ;; All of SECURITY-PACKAGES updated at once, together with their dependents.
+  (manifest
+   (with-store store
+     (let ((update-all (package-input-rewriting
+                        (map (lambda (package)
+                               `(,package . ,(latest-version package)))
+                             security-packages))))
+       (map (lambda (package)
+              (manifest-entry
+                (inherit (package->manifest-entry
+                          (update-all package)))
+                (name (string-append (package-name package) "-full-upgrade"))))
+            (dependents store security-packages 2))))))
+
+(concatenate-manifests
+ (list individual-security-upgrades joint-security-upgrades))
diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm
index d1368f7e48..0080f1fe58 100644
--- a/guix/scripts/build.scm
+++ b/guix/scripts/build.scm
@@ -63,6 +63,8 @@ (define-module (guix scripts build)
             show-cross-build-options-help
             show-native-build-options-help
 
+            dependents
+
             guix-build
             register-root
             register-root*))
-- 
2.46.0