[bug#60629] [PATCH 0/3] Add '--with-version' package transformation

[bug#60629] [PATCH 0/3] Add '--with-version' package transformation

[Ludovic Courtès]

Hello Guix!

Now that updaters can (to some extent) import any upstream version, we
can provide a ‘--with-version’ package transformation option that picks
the given upstream version.

The nice thing is that it’s just a dozen new lines of code in
(guix transformations)!

Feedback welcome!

Ludo’.

Ludovic Courtès (3):
  transformations: Add '--with-version'.
  transformations: Let users know when '--with-latest' has no effect.
  DRAFT news: Add entry for '--with-version'.

 doc/guix.texi             | 19 +++++++--
 etc/news.scm              | 36 ++++++++++++++++-
 guix/transformations.scm  | 81 ++++++++++++++++++++++++++++-----------
 tests/transformations.scm | 19 ++++++++-
 4 files changed, 127 insertions(+), 28 deletions(-)


base-commit: cce9ff2d5ada3d1e26e1c70205f21271a0065482
-- 
2.38.1

[bug#60629] [PATCH 1/3] transformations: Add '--with-version'.

[Ludovic Courtès]

This is a followup to 8aeccc6240ec45f0bc7bed655e0c8149ae4253eb.

* guix/transformations.scm (package-with-upstream-version): New procedure.
(transform-package-latest)[package-with-latest-upstream]: Remove.
Use 'package-with-upstream-version' instead.
(transform-package-version): New procedure.
(%transformations, %transformation-options)
(show-transformation-options-help/detailed): Add '-with-version'.
* tests/transformations.scm ("options->transformation, with-version"):
New test.
* doc/guix.texi (Package Transformation Options): Document '--with-version'.
(Defining Package Variants): Mention it.
---
 doc/guix.texi             | 19 ++++++++--
 guix/transformations.scm  | 78 +++++++++++++++++++++++++++------------
 tests/transformations.scm | 19 +++++++++-
 3 files changed, 89 insertions(+), 27 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 293c3016aa..d3af813b5a 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -8072,8 +8072,9 @@ vintage!):
                 "0lappv4slgb5spyqbh6yl5r013zv72yqg2pcl30mginf3wdqd8k9"))))))
 @end lisp
 
-The example above corresponds to what the @option{--with-source} package
-transformation option does.  Essentially @code{hello-2.2} preserves all
+The example above corresponds to what the @option{--with-version}
+or @option{--with-source} package transformations option do.
+Essentially @code{hello-2.2} preserves all
 the fields of @code{hello}, except @code{version} and @code{source},
 which it overrides.  Note that the original @code{hello} variable is
 still there, in the @code{(gnu packages base)} module, unchanged.  When
@@ -12735,7 +12736,9 @@ Coreutils in the dependency graph is rebuilt.
 
 @cindex upstream, latest version
 @item --with-latest=@var{package}
-So you like living on the bleeding edge?  This option is for you!  It
+@itemx --with-version=@var{package}=@var{version}
+So you like living on the bleeding edge?  The @option{--with-latest}
+option is for you!  It
 replaces occurrences of @var{package} in the dependency graph with its
 latest upstream version, as reported by @command{guix refresh}
 (@pxref{Invoking guix refresh}).
@@ -12751,6 +12754,16 @@ of Guile-JSON:
 guix build guix --with-latest=guile-json
 @end example
 
+The @option{--with-version} works similarly except that it lets you
+specify that you want precisely @var{version}, assuming that version
+exists upstream.  For example, to spawn a development environment with
+SciPy built against version 1.22.4 of NumPy (skipping its test suite
+because hey, we're not gonna wait this long), you would run:
+
+@example
+guix shell python python-scipy --with-version=python-numpy=1.22.4
+@end example
+
 There are limitations.  First, in cases where the tool cannot or does
 not know how to authenticate source code, you are at risk of running
 malicious code; a warning is emitted in this case.  Second, this option
diff --git a/guix/transformations.scm b/guix/transformations.scm
index bf9639020b..d6a2ef0708 100644
--- a/guix/transformations.scm
+++ b/guix/transformations.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2016-2022 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2016-2023 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2021 Marius Bakke <marius@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
@@ -757,35 +757,61 @@ (define rewrite
         (rewrite obj)
         obj)))
 
+(define* (package-with-upstream-version p #:optional version)
+  "Return package P changed to use the given upstream VERSION or, if VERSION
+is #f, the latest known upstream version."
+  (let ((source (package-latest-release p #:version version)))
+    (cond ((not source)
+           (if version
+               (warning
+                (G_ "could not find version ~a of '~a' upstream~%")
+                version (package-name p))
+               (warning
+                (G_ "could not determine latest upstream release of '~a'~%")
+                (package-name p)))
+           p)
+          ((string=? (upstream-source-version source)
+                     (package-version p))
+           p)
+          (else
+           (unless (pair? (upstream-source-signature-urls source))
+             (warning (G_ "cannot authenticate source of '~a', version ~a~%")
+                      (package-name p)
+                      (upstream-source-version source)))
+
+           ;; TODO: Take 'upstream-source-input-changes' into account.
+           (package
+             (inherit p)
+             (version (upstream-source-version source))
+             (source source))))))
+
 (define (transform-package-latest specs)
   "Return a procedure that rewrites package graphs such that those in SPECS
 are replaced by their latest upstream version."
-  (define (package-with-latest-upstream p)
-    (let ((source (package-latest-release p)))
-      (cond ((not source)
-             (warning
-              (G_ "could not determine latest upstream release of '~a'~%")
-              (package-name p))
-             p)
-            ((string=? (upstream-source-version source)
-                       (package-version p))
-             p)
-            (else
-             (unless (pair? (upstream-source-signature-urls source))
-               (warning (G_ "cannot authenticate source of '~a', version ~a~%")
-                        (package-name p)
-                        (upstream-source-version source)))
+  (define rewrite
+    (package-input-rewriting/spec
+     (map (lambda (spec)
+            (cons spec package-with-upstream-version))
+          specs)))
 
-             ;; TODO: Take 'upstream-source-input-changes' into account.
-             (package
-               (inherit p)
-               (version (upstream-source-version source))
-               (source source))))))
+  (lambda (obj)
+    (if (package? obj)
+        (rewrite obj)
+        obj)))
 
+(define (transform-package-version specs)
+  "Return a procedure that rewrites package graphs such that those in SPECS
+are replaced by the specified upstream version."
   (define rewrite
     (package-input-rewriting/spec
      (map (lambda (spec)
-            (cons spec package-with-latest-upstream))
+            (match (string-tokenize spec %not-equal)
+              ((spec version)
+               (cons spec (cut package-with-upstream-version <> version)))
+              (_
+               (raise (formatted-message
+                       (G_ "~a: invalid upstream version specification")
+                       spec)))))
           specs)))
 
   (lambda (obj)
@@ -809,7 +835,8 @@ (define %transformations
     (with-debug-info . ,transform-package-with-debug-info)
     (without-tests . ,transform-package-tests)
     (with-patch  . ,transform-package-patches)
-    (with-latest . ,transform-package-latest)))
+    (with-latest . ,transform-package-latest)
+    (with-version . ,transform-package-version)))
 
 (define (transformation-procedure key)
   "Return the transformation procedure associated with KEY, a symbol such as
@@ -881,6 +908,8 @@ (define micro-architecture
                   (parser 'with-patch))
           (option '("with-latest") #t #f
                   (parser 'with-latest))
+          (option '("with-version") #t #f
+                  (parser 'with-version))
 
           (option '("help-transform") #f #f
                   (lambda _
@@ -915,6 +944,9 @@ (define (show-transformation-options-help/detailed)
   (display (G_ "
       --with-latest=PACKAGE
                          use the latest upstream release of PACKAGE"))
+  (display (G_ "
+      --with-version=PACKAGE=VERSION
+                         use the given upstream VERSION of PACKAGE"))
   (display (G_ "
       --with-c-toolchain=PACKAGE=TOOLCHAIN
                          build PACKAGE and its dependents with TOOLCHAIN"))
diff --git a/tests/transformations.scm b/tests/transformations.scm
index 5c136e1d48..1fa2c0bba8 100644
--- a/tests/transformations.scm
+++ b/tests/transformations.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2016-2017, 2019-2022 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2016-2017, 2019-2023 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2021 Marius Bakke <marius@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
@@ -497,6 +497,23 @@ (define (package-name* obj)
                    `((with-latest . "foo")))))
           (package-version (t p)))))
 
+(test-equal "options->transformation, with-version"
+  "1.0"
+  (mock ((guix upstream) %updaters
+         (delay (list (upstream-updater
+                       (name 'dummy)
+                       (pred (const #t))
+                       (description "")
+                       (import (const (upstream-source
+                                         (package "foo")
+                                         (version "1.0")
+                                         (urls '("http://example.org")))))))))
+        (let* ((p0 (dummy-package "foo" (version "7.7")))
+               (p1 (dummy-package "bar" (inputs (list p0))))
+               (t  (options->transformation
+                    `((with-version . "foo=1.0")))))
+          (package-version (lookup-package-input (t p1) "foo")))))
+
 (test-equal "options->transformation, tune"
   '(cpu-tuning . "superfast")
   (let* ((p0 (dummy-package "p0"))
-- 
2.38.1

[bug#60629] [PATCH 2/3] transformations: Let users know when '--with-latest' has no effect.

[Ludovic Courtès]

* guix/transformations.scm (package-with-upstream-version): Print a
message when VERSION is false and SOURCE has the same version as P.
---
 guix/transformations.scm | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/guix/transformations.scm b/guix/transformations.scm
index d6a2ef0708..b8e2b3d52a 100644
--- a/guix/transformations.scm
+++ b/guix/transformations.scm
@@ -772,6 +772,9 @@ (define* (package-with-upstream-version p #:optional version)
            p)
           ((string=? (upstream-source-version source)
                      (package-version p))
+           (unless version
+             (info (G_ "~a is already the latest version of '~a'~%")
+                   (package-version p) (package-name p)))
            p)
           (else
            (unless (pair? (upstream-source-signature-urls source))
-- 
2.38.1

[bug#60629] [PATCH 3/3] DRAFT news: Add entry for '--with-version'.

[Ludovic Courtès]

DRAFT: Update commit.

* etc/news.scm: Add entry.
---
 etc/news.scm | 36 +++++++++++++++++++++++++++++++++++-
 1 file changed, 35 insertions(+), 1 deletion(-)

diff --git a/etc/news.scm b/etc/news.scm
index 86451495fd..aa205ceffe 100644
--- a/etc/news.scm
+++ b/etc/news.scm
@@ -1,6 +1,6 @@
 ;; GNU Guix news, for use by 'guix pull'.
 ;;
-;; Copyright © 2019-2022 Ludovic Courtès <ludo@gnu.org>
+;; Copyright © 2019-2023 Ludovic Courtès <ludo@gnu.org>
 ;; Copyright © 2019–2021 Tobias Geerinckx-Rice <me@tobias.gr>
 ;; Copyright © 2019, 2020 Miguel Ángel Arruga Vivas <rosen644835@gmail.com>
 ;; Copyright © 2019, 2020 Konrad Hinsen <konrad.hinsen@fastmail.net>
@@ -26,6 +26,40 @@
 (channel-news
  (version 0)
 
+ (entry (commit "XXX")
+        (title
+         (en "New @option{--with-version} package transformation option")
+         (fr "Nouvelle option de transformation @option{--with-version}"))
+        (body
+         (en "The new @option{--with-version} package transformation option
+generalizes @option{--with-latest}: it gets the specified upstream release of
+a package and uses it instead of the currently-packaged version.
+
+For example, the command below would spawn GNOME Clocks built against GTK
+4.7.0, skipping its test suite:
+
+@example
+guix shell gnome-clocks --with-version=gtk=4.7.0 \
+  --without-tests=gtk -- gnome-clocks
+@end example
+
+Run @command{info \"(guix) Package Transformation Options\"} for more info.")
+         (fr "La nouvelle option de transformation de paquets
+@option{--with-version} généralise @option{--with-latest} : elle permet de
+spécifier quelle version amont d'un logiciel utiliser à la place de celle
+actuellement fournie.
+
+Par exemple, la commande ci-dessous démarre GNOME Clocks construit avec GTK
+4.7.0, sans lancer sa suite de tests :
+
+@example
+guix shell gnome-clocks --with-version=gtk=4.7.0 \
+  --without-tests=gtk -- gnome-clocks
+@end example
+
+Voir @command{info \"(guix.fr) Options de transformation de paquets\"} pour
+plus de détails.")))
+
  (entry (commit "dfc6957a5af7d179d4618eb19d4f555c519bc6f2")
         (title
          (en "New @code{customize-linux} procedure")
-- 
2.38.1

[bug#60629] [PATCH 0/3] Add '--with-version' package transformation

[Hartmut Goebel]

Am 07.01.23 um 16:04 schrieb Ludovic Courtès:
> Now that updaters can (to some extent) import any upstream version, we
> can provide a ‘--with-version’ package transformation option that picks
> the given upstream version.

Cool! (Can't judge on the code, though)

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

[bug#60629] [PATCH 3/3] DRAFT news: Add entry for '--with-version'.

[pelzflorian (Florian Pelz)]

Hello Ludo, excuse the delay.

Could you add this German translation?

Ludovic Courtès <ludo@gnu.org> writes:
> + (entry (commit "XXX")
> +        (title
> +         (en "New @option{--with-version} package transformation
> option")

         (de "Neue Paketumwandlungsoption @option{--with-version}")

> +        (body
> +         (en "The new @option{--with-version} package transformation option
> +generalizes @option{--with-latest}: it gets the specified upstream release of
> +a package and uses it instead of the currently-packaged version.
> +
> +For example, the command below would spawn GNOME Clocks built against GTK
> +4.7.0, skipping its test suite:
> +
> +@example
> +guix shell gnome-clocks --with-version=gtk=4.7.0 \
> +  --without-tests=gtk -- gnome-clocks
> +@end example
> +
> +Run @command{info \"(guix) Package Transformation Options\"} for more info.")

         (de "Die neue Paketumwandlungsoption @option{--with-version}
verallgemeinert @option{--with-latest}: Mit ihr kann man angeben, welche
vom Anbieter veröffentlichte Version man anstelle der derzeit im Paket
vorgegebenen haben möchte.

Zum Beispiel kann mit folgendem Befehl ein für die GTK-Version 4.7.0
erstelltes GNOME Clocks aufgerufen werden, wobei der Testkatalog dafür
übersprungen wird.

@example
guix shell gnome-clocks --with-version=gtk=4.7.0 \
  --without-tests=gtk -- gnome-clocks
@end example

Führen Sie für mehr Informationen @command{info \"(guix.de)
Paketumwandlungsoptionen\"} aus.")



Regards,
Florian

[bug#60629] [PATCH 0/3] Add '--with-version' package transformation

[Simon Tournier]

Hi,

The code LGTM.  Minor the comment below. :-)


On sam., 07 janv. 2023 at 16:04, Ludovic Courtès <ludo@gnu.org> wrote:

> Now that updaters can (to some extent) import any upstream version, we
> can provide a ‘--with-version’ package transformation option that picks
> the given upstream version.

Well, transformation bypasses the checksum control and so it is not
really a fixed-output, right?

Therefore, the result depends on the state of the world and thus it is
not necessary reproducible.

While --with-latest (or other transformation) makes this potential
unreproducibility clear, that’s not the case with --with-version.

For instance, Alice runs

  guix shell python python-scipy --with-version=python-numpy=1.22.4

and the result depends on what upstream provides for Numpy@1.22.4.  Even
if Alice provide a channels.scm file pinning Guix, then Bob does not get
necessary the same as Alice with:

  guix time-machine -C channels.scm \
       -- shell python python-scipy --with-version=python-numpy=1.22.4

Moreover, with my hat dealing with scientific folk, I already see their
confusion.  For instance, they will write some manifest such that,

--8<---------------cut here---------------start------------->8---
(use-modules (guix transformations))

(define transform1
  (options->transformation
    '((with-version . "python-numpy=1.22.4"))))

(packages->manifest
  (list (transform1 (specification->package "python"))
        (transform1
          (specification->package "python-scipy"))))
--8<---------------cut here---------------end--------------->8---

which looks very similar to some requirements.txt or other YAML variants
consumed by well-known other package managers. ;-)

It is confusing because the common motto is «Guix is reproducible by
design» but then here it is not.

And what happens if the upstream content pointed by
--with-version=python-numpy=1.22.4 is not available?  It is not possible
to fallback.  All fall downss.

Well, my concerns are also valid for other transformations but their
names are clear enough to ring some bell for reproducing.  The name
’with-version’ appears to me a trap because it eases incorrect
description of the computational environment while thinking doing it
right because using Guix.  The issue is the connotation of the term
“version” (*).

    (*) term “version”: people are often confused with the difference
    between source code version and produced binary version containing
    compilation options and dependencies; or also confused with the
    difference between extrinsic reference as label or tag and intrinsic
    reference as checksum.


All that said, this option is cool to have. :-) It is missing some
guards, IMHO.  Maybe:

 1. Wording some warning for the manual
 2. Add comments about reproducibility when exporting manifest with
    transformations; for example,

    (define transform1
      ;; WARNING: some transformation might lead to non-reproducible profile.
      ;; For more details, please see the Package Transformation Options section.
      (options->transformation
        '((with-version . "python-numpy=1.22.4")
          (with-c-toolchain . "python-scipy=gcc-toolchain@7"))))


 3. Add a warning.  Similarly, to this:

   warning: cannot authenticate source of 'python-numpy', version 1.22.4

   For instance,

   warning: transformation for 'python-numpy' with version 1.22.4 might be non-reproducible


Cheers,
simon

[bug#60629] [PATCH 0/3] Add '--with-version' package transformation

[Ludovic Courtès]

Hello!

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> On sam., 07 janv. 2023 at 16:04, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> Now that updaters can (to some extent) import any upstream version, we
>> can provide a ‘--with-version’ package transformation option that picks
>> the given upstream version.
>
> Well, transformation bypasses the checksum control and so it is not
> really a fixed-output, right?

It’s not fixed-output at all because we don’t know in advance what we’ll
get from that download.

> Therefore, the result depends on the state of the world and thus it is
> not necessary reproducible.

Absolutely.

[...]

> All that said, this option is cool to have. :-) It is missing some
> guards, IMHO.  Maybe:
>
>  1. Wording some warning for the manual

Will do, that makes a lot of sense to me, I share your concerns.

>  2. Add comments about reproducibility when exporting manifest with
>     transformations; for example,
>
>     (define transform1
>       ;; WARNING: some transformation might lead to non-reproducible profile.
>       ;; For more details, please see the Package Transformation Options section.
>       (options->transformation
>         '((with-version . "python-numpy=1.22.4")
>           (with-c-toolchain . "python-scipy=gcc-toolchain@7"))))

This one’s a bit more involved technically because we don’t use (guix
read-print) yet here, so I prefer to do that separately.

>  3. Add a warning.  Similarly, to this:
>
>    warning: cannot authenticate source of 'python-numpy', version 1.22.4
>
>    For instance,
>
>    warning: transformation for 'python-numpy' with version 1.22.4 might be non-reproducible

I wonder about this one because it would be displayed unconditionally,
there’s no “I understand the risks” button one can click to make it
disappear.  I’d lean towards making a prominent warning in the manual
and not have a warning message every time here, dunno.

Thoughts?

Thanks for your feedback!

Ludo’.

[bug#60629] [PATCH 0/3] Add '--with-version' package transformation

[Simon Tournier]

Hi Ludo,

On ven., 13 janv. 2023 at 12:12, Ludovic Courtès <ludo@gnu.org> wrote:

> I wonder about this one because it would be displayed unconditionally,
> there’s no “I understand the risks” button one can click to make it
> disappear.  I’d lean towards making a prominent warning in the manual
> and not have a warning message every time here, dunno.

Yeah, I agree it will be visually polluting. :-)

Well, this patch set exposes the concern about reproducibility and some
package transformation.  Therefore, it could be addressed and discussed
separately since it is unique to this --with-version.

A first idea. :-)

Maybe, this warning could appears the first time the transformation is
called; similarly as “guix shell --check”.

Or if we want to be really stringent, the first time of transformation
calling, we simply raise ad message explaining it will not be
reproducible, ask an Yes/No “I understand the risk”, store the result in
a cache.  This makes sure the information is spread.

However, it could be annoying when launching in some non-interactive
mode.  The trick would be to set this cache but that’s not super handy.

Bah, it needs more thoughts. :-)


Cheers,
simon

bug#60629: [PATCH 0/3] Add '--with-version' package transformation

[Ludovic Courtès]

Hello,

Pushed!

  2e652e687e master origin/master news: Add entry for '--with-version'.
  20c923c40d transformations: Let users know when '--with-latest' has no effect.
  137b91f03b transformations: Add '--with-version'.

I added a warning in the manual as you suggested, Simon; I also added a
warning in ‘package-with-upstream-version’ about downgrades.  I included
Florian’s translation of the news entry too.

Thanks everyone!

Ludo’.