Re: A plan for parameterized packages

Re: A plan for parameterized packages

[Stephen Christie]

[-- Attachment #1: Type: text/plain, Size: 2661 bytes --]



> Hi Pierre,

> Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> > One of the biggest struggle we had when discussing it was figuring out
> > what to do about parameter propagation across dependencies.

> > For instance, what if we want to build "all packages without X support"?
> > This means that the parameter must traverse all inputs recursively if we
> > don't want to drag X indirectly.

> > If I understand your change correctly, the patch is only applying
> > parameters to the given package and it's not propagated to the inputs,
> > is that correct?

> That’s correct: in this patch set parameters are per-package, and I
think it’s easier to start simple, but we could implement what you
describe without too much hassle I think.

> For example, you’d type ‘--with-parameters=x11=false’, and that’d be
applied to all the packages that have an ‘x11’ parameter.

> Ludo’.

I have done a lot of work with the Conan package manager, a c++ language package manager, that has grown in capability. It is not fully functional, but works on the hash of the key parameters of the package (name, version, etc.) to find the right reproducible binary. Two important parameters are "options" and "settings".

Options are per-package, and generally affect none below it. You can specify defaults for the options in the package, and also call for specific options on dependencies in package "recipes". There are also ways to define incompatibilities and substitutes. On the command line, you can specify options with -o,--options, with no namespace needed for the package you are installing, and package:option to specify for other packages pulled in. I prefer this syntax to all the equal signs you proposed (though I defer if this is standard throughout Guix/Guile)

Settings are more "system-wide", though being a language package manager, it does not have a "system". The same settings are applied to the whole tree during an install, and are usually things like compiler, architecture, and build type. These settings are chosen through a profile file, of which there is a default generated for a given computer. Settings can also be set at the command line during install with -s,--settings, but of course there is no namespacing.

https://docs.conan.io/en/latest/mastering/conditional.html

I think there is a lot of good stuff in Conan that Guix could learn from. It's a lot closer in architecture than any of the traditional system package managers.

If you would, please forward this to the mailing list that this message was from, as I am not currently on that list.

Sincerely,
Stephen Christie

[-- Attachment #2: Type: text/html, Size: 2883 bytes --]

Re: A plan for parameterized packages

[Ludovic Courtès]

Hi Stephen,

(+Cc: guix-devel@gnu.org.  You can post without being subscribed.)

Stephen Christie <spc@e.email> skribis:

> I have done a lot of work with the Conan package manager, a c++
> language package manager, that has grown in capability. It is not
> fully functional, but works on the hash of the key parameters of the
> package (name, version, etc.) to find the right reproducible
> binary. Two important parameters are "options" and "settings".
>
> Options are per-package, and generally affect none below it. You can
> specify defaults for the options in the package, and also call for
> specific options on dependencies in package "recipes". There are also
> ways to define incompatibilities and substitutes. On the command line,
> you can specify options with -o,--options, with no namespace needed
> for the package you are installing, and package:option to specify for
> other packages pulled in. I prefer this syntax to all the equal signs
> you proposed (though I defer if this is standard throughout
> Guix/Guile)
>
> Settings are more "system-wide", though being a language package
> manager, it does not have a "system". The same settings are applied to
> the whole tree during an install, and are usually things like
> compiler, architecture, and build type. These settings are chosen
> through a profile file, of which there is a default generated for a
> given computer. Settings can also be set at the command line during
> install with -s,--settings, but of course there is no namespacing.
>
> https://docs.conan.io/en/latest/mastering/conditional.html
>
> I think there is a lot of good stuff in Conan that Guix could learn
> from. It's a lot closer in architecture than any of the traditional
> system package managers.

Thanks for sharing Conan’s perspective on these issues!

The settings/options distinction looks like a useful one.  Like Pierre
suggested, it’d be nice to have options that apply to the whole graph in
addition to per-package options like I was focusing on.

Conan’s approach to conflicting options may not be applicable to Guix.
For instance, the manual above has this example:

  def configure(self):
    # …
    if self.settings.os == "Windows":
       self.options["openssl"].shared = True

  def requirements(self):
      # Or add a new requirement!
      if self.options.testing:
         self.requires("OpenSSL/2.1@memsharded/testing")
      else:
         self.requires("openssl/1.0.2u")

In Guix, instead of stating that OpenSSL 1.0.2u is required or that it
needs to include shared libraries, you’d actually depend on a variant of
OpenSSL that fulfills these constraints; by construction, you can be
sure you have the intended OpenSSL variant (generally speaking, a Guix
package dependency graph has zero degrees of liberty, unlike an
apt/Spack/Conan graph.)

As for the syntax… yeah, we could find a shorthand.  :-)  The verbosity
in the examples I gave partly stems from the fact that these are
per-package parameters, so you need to specify which package it applies
to.  With “global” parameters, we could have, say:

  guix install -P x11=false emacs

meaning that the ‘x11’ parameter will be set to #f in all the packages
that have such a parameter.

Thanks,
Ludo’.

Re: A plan for parameterized packages

[Stephen Christie]

[-- Attachment #1: Type: text/plain, Size: 4324 bytes --]

Ludo,

The self.options["openssl"].shared does set the option for openssl, but this is not different than providing `-o openssl:shared` on the command line; it uses this to determine which package was actually requested. Conan packages are also non-configurable in the end, stored as the hash of it's name and parameters. This is probably done while calculating the tree before installing/building everything, but once that tree is calculated, it is fixed.

Besides parameters determining the hash of the package, OpenSSL/2.1@memsharded/testing and openssl/1.0.2u are most certainly separate packages. Conan has this name/version@username/channel structure to help find alternatives (semver is supported), but otherwise those are completely different.

On November 17, 2020 10:31:12 AM EST, "Ludovic Courtès" <ludo@gnu.org> wrote:
>Hi Stephen,
>
>(+Cc: guix-devel@gnu.org.  You can post without being subscribed.)
>
>Stephen Christie <spc@e.email> skribis:
>
>> I have done a lot of work with the Conan package manager, a c++
>> language package manager, that has grown in capability. It is not
>> fully functional, but works on the hash of the key parameters of the
>> package (name, version, etc.) to find the right reproducible
>> binary. Two important parameters are "options" and "settings".
>>
>> Options are per-package, and generally affect none below it. You can
>> specify defaults for the options in the package, and also call for
>> specific options on dependencies in package "recipes". There are also
>> ways to define incompatibilities and substitutes. On the command
>line,
>> you can specify options with -o,--options, with no namespace needed
>> for the package you are installing, and package:option to specify for
>> other packages pulled in. I prefer this syntax to all the equal signs
>> you proposed (though I defer if this is standard throughout
>> Guix/Guile)
>>
>> Settings are more "system-wide", though being a language package
>> manager, it does not have a "system". The same settings are applied
>to
>> the whole tree during an install, and are usually things like
>> compiler, architecture, and build type. These settings are chosen
>> through a profile file, of which there is a default generated for a
>> given computer. Settings can also be set at the command line during
>> install with -s,--settings, but of course there is no namespacing.
>>
>> https://docs.conan.io/en/latest/mastering/conditional.html
>>
>> I think there is a lot of good stuff in Conan that Guix could learn
>> from. It's a lot closer in architecture than any of the traditional
>> system package managers.
>
>Thanks for sharing Conan’s perspective on these issues!
>
>The settings/options distinction looks like a useful one.  Like Pierre
>suggested, it’d be nice to have options that apply to the whole graph
>in
>addition to per-package options like I was focusing on.
>
>Conan’s approach to conflicting options may not be applicable to Guix.
>For instance, the manual above has this example:
>
>  def configure(self):
>    # …
>    if self.settings.os == "Windows":
>       self.options["openssl"].shared = True
>
>  def requirements(self):
>      # Or add a new requirement!
>      if self.options.testing:
>         self.requires("OpenSSL/2.1@memsharded/testing")
>      else:
>         self.requires("openssl/1.0.2u")
>
>In Guix, instead of stating that OpenSSL 1.0.2u is required or that it
>needs to include shared libraries, you’d actually depend on a variant
>of
>OpenSSL that fulfills these constraints; by construction, you can be
>sure you have the intended OpenSSL variant (generally speaking, a Guix
>package dependency graph has zero degrees of liberty, unlike an
>apt/Spack/Conan graph.)
>
>As for the syntax… yeah, we could find a shorthand.  :-)  The verbosity
>in the examples I gave partly stems from the fact that these are
>per-package parameters, so you need to specify which package it applies
>to.  With “global” parameters, we could have, say:
>
>  guix install -P x11=false emacs
>
>meaning that the ‘x11’ parameter will be set to #f in all the packages
>that have such a parameter.
>
>Thanks,
>Ludo’.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[-- Attachment #2: Type: text/html, Size: 4833 bytes --]

A plan for parameterized packages

[Ludovic Courtès]

[-- Attachment #1.1: Type: text/plain, Size: 4426 bytes --]

Hello Guix!

For some time we’ve discussed ways to achieve “parameterized
packages”—packages where one can from the command line or from Scheme
configure optional build-time features, similar to Gentoo “USE flags”.

I still have mixed feeling about this feature: on one hand it can bring
much welcome flexibility, but on the other hand it can also lead us to
the wild west of untested package configurations and combinatorial
explosion.  It could also be argued that we achieve something similar
today by simply defining package variants when we have to:

  https://guix.gnu.org/manual/devel/en/html_node/Defining-Package-Variants.html

That said, this message is about a possible implementation of package
parameters, so here we go.  :-)

To me the requirements for package parameters are:

  1. it must be possible to discover them and choose them from the UI;

  2. they must contain on-line internationalized documentation such that
     the UI can list a package’s parameters and their type;

  3. the chosen parameters when installing a package in a profile must
     be preserved;

  4. it must be possible to enumerate all the possible values of a
     parameter, and thus to build the Cartesian product of all the
     possible parameter combinations of a package (or of a package
     graph!), so we can test those combinations as much as possible.

This leads to the patches below.  The last one gives an example use for
Bitlbee: it adds a “libpurple” parameter that allows users to choose
whether or not to enable the optional libpurple dependency:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libpurple=true -n
The following derivation would be built:
   /gnu/store/mkknqgjsa93ajcl5d2krngizb11j1b0q-bitlbee-3.6.drv
$ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libpurple=false -n
/gnu/store/c2ckg51ffwgs6jni3l549k06w3jd3b7a-bitlbee-3.6
$ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libpurple=wat? -n
guix build: error: wrong value
$ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libviolet=true -n
gnu/packages/messaging.scm:283:5: error: libviolet: no such package parameter
$ ./pre-inst-env guix show bitlbee
name: bitlbee
version: 3.6
outputs: out
parameters: libpurple
systems: x86_64-linux i686-linux
dependencies: check@0.12.0 glib@2.62.6 gnutls@3.6.12 libotr@4.1.1 perl@5.30.2 pkg-config@0.29.2 python@3.8.2
location: gnu/packages/messaging.scm:243:2
homepage: https://www.bitlbee.org/
license: GPL 2+, FreeBSD
synopsis: IRC to instant messaging gateway  
description: BitlBee brings IM (instant messaging) to IRC clients, for people who have an IRC client running all
+ the time and don't want to run an additional IM client.  BitlBee currently supports XMPP/Jabber (including Google
+ Talk), MSN Messenger, Yahoo! Messenger, AIM and ICQ, and the Twitter microblogging network (plus all other Twitter
+ API compatible services like identi.ca and status.net).

$ ./pre-inst-env guix install bitlbee --with-parameter=bitlbee=libpurple=true -p /tmp/test-bitlbee
The following package will be installed:
   bitlbee 3.6

The following derivation will be built:
   /gnu/store/clvs5521v5ybdw1z1yh97z2ky1dxm4d9-bitlbee-3.6.drv

[...]

$ cat /tmp/test-bitlbee/manifest
;; This file was automatically generated and is for internal use only.
;; It cannot be passed to the '--manifest' option.

(manifest
  (version 3)
  (packages
    (("bitlbee"
      "3.6"
      "out"
      "/gnu/store/d67r9k5hwfm5hkd1d3pbhg49fcc2jj4q-bitlbee-3.6"
      (propagated-inputs ())
      (search-paths ())
      (properties
        (transformations
          (with-parameter . "bitlbee=libpurple=true")))))))
--8<---------------cut here---------------end--------------->8---

That’s it!

We would need more things, like a ‘guix parameters’ command to show the
available parameters of a package and an ‘--all-parameter-values’ option
to ‘guix build’ to build all the variants of a given package.

An important question: do we have examples of packages for which we’d
like to have parameters?  I’d grepped for “inherit” and that yields a
few potential candidates, but also maybe a few potential non-candidates.
Would this be a good fit for them?

Thoughts?

Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0003-ui-package-recutils-emits-parameters-field.patch --]
[-- Type: text/x-patch, Size: 1414 bytes --]

From 9155411f2e8e78922e1e46d92068ac8f652ff0a5 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Sun, 15 Nov 2020 17:01:14 +0100
Subject: [PATCH 3/4] ui: 'package->recutils' emits "parameters" field.

* guix/ui.scm (package->recutils): Add "parameters" recutils field.
---
 guix/ui.scm | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/guix/ui.scm b/guix/ui.scm
index 4e686297e8..2485400cc9 100644
--- a/guix/ui.scm
+++ b/guix/ui.scm
@@ -60,6 +60,7 @@
                 #:hide (package-name->name+version
                         ;; Avoid "overrides core binding" warning.
                         delete))
+  #:autoload   (guix parameters) (package-parameters package-parameter-name)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-9 gnu)
   #:use-module (srfi srfi-11)
@@ -1529,6 +1530,10 @@ HYPERLINKS? is true, emit hyperlink escape sequences when appropriate."
   (format port "name: ~a~%" (package-name p))
   (format port "version: ~a~%" (package-version p))
   (format port "outputs: ~a~%" (string-join (package-outputs p)))
+  (match (package-parameters p)
+    (() #t)
+    (lst (format port "parameters:~{ ~a~}~%"
+                 (map package-parameter-name lst))))
   (format port "systems: ~a~%"
           (string-join (package-transitive-supported-systems p)))
   (format port "dependencies: ~a~%"
-- 
2.29.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.3: 0004-gnu-bitlbee-Add-libpurple-parameter.patch --]
[-- Type: text/x-patch, Size: 3823 bytes --]

From 49d7746ada4d4674acbbfd2606ad9bff4f6207eb Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Sun, 15 Nov 2020 17:04:18 +0100
Subject: [PATCH 4/4] gnu: bitlbee: Add "libpurple" parameter.

* gnu/packages/messaging.scm (bitlbee)[inputs]: Add optional PIDGIN
input.
[properties]: New field.
(bitlbee-purple): Mark as deprecated.
---
 gnu/packages/messaging.scm | 43 +++++++++++++++++++-------------------
 1 file changed, 22 insertions(+), 21 deletions(-)

diff --git a/gnu/packages/messaging.scm b/gnu/packages/messaging.scm
index b462504894..2f0f44d10d 100644
--- a/gnu/packages/messaging.scm
+++ b/gnu/packages/messaging.scm
@@ -120,6 +120,7 @@
   #:use-module (guix build-system trivial)
   #:use-module (guix download)
   #:use-module (guix git-download)
+  #:use-module (guix parameters)
   #:use-module (guix hg-download)
   #:use-module ((guix licenses) #:prefix license:)
   #:use-module (guix packages)
@@ -256,7 +257,8 @@ end-to-end encryption.")
               ("libotr" ,libotr)
               ("gnutls" ,gnutls)
               ("python" ,python)
-              ("perl" ,perl)))
+              ("perl" ,perl)
+              ,@(optionally 'libpurple? `("purple" ,pidgin))))
     (arguments
      `(#:phases
        (modify-phases %standard-phases
@@ -275,7 +277,21 @@ end-to-end encryption.")
              (invoke "./configure"
                      (string-append "--prefix="
                                     (assoc-ref outputs "out"))
-                     "--otr=1"))))))
+                     "--otr=1"
+                     ,@(optionally 'libpurple? "--purple=1")))))
+
+       ;; XXX: Tests fail to link, and ./configure says that it's "supported
+       ;; on a best-effort basis" anyway.
+       #:tests? ,(not (assq-ref (package-properties this-package)
+                                'libpurple?))))
+    (properties
+     `((parameters
+        ,(package-parameter (name "libpurple")
+                            (description
+                             "Whether to enable libpurple (Pidgin)
+support.")
+                            (property 'libpurple?)
+                            (type boolean)))))
     (synopsis "IRC to instant messaging gateway")
     (description "BitlBee brings IM (instant messaging) to IRC clients, for
 people who have an IRC client running all the time and don't want to run an
@@ -289,25 +305,10 @@ identi.ca and status.net).")
 (define-public bitlbee-purple
   ;; This variant uses libpurple, which provides support for more protocols at
   ;; the expense of a much bigger closure.
-  (package/inherit bitlbee
-    (name "bitlbee-purple")
-    (synopsis "IRC to instant messaging gateway (using Pidgin's libpurple)")
-    (inputs `(("purple" ,pidgin)
-              ,@(package-inputs bitlbee)))
-    (arguments
-     (substitute-keyword-arguments (package-arguments bitlbee)
-       ((#:phases phases '%standard-phases)
-        `(modify-phases ,phases
-           (replace 'configure                    ;add "--purple=1"
-             (lambda* (#:key outputs #:allow-other-keys)
-               (invoke "./configure"
-                       (string-append "--prefix="
-                                      (assoc-ref outputs "out"))
-                       "--otr=1" "--purple=1")))))
-       ((#:tests? _ #t)
-        ;; XXX: Tests fail to link, and ./configure says that it's "supported
-        ;; on a best-effort basis" anyway.
-        #f)))))
+  (deprecated-package "bitlbee-purple"
+                      (package/inherit bitlbee
+                        (properties `((libpurple? . #t)
+                                      ,@(package-properties bitlbee))))))
 
 (define-public bitlbee-discord
   (package
-- 
2.29.2


[-- Attachment #1.4: 0001-DRAFT-Add-guix-parameters.patch --]
[-- Type: text/x-patch, Size: 6318 bytes --]

From f42b68499c4e2a9bd368fe6a516932f5afa7a189 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Sun, 15 Nov 2020 16:58:52 +0100
Subject: [PATCH 1/4] DRAFT Add (guix parameters).

DRAFT: Missing tests & doc.

* guix/parameters.scm: New file.
* Makefile.am (MODULES): Add it.
---
 Makefile.am         |   1 +
 guix/parameters.scm | 131 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 132 insertions(+)
 create mode 100644 guix/parameters.scm

diff --git a/Makefile.am b/Makefile.am
index e7053ee4f4..72f955360d 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -235,6 +235,7 @@ MODULES =					\
   guix/build/make-bootstrap.scm			\
   guix/search-paths.scm				\
   guix/packages.scm				\
+  guix/parameters.scm				\
   guix/import/cabal.scm				\
   guix/import/cpan.scm				\
   guix/import/cran.scm				\
diff --git a/guix/parameters.scm b/guix/parameters.scm
new file mode 100644
index 0000000000..e4f8240aa4
--- /dev/null
+++ b/guix/parameters.scm
@@ -0,0 +1,131 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2020 Ludovic Courtès <ludo@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix parameters)
+  #:use-module (guix packages)
+  #:use-module (guix records)
+  #:use-module (guix diagnostics)
+  #:use-module (guix i18n)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-34)
+  #:use-module (srfi srfi-35)
+  #:use-module (ice-9 match)
+  #:export (package-parameter
+            package-parameter?
+            package-parameter-name
+            package-parameter-property
+            package-parameter-type
+            package-parameter-description
+
+            boolean
+            optionally
+
+            package-parameters
+            lookup-package-parameter
+            package-parameter-value
+            set-package-parameter-value))
+
+;;; Commentary:
+;;;
+;;; This module provides a way to express high-level "package parameters",
+;;; which allow users to customize how packages are built.  Parameters are an
+;;; interface that package developers define, where each parameter has a name
+;;; and type.  The user interface then converts parameter values from string
+;;; to Scheme values and records them in the package properties.
+;;;
+;;; Package parameters are discoverable; their description is
+;;; internationalized.  The possible values of a parameter can be enumerated,
+;;; and thus the Cartesian product of all possible parameter values for a
+;;; package can be enumerated as well.
+;;;
+;;; Code:
+
+;; Package parameter interface.
+(define-record-type* <package-parameter> package-parameter
+  make-package-parameter
+  package-parameter?
+  (name          package-parameter-name)
+  (property      package-parameter-property (default (string->symbol name)))
+  (type          package-parameter-type)
+  (description   package-parameter-description))
+
+;; Type of a package parameter.
+(define-record-type* <parameter-type> parameter-type
+  make-parameter-type
+  parameter-type?
+  (name          parameter-type-name)              ;debugging purposes only!
+  (string->value parameter-type-string->value)
+  (value->string parameter-type-value->string)
+  (universe      parameter-type-universe))
+
+(define boolean
+  ;; The Boolean parameter type.
+  (parameter-type (name 'boolean)
+                  (universe '(#true #false))
+                  (value->string
+                   (match-lambda
+                     (#f "false")
+                     (#t "true")))
+                  (string->value
+                   (lambda (str)
+                     (cond ((string-ci=? str "true")
+                            #t)
+                           ((string-ci=? str "false")
+                            #f)
+                           (else
+                            (raise (condition
+                                    (&message (message "wrong value"))))))))))
+
+(define (package-parameters package)
+  (or (assq-ref (package-properties package) 'parameters)
+      '()))
+
+(define (package-parameter-value package parameter)
+  (assq-ref (package-properties package)
+            (package-parameter-property parameter)))
+
+(define (lookup-package-parameter package name)
+  (find (lambda (parameter)
+          (string=? (package-parameter-name parameter) name))
+        (package-parameters package)))
+
+(define (set-package-parameter-value package name value)
+  (let ((parameter (lookup-package-parameter package name))
+        (location  (package-field-location package 'properties)))
+    (unless parameter
+      (raise (apply make-compound-condition
+                    (formatted-message
+                     (G_ "~a: no such package parameter")
+                     name)
+                    (if location
+                        (list (condition
+                               (&error-location (location location))))
+                        '()))))
+    (let* ((property (package-parameter-property parameter))
+           (type     (package-parameter-type parameter))
+           (value    ((parameter-type-string->value type) value)))
+      (package/inherit package
+        (properties
+         (alist-cons property value
+                     (alist-delete property (package-properties package)
+                                   eq?)))))))
+
+(define-syntax-rule (optionally property exp)
+  (if (assq-ref (package-properties this-package) property)
+      (list exp)
+      '()))
-- 
2.29.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.5: 0002-DRAFT-transformations-Add-with-parameter.patch --]
[-- Type: text/x-patch, Size: 3849 bytes --]

From 5d6d81e4c3e37de53fe7f62ff7ef94da8b2df033 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Sun, 15 Nov 2020 16:59:48 +0100
Subject: [PATCH 2/4] DRAFT transformations: Add '--with-parameter'.

DRAFT: Missing tests & doc.

* guix/transformations.scm (evaluate-parameter-specs)
(transform-package-parameters): New procedures.
(%transformations, %transformation-options): Add 'with-parameter'.
---
 guix/transformations.scm | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/guix/transformations.scm b/guix/transformations.scm
index 30142dd059..0f83eb470d 100644
--- a/guix/transformations.scm
+++ b/guix/transformations.scm
@@ -25,6 +25,7 @@
   #:autoload   (guix download) (download-to-store)
   #:autoload   (guix git-download) (git-reference? git-reference-url)
   #:autoload   (guix git) (git-checkout git-checkout? git-checkout-url)
+  #:autoload   (guix parameters) (set-package-parameter-value)
   #:use-module (guix utils)
   #:use-module (guix memoization)
   #:use-module (guix gexp)
@@ -324,6 +325,41 @@ a checkout of the Git repository at the given URL."
         (rewrite obj)
         obj)))
 
+(define (evaluate-parameter-specs specs proc)
+  "Parse SPECS, a list of strings like \"bitlbee=purple=true\", and return a
+list of spec/procedure pairs, where (PROC PACKAGE PARAMETER VALUE) is called
+to return the replacement package.  Raise an error if an element of SPECS uses
+invalid syntax, or if a package it refers to could not be found."
+  (map (lambda (spec)
+         (match (string-tokenize spec %not-equal)
+           ((spec name value)
+            (define (replace old)
+              (proc old name value))
+
+            (cons spec replace))
+           (_
+            (raise
+             (formatted-message
+              (G_ "invalid package parameter specification: ~s")
+              spec)))))
+       specs))
+
+(define (transform-package-parameters replacement-specs)
+  "Return a procedure that, when passed a package, replaces its direct
+dependencies according to REPLACEMENT-SPECS.  REPLACEMENT-SPECS is a list of
+strings like \"guile-next=stable-3.0\" meaning that packages are built using
+'guile-next' from the latest commit on its 'stable-3.0' branch."
+  (define (replace old name value)
+    (set-package-parameter-value old name value))
+
+  (let* ((replacements (evaluate-parameter-specs replacement-specs
+                                                 replace))
+         (rewrite      (package-input-rewriting/spec replacements)))
+    (lambda (obj)
+      (if (package? obj)
+          (rewrite obj)
+          obj))))
+
 (define (package-dependents/spec top bottom)
   "Return the list of dependents of BOTTOM, a spec string, that are also
 dependencies of TOP, a package."
@@ -467,6 +503,7 @@ to the same package but with #:strip-binaries? #f in its 'arguments' field."
     (with-branch . ,transform-package-source-branch)
     (with-commit . ,transform-package-source-commit)
     (with-git-url . ,transform-package-source-git-url)
+    (with-parameter . ,transform-package-parameters)
     (with-c-toolchain . ,transform-package-toolchain)
     (with-debug-info . ,transform-package-with-debug-info)
     (without-tests . ,transform-package-tests)))
@@ -503,6 +540,8 @@ to the same package but with #:strip-binaries? #f in its 'arguments' field."
                   (parser 'with-commit))
           (option '("with-git-url") #t #f
                   (parser 'with-git-url))
+          (option '("with-parameter") #t #f
+                  (parser 'with-parameter))
           (option '("with-c-toolchain") #t #f
                   (parser 'with-c-toolchain))
           (option '("with-debug-info") #t #f
-- 
2.29.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 853 bytes --]

Re: A plan for parameterized packages

[Nicolò Balzarotti]

Neat! Thanks for working on it :)

> An important question: do we have examples of packages for which we’d
> like to have parameters?  I’d grepped for “inherit” and that yields a
> few potential candidates, but also maybe a few potential non-candidates.
> Would this be a good fit for them?

What about evince?  eps/ps/dvi rendering is disabled, according to this
thread [1] for security reasons, so it makes sense to have it disabled
by default.  But I'd like to be able to enable it easily.

WDYT?

Nicolò

[1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=39674

Ludovic Courtès <ludo@gnu.org> writes:

> Hello Guix!
>
> For some time we’ve discussed ways to achieve “parameterized
> packages”—packages where one can from the command line or from Scheme
> configure optional build-time features, similar to Gentoo “USE flags”.
>
> I still have mixed feeling about this feature: on one hand it can bring
> much welcome flexibility, but on the other hand it can also lead us to
> the wild west of untested package configurations and combinatorial
> explosion.  It could also be argued that we achieve something similar
> today by simply defining package variants when we have to:
>
>   https://guix.gnu.org/manual/devel/en/html_node/Defining-Package-Variants.html
>
> That said, this message is about a possible implementation of package
> parameters, so here we go.  :-)
>
> To me the requirements for package parameters are:
>
>   1. it must be possible to discover them and choose them from the UI;
>
>   2. they must contain on-line internationalized documentation such that
>      the UI can list a package’s parameters and their type;
>
>   3. the chosen parameters when installing a package in a profile must
>      be preserved;
>
>   4. it must be possible to enumerate all the possible values of a
>      parameter, and thus to build the Cartesian product of all the
>      possible parameter combinations of a package (or of a package
>      graph!), so we can test those combinations as much as possible.
>
> This leads to the patches below.  The last one gives an example use for
> Bitlbee: it adds a “libpurple” parameter that allows users to choose
> whether or not to enable the optional libpurple dependency:
>
> --8<---------------cut here---------------start------------->8---
> $ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libpurple=true -n
> The following derivation would be built:
>    /gnu/store/mkknqgjsa93ajcl5d2krngizb11j1b0q-bitlbee-3.6.drv
> $ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libpurple=false -n
> /gnu/store/c2ckg51ffwgs6jni3l549k06w3jd3b7a-bitlbee-3.6
> $ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libpurple=wat? -n
> guix build: error: wrong value
> $ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libviolet=true -n
> gnu/packages/messaging.scm:283:5: error: libviolet: no such package parameter
> $ ./pre-inst-env guix show bitlbee
> name: bitlbee
> version: 3.6
> outputs: out
> parameters: libpurple
> systems: x86_64-linux i686-linux
> dependencies: check@0.12.0 glib@2.62.6 gnutls@3.6.12 libotr@4.1.1 perl@5.30.2 pkg-config@0.29.2 python@3.8.2
> location: gnu/packages/messaging.scm:243:2
> homepage: https://www.bitlbee.org/
> license: GPL 2+, FreeBSD
> synopsis: IRC to instant messaging gateway  
> description: BitlBee brings IM (instant messaging) to IRC clients, for people who have an IRC client running all
> + the time and don't want to run an additional IM client.  BitlBee currently supports XMPP/Jabber (including Google
> + Talk), MSN Messenger, Yahoo! Messenger, AIM and ICQ, and the Twitter microblogging network (plus all other Twitter
> + API compatible services like identi.ca and status.net).
>
> $ ./pre-inst-env guix install bitlbee --with-parameter=bitlbee=libpurple=true -p /tmp/test-bitlbee
> The following package will be installed:
>    bitlbee 3.6
>
> The following derivation will be built:
>    /gnu/store/clvs5521v5ybdw1z1yh97z2ky1dxm4d9-bitlbee-3.6.drv
>
> [...]
>
> $ cat /tmp/test-bitlbee/manifest
> ;; This file was automatically generated and is for internal use only.
> ;; It cannot be passed to the '--manifest' option.
>
> (manifest
>   (version 3)
>   (packages
>     (("bitlbee"
>       "3.6"
>       "out"
>       "/gnu/store/d67r9k5hwfm5hkd1d3pbhg49fcc2jj4q-bitlbee-3.6"
>       (propagated-inputs ())
>       (search-paths ())
>       (properties
>         (transformations
>           (with-parameter . "bitlbee=libpurple=true")))))))
> --8<---------------cut here---------------end--------------->8---
>
> That’s it!
>
> We would need more things, like a ‘guix parameters’ command to show the
> available parameters of a package and an ‘--all-parameter-values’ option
> to ‘guix build’ to build all the variants of a given package.
>
> An important question: do we have examples of packages for which we’d
> like to have parameters?  I’d grepped for “inherit” and that yields a
> few potential candidates, but also maybe a few potential non-candidates.
> Would this be a good fit for them?
>
> Thoughts?
>
> Ludo’.
>
> From 9155411f2e8e78922e1e46d92068ac8f652ff0a5 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
> Date: Sun, 15 Nov 2020 17:01:14 +0100
> Subject: [PATCH 3/4] ui: 'package->recutils' emits "parameters" field.
>
> * guix/ui.scm (package->recutils): Add "parameters" recutils field.
> ---
>  guix/ui.scm | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/guix/ui.scm b/guix/ui.scm
> index 4e686297e8..2485400cc9 100644
> --- a/guix/ui.scm
> +++ b/guix/ui.scm
> @@ -60,6 +60,7 @@
>                  #:hide (package-name->name+version
>                          ;; Avoid "overrides core binding" warning.
>                          delete))
> +  #:autoload   (guix parameters) (package-parameters package-parameter-name)
>    #:use-module (srfi srfi-1)
>    #:use-module (srfi srfi-9 gnu)
>    #:use-module (srfi srfi-11)
> @@ -1529,6 +1530,10 @@ HYPERLINKS? is true, emit hyperlink escape sequences when appropriate."
>    (format port "name: ~a~%" (package-name p))
>    (format port "version: ~a~%" (package-version p))
>    (format port "outputs: ~a~%" (string-join (package-outputs p)))
> +  (match (package-parameters p)
> +    (() #t)
> +    (lst (format port "parameters:~{ ~a~}~%"
> +                 (map package-parameter-name lst))))
>    (format port "systems: ~a~%"
>            (string-join (package-transitive-supported-systems p)))
>    (format port "dependencies: ~a~%"
> -- 
> 2.29.2
>
> From 49d7746ada4d4674acbbfd2606ad9bff4f6207eb Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
> Date: Sun, 15 Nov 2020 17:04:18 +0100
> Subject: [PATCH 4/4] gnu: bitlbee: Add "libpurple" parameter.
>
> * gnu/packages/messaging.scm (bitlbee)[inputs]: Add optional PIDGIN
> input.
> [properties]: New field.
> (bitlbee-purple): Mark as deprecated.
> ---
>  gnu/packages/messaging.scm | 43 +++++++++++++++++++-------------------
>  1 file changed, 22 insertions(+), 21 deletions(-)
>
> diff --git a/gnu/packages/messaging.scm b/gnu/packages/messaging.scm
> index b462504894..2f0f44d10d 100644
> --- a/gnu/packages/messaging.scm
> +++ b/gnu/packages/messaging.scm
> @@ -120,6 +120,7 @@
>    #:use-module (guix build-system trivial)
>    #:use-module (guix download)
>    #:use-module (guix git-download)
> +  #:use-module (guix parameters)
>    #:use-module (guix hg-download)
>    #:use-module ((guix licenses) #:prefix license:)
>    #:use-module (guix packages)
> @@ -256,7 +257,8 @@ end-to-end encryption.")
>                ("libotr" ,libotr)
>                ("gnutls" ,gnutls)
>                ("python" ,python)
> -              ("perl" ,perl)))
> +              ("perl" ,perl)
> +              ,@(optionally 'libpurple? `("purple" ,pidgin))))
>      (arguments
>       `(#:phases
>         (modify-phases %standard-phases
> @@ -275,7 +277,21 @@ end-to-end encryption.")
>               (invoke "./configure"
>                       (string-append "--prefix="
>                                      (assoc-ref outputs "out"))
> -                     "--otr=1"))))))
> +                     "--otr=1"
> +                     ,@(optionally 'libpurple? "--purple=1")))))
> +
> +       ;; XXX: Tests fail to link, and ./configure says that it's "supported
> +       ;; on a best-effort basis" anyway.
> +       #:tests? ,(not (assq-ref (package-properties this-package)
> +                                'libpurple?))))
> +    (properties
> +     `((parameters
> +        ,(package-parameter (name "libpurple")
> +                            (description
> +                             "Whether to enable libpurple (Pidgin)
> +support.")
> +                            (property 'libpurple?)
> +                            (type boolean)))))
>      (synopsis "IRC to instant messaging gateway")
>      (description "BitlBee brings IM (instant messaging) to IRC clients, for
>  people who have an IRC client running all the time and don't want to run an
> @@ -289,25 +305,10 @@ identi.ca and status.net).")
>  (define-public bitlbee-purple
>    ;; This variant uses libpurple, which provides support for more protocols at
>    ;; the expense of a much bigger closure.
> -  (package/inherit bitlbee
> -    (name "bitlbee-purple")
> -    (synopsis "IRC to instant messaging gateway (using Pidgin's libpurple)")
> -    (inputs `(("purple" ,pidgin)
> -              ,@(package-inputs bitlbee)))
> -    (arguments
> -     (substitute-keyword-arguments (package-arguments bitlbee)
> -       ((#:phases phases '%standard-phases)
> -        `(modify-phases ,phases
> -           (replace 'configure                    ;add "--purple=1"
> -             (lambda* (#:key outputs #:allow-other-keys)
> -               (invoke "./configure"
> -                       (string-append "--prefix="
> -                                      (assoc-ref outputs "out"))
> -                       "--otr=1" "--purple=1")))))
> -       ((#:tests? _ #t)
> -        ;; XXX: Tests fail to link, and ./configure says that it's "supported
> -        ;; on a best-effort basis" anyway.
> -        #f)))))
> +  (deprecated-package "bitlbee-purple"
> +                      (package/inherit bitlbee
> +                        (properties `((libpurple? . #t)
> +                                      ,@(package-properties bitlbee))))))
>  
>  (define-public bitlbee-discord
>    (package
> -- 
> 2.29.2
>
> From f42b68499c4e2a9bd368fe6a516932f5afa7a189 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
> Date: Sun, 15 Nov 2020 16:58:52 +0100
> Subject: [PATCH 1/4] DRAFT Add (guix parameters).
>
> DRAFT: Missing tests & doc.
>
> * guix/parameters.scm: New file.
> * Makefile.am (MODULES): Add it.
> ---
>  Makefile.am         |   1 +
>  guix/parameters.scm | 131 ++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 132 insertions(+)
>  create mode 100644 guix/parameters.scm
>
> diff --git a/Makefile.am b/Makefile.am
> index e7053ee4f4..72f955360d 100644
> --- a/Makefile.am
> +++ b/Makefile.am
> @@ -235,6 +235,7 @@ MODULES =					\
>    guix/build/make-bootstrap.scm			\
>    guix/search-paths.scm				\
>    guix/packages.scm				\
> +  guix/parameters.scm				\
>    guix/import/cabal.scm				\
>    guix/import/cpan.scm				\
>    guix/import/cran.scm				\
> diff --git a/guix/parameters.scm b/guix/parameters.scm
> new file mode 100644
> index 0000000000..e4f8240aa4
> --- /dev/null
> +++ b/guix/parameters.scm
> @@ -0,0 +1,131 @@
> +;;; GNU Guix --- Functional package management for GNU
> +;;; Copyright © 2020 Ludovic Courtès <ludo@gnu.org>
> +;;;
> +;;; This file is part of GNU Guix.
> +;;;
> +;;; GNU Guix is free software; you can redistribute it and/or modify it
> +;;; under the terms of the GNU General Public License as published by
> +;;; the Free Software Foundation; either version 3 of the License, or (at
> +;;; your option) any later version.
> +;;;
> +;;; GNU Guix is distributed in the hope that it will be useful, but
> +;;; WITHOUT ANY WARRANTY; without even the implied warranty of
> +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +;;; GNU General Public License for more details.
> +;;;
> +;;; You should have received a copy of the GNU General Public License
> +;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
> +
> +(define-module (guix parameters)
> +  #:use-module (guix packages)
> +  #:use-module (guix records)
> +  #:use-module (guix diagnostics)
> +  #:use-module (guix i18n)
> +  #:use-module (srfi srfi-1)
> +  #:use-module (srfi srfi-34)
> +  #:use-module (srfi srfi-35)
> +  #:use-module (ice-9 match)
> +  #:export (package-parameter
> +            package-parameter?
> +            package-parameter-name
> +            package-parameter-property
> +            package-parameter-type
> +            package-parameter-description
> +
> +            boolean
> +            optionally
> +
> +            package-parameters
> +            lookup-package-parameter
> +            package-parameter-value
> +            set-package-parameter-value))
> +
> +;;; Commentary:
> +;;;
> +;;; This module provides a way to express high-level "package parameters",
> +;;; which allow users to customize how packages are built.  Parameters are an
> +;;; interface that package developers define, where each parameter has a name
> +;;; and type.  The user interface then converts parameter values from string
> +;;; to Scheme values and records them in the package properties.
> +;;;
> +;;; Package parameters are discoverable; their description is
> +;;; internationalized.  The possible values of a parameter can be enumerated,
> +;;; and thus the Cartesian product of all possible parameter values for a
> +;;; package can be enumerated as well.
> +;;;
> +;;; Code:
> +
> +;; Package parameter interface.
> +(define-record-type* <package-parameter> package-parameter
> +  make-package-parameter
> +  package-parameter?
> +  (name          package-parameter-name)
> +  (property      package-parameter-property (default (string->symbol name)))
> +  (type          package-parameter-type)
> +  (description   package-parameter-description))
> +
> +;; Type of a package parameter.
> +(define-record-type* <parameter-type> parameter-type
> +  make-parameter-type
> +  parameter-type?
> +  (name          parameter-type-name)              ;debugging purposes only!
> +  (string->value parameter-type-string->value)
> +  (value->string parameter-type-value->string)
> +  (universe      parameter-type-universe))
> +
> +(define boolean
> +  ;; The Boolean parameter type.
> +  (parameter-type (name 'boolean)
> +                  (universe '(#true #false))
> +                  (value->string
> +                   (match-lambda
> +                     (#f "false")
> +                     (#t "true")))
> +                  (string->value
> +                   (lambda (str)
> +                     (cond ((string-ci=? str "true")
> +                            #t)
> +                           ((string-ci=? str "false")
> +                            #f)
> +                           (else
> +                            (raise (condition
> +                                    (&message (message "wrong value"))))))))))
> +
> +(define (package-parameters package)
> +  (or (assq-ref (package-properties package) 'parameters)
> +      '()))
> +
> +(define (package-parameter-value package parameter)
> +  (assq-ref (package-properties package)
> +            (package-parameter-property parameter)))
> +
> +(define (lookup-package-parameter package name)
> +  (find (lambda (parameter)
> +          (string=? (package-parameter-name parameter) name))
> +        (package-parameters package)))
> +
> +(define (set-package-parameter-value package name value)
> +  (let ((parameter (lookup-package-parameter package name))
> +        (location  (package-field-location package 'properties)))
> +    (unless parameter
> +      (raise (apply make-compound-condition
> +                    (formatted-message
> +                     (G_ "~a: no such package parameter")
> +                     name)
> +                    (if location
> +                        (list (condition
> +                               (&error-location (location location))))
> +                        '()))))
> +    (let* ((property (package-parameter-property parameter))
> +           (type     (package-parameter-type parameter))
> +           (value    ((parameter-type-string->value type) value)))
> +      (package/inherit package
> +        (properties
> +         (alist-cons property value
> +                     (alist-delete property (package-properties package)
> +                                   eq?)))))))
> +
> +(define-syntax-rule (optionally property exp)
> +  (if (assq-ref (package-properties this-package) property)
> +      (list exp)
> +      '()))
> -- 
> 2.29.2
>
> From 5d6d81e4c3e37de53fe7f62ff7ef94da8b2df033 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
> Date: Sun, 15 Nov 2020 16:59:48 +0100
> Subject: [PATCH 2/4] DRAFT transformations: Add '--with-parameter'.
>
> DRAFT: Missing tests & doc.
>
> * guix/transformations.scm (evaluate-parameter-specs)
> (transform-package-parameters): New procedures.
> (%transformations, %transformation-options): Add 'with-parameter'.
> ---
>  guix/transformations.scm | 39 +++++++++++++++++++++++++++++++++++++++
>  1 file changed, 39 insertions(+)
>
> diff --git a/guix/transformations.scm b/guix/transformations.scm
> index 30142dd059..0f83eb470d 100644
> --- a/guix/transformations.scm
> +++ b/guix/transformations.scm
> @@ -25,6 +25,7 @@
>    #:autoload   (guix download) (download-to-store)
>    #:autoload   (guix git-download) (git-reference? git-reference-url)
>    #:autoload   (guix git) (git-checkout git-checkout? git-checkout-url)
> +  #:autoload   (guix parameters) (set-package-parameter-value)
>    #:use-module (guix utils)
>    #:use-module (guix memoization)
>    #:use-module (guix gexp)
> @@ -324,6 +325,41 @@ a checkout of the Git repository at the given URL."
>          (rewrite obj)
>          obj)))
>  
> +(define (evaluate-parameter-specs specs proc)
> +  "Parse SPECS, a list of strings like \"bitlbee=purple=true\", and return a
> +list of spec/procedure pairs, where (PROC PACKAGE PARAMETER VALUE) is called
> +to return the replacement package.  Raise an error if an element of SPECS uses
> +invalid syntax, or if a package it refers to could not be found."
> +  (map (lambda (spec)
> +         (match (string-tokenize spec %not-equal)
> +           ((spec name value)
> +            (define (replace old)
> +              (proc old name value))
> +
> +            (cons spec replace))
> +           (_
> +            (raise
> +             (formatted-message
> +              (G_ "invalid package parameter specification: ~s")
> +              spec)))))
> +       specs))
> +
> +(define (transform-package-parameters replacement-specs)
> +  "Return a procedure that, when passed a package, replaces its direct
> +dependencies according to REPLACEMENT-SPECS.  REPLACEMENT-SPECS is a list of
> +strings like \"guile-next=stable-3.0\" meaning that packages are built using
> +'guile-next' from the latest commit on its 'stable-3.0' branch."
> +  (define (replace old name value)
> +    (set-package-parameter-value old name value))
> +
> +  (let* ((replacements (evaluate-parameter-specs replacement-specs
> +                                                 replace))
> +         (rewrite      (package-input-rewriting/spec replacements)))
> +    (lambda (obj)
> +      (if (package? obj)
> +          (rewrite obj)
> +          obj))))
> +
>  (define (package-dependents/spec top bottom)
>    "Return the list of dependents of BOTTOM, a spec string, that are also
>  dependencies of TOP, a package."
> @@ -467,6 +503,7 @@ to the same package but with #:strip-binaries? #f in its 'arguments' field."
>      (with-branch . ,transform-package-source-branch)
>      (with-commit . ,transform-package-source-commit)
>      (with-git-url . ,transform-package-source-git-url)
> +    (with-parameter . ,transform-package-parameters)
>      (with-c-toolchain . ,transform-package-toolchain)
>      (with-debug-info . ,transform-package-with-debug-info)
>      (without-tests . ,transform-package-tests)))
> @@ -503,6 +540,8 @@ to the same package but with #:strip-binaries? #f in its 'arguments' field."
>                    (parser 'with-commit))
>            (option '("with-git-url") #t #f
>                    (parser 'with-git-url))
> +          (option '("with-parameter") #t #f
> +                  (parser 'with-parameter))
>            (option '("with-c-toolchain") #t #f
>                    (parser 'with-c-toolchain))
>            (option '("with-debug-info") #t #f
> -- 
> 2.29.2

Re: A plan for parameterized packages

[Nicolò Balzarotti]

More in general, grepping for `--enable` or `--with` flags shows
a lot of potential packages (but also many false positives).


Nicolò Balzarotti <anothersms@gmail.com> writes:

> Neat! Thanks for working on it :)
>
>> An important question: do we have examples of packages for which we’d
>> like to have parameters?  I’d grepped for “inherit” and that yields a
>> few potential candidates, but also maybe a few potential non-candidates.
>> Would this be a good fit for them?
>
> What about evince?  eps/ps/dvi rendering is disabled, according to this
> thread [1] for security reasons, so it makes sense to have it disabled
> by default.  But I'd like to be able to enable it easily.
>
> WDYT?
>
> Nicolò
>
> [1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=39674
>
> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Hello Guix!
>>
>> For some time we’ve discussed ways to achieve “parameterized
>> packages”—packages where one can from the command line or from Scheme
>> configure optional build-time features, similar to Gentoo “USE flags”.
>>
>> I still have mixed feeling about this feature: on one hand it can bring
>> much welcome flexibility, but on the other hand it can also lead us to
>> the wild west of untested package configurations and combinatorial
>> explosion.  It could also be argued that we achieve something similar
>> today by simply defining package variants when we have to:
>>
>>   https://guix.gnu.org/manual/devel/en/html_node/Defining-Package-Variants.html
>>
>> That said, this message is about a possible implementation of package
>> parameters, so here we go.  :-)
>>
>> To me the requirements for package parameters are:
>>
>>   1. it must be possible to discover them and choose them from the UI;
>>
>>   2. they must contain on-line internationalized documentation such that
>>      the UI can list a package’s parameters and their type;
>>
>>   3. the chosen parameters when installing a package in a profile must
>>      be preserved;
>>
>>   4. it must be possible to enumerate all the possible values of a
>>      parameter, and thus to build the Cartesian product of all the
>>      possible parameter combinations of a package (or of a package
>>      graph!), so we can test those combinations as much as possible.
>>
>> This leads to the patches below.  The last one gives an example use for
>> Bitlbee: it adds a “libpurple” parameter that allows users to choose
>> whether or not to enable the optional libpurple dependency:
>>
>> --8<---------------cut here---------------start------------->8---
>> $ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libpurple=true -n
>> The following derivation would be built:
>>    /gnu/store/mkknqgjsa93ajcl5d2krngizb11j1b0q-bitlbee-3.6.drv
>> $ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libpurple=false -n
>> /gnu/store/c2ckg51ffwgs6jni3l549k06w3jd3b7a-bitlbee-3.6
>> $ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libpurple=wat? -n
>> guix build: error: wrong value
>> $ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libviolet=true -n
>> gnu/packages/messaging.scm:283:5: error: libviolet: no such package parameter
>> $ ./pre-inst-env guix show bitlbee
>> name: bitlbee
>> version: 3.6
>> outputs: out
>> parameters: libpurple
>> systems: x86_64-linux i686-linux
>> dependencies: check@0.12.0 glib@2.62.6 gnutls@3.6.12 libotr@4.1.1 perl@5.30.2 pkg-config@0.29.2 python@3.8.2
>> location: gnu/packages/messaging.scm:243:2
>> homepage: https://www.bitlbee.org/
>> license: GPL 2+, FreeBSD
>> synopsis: IRC to instant messaging gateway  
>> description: BitlBee brings IM (instant messaging) to IRC clients, for people who have an IRC client running all
>> + the time and don't want to run an additional IM client.  BitlBee currently supports XMPP/Jabber (including Google
>> + Talk), MSN Messenger, Yahoo! Messenger, AIM and ICQ, and the Twitter microblogging network (plus all other Twitter
>> + API compatible services like identi.ca and status.net).
>>
>> $ ./pre-inst-env guix install bitlbee --with-parameter=bitlbee=libpurple=true -p /tmp/test-bitlbee
>> The following package will be installed:
>>    bitlbee 3.6
>>
>> The following derivation will be built:
>>    /gnu/store/clvs5521v5ybdw1z1yh97z2ky1dxm4d9-bitlbee-3.6.drv
>>
>> [...]
>>
>> $ cat /tmp/test-bitlbee/manifest
>> ;; This file was automatically generated and is for internal use only.
>> ;; It cannot be passed to the '--manifest' option.
>>
>> (manifest
>>   (version 3)
>>   (packages
>>     (("bitlbee"
>>       "3.6"
>>       "out"
>>       "/gnu/store/d67r9k5hwfm5hkd1d3pbhg49fcc2jj4q-bitlbee-3.6"
>>       (propagated-inputs ())
>>       (search-paths ())
>>       (properties
>>         (transformations
>>           (with-parameter . "bitlbee=libpurple=true")))))))
>> --8<---------------cut here---------------end--------------->8---
>>
>> That’s it!
>>
>> We would need more things, like a ‘guix parameters’ command to show the
>> available parameters of a package and an ‘--all-parameter-values’ option
>> to ‘guix build’ to build all the variants of a given package.
>>
>> An important question: do we have examples of packages for which we’d
>> like to have parameters?  I’d grepped for “inherit” and that yields a
>> few potential candidates, but also maybe a few potential non-candidates.
>> Would this be a good fit for them?
>>
>> Thoughts?
>>
>> Ludo’.
>>
>> From 9155411f2e8e78922e1e46d92068ac8f652ff0a5 Mon Sep 17 00:00:00 2001
>> From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
>> Date: Sun, 15 Nov 2020 17:01:14 +0100
>> Subject: [PATCH 3/4] ui: 'package->recutils' emits "parameters" field.
>>
>> * guix/ui.scm (package->recutils): Add "parameters" recutils field.
>> ---
>>  guix/ui.scm | 5 +++++
>>  1 file changed, 5 insertions(+)
>>
>> diff --git a/guix/ui.scm b/guix/ui.scm
>> index 4e686297e8..2485400cc9 100644
>> --- a/guix/ui.scm
>> +++ b/guix/ui.scm
>> @@ -60,6 +60,7 @@
>>                  #:hide (package-name->name+version
>>                          ;; Avoid "overrides core binding" warning.
>>                          delete))
>> +  #:autoload   (guix parameters) (package-parameters package-parameter-name)
>>    #:use-module (srfi srfi-1)
>>    #:use-module (srfi srfi-9 gnu)
>>    #:use-module (srfi srfi-11)
>> @@ -1529,6 +1530,10 @@ HYPERLINKS? is true, emit hyperlink escape sequences when appropriate."
>>    (format port "name: ~a~%" (package-name p))
>>    (format port "version: ~a~%" (package-version p))
>>    (format port "outputs: ~a~%" (string-join (package-outputs p)))
>> +  (match (package-parameters p)
>> +    (() #t)
>> +    (lst (format port "parameters:~{ ~a~}~%"
>> +                 (map package-parameter-name lst))))
>>    (format port "systems: ~a~%"
>>            (string-join (package-transitive-supported-systems p)))
>>    (format port "dependencies: ~a~%"
>> -- 
>> 2.29.2
>>
>> From 49d7746ada4d4674acbbfd2606ad9bff4f6207eb Mon Sep 17 00:00:00 2001
>> From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
>> Date: Sun, 15 Nov 2020 17:04:18 +0100
>> Subject: [PATCH 4/4] gnu: bitlbee: Add "libpurple" parameter.
>>
>> * gnu/packages/messaging.scm (bitlbee)[inputs]: Add optional PIDGIN
>> input.
>> [properties]: New field.
>> (bitlbee-purple): Mark as deprecated.
>> ---
>>  gnu/packages/messaging.scm | 43 +++++++++++++++++++-------------------
>>  1 file changed, 22 insertions(+), 21 deletions(-)
>>
>> diff --git a/gnu/packages/messaging.scm b/gnu/packages/messaging.scm
>> index b462504894..2f0f44d10d 100644
>> --- a/gnu/packages/messaging.scm
>> +++ b/gnu/packages/messaging.scm
>> @@ -120,6 +120,7 @@
>>    #:use-module (guix build-system trivial)
>>    #:use-module (guix download)
>>    #:use-module (guix git-download)
>> +  #:use-module (guix parameters)
>>    #:use-module (guix hg-download)
>>    #:use-module ((guix licenses) #:prefix license:)
>>    #:use-module (guix packages)
>> @@ -256,7 +257,8 @@ end-to-end encryption.")
>>                ("libotr" ,libotr)
>>                ("gnutls" ,gnutls)
>>                ("python" ,python)
>> -              ("perl" ,perl)))
>> +              ("perl" ,perl)
>> +              ,@(optionally 'libpurple? `("purple" ,pidgin))))
>>      (arguments
>>       `(#:phases
>>         (modify-phases %standard-phases
>> @@ -275,7 +277,21 @@ end-to-end encryption.")
>>               (invoke "./configure"
>>                       (string-append "--prefix="
>>                                      (assoc-ref outputs "out"))
>> -                     "--otr=1"))))))
>> +                     "--otr=1"
>> +                     ,@(optionally 'libpurple? "--purple=1")))))
>> +
>> +       ;; XXX: Tests fail to link, and ./configure says that it's "supported
>> +       ;; on a best-effort basis" anyway.
>> +       #:tests? ,(not (assq-ref (package-properties this-package)
>> +                                'libpurple?))))
>> +    (properties
>> +     `((parameters
>> +        ,(package-parameter (name "libpurple")
>> +                            (description
>> +                             "Whether to enable libpurple (Pidgin)
>> +support.")
>> +                            (property 'libpurple?)
>> +                            (type boolean)))))
>>      (synopsis "IRC to instant messaging gateway")
>>      (description "BitlBee brings IM (instant messaging) to IRC clients, for
>>  people who have an IRC client running all the time and don't want to run an
>> @@ -289,25 +305,10 @@ identi.ca and status.net).")
>>  (define-public bitlbee-purple
>>    ;; This variant uses libpurple, which provides support for more protocols at
>>    ;; the expense of a much bigger closure.
>> -  (package/inherit bitlbee
>> -    (name "bitlbee-purple")
>> -    (synopsis "IRC to instant messaging gateway (using Pidgin's libpurple)")
>> -    (inputs `(("purple" ,pidgin)
>> -              ,@(package-inputs bitlbee)))
>> -    (arguments
>> -     (substitute-keyword-arguments (package-arguments bitlbee)
>> -       ((#:phases phases '%standard-phases)
>> -        `(modify-phases ,phases
>> -           (replace 'configure                    ;add "--purple=1"
>> -             (lambda* (#:key outputs #:allow-other-keys)
>> -               (invoke "./configure"
>> -                       (string-append "--prefix="
>> -                                      (assoc-ref outputs "out"))
>> -                       "--otr=1" "--purple=1")))))
>> -       ((#:tests? _ #t)
>> -        ;; XXX: Tests fail to link, and ./configure says that it's "supported
>> -        ;; on a best-effort basis" anyway.
>> -        #f)))))
>> +  (deprecated-package "bitlbee-purple"
>> +                      (package/inherit bitlbee
>> +                        (properties `((libpurple? . #t)
>> +                                      ,@(package-properties bitlbee))))))
>>  
>>  (define-public bitlbee-discord
>>    (package
>> -- 
>> 2.29.2
>>
>> From f42b68499c4e2a9bd368fe6a516932f5afa7a189 Mon Sep 17 00:00:00 2001
>> From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
>> Date: Sun, 15 Nov 2020 16:58:52 +0100
>> Subject: [PATCH 1/4] DRAFT Add (guix parameters).
>>
>> DRAFT: Missing tests & doc.
>>
>> * guix/parameters.scm: New file.
>> * Makefile.am (MODULES): Add it.
>> ---
>>  Makefile.am         |   1 +
>>  guix/parameters.scm | 131 ++++++++++++++++++++++++++++++++++++++++++++
>>  2 files changed, 132 insertions(+)
>>  create mode 100644 guix/parameters.scm
>>
>> diff --git a/Makefile.am b/Makefile.am
>> index e7053ee4f4..72f955360d 100644
>> --- a/Makefile.am
>> +++ b/Makefile.am
>> @@ -235,6 +235,7 @@ MODULES =					\
>>    guix/build/make-bootstrap.scm			\
>>    guix/search-paths.scm				\
>>    guix/packages.scm				\
>> +  guix/parameters.scm				\
>>    guix/import/cabal.scm				\
>>    guix/import/cpan.scm				\
>>    guix/import/cran.scm				\
>> diff --git a/guix/parameters.scm b/guix/parameters.scm
>> new file mode 100644
>> index 0000000000..e4f8240aa4
>> --- /dev/null
>> +++ b/guix/parameters.scm
>> @@ -0,0 +1,131 @@
>> +;;; GNU Guix --- Functional package management for GNU
>> +;;; Copyright © 2020 Ludovic Courtès <ludo@gnu.org>
>> +;;;
>> +;;; This file is part of GNU Guix.
>> +;;;
>> +;;; GNU Guix is free software; you can redistribute it and/or modify it
>> +;;; under the terms of the GNU General Public License as published by
>> +;;; the Free Software Foundation; either version 3 of the License, or (at
>> +;;; your option) any later version.
>> +;;;
>> +;;; GNU Guix is distributed in the hope that it will be useful, but
>> +;;; WITHOUT ANY WARRANTY; without even the implied warranty of
>> +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> +;;; GNU General Public License for more details.
>> +;;;
>> +;;; You should have received a copy of the GNU General Public License
>> +;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
>> +
>> +(define-module (guix parameters)
>> +  #:use-module (guix packages)
>> +  #:use-module (guix records)
>> +  #:use-module (guix diagnostics)
>> +  #:use-module (guix i18n)
>> +  #:use-module (srfi srfi-1)
>> +  #:use-module (srfi srfi-34)
>> +  #:use-module (srfi srfi-35)
>> +  #:use-module (ice-9 match)
>> +  #:export (package-parameter
>> +            package-parameter?
>> +            package-parameter-name
>> +            package-parameter-property
>> +            package-parameter-type
>> +            package-parameter-description
>> +
>> +            boolean
>> +            optionally
>> +
>> +            package-parameters
>> +            lookup-package-parameter
>> +            package-parameter-value
>> +            set-package-parameter-value))
>> +
>> +;;; Commentary:
>> +;;;
>> +;;; This module provides a way to express high-level "package parameters",
>> +;;; which allow users to customize how packages are built.  Parameters are an
>> +;;; interface that package developers define, where each parameter has a name
>> +;;; and type.  The user interface then converts parameter values from string
>> +;;; to Scheme values and records them in the package properties.
>> +;;;
>> +;;; Package parameters are discoverable; their description is
>> +;;; internationalized.  The possible values of a parameter can be enumerated,
>> +;;; and thus the Cartesian product of all possible parameter values for a
>> +;;; package can be enumerated as well.
>> +;;;
>> +;;; Code:
>> +
>> +;; Package parameter interface.
>> +(define-record-type* <package-parameter> package-parameter
>> +  make-package-parameter
>> +  package-parameter?
>> +  (name          package-parameter-name)
>> +  (property      package-parameter-property (default (string->symbol name)))
>> +  (type          package-parameter-type)
>> +  (description   package-parameter-description))
>> +
>> +;; Type of a package parameter.
>> +(define-record-type* <parameter-type> parameter-type
>> +  make-parameter-type
>> +  parameter-type?
>> +  (name          parameter-type-name)              ;debugging purposes only!
>> +  (string->value parameter-type-string->value)
>> +  (value->string parameter-type-value->string)
>> +  (universe      parameter-type-universe))
>> +
>> +(define boolean
>> +  ;; The Boolean parameter type.
>> +  (parameter-type (name 'boolean)
>> +                  (universe '(#true #false))
>> +                  (value->string
>> +                   (match-lambda
>> +                     (#f "false")
>> +                     (#t "true")))
>> +                  (string->value
>> +                   (lambda (str)
>> +                     (cond ((string-ci=? str "true")
>> +                            #t)
>> +                           ((string-ci=? str "false")
>> +                            #f)
>> +                           (else
>> +                            (raise (condition
>> +                                    (&message (message "wrong value"))))))))))
>> +
>> +(define (package-parameters package)
>> +  (or (assq-ref (package-properties package) 'parameters)
>> +      '()))
>> +
>> +(define (package-parameter-value package parameter)
>> +  (assq-ref (package-properties package)
>> +            (package-parameter-property parameter)))
>> +
>> +(define (lookup-package-parameter package name)
>> +  (find (lambda (parameter)
>> +          (string=? (package-parameter-name parameter) name))
>> +        (package-parameters package)))
>> +
>> +(define (set-package-parameter-value package name value)
>> +  (let ((parameter (lookup-package-parameter package name))
>> +        (location  (package-field-location package 'properties)))
>> +    (unless parameter
>> +      (raise (apply make-compound-condition
>> +                    (formatted-message
>> +                     (G_ "~a: no such package parameter")
>> +                     name)
>> +                    (if location
>> +                        (list (condition
>> +                               (&error-location (location location))))
>> +                        '()))))
>> +    (let* ((property (package-parameter-property parameter))
>> +           (type     (package-parameter-type parameter))
>> +           (value    ((parameter-type-string->value type) value)))
>> +      (package/inherit package
>> +        (properties
>> +         (alist-cons property value
>> +                     (alist-delete property (package-properties package)
>> +                                   eq?)))))))
>> +
>> +(define-syntax-rule (optionally property exp)
>> +  (if (assq-ref (package-properties this-package) property)
>> +      (list exp)
>> +      '()))
>> -- 
>> 2.29.2
>>
>> From 5d6d81e4c3e37de53fe7f62ff7ef94da8b2df033 Mon Sep 17 00:00:00 2001
>> From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
>> Date: Sun, 15 Nov 2020 16:59:48 +0100
>> Subject: [PATCH 2/4] DRAFT transformations: Add '--with-parameter'.
>>
>> DRAFT: Missing tests & doc.
>>
>> * guix/transformations.scm (evaluate-parameter-specs)
>> (transform-package-parameters): New procedures.
>> (%transformations, %transformation-options): Add 'with-parameter'.
>> ---
>>  guix/transformations.scm | 39 +++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 39 insertions(+)
>>
>> diff --git a/guix/transformations.scm b/guix/transformations.scm
>> index 30142dd059..0f83eb470d 100644
>> --- a/guix/transformations.scm
>> +++ b/guix/transformations.scm
>> @@ -25,6 +25,7 @@
>>    #:autoload   (guix download) (download-to-store)
>>    #:autoload   (guix git-download) (git-reference? git-reference-url)
>>    #:autoload   (guix git) (git-checkout git-checkout? git-checkout-url)
>> +  #:autoload   (guix parameters) (set-package-parameter-value)
>>    #:use-module (guix utils)
>>    #:use-module (guix memoization)
>>    #:use-module (guix gexp)
>> @@ -324,6 +325,41 @@ a checkout of the Git repository at the given URL."
>>          (rewrite obj)
>>          obj)))
>>  
>> +(define (evaluate-parameter-specs specs proc)
>> +  "Parse SPECS, a list of strings like \"bitlbee=purple=true\", and return a
>> +list of spec/procedure pairs, where (PROC PACKAGE PARAMETER VALUE) is called
>> +to return the replacement package.  Raise an error if an element of SPECS uses
>> +invalid syntax, or if a package it refers to could not be found."
>> +  (map (lambda (spec)
>> +         (match (string-tokenize spec %not-equal)
>> +           ((spec name value)
>> +            (define (replace old)
>> +              (proc old name value))
>> +
>> +            (cons spec replace))
>> +           (_
>> +            (raise
>> +             (formatted-message
>> +              (G_ "invalid package parameter specification: ~s")
>> +              spec)))))
>> +       specs))
>> +
>> +(define (transform-package-parameters replacement-specs)
>> +  "Return a procedure that, when passed a package, replaces its direct
>> +dependencies according to REPLACEMENT-SPECS.  REPLACEMENT-SPECS is a list of
>> +strings like \"guile-next=stable-3.0\" meaning that packages are built using
>> +'guile-next' from the latest commit on its 'stable-3.0' branch."
>> +  (define (replace old name value)
>> +    (set-package-parameter-value old name value))
>> +
>> +  (let* ((replacements (evaluate-parameter-specs replacement-specs
>> +                                                 replace))
>> +         (rewrite      (package-input-rewriting/spec replacements)))
>> +    (lambda (obj)
>> +      (if (package? obj)
>> +          (rewrite obj)
>> +          obj))))
>> +
>>  (define (package-dependents/spec top bottom)
>>    "Return the list of dependents of BOTTOM, a spec string, that are also
>>  dependencies of TOP, a package."
>> @@ -467,6 +503,7 @@ to the same package but with #:strip-binaries? #f in its 'arguments' field."
>>      (with-branch . ,transform-package-source-branch)
>>      (with-commit . ,transform-package-source-commit)
>>      (with-git-url . ,transform-package-source-git-url)
>> +    (with-parameter . ,transform-package-parameters)
>>      (with-c-toolchain . ,transform-package-toolchain)
>>      (with-debug-info . ,transform-package-with-debug-info)
>>      (without-tests . ,transform-package-tests)))
>> @@ -503,6 +540,8 @@ to the same package but with #:strip-binaries? #f in its 'arguments' field."
>>                    (parser 'with-commit))
>>            (option '("with-git-url") #t #f
>>                    (parser 'with-git-url))
>> +          (option '("with-parameter") #t #f
>> +                  (parser 'with-parameter))
>>            (option '("with-c-toolchain") #t #f
>>                    (parser 'with-c-toolchain))
>>            (option '("with-debug-info") #t #f
>> -- 
>> 2.29.2

Re: A plan for parameterized packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 549 bytes --]

Fantastic!

One of the biggest struggle we had when discussing it was figuring out
what to do about parameter propagation across dependencies.

For instance, what if we want to build "all packages without X support"?
This means that the parameter must traverse all inputs recursively if we
don't want to drag X indirectly.

If I understand your change correctly, the patch is only applying
parameters to the given package and it's not propagated to the inputs,
is that correct?

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

Re: A plan for parameterized packages

[zimoun]

Hi Pierre,

On Sun, 15 Nov 2020 at 18:44, Pierre Neidhardt <mail@ambrevar.xyz> wrote:

> For instance, what if we want to build "all packages without X support"?
> This means that the parameter must traverse all inputs recursively if we
> don't want to drag X indirectly.

It means: add this new ’optionally’ to all the packages supporting X to
be able to turn it off with the ’boolean’ type.  IIUC.


> If I understand your change correctly, the patch is only applying
> parameters to the given package and it's not propagated to the inputs,
> is that correct?

Because of ’package-input-rewriting/spec’, it is applied to all the
graph,  IIUC.

Cheers,
simon

Re: A plan for parameterized packages

[Ludovic Courtès]

Hi Pierre,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> One of the biggest struggle we had when discussing it was figuring out
> what to do about parameter propagation across dependencies.
>
> For instance, what if we want to build "all packages without X support"?
> This means that the parameter must traverse all inputs recursively if we
> don't want to drag X indirectly.
>
> If I understand your change correctly, the patch is only applying
> parameters to the given package and it's not propagated to the inputs,
> is that correct?

That’s correct: in this patch set parameters are per-package, and I
think it’s easier to start simple, but we could implement what you
describe without too much hassle I think.

For example, you’d type ‘--with-parameters=x11=false’, and that’d be
applied to all the packages that have an ‘x11’ parameter.

Ludo’.

Re: A plan for parameterized packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 540 bytes --]

Thanks for confirming.

Indeed, we should start easy!

I believe it's important to foresee the obstacles that are looming at
the horizon nonetheless, because "without too much hassle" might be a
bit optimistic ;)

See point 7. about conflicts:

https://lists.gnu.org/archive/html/guix-devel/2020-01/msg00026.html

and comments:

https://lists.gnu.org/archive/html/guix-devel/2020-01/msg00177.html
https://lists.gnu.org/archive/html/guix-devel/2020-01/msg00181.html

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

Re: A plan for parameterized packages

[zimoun]

Hi,

On Mon, 16 Nov 2020 at 13:03, Pierre Neidhardt <mail@ambrevar.xyz> wrote:

> See point 7. about conflicts:
>
> https://lists.gnu.org/archive/html/guix-devel/2020-01/msg00026.html

Quoting for instance the 2 examples:

        For instance, use guile-2.2.4 instead of guile for all guile
        libraries, or use pulseaudio everywhere, including in
        dependencies that are not explicitly installed to the user
        profile.

From my understanding, the first case (guile) is now covered by the
“package rewriting” transformation (see package-input-rewriting IIUC).

The issue with the second case is below.


> and comments:
>
> https://lists.gnu.org/archive/html/guix-devel/2020-01/msg00177.html

From my understanding of the Ludo’s patch and from the Marius’s message,
the both looks really similar. :-)


> https://lists.gnu.org/archive/html/guix-devel/2020-01/msg00181.html

Quoting:

        To be usable, we would need something to say "build bar and all
        its inputs without pulseaudio, except for some given packages".

        While this is OK with 1 parameter, it's quickly gets much more
        complicated when packages have multiple parameters that maybe conflict
        with one another.

Is the combinatorial conflict solvable in advance?  Well, from my point
of view, it cannot be via package transformation.

For example, let’s consider the Emacs packages and #41732.  Quoting [1]:

        > Perhaps then,
        >
        > --8<---------------cut here---------------start------------->8---
        > guix build -m manifest.scm --with-input=emacs-minimal=emacs-next \
        >      --with-input=emacs=emacs-next
        > --8<---------------cut here---------------end--------------->8---

        Possibly. And then Magit uses emacs-no-x as an input, so we may
        need to also add --with-input=emacs-no-x=emacs-next to the
        command.

        I'm just pointing out that the process is not as straightforward
        as it might seem. So, it doesn't sound right to simply suggest

and [2]:

        For example, the package emacs-magit drags emacs-no-x because of
        emacs-libgit, why is emacs-minimal not enough here?

        Well, the emacs-build-system depends (implicitly) on emacs-minimal,
        only.  And the initial patch `package-with-emacs-next' was changing
        this, only.  However, the package emacs-libgit is cmake-build-system
        and the package emacs-no-x is an explicit dependency; which is another
        story. :-)

these both examples show that it is already complex enough just to
rebuild all the Emacs packages using another Emacs VM (emacs-next or
REmacs, etc.).

To be clear, I am not convinced that the current package recipes are
functional enough to be able to keep under control the combinatorial
conflict; in the general case.

1: <http://issues.guix.gnu.org/issue/41732#14>
2: <http://issues.guix.gnu.org/issue/41732>

All the best,
simon

Re: A plan for parameterized packages

[zimoun]

Hi Lduo,

On Sun, 15 Nov 2020 at 17:33, Ludovic Courtès <ludo@gnu.org> wrote:

> That said, this message is about a possible implementation of package
> parameters, so here we go.  :-)

Cool!


> To me the requirements for package parameters are:
>
>   1. it must be possible to discover them and choose them from the UI;
>
>   2. they must contain on-line internationalized documentation such that
>      the UI can list a package’s parameters and their type;

Except ’boolean’, which kind of type do you have in mind?  Aside that
you did not find examples of packages requiring parameters. ;-)

The answer leads to your point #4.


>   3. the chosen parameters when installing a package in a profile must
>      be preserved;

You mean track the parameters with ’properties’ in <profile>/manifest,
right?


>   4. it must be possible to enumerate all the possible values of a
>      parameter, and thus to build the Cartesian product of all the
>      possible parameter combinations of a package (or of a package
>      graph!), so we can test those combinations as much as possible.

The values of the option are therefore known at package time, right?
However, this implies restricted possibility for the type, right?


> Subject: [PATCH 1/4] DRAFT Add (guix parameters).
>
> DRAFT: Missing tests & doc.
>
> * guix/parameters.scm: New file.
> * Makefile.am (MODULES): Add it.
> ---

[...]

> diff --git a/guix/parameters.scm b/guix/parameters.scm
> +
> +;; Type of a package parameter.
> +(define-record-type* <parameter-type> parameter-type
> +  make-parameter-type
> +  parameter-type?
> +  (name          parameter-type-name)              ;debugging purposes only!
> +  (string->value parameter-type-string->value)
> +  (value->string parameter-type-value->string)
> +  (universe      parameter-type-universe))
> +
> +(define boolean
> +  ;; The Boolean parameter type.
> +  (parameter-type (name 'boolean)
> +                  (universe '(#true #false))
> +                  (value->string
> +                   (match-lambda
> +                     (#f "false")
> +                     (#t "true")))
> +                  (string->value
> +                   (lambda (str)
> +                     (cond ((string-ci=? str "true")
> +                            #t)
> +                           ((string-ci=? str "false")
> +                            #f)
> +                           (else
> +                            (raise (condition
> +                                    (&message (message "wrong value"))))))))))

The types will be “hard-coded“ here, right?  Boolean being the simplest
example and imagination just needs to be released, right? :-)



> Subject: [PATCH 2/4] DRAFT transformations: Add '--with-parameter'.
>
> DRAFT: Missing tests & doc.
>
> * guix/transformations.scm (evaluate-parameter-specs)
> (transform-package-parameters): New procedures.
> (%transformations, %transformation-options): Add 'with-parameter'.
> ---
>  guix/transformations.scm | 39 +++++++++++++++++++++++++++++++++++++++
>  1 file changed, 39 insertions(+)
>
> diff --git a/guix/transformations.scm b/guix/transformations.scm

[...]

> +(define (evaluate-parameter-specs specs proc)
> +  "Parse SPECS, a list of strings like \"bitlbee=purple=true\", and return a
> +list of spec/procedure pairs, where (PROC PACKAGE PARAMETER VALUE) is called
> +to return the replacement package.  Raise an error if an element of SPECS uses
> +invalid syntax, or if a package it refers to could not be found."
> +  (map (lambda (spec)
> +         (match (string-tokenize spec %not-equal)
> +           ((spec name value)
> +            (define (replace old)
> +              (proc old name value))
> +
> +            (cons spec replace))
> +           (_
> +            (raise
> +             (formatted-message
> +              (G_ "invalid package parameter specification: ~s")
> +              spec)))))
> +       specs))

Here ’proc’ could be anything, right?  But then…

> +(define (transform-package-parameters replacement-specs)
> +  "Return a procedure that, when passed a package, replaces its direct
> +dependencies according to REPLACEMENT-SPECS.  REPLACEMENT-SPECS is a list of
> +strings like \"guile-next=stable-3.0\" meaning that packages are built using
> +'guile-next' from the latest commit on its 'stable-3.0' branch."
> +  (define (replace old name value)
> +    (set-package-parameter-value old name value))
> +
> +  (let* ((replacements (evaluate-parameter-specs replacement-specs
> +                                                 replace))
> +         (rewrite      (package-input-rewriting/spec replacements)))
> +    (lambda (obj)
> +      (if (package? obj)
> +          (rewrite obj)
> +          obj))))

… it is ’set-package-parameter-value’.  It is not clear in my mind.
Does this constrain the hypothetical types?


Cheers,
simon

Re: A plan for parameterized packages

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

>> To me the requirements for package parameters are:
>>
>>   1. it must be possible to discover them and choose them from the UI;
>>
>>   2. they must contain on-line internationalized documentation such that
>>      the UI can list a package’s parameters and their type;
>
> Except ’boolean’, which kind of type do you have in mind?  Aside that
> you did not find examples of packages requiring parameters. ;-)

Another example would be enumerated types.

>>   3. the chosen parameters when installing a package in a profile must
>>      be preserved;
>
> You mean track the parameters with ’properties’ in <profile>/manifest,
> right?

Yup!

>>   4. it must be possible to enumerate all the possible values of a
>>      parameter, and thus to build the Cartesian product of all the
>>      possible parameter combinations of a package (or of a package
>>      graph!), so we can test those combinations as much as possible.
>
> The values of the option are therefore known at package time, right?
> However, this implies restricted possibility for the type, right?

The <package-parameter> record has to specify a type, and the type must
have a finite universe.  So there cannot be an “integer” type, for
instance, but there can be “integer between 10 and 42”.

>> +(define (evaluate-parameter-specs specs proc)
>> +  "Parse SPECS, a list of strings like \"bitlbee=purple=true\", and return a
>> +list of spec/procedure pairs, where (PROC PACKAGE PARAMETER VALUE) is called
>> +to return the replacement package.  Raise an error if an element of SPECS uses
>> +invalid syntax, or if a package it refers to could not be found."
>> +  (map (lambda (spec)
>> +         (match (string-tokenize spec %not-equal)
>> +           ((spec name value)
>> +            (define (replace old)
>> +              (proc old name value))
>> +
>> +            (cons spec replace))
>> +           (_
>> +            (raise
>> +             (formatted-message
>> +              (G_ "invalid package parameter specification: ~s")
>> +              spec)))))
>> +       specs))
>
> Here ’proc’ could be anything, right?  But then…
>
>> +(define (transform-package-parameters replacement-specs)
>> +  "Return a procedure that, when passed a package, replaces its direct
>> +dependencies according to REPLACEMENT-SPECS.  REPLACEMENT-SPECS is a list of
>> +strings like \"guile-next=stable-3.0\" meaning that packages are built using
>> +'guile-next' from the latest commit on its 'stable-3.0' branch."
>> +  (define (replace old name value)
>> +    (set-package-parameter-value old name value))
>> +
>> +  (let* ((replacements (evaluate-parameter-specs replacement-specs
>> +                                                 replace))
>> +         (rewrite      (package-input-rewriting/spec replacements)))
>> +    (lambda (obj)
>> +      (if (package? obj)
>> +          (rewrite obj)
>> +          obj))))
>
> … it is ’set-package-parameter-value’.  It is not clear in my mind.

Yes I could have used ‘set-package-parameter-value’ directly instead of
adding this ‘proc’ parameter; it would have been more readable I guess!

Thanks,
Ludo’.

Re: A plan for parameterized packages

[Taylan Kammer]

> An important question: do we have examples of packages for which we’d
> like to have parameters?  I’d grepped for “inherit” and that yields a
> few potential candidates, but also maybe a few potential non-candidates.
> Would this be a good fit for them?

I suppose Emacs would be an obvious candidate, with its "no-x" as well 
as "no-x-toolkit" variants.


Taylan

Re: A plan for parameterized packages

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 2374 bytes --]

Hi Ludo,

nice feature!

On Sun, 15 Nov 2020 17:33:28 +0100
Ludovic Courtès <ludo@gnu.org> wrote:

> An important question: do we have examples of packages for which we’d
> like to have parameters?

For the embedded/flash rom side:

* Enable/disable building the documentation.  I really don't need a 40 MiB
manual stored onto a 16 MiB firmware flash chip.  If that's better done as an
extra output, fair enough.

* Enable/disable obscure dependencies: Library packages that pull in 400 MiB Qt
into the closure for a thing no one (TM) uses should probably provide a switch to
disable that GUI.  Sometimes a package provides multiple GUIs for different
toolkits, in which case one should be able to choose one toolkit and not
build for the others.

* No, even in 2020, I won't start using AmigaFS, NFSv3 and whatever old
protocol/format is still around and superseded by other protocols.

The gexp-functionality of being able to select individual files of a package
is already very useful for use cases an embedded developer would have
(and that's there for a long time already).  So that's nice!

For the kind of feature flags I have in mind, it usually means I don't want
to have the feature *anywhere*--for example, if I don't want to have Qt or
Kerberos or whatever, that's because I want to save the space and thus
it should be able to be *globally* specified--at least per profile.

It doesn't help one bit for space savings if package A doesn't pull in
Kerberos or latex or or Haskell bindings for a Raspberry Pi serial port
library (guess why that's oddly specific ;) ), but package B does.

However, sometimes using static libraries is better and there's no reason to
build the shared libraries.  But that's very much local to whatever I'm
trying to do (at least local to a profile if not package).

I would advise against doing a grep -r -- --enable and introducing all those
as parameters.  Rather I would check the closure of stuff and if the closure
goes from 1200 MiB to 50 MiB, chances are a parameter would be nice there :)

I guess the kind of flags I envision would be set at profile level.

From experience with Gentoo before I can tell you that the combinatory
explosion is a real problem and most of the "more advanced" (toggled more
switches :) ) combinations did not work the majority of the time.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: A plan for parameterized packages

[zimoun]

Dear,

On Sun, 15 Nov 2020 at 21:46, Danny Milosavljevic <dannym@scratchpost.org> wrote:

> * Enable/disable building the documentation.  I really don't need a 40 MiB
> manual stored onto a 16 MiB firmware flash chip.  If that's better done as an
> extra output, fair enough.

Related (I hope) is: build packages with several outputs.  For instance,
’git’ has several 'outputs' ("send-email", "svn", etc.), so the list of
"inputs" provides e.g., "subversion" even if I am only interested by
e.g., "git:send-email".  This matters about closure.

And it is maybe an occasion to revisit the museum, i.e., the TODO file:

** extend ‘propagated-build-inputs’ with support for multiple outputs

#+BEGIN_SRC scheme
  (outputs '("out" "include"))
  (propagated-build-inputs
    `(((("i1" ,p1 "o1")
        ("i2" ,p2))
       => "include")
      ("i3" ,p3)))
#+END_SRC

For one reference:

  <https://lists.gnu.org/archive/html/guix-devel/2020-06/msg00284.html>


All the best,
simon

Re: A plan for parameterized packages

[raingloom]

On Sun, 15 Nov 2020 21:46:58 +0100
Danny Milosavljevic <dannym@scratchpost.org> wrote:

> Hi Ludo,
> 
> nice feature!
> 
> On Sun, 15 Nov 2020 17:33:28 +0100
> Ludovic Courtès <ludo@gnu.org> wrote:
> 
> > An important question: do we have examples of packages for which
> > we’d like to have parameters?
> 
> For the embedded/flash rom side:
> 
> * Enable/disable building the documentation.  I really don't need a
> 40 MiB manual stored onto a 16 MiB firmware flash chip.  If that's
> better done as an extra output, fair enough.
> ......

Alpine already achieves an incredibly tiny install size by splitting
packages into many outputs. We could and should do the same.
As far as I know, they do not have parameterized packages.

Re: A plan for parameterized packages

[Ryan Prior]

[-- Attachment #1: Type: text/plain, Size: 763 bytes --]

On November 15, 2020, raingloom <raingloom@riseup.net> wrote:
> Alpine already achieves an incredibly tiny install size by splitting
> packages into many outputs. We could and should do the same.
> As far as I know, they do not have parameterized packages.

I definitely support more package-splitting and dependency tree-shaking
where possible. Getting a core Guix System down to Alpine size would be
fantastic.

For example, `docker size python` shows a 171.4mb bundle size just for
Python alone, while `docker images python:alpine` shows a 44.3mb system
image, which includes a bunch of system stuff in addition to Python. It
would be great to have tools to understand what's in our big Guix
packages and how we could ship light bundles that get the job done.

[-- Attachment #2: Type: text/html, Size: 3073 bytes --]

Re: A plan for parameterized packages

[Denis 'GNUtoo' Carikli]

[-- Attachment #1: Type: text/plain, Size: 2850 bytes --]

On Sun, 15 Nov 2020 22:24:29 +0100
raingloom <raingloom@riseup.net> wrote:
> Alpine already achieves an incredibly tiny install size by splitting
> packages into many outputs. We could and should do the same.
> As far as I know, they do not have parameterized packages.
That also depends on how far you want to go.

Last time I looked into how LibreCMC/OpenWRT did that, they had much
more optimization than that. If I recall well, they use at least:
- sstrip to strip binaries as much as they could. sstrip produces
  smaller binaries than with strip.
- compilation flags like -Os
- a read-only compressed filesystem with an overlay to store the
  changes

The issue is that despite all that, the size of the images tend to
increase too rapidly over time[1].

If we manage to shrink Guix enough, it might be possible to use it on
way more devices, including RYF compliant devices or potentially
certifiable devices:
- The Talos II BMC has 32M according to both the wiki[2] and the image
  sizes[4]. Its architecture is ARM. So once we have the PPC64
  architecture working, it would be great to be able to run Guix
  both in the BMC and on the PowerPC CPU.

  That BMC is also available on other mainboards like the D16 which is
  supported by Libreboot, but the flash size is probably even smaller
  there.
- Many WiFi access point have very few flash space. It can boils down
  to as low as 16M for LibreCMC/OpenWRT compatible devices, or even 8M
  for older devices. However they typically use the MIPS architecture
  which isn't supported yet in Guix.
- There is a GNU/Linux distribution[6] that runs inside the flash chip
  where Libreboot or Coreboot typically runs. The goal is to enable
  more flexible and/or secure booting by using GNU/Linux to boot
  GNU/Linux. Here too the flash chip of computers supported by
  Libreboot can be quite small, like 8M for Thinkpads with GM45
  chipsets.

In some case it might be possible to increase the flash chip size
(sometimes you don't need soldering for that), but at least with x86
mainboards, the chipset has limits on the size of the flash chip that it
can see. And the size cannot be increased that much: The biggest flash
chip that flashrom supports is 256M.

References:
-----------
[1]https://openwrt.org/supported_devices/864_warning
[2]The wiki[3] mention a MX25L25635F/MX25L25645E/MX25L25665E flash chip
   which is 32M according to flashrom -L
[3]https://wiki.raptorcs.com/wiki/Debricking_the_BMC#Flash_new_BMC_firmware_via_serial_port_.28Open_Source_Method.29
[4]Once uncompressed the image[5] size (for installation through the
   shell) is 32M.
[5]https://wiki.raptorcs.com/wiki/File:Talos-ii-openbmc-v2.00-bundle.tar
[6]https://github.com/osresearch/heads/
[7]https://github.com/osresearch/heads/tree/master/config

Denis.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: A plan for parameterized packages

[Ludovic Courtès]

Hi,

Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org> skribis:

> Last time I looked into how LibreCMC/OpenWRT did that, they had much
> more optimization than that. If I recall well, they use at least:
> - sstrip to strip binaries as much as they could. sstrip produces
>   smaller binaries than with strip.
> - compilation flags like -Os
> - a read-only compressed filesystem with an overlay to store the
>   changes

To me this looks like the ultimate size optimization level.  Before we
get there, we should first see how to get closer to package sizes
typically found on Debian and that alone is a real challenge.

> The issue is that despite all that, the size of the images tend to
> increase too rapidly over time[1].

Yeah, that’s also the problem here: we have ‘guix size’ to profile a
package at one point in time, but it’s easy to unwillingly increase its
closure size the next day without noticing.

Chris: does the Data Service track store item sizes (and more generally
everything ‘query-path-info’ returns)?  It’d be great to be able to
visualize size plots over time!

Ludo’.

Re: A plan for parameterized packages

[zimoun]

Hi,

On Fri, 20 Nov 2020 at 12:40, Ludovic Courtès <ludo@gnu.org> wrote:

> > The issue is that despite all that, the size of the images tend to
> > increase too rapidly over time[1].
>
> Yeah, that’s also the problem here: we have ‘guix size’ to profile a
> package at one point in time, but it’s easy to unwillingly increase its
> closure size the next day without noticing.

I think that should be part of the tooling we need to ease the
"release process".  I mean, that's what I have tried to describe as
one starting point at the BoF discussion: tools that help to know how
is the shape of Guix at one point in time (via scripts and repl or via
commands and options) and then time-machine does the rest.

What is currently missing (from my little experience):

  - check reproducibility
  - check size
  - check substitutes per build system, list the "essentials" (the
ones we *absolutly* want to be substituable), maybe per groupes by
topics


> Chris: does the Data Service track store item sizes (and more generally
> everything ‘query-path-info’ returns)?  It’d be great to be able to
> visualize size plots over time!

Yeah, plot could be really useful.


All the best,
simon

Re: A plan for parameterized packages

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 979 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org> skribis:
>
>> The issue is that despite all that, the size of the images tend to
>> increase too rapidly over time[1].
>
> Yeah, that’s also the problem here: we have ‘guix size’ to profile a
> package at one point in time, but it’s easy to unwillingly increase its
> closure size the next day without noticing.
>
> Chris: does the Data Service track store item sizes (and more generally
> everything ‘query-path-info’ returns)?  It’d be great to be able to
> visualize size plots over time!

The Guix Data Service can store narinfos, which I believe contain all
the information you need to do something similar to what `guix size`
does. I believe you'd be able to track the size of individual items, as
well as closure size through the reference information.

So, yeah, I think it's definately possible to do what you describe.

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: A plan for parameterized packages

[Ludovic Courtès]

Hi,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> For the embedded/flash rom side:

Note: it’s great if it can help reduce closure size, but that’s not the
goal.

> * Enable/disable building the documentation.  I really don't need a 40 MiB
> manual stored onto a 16 MiB firmware flash chip.  If that's better done as an
> extra output, fair enough.

Extra output is better, yup.  :-)

> * Enable/disable obscure dependencies: Library packages that pull in 400 MiB Qt
> into the closure for a thing no one (TM) uses should probably provide a switch to
> disable that GUI.  Sometimes a package provides multiple GUIs for different
> toolkits, in which case one should be able to choose one toolkit and not
> build for the others.

Yes, that sounds like a good use case.

> * No, even in 2020, I won't start using AmigaFS, NFSv3 and whatever old
> protocol/format is still around and superseded by other protocols.
>
> The gexp-functionality of being able to select individual files of a package
> is already very useful for use cases an embedded developer would have
> (and that's there for a long time already).  So that's nice!
>
> For the kind of feature flags I have in mind, it usually means I don't want
> to have the feature *anywhere*--for example, if I don't want to have Qt or
> Kerberos or whatever, that's because I want to save the space and thus
> it should be able to be *globally* specified--at least per profile.

OK, so that’s similar to what Pierre was suggesting: “global”
parameters, or at least parameters that apply to all the packages that
know about it, not just to one package.

Sounds doable, but again, the challenge will be to build all the
combinations.

> I would advise against doing a grep -r -- --enable and introducing all those
> as parameters.  Rather I would check the closure of stuff and if the closure
> goes from 1200 MiB to 50 MiB, chances are a parameter would be nice there :)

Heh, sounds like a valid criterion.

I guess initially we’ll have to use them sparsely (in Guix proper) so we
can gain experience with them.

> From experience with Gentoo before I can tell you that the combinatory
> explosion is a real problem and most of the "more advanced" (toggled more
> switches :) ) combinations did not work the majority of the time.

Right, and that’s precisely what I’d like to avoid, at least for the
packages in Guix proper (authors of external channels might have a
different policy.)  So I suppose in Guix we’d have a policy of using
them sparsely, and only/primarily in leaf packages.

Thanks,
Ludo’.