Re: A plan for parameterized packages

["Nicolò Balzarotti" <anothersms@gmail.com>]

Neat! Thanks for working on it :)

> An important question: do we have examples of packages for which we’d
> like to have parameters?  I’d grepped for “inherit” and that yields a
> few potential candidates, but also maybe a few potential non-candidates.
> Would this be a good fit for them?

What about evince?  eps/ps/dvi rendering is disabled, according to this
thread [1] for security reasons, so it makes sense to have it disabled
by default.  But I'd like to be able to enable it easily.

WDYT?

Nicolò

[1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=39674

Ludovic Courtès <ludo@gnu.org> writes:

> Hello Guix!
>
> For some time we’ve discussed ways to achieve “parameterized
> packages”—packages where one can from the command line or from Scheme
> configure optional build-time features, similar to Gentoo “USE flags”.
>
> I still have mixed feeling about this feature: on one hand it can bring
> much welcome flexibility, but on the other hand it can also lead us to
> the wild west of untested package configurations and combinatorial
> explosion.  It could also be argued that we achieve something similar
> today by simply defining package variants when we have to:
>
>   https://guix.gnu.org/manual/devel/en/html_node/Defining-Package-Variants.html
>
> That said, this message is about a possible implementation of package
> parameters, so here we go.  :-)
>
> To me the requirements for package parameters are:
>
>   1. it must be possible to discover them and choose them from the UI;
>
>   2. they must contain on-line internationalized documentation such that
>      the UI can list a package’s parameters and their type;
>
>   3. the chosen parameters when installing a package in a profile must
>      be preserved;
>
>   4. it must be possible to enumerate all the possible values of a
>      parameter, and thus to build the Cartesian product of all the
>      possible parameter combinations of a package (or of a package
>      graph!), so we can test those combinations as much as possible.
>
> This leads to the patches below.  The last one gives an example use for
> Bitlbee: it adds a “libpurple” parameter that allows users to choose
> whether or not to enable the optional libpurple dependency:
>
> --8<---------------cut here---------------start------------->8---
> $ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libpurple=true -n
> The following derivation would be built:
>    /gnu/store/mkknqgjsa93ajcl5d2krngizb11j1b0q-bitlbee-3.6.drv
> $ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libpurple=false -n
> /gnu/store/c2ckg51ffwgs6jni3l549k06w3jd3b7a-bitlbee-3.6
> $ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libpurple=wat? -n
> guix build: error: wrong value
> $ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libviolet=true -n
> gnu/packages/messaging.scm:283:5: error: libviolet: no such package parameter
> $ ./pre-inst-env guix show bitlbee
> name: bitlbee
> version: 3.6
> outputs: out
> parameters: libpurple
> systems: x86_64-linux i686-linux
> dependencies: check@0.12.0 glib@2.62.6 gnutls@3.6.12 libotr@4.1.1 perl@5.30.2 pkg-config@0.29.2 python@3.8.2
> location: gnu/packages/messaging.scm:243:2
> homepage: https://www.bitlbee.org/
> license: GPL 2+, FreeBSD
> synopsis: IRC to instant messaging gateway  
> description: BitlBee brings IM (instant messaging) to IRC clients, for people who have an IRC client running all
> + the time and don't want to run an additional IM client.  BitlBee currently supports XMPP/Jabber (including Google
> + Talk), MSN Messenger, Yahoo! Messenger, AIM and ICQ, and the Twitter microblogging network (plus all other Twitter
> + API compatible services like identi.ca and status.net).
>
> $ ./pre-inst-env guix install bitlbee --with-parameter=bitlbee=libpurple=true -p /tmp/test-bitlbee
> The following package will be installed:
>    bitlbee 3.6
>
> The following derivation will be built:
>    /gnu/store/clvs5521v5ybdw1z1yh97z2ky1dxm4d9-bitlbee-3.6.drv
>
> [...]
>
> $ cat /tmp/test-bitlbee/manifest
> ;; This file was automatically generated and is for internal use only.
> ;; It cannot be passed to the '--manifest' option.
>
> (manifest
>   (version 3)
>   (packages
>     (("bitlbee"
>       "3.6"
>       "out"
>       "/gnu/store/d67r9k5hwfm5hkd1d3pbhg49fcc2jj4q-bitlbee-3.6"
>       (propagated-inputs ())
>       (search-paths ())
>       (properties
>         (transformations
>           (with-parameter . "bitlbee=libpurple=true")))))))
> --8<---------------cut here---------------end--------------->8---
>
> That’s it!
>
> We would need more things, like a ‘guix parameters’ command to show the
> available parameters of a package and an ‘--all-parameter-values’ option
> to ‘guix build’ to build all the variants of a given package.
>
> An important question: do we have examples of packages for which we’d
> like to have parameters?  I’d grepped for “inherit” and that yields a
> few potential candidates, but also maybe a few potential non-candidates.
> Would this be a good fit for them?
>
> Thoughts?
>
> Ludo’.
>
> From 9155411f2e8e78922e1e46d92068ac8f652ff0a5 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
> Date: Sun, 15 Nov 2020 17:01:14 +0100
> Subject: [PATCH 3/4] ui: 'package->recutils' emits "parameters" field.
>
> * guix/ui.scm (package->recutils): Add "parameters" recutils field.
> ---
>  guix/ui.scm | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/guix/ui.scm b/guix/ui.scm
> index 4e686297e8..2485400cc9 100644
> --- a/guix/ui.scm
> +++ b/guix/ui.scm
> @@ -60,6 +60,7 @@
>                  #:hide (package-name->name+version
>                          ;; Avoid "overrides core binding" warning.
>                          delete))
> +  #:autoload   (guix parameters) (package-parameters package-parameter-name)
>    #:use-module (srfi srfi-1)
>    #:use-module (srfi srfi-9 gnu)
>    #:use-module (srfi srfi-11)
> @@ -1529,6 +1530,10 @@ HYPERLINKS? is true, emit hyperlink escape sequences when appropriate."
>    (format port "name: ~a~%" (package-name p))
>    (format port "version: ~a~%" (package-version p))
>    (format port "outputs: ~a~%" (string-join (package-outputs p)))
> +  (match (package-parameters p)
> +    (() #t)
> +    (lst (format port "parameters:~{ ~a~}~%"
> +                 (map package-parameter-name lst))))
>    (format port "systems: ~a~%"
>            (string-join (package-transitive-supported-systems p)))
>    (format port "dependencies: ~a~%"
> -- 
> 2.29.2
>
> From 49d7746ada4d4674acbbfd2606ad9bff4f6207eb Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
> Date: Sun, 15 Nov 2020 17:04:18 +0100
> Subject: [PATCH 4/4] gnu: bitlbee: Add "libpurple" parameter.
>
> * gnu/packages/messaging.scm (bitlbee)[inputs]: Add optional PIDGIN
> input.
> [properties]: New field.
> (bitlbee-purple): Mark as deprecated.
> ---
>  gnu/packages/messaging.scm | 43 +++++++++++++++++++-------------------
>  1 file changed, 22 insertions(+), 21 deletions(-)
>
> diff --git a/gnu/packages/messaging.scm b/gnu/packages/messaging.scm
> index b462504894..2f0f44d10d 100644
> --- a/gnu/packages/messaging.scm
> +++ b/gnu/packages/messaging.scm
> @@ -120,6 +120,7 @@
>    #:use-module (guix build-system trivial)
>    #:use-module (guix download)
>    #:use-module (guix git-download)
> +  #:use-module (guix parameters)
>    #:use-module (guix hg-download)
>    #:use-module ((guix licenses) #:prefix license:)
>    #:use-module (guix packages)
> @@ -256,7 +257,8 @@ end-to-end encryption.")
>                ("libotr" ,libotr)
>                ("gnutls" ,gnutls)
>                ("python" ,python)
> -              ("perl" ,perl)))
> +              ("perl" ,perl)
> +              ,@(optionally 'libpurple? `("purple" ,pidgin))))
>      (arguments
>       `(#:phases
>         (modify-phases %standard-phases
> @@ -275,7 +277,21 @@ end-to-end encryption.")
>               (invoke "./configure"
>                       (string-append "--prefix="
>                                      (assoc-ref outputs "out"))
> -                     "--otr=1"))))))
> +                     "--otr=1"
> +                     ,@(optionally 'libpurple? "--purple=1")))))
> +
> +       ;; XXX: Tests fail to link, and ./configure says that it's "supported
> +       ;; on a best-effort basis" anyway.
> +       #:tests? ,(not (assq-ref (package-properties this-package)
> +                                'libpurple?))))
> +    (properties
> +     `((parameters
> +        ,(package-parameter (name "libpurple")
> +                            (description
> +                             "Whether to enable libpurple (Pidgin)
> +support.")
> +                            (property 'libpurple?)
> +                            (type boolean)))))
>      (synopsis "IRC to instant messaging gateway")
>      (description "BitlBee brings IM (instant messaging) to IRC clients, for
>  people who have an IRC client running all the time and don't want to run an
> @@ -289,25 +305,10 @@ identi.ca and status.net).")
>  (define-public bitlbee-purple
>    ;; This variant uses libpurple, which provides support for more protocols at
>    ;; the expense of a much bigger closure.
> -  (package/inherit bitlbee
> -    (name "bitlbee-purple")
> -    (synopsis "IRC to instant messaging gateway (using Pidgin's libpurple)")
> -    (inputs `(("purple" ,pidgin)
> -              ,@(package-inputs bitlbee)))
> -    (arguments
> -     (substitute-keyword-arguments (package-arguments bitlbee)
> -       ((#:phases phases '%standard-phases)
> -        `(modify-phases ,phases
> -           (replace 'configure                    ;add "--purple=1"
> -             (lambda* (#:key outputs #:allow-other-keys)
> -               (invoke "./configure"
> -                       (string-append "--prefix="
> -                                      (assoc-ref outputs "out"))
> -                       "--otr=1" "--purple=1")))))
> -       ((#:tests? _ #t)
> -        ;; XXX: Tests fail to link, and ./configure says that it's "supported
> -        ;; on a best-effort basis" anyway.
> -        #f)))))
> +  (deprecated-package "bitlbee-purple"
> +                      (package/inherit bitlbee
> +                        (properties `((libpurple? . #t)
> +                                      ,@(package-properties bitlbee))))))
>  
>  (define-public bitlbee-discord
>    (package
> -- 
> 2.29.2
>
> From f42b68499c4e2a9bd368fe6a516932f5afa7a189 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
> Date: Sun, 15 Nov 2020 16:58:52 +0100
> Subject: [PATCH 1/4] DRAFT Add (guix parameters).
>
> DRAFT: Missing tests & doc.
>
> * guix/parameters.scm: New file.
> * Makefile.am (MODULES): Add it.
> ---
>  Makefile.am         |   1 +
>  guix/parameters.scm | 131 ++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 132 insertions(+)
>  create mode 100644 guix/parameters.scm
>
> diff --git a/Makefile.am b/Makefile.am
> index e7053ee4f4..72f955360d 100644
> --- a/Makefile.am
> +++ b/Makefile.am
> @@ -235,6 +235,7 @@ MODULES =					\
>    guix/build/make-bootstrap.scm			\
>    guix/search-paths.scm				\
>    guix/packages.scm				\
> +  guix/parameters.scm				\
>    guix/import/cabal.scm				\
>    guix/import/cpan.scm				\
>    guix/import/cran.scm				\
> diff --git a/guix/parameters.scm b/guix/parameters.scm
> new file mode 100644
> index 0000000000..e4f8240aa4
> --- /dev/null
> +++ b/guix/parameters.scm
> @@ -0,0 +1,131 @@
> +;;; GNU Guix --- Functional package management for GNU
> +;;; Copyright © 2020 Ludovic Courtès <ludo@gnu.org>
> +;;;
> +;;; This file is part of GNU Guix.
> +;;;
> +;;; GNU Guix is free software; you can redistribute it and/or modify it
> +;;; under the terms of the GNU General Public License as published by
> +;;; the Free Software Foundation; either version 3 of the License, or (at
> +;;; your option) any later version.
> +;;;
> +;;; GNU Guix is distributed in the hope that it will be useful, but
> +;;; WITHOUT ANY WARRANTY; without even the implied warranty of
> +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +;;; GNU General Public License for more details.
> +;;;
> +;;; You should have received a copy of the GNU General Public License
> +;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
> +
> +(define-module (guix parameters)
> +  #:use-module (guix packages)
> +  #:use-module (guix records)
> +  #:use-module (guix diagnostics)
> +  #:use-module (guix i18n)
> +  #:use-module (srfi srfi-1)
> +  #:use-module (srfi srfi-34)
> +  #:use-module (srfi srfi-35)
> +  #:use-module (ice-9 match)
> +  #:export (package-parameter
> +            package-parameter?
> +            package-parameter-name
> +            package-parameter-property
> +            package-parameter-type
> +            package-parameter-description
> +
> +            boolean
> +            optionally
> +
> +            package-parameters
> +            lookup-package-parameter
> +            package-parameter-value
> +            set-package-parameter-value))
> +
> +;;; Commentary:
> +;;;
> +;;; This module provides a way to express high-level "package parameters",
> +;;; which allow users to customize how packages are built.  Parameters are an
> +;;; interface that package developers define, where each parameter has a name
> +;;; and type.  The user interface then converts parameter values from string
> +;;; to Scheme values and records them in the package properties.
> +;;;
> +;;; Package parameters are discoverable; their description is
> +;;; internationalized.  The possible values of a parameter can be enumerated,
> +;;; and thus the Cartesian product of all possible parameter values for a
> +;;; package can be enumerated as well.
> +;;;
> +;;; Code:
> +
> +;; Package parameter interface.
> +(define-record-type* <package-parameter> package-parameter
> +  make-package-parameter
> +  package-parameter?
> +  (name          package-parameter-name)
> +  (property      package-parameter-property (default (string->symbol name)))
> +  (type          package-parameter-type)
> +  (description   package-parameter-description))
> +
> +;; Type of a package parameter.
> +(define-record-type* <parameter-type> parameter-type
> +  make-parameter-type
> +  parameter-type?
> +  (name          parameter-type-name)              ;debugging purposes only!
> +  (string->value parameter-type-string->value)
> +  (value->string parameter-type-value->string)
> +  (universe      parameter-type-universe))
> +
> +(define boolean
> +  ;; The Boolean parameter type.
> +  (parameter-type (name 'boolean)
> +                  (universe '(#true #false))
> +                  (value->string
> +                   (match-lambda
> +                     (#f "false")
> +                     (#t "true")))
> +                  (string->value
> +                   (lambda (str)
> +                     (cond ((string-ci=? str "true")
> +                            #t)
> +                           ((string-ci=? str "false")
> +                            #f)
> +                           (else
> +                            (raise (condition
> +                                    (&message (message "wrong value"))))))))))
> +
> +(define (package-parameters package)
> +  (or (assq-ref (package-properties package) 'parameters)
> +      '()))
> +
> +(define (package-parameter-value package parameter)
> +  (assq-ref (package-properties package)
> +            (package-parameter-property parameter)))
> +
> +(define (lookup-package-parameter package name)
> +  (find (lambda (parameter)
> +          (string=? (package-parameter-name parameter) name))
> +        (package-parameters package)))
> +
> +(define (set-package-parameter-value package name value)
> +  (let ((parameter (lookup-package-parameter package name))
> +        (location  (package-field-location package 'properties)))
> +    (unless parameter
> +      (raise (apply make-compound-condition
> +                    (formatted-message
> +                     (G_ "~a: no such package parameter")
> +                     name)
> +                    (if location
> +                        (list (condition
> +                               (&error-location (location location))))
> +                        '()))))
> +    (let* ((property (package-parameter-property parameter))
> +           (type     (package-parameter-type parameter))
> +           (value    ((parameter-type-string->value type) value)))
> +      (package/inherit package
> +        (properties
> +         (alist-cons property value
> +                     (alist-delete property (package-properties package)
> +                                   eq?)))))))
> +
> +(define-syntax-rule (optionally property exp)
> +  (if (assq-ref (package-properties this-package) property)
> +      (list exp)
> +      '()))
> -- 
> 2.29.2
>
> From 5d6d81e4c3e37de53fe7f62ff7ef94da8b2df033 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
> Date: Sun, 15 Nov 2020 16:59:48 +0100
> Subject: [PATCH 2/4] DRAFT transformations: Add '--with-parameter'.
>
> DRAFT: Missing tests & doc.
>
> * guix/transformations.scm (evaluate-parameter-specs)
> (transform-package-parameters): New procedures.
> (%transformations, %transformation-options): Add 'with-parameter'.
> ---
>  guix/transformations.scm | 39 +++++++++++++++++++++++++++++++++++++++
>  1 file changed, 39 insertions(+)
>
> diff --git a/guix/transformations.scm b/guix/transformations.scm
> index 30142dd059..0f83eb470d 100644
> --- a/guix/transformations.scm
> +++ b/guix/transformations.scm
> @@ -25,6 +25,7 @@
>    #:autoload   (guix download) (download-to-store)
>    #:autoload   (guix git-download) (git-reference? git-reference-url)
>    #:autoload   (guix git) (git-checkout git-checkout? git-checkout-url)
> +  #:autoload   (guix parameters) (set-package-parameter-value)
>    #:use-module (guix utils)
>    #:use-module (guix memoization)
>    #:use-module (guix gexp)
> @@ -324,6 +325,41 @@ a checkout of the Git repository at the given URL."
>          (rewrite obj)
>          obj)))
>  
> +(define (evaluate-parameter-specs specs proc)
> +  "Parse SPECS, a list of strings like \"bitlbee=purple=true\", and return a
> +list of spec/procedure pairs, where (PROC PACKAGE PARAMETER VALUE) is called
> +to return the replacement package.  Raise an error if an element of SPECS uses
> +invalid syntax, or if a package it refers to could not be found."
> +  (map (lambda (spec)
> +         (match (string-tokenize spec %not-equal)
> +           ((spec name value)
> +            (define (replace old)
> +              (proc old name value))
> +
> +            (cons spec replace))
> +           (_
> +            (raise
> +             (formatted-message
> +              (G_ "invalid package parameter specification: ~s")
> +              spec)))))
> +       specs))
> +
> +(define (transform-package-parameters replacement-specs)
> +  "Return a procedure that, when passed a package, replaces its direct
> +dependencies according to REPLACEMENT-SPECS.  REPLACEMENT-SPECS is a list of
> +strings like \"guile-next=stable-3.0\" meaning that packages are built using
> +'guile-next' from the latest commit on its 'stable-3.0' branch."
> +  (define (replace old name value)
> +    (set-package-parameter-value old name value))
> +
> +  (let* ((replacements (evaluate-parameter-specs replacement-specs
> +                                                 replace))
> +         (rewrite      (package-input-rewriting/spec replacements)))
> +    (lambda (obj)
> +      (if (package? obj)
> +          (rewrite obj)
> +          obj))))
> +
>  (define (package-dependents/spec top bottom)
>    "Return the list of dependents of BOTTOM, a spec string, that are also
>  dependencies of TOP, a package."
> @@ -467,6 +503,7 @@ to the same package but with #:strip-binaries? #f in its 'arguments' field."
>      (with-branch . ,transform-package-source-branch)
>      (with-commit . ,transform-package-source-commit)
>      (with-git-url . ,transform-package-source-git-url)
> +    (with-parameter . ,transform-package-parameters)
>      (with-c-toolchain . ,transform-package-toolchain)
>      (with-debug-info . ,transform-package-with-debug-info)
>      (without-tests . ,transform-package-tests)))
> @@ -503,6 +540,8 @@ to the same package but with #:strip-binaries? #f in its 'arguments' field."
>                    (parser 'with-commit))
>            (option '("with-git-url") #t #f
>                    (parser 'with-git-url))
> +          (option '("with-parameter") #t #f
> +                  (parser 'with-parameter))
>            (option '("with-c-toolchain") #t #f
>                    (parser 'with-c-toolchain))
>            (option '("with-debug-info") #t #f
> -- 
> 2.29.2