Re: A plan for parameterized packages

["Nicolò Balzarotti" <anothersms@gmail.com>]

More in general, grepping for `--enable` or `--with` flags shows
a lot of potential packages (but also many false positives).


Nicolò Balzarotti <anothersms@gmail.com> writes:

> Neat! Thanks for working on it :)
>
>> An important question: do we have examples of packages for which we’d
>> like to have parameters?  I’d grepped for “inherit” and that yields a
>> few potential candidates, but also maybe a few potential non-candidates.
>> Would this be a good fit for them?
>
> What about evince?  eps/ps/dvi rendering is disabled, according to this
> thread [1] for security reasons, so it makes sense to have it disabled
> by default.  But I'd like to be able to enable it easily.
>
> WDYT?
>
> Nicolò
>
> [1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=39674
>
> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Hello Guix!
>>
>> For some time we’ve discussed ways to achieve “parameterized
>> packages”—packages where one can from the command line or from Scheme
>> configure optional build-time features, similar to Gentoo “USE flags”.
>>
>> I still have mixed feeling about this feature: on one hand it can bring
>> much welcome flexibility, but on the other hand it can also lead us to
>> the wild west of untested package configurations and combinatorial
>> explosion.  It could also be argued that we achieve something similar
>> today by simply defining package variants when we have to:
>>
>>   https://guix.gnu.org/manual/devel/en/html_node/Defining-Package-Variants.html
>>
>> That said, this message is about a possible implementation of package
>> parameters, so here we go.  :-)
>>
>> To me the requirements for package parameters are:
>>
>>   1. it must be possible to discover them and choose them from the UI;
>>
>>   2. they must contain on-line internationalized documentation such that
>>      the UI can list a package’s parameters and their type;
>>
>>   3. the chosen parameters when installing a package in a profile must
>>      be preserved;
>>
>>   4. it must be possible to enumerate all the possible values of a
>>      parameter, and thus to build the Cartesian product of all the
>>      possible parameter combinations of a package (or of a package
>>      graph!), so we can test those combinations as much as possible.
>>
>> This leads to the patches below.  The last one gives an example use for
>> Bitlbee: it adds a “libpurple” parameter that allows users to choose
>> whether or not to enable the optional libpurple dependency:
>>
>> --8<---------------cut here---------------start------------->8---
>> $ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libpurple=true -n
>> The following derivation would be built:
>>    /gnu/store/mkknqgjsa93ajcl5d2krngizb11j1b0q-bitlbee-3.6.drv
>> $ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libpurple=false -n
>> /gnu/store/c2ckg51ffwgs6jni3l549k06w3jd3b7a-bitlbee-3.6
>> $ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libpurple=wat? -n
>> guix build: error: wrong value
>> $ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libviolet=true -n
>> gnu/packages/messaging.scm:283:5: error: libviolet: no such package parameter
>> $ ./pre-inst-env guix show bitlbee
>> name: bitlbee
>> version: 3.6
>> outputs: out
>> parameters: libpurple
>> systems: x86_64-linux i686-linux
>> dependencies: check@0.12.0 glib@2.62.6 gnutls@3.6.12 libotr@4.1.1 perl@5.30.2 pkg-config@0.29.2 python@3.8.2
>> location: gnu/packages/messaging.scm:243:2
>> homepage: https://www.bitlbee.org/
>> license: GPL 2+, FreeBSD
>> synopsis: IRC to instant messaging gateway  
>> description: BitlBee brings IM (instant messaging) to IRC clients, for people who have an IRC client running all
>> + the time and don't want to run an additional IM client.  BitlBee currently supports XMPP/Jabber (including Google
>> + Talk), MSN Messenger, Yahoo! Messenger, AIM and ICQ, and the Twitter microblogging network (plus all other Twitter
>> + API compatible services like identi.ca and status.net).
>>
>> $ ./pre-inst-env guix install bitlbee --with-parameter=bitlbee=libpurple=true -p /tmp/test-bitlbee
>> The following package will be installed:
>>    bitlbee 3.6
>>
>> The following derivation will be built:
>>    /gnu/store/clvs5521v5ybdw1z1yh97z2ky1dxm4d9-bitlbee-3.6.drv
>>
>> [...]
>>
>> $ cat /tmp/test-bitlbee/manifest
>> ;; This file was automatically generated and is for internal use only.
>> ;; It cannot be passed to the '--manifest' option.
>>
>> (manifest
>>   (version 3)
>>   (packages
>>     (("bitlbee"
>>       "3.6"
>>       "out"
>>       "/gnu/store/d67r9k5hwfm5hkd1d3pbhg49fcc2jj4q-bitlbee-3.6"
>>       (propagated-inputs ())
>>       (search-paths ())
>>       (properties
>>         (transformations
>>           (with-parameter . "bitlbee=libpurple=true")))))))
>> --8<---------------cut here---------------end--------------->8---
>>
>> That’s it!
>>
>> We would need more things, like a ‘guix parameters’ command to show the
>> available parameters of a package and an ‘--all-parameter-values’ option
>> to ‘guix build’ to build all the variants of a given package.
>>
>> An important question: do we have examples of packages for which we’d
>> like to have parameters?  I’d grepped for “inherit” and that yields a
>> few potential candidates, but also maybe a few potential non-candidates.
>> Would this be a good fit for them?
>>
>> Thoughts?
>>
>> Ludo’.
>>
>> From 9155411f2e8e78922e1e46d92068ac8f652ff0a5 Mon Sep 17 00:00:00 2001
>> From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
>> Date: Sun, 15 Nov 2020 17:01:14 +0100
>> Subject: [PATCH 3/4] ui: 'package->recutils' emits "parameters" field.
>>
>> * guix/ui.scm (package->recutils): Add "parameters" recutils field.
>> ---
>>  guix/ui.scm | 5 +++++
>>  1 file changed, 5 insertions(+)
>>
>> diff --git a/guix/ui.scm b/guix/ui.scm
>> index 4e686297e8..2485400cc9 100644
>> --- a/guix/ui.scm
>> +++ b/guix/ui.scm
>> @@ -60,6 +60,7 @@
>>                  #:hide (package-name->name+version
>>                          ;; Avoid "overrides core binding" warning.
>>                          delete))
>> +  #:autoload   (guix parameters) (package-parameters package-parameter-name)
>>    #:use-module (srfi srfi-1)
>>    #:use-module (srfi srfi-9 gnu)
>>    #:use-module (srfi srfi-11)
>> @@ -1529,6 +1530,10 @@ HYPERLINKS? is true, emit hyperlink escape sequences when appropriate."
>>    (format port "name: ~a~%" (package-name p))
>>    (format port "version: ~a~%" (package-version p))
>>    (format port "outputs: ~a~%" (string-join (package-outputs p)))
>> +  (match (package-parameters p)
>> +    (() #t)
>> +    (lst (format port "parameters:~{ ~a~}~%"
>> +                 (map package-parameter-name lst))))
>>    (format port "systems: ~a~%"
>>            (string-join (package-transitive-supported-systems p)))
>>    (format port "dependencies: ~a~%"
>> -- 
>> 2.29.2
>>
>> From 49d7746ada4d4674acbbfd2606ad9bff4f6207eb Mon Sep 17 00:00:00 2001
>> From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
>> Date: Sun, 15 Nov 2020 17:04:18 +0100
>> Subject: [PATCH 4/4] gnu: bitlbee: Add "libpurple" parameter.
>>
>> * gnu/packages/messaging.scm (bitlbee)[inputs]: Add optional PIDGIN
>> input.
>> [properties]: New field.
>> (bitlbee-purple): Mark as deprecated.
>> ---
>>  gnu/packages/messaging.scm | 43 +++++++++++++++++++-------------------
>>  1 file changed, 22 insertions(+), 21 deletions(-)
>>
>> diff --git a/gnu/packages/messaging.scm b/gnu/packages/messaging.scm
>> index b462504894..2f0f44d10d 100644
>> --- a/gnu/packages/messaging.scm
>> +++ b/gnu/packages/messaging.scm
>> @@ -120,6 +120,7 @@
>>    #:use-module (guix build-system trivial)
>>    #:use-module (guix download)
>>    #:use-module (guix git-download)
>> +  #:use-module (guix parameters)
>>    #:use-module (guix hg-download)
>>    #:use-module ((guix licenses) #:prefix license:)
>>    #:use-module (guix packages)
>> @@ -256,7 +257,8 @@ end-to-end encryption.")
>>                ("libotr" ,libotr)
>>                ("gnutls" ,gnutls)
>>                ("python" ,python)
>> -              ("perl" ,perl)))
>> +              ("perl" ,perl)
>> +              ,@(optionally 'libpurple? `("purple" ,pidgin))))
>>      (arguments
>>       `(#:phases
>>         (modify-phases %standard-phases
>> @@ -275,7 +277,21 @@ end-to-end encryption.")
>>               (invoke "./configure"
>>                       (string-append "--prefix="
>>                                      (assoc-ref outputs "out"))
>> -                     "--otr=1"))))))
>> +                     "--otr=1"
>> +                     ,@(optionally 'libpurple? "--purple=1")))))
>> +
>> +       ;; XXX: Tests fail to link, and ./configure says that it's "supported
>> +       ;; on a best-effort basis" anyway.
>> +       #:tests? ,(not (assq-ref (package-properties this-package)
>> +                                'libpurple?))))
>> +    (properties
>> +     `((parameters
>> +        ,(package-parameter (name "libpurple")
>> +                            (description
>> +                             "Whether to enable libpurple (Pidgin)
>> +support.")
>> +                            (property 'libpurple?)
>> +                            (type boolean)))))
>>      (synopsis "IRC to instant messaging gateway")
>>      (description "BitlBee brings IM (instant messaging) to IRC clients, for
>>  people who have an IRC client running all the time and don't want to run an
>> @@ -289,25 +305,10 @@ identi.ca and status.net).")
>>  (define-public bitlbee-purple
>>    ;; This variant uses libpurple, which provides support for more protocols at
>>    ;; the expense of a much bigger closure.
>> -  (package/inherit bitlbee
>> -    (name "bitlbee-purple")
>> -    (synopsis "IRC to instant messaging gateway (using Pidgin's libpurple)")
>> -    (inputs `(("purple" ,pidgin)
>> -              ,@(package-inputs bitlbee)))
>> -    (arguments
>> -     (substitute-keyword-arguments (package-arguments bitlbee)
>> -       ((#:phases phases '%standard-phases)
>> -        `(modify-phases ,phases
>> -           (replace 'configure                    ;add "--purple=1"
>> -             (lambda* (#:key outputs #:allow-other-keys)
>> -               (invoke "./configure"
>> -                       (string-append "--prefix="
>> -                                      (assoc-ref outputs "out"))
>> -                       "--otr=1" "--purple=1")))))
>> -       ((#:tests? _ #t)
>> -        ;; XXX: Tests fail to link, and ./configure says that it's "supported
>> -        ;; on a best-effort basis" anyway.
>> -        #f)))))
>> +  (deprecated-package "bitlbee-purple"
>> +                      (package/inherit bitlbee
>> +                        (properties `((libpurple? . #t)
>> +                                      ,@(package-properties bitlbee))))))
>>  
>>  (define-public bitlbee-discord
>>    (package
>> -- 
>> 2.29.2
>>
>> From f42b68499c4e2a9bd368fe6a516932f5afa7a189 Mon Sep 17 00:00:00 2001
>> From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
>> Date: Sun, 15 Nov 2020 16:58:52 +0100
>> Subject: [PATCH 1/4] DRAFT Add (guix parameters).
>>
>> DRAFT: Missing tests & doc.
>>
>> * guix/parameters.scm: New file.
>> * Makefile.am (MODULES): Add it.
>> ---
>>  Makefile.am         |   1 +
>>  guix/parameters.scm | 131 ++++++++++++++++++++++++++++++++++++++++++++
>>  2 files changed, 132 insertions(+)
>>  create mode 100644 guix/parameters.scm
>>
>> diff --git a/Makefile.am b/Makefile.am
>> index e7053ee4f4..72f955360d 100644
>> --- a/Makefile.am
>> +++ b/Makefile.am
>> @@ -235,6 +235,7 @@ MODULES =					\
>>    guix/build/make-bootstrap.scm			\
>>    guix/search-paths.scm				\
>>    guix/packages.scm				\
>> +  guix/parameters.scm				\
>>    guix/import/cabal.scm				\
>>    guix/import/cpan.scm				\
>>    guix/import/cran.scm				\
>> diff --git a/guix/parameters.scm b/guix/parameters.scm
>> new file mode 100644
>> index 0000000000..e4f8240aa4
>> --- /dev/null
>> +++ b/guix/parameters.scm
>> @@ -0,0 +1,131 @@
>> +;;; GNU Guix --- Functional package management for GNU
>> +;;; Copyright © 2020 Ludovic Courtès <ludo@gnu.org>
>> +;;;
>> +;;; This file is part of GNU Guix.
>> +;;;
>> +;;; GNU Guix is free software; you can redistribute it and/or modify it
>> +;;; under the terms of the GNU General Public License as published by
>> +;;; the Free Software Foundation; either version 3 of the License, or (at
>> +;;; your option) any later version.
>> +;;;
>> +;;; GNU Guix is distributed in the hope that it will be useful, but
>> +;;; WITHOUT ANY WARRANTY; without even the implied warranty of
>> +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> +;;; GNU General Public License for more details.
>> +;;;
>> +;;; You should have received a copy of the GNU General Public License
>> +;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
>> +
>> +(define-module (guix parameters)
>> +  #:use-module (guix packages)
>> +  #:use-module (guix records)
>> +  #:use-module (guix diagnostics)
>> +  #:use-module (guix i18n)
>> +  #:use-module (srfi srfi-1)
>> +  #:use-module (srfi srfi-34)
>> +  #:use-module (srfi srfi-35)
>> +  #:use-module (ice-9 match)
>> +  #:export (package-parameter
>> +            package-parameter?
>> +            package-parameter-name
>> +            package-parameter-property
>> +            package-parameter-type
>> +            package-parameter-description
>> +
>> +            boolean
>> +            optionally
>> +
>> +            package-parameters
>> +            lookup-package-parameter
>> +            package-parameter-value
>> +            set-package-parameter-value))
>> +
>> +;;; Commentary:
>> +;;;
>> +;;; This module provides a way to express high-level "package parameters",
>> +;;; which allow users to customize how packages are built.  Parameters are an
>> +;;; interface that package developers define, where each parameter has a name
>> +;;; and type.  The user interface then converts parameter values from string
>> +;;; to Scheme values and records them in the package properties.
>> +;;;
>> +;;; Package parameters are discoverable; their description is
>> +;;; internationalized.  The possible values of a parameter can be enumerated,
>> +;;; and thus the Cartesian product of all possible parameter values for a
>> +;;; package can be enumerated as well.
>> +;;;
>> +;;; Code:
>> +
>> +;; Package parameter interface.
>> +(define-record-type* <package-parameter> package-parameter
>> +  make-package-parameter
>> +  package-parameter?
>> +  (name          package-parameter-name)
>> +  (property      package-parameter-property (default (string->symbol name)))
>> +  (type          package-parameter-type)
>> +  (description   package-parameter-description))
>> +
>> +;; Type of a package parameter.
>> +(define-record-type* <parameter-type> parameter-type
>> +  make-parameter-type
>> +  parameter-type?
>> +  (name          parameter-type-name)              ;debugging purposes only!
>> +  (string->value parameter-type-string->value)
>> +  (value->string parameter-type-value->string)
>> +  (universe      parameter-type-universe))
>> +
>> +(define boolean
>> +  ;; The Boolean parameter type.
>> +  (parameter-type (name 'boolean)
>> +                  (universe '(#true #false))
>> +                  (value->string
>> +                   (match-lambda
>> +                     (#f "false")
>> +                     (#t "true")))
>> +                  (string->value
>> +                   (lambda (str)
>> +                     (cond ((string-ci=? str "true")
>> +                            #t)
>> +                           ((string-ci=? str "false")
>> +                            #f)
>> +                           (else
>> +                            (raise (condition
>> +                                    (&message (message "wrong value"))))))))))
>> +
>> +(define (package-parameters package)
>> +  (or (assq-ref (package-properties package) 'parameters)
>> +      '()))
>> +
>> +(define (package-parameter-value package parameter)
>> +  (assq-ref (package-properties package)
>> +            (package-parameter-property parameter)))
>> +
>> +(define (lookup-package-parameter package name)
>> +  (find (lambda (parameter)
>> +          (string=? (package-parameter-name parameter) name))
>> +        (package-parameters package)))
>> +
>> +(define (set-package-parameter-value package name value)
>> +  (let ((parameter (lookup-package-parameter package name))
>> +        (location  (package-field-location package 'properties)))
>> +    (unless parameter
>> +      (raise (apply make-compound-condition
>> +                    (formatted-message
>> +                     (G_ "~a: no such package parameter")
>> +                     name)
>> +                    (if location
>> +                        (list (condition
>> +                               (&error-location (location location))))
>> +                        '()))))
>> +    (let* ((property (package-parameter-property parameter))
>> +           (type     (package-parameter-type parameter))
>> +           (value    ((parameter-type-string->value type) value)))
>> +      (package/inherit package
>> +        (properties
>> +         (alist-cons property value
>> +                     (alist-delete property (package-properties package)
>> +                                   eq?)))))))
>> +
>> +(define-syntax-rule (optionally property exp)
>> +  (if (assq-ref (package-properties this-package) property)
>> +      (list exp)
>> +      '()))
>> -- 
>> 2.29.2
>>
>> From 5d6d81e4c3e37de53fe7f62ff7ef94da8b2df033 Mon Sep 17 00:00:00 2001
>> From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
>> Date: Sun, 15 Nov 2020 16:59:48 +0100
>> Subject: [PATCH 2/4] DRAFT transformations: Add '--with-parameter'.
>>
>> DRAFT: Missing tests & doc.
>>
>> * guix/transformations.scm (evaluate-parameter-specs)
>> (transform-package-parameters): New procedures.
>> (%transformations, %transformation-options): Add 'with-parameter'.
>> ---
>>  guix/transformations.scm | 39 +++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 39 insertions(+)
>>
>> diff --git a/guix/transformations.scm b/guix/transformations.scm
>> index 30142dd059..0f83eb470d 100644
>> --- a/guix/transformations.scm
>> +++ b/guix/transformations.scm
>> @@ -25,6 +25,7 @@
>>    #:autoload   (guix download) (download-to-store)
>>    #:autoload   (guix git-download) (git-reference? git-reference-url)
>>    #:autoload   (guix git) (git-checkout git-checkout? git-checkout-url)
>> +  #:autoload   (guix parameters) (set-package-parameter-value)
>>    #:use-module (guix utils)
>>    #:use-module (guix memoization)
>>    #:use-module (guix gexp)
>> @@ -324,6 +325,41 @@ a checkout of the Git repository at the given URL."
>>          (rewrite obj)
>>          obj)))
>>  
>> +(define (evaluate-parameter-specs specs proc)
>> +  "Parse SPECS, a list of strings like \"bitlbee=purple=true\", and return a
>> +list of spec/procedure pairs, where (PROC PACKAGE PARAMETER VALUE) is called
>> +to return the replacement package.  Raise an error if an element of SPECS uses
>> +invalid syntax, or if a package it refers to could not be found."
>> +  (map (lambda (spec)
>> +         (match (string-tokenize spec %not-equal)
>> +           ((spec name value)
>> +            (define (replace old)
>> +              (proc old name value))
>> +
>> +            (cons spec replace))
>> +           (_
>> +            (raise
>> +             (formatted-message
>> +              (G_ "invalid package parameter specification: ~s")
>> +              spec)))))
>> +       specs))
>> +
>> +(define (transform-package-parameters replacement-specs)
>> +  "Return a procedure that, when passed a package, replaces its direct
>> +dependencies according to REPLACEMENT-SPECS.  REPLACEMENT-SPECS is a list of
>> +strings like \"guile-next=stable-3.0\" meaning that packages are built using
>> +'guile-next' from the latest commit on its 'stable-3.0' branch."
>> +  (define (replace old name value)
>> +    (set-package-parameter-value old name value))
>> +
>> +  (let* ((replacements (evaluate-parameter-specs replacement-specs
>> +                                                 replace))
>> +         (rewrite      (package-input-rewriting/spec replacements)))
>> +    (lambda (obj)
>> +      (if (package? obj)
>> +          (rewrite obj)
>> +          obj))))
>> +
>>  (define (package-dependents/spec top bottom)
>>    "Return the list of dependents of BOTTOM, a spec string, that are also
>>  dependencies of TOP, a package."
>> @@ -467,6 +503,7 @@ to the same package but with #:strip-binaries? #f in its 'arguments' field."
>>      (with-branch . ,transform-package-source-branch)
>>      (with-commit . ,transform-package-source-commit)
>>      (with-git-url . ,transform-package-source-git-url)
>> +    (with-parameter . ,transform-package-parameters)
>>      (with-c-toolchain . ,transform-package-toolchain)
>>      (with-debug-info . ,transform-package-with-debug-info)
>>      (without-tests . ,transform-package-tests)))
>> @@ -503,6 +540,8 @@ to the same package but with #:strip-binaries? #f in its 'arguments' field."
>>                    (parser 'with-commit))
>>            (option '("with-git-url") #t #f
>>                    (parser 'with-git-url))
>> +          (option '("with-parameter") #t #f
>> +                  (parser 'with-parameter))
>>            (option '("with-c-toolchain") #t #f
>>                    (parser 'with-c-toolchain))
>>            (option '("with-debug-info") #t #f
>> -- 
>> 2.29.2