Parameterized packages

Parameterized packages

[Ludovic Courtès]

Hello Guix!

While thinking about <https://issues.guix.info/issue/35671> and looking
for ways to allow users to install just the locales they need right from
‘guix package’, I realized that “parameterized packages” are a
low-hanging fruit (by “parameterized packages” I mean something kinda
like Gentoo USE flags, or like a procedure if you will, but with a
command-line interface.)

Namely, one could write:

  (package
    (name "glibc-utf8-locales")
    ;; …
    (arguments
      `(#:phases … ,(assoc-ref (package-properties this-package)
                               'locales)
        …)))


and then we’d have a package transformation option like, say,
‘--with-argument=glibc-utf8-locales:locales=zh_CN.utf8’, which would
add the right property, like so:

  (package
    (inherit glibc-utf8-locales)
    (properties `((locales . ("zh_CN.utf8")))))

and tadaam! we have a parameterized package.

There’s a couple of gotchas:

  • We’d need to store more info in manifest entries so that
    transformation options are preserved upon ‘guix upgrade’.

  • We might need to expose the package parameters somehow, so that if
    one types ‘--with-argument=foobar=baz’, they get a correct error
    message saying that “foobar” is not a known parameter.

  • We might want to make the CLI option less verbose too, but how?

Thoughts, comrades?

Thanks,
Ludo’.

Re: Parameterized packages

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 2275 bytes --]

Ludo',

Ludovic Courtès wrote:
> While thinking about <https://issues.guix.info/issue/35671> and 
> looking
> for ways to allow users to install just the locales they need 
> right from
> ‘guix package’, I realized that “parameterized packages” are a
> low-hanging fruit

Wow.  Unexpected turn…

I'm still thinking about this, so all this is just me doing it out 
loud:

>   (package
>     (inherit glibc-utf8-locales)
>     (properties `((locales . ("zh_CN.utf8")))))
>
> and tadaam! we have a parameterized package.
>
> There’s a couple of gotchas:
>
>   • We’d need to store more info in manifest entries so that
>     transformation options are preserved upon ‘guix upgrade’.
>
>   • We might need to expose the package parameters somehow, so 
>   that if
>     one types ‘--with-argument=foobar=baz’, they get a correct 
>     error
>     message saying that “foobar” is not a known parameter.

Interesting idea to piggy-back on the properties field, and it 
might save some code (at least initially), but I don't see the 
overlap here.

None of the current properties (superseded, upstream-name, 
*-variant, cpe-name, hidden?) make much sense to expose in this 
way; they're semimmutable facts about the package.

While a dedicated field can provide richer metadata, such as 
description, basic ‘types’ & sane/permitted values, to be used by 
the UI.

Looking at Exherbo/ntoo, this will inevitably lead to factoring 
out very common option metadata so one doesn't have to keep 
writing ‘x11 bool "Add support for the X Window System"’ in 
every. single. package.

Then come the people asking how to set CFLAGS for all packages and 
you'll know you've made it.

>   • We might want to make the CLI option less verbose too, but 
>   how?

Possibly relevant: do you see package options¹ being added to 
package specifications?  That seems to be the classical approach.

Kind regards,

T G-R

[1]: I use the term ‘option’ because it's what my last FHS 
distribution called them, and it was a good distribution, and 
because ‘arguments’ is already an unrelated field in package 
expressions, and flags are boolean, but substitute whatever name 
you prefer.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Parameterized packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 628 bytes --]

Hi,

This would be an awesome feature!

It seems to be one of the very few features that's missing compared to,
say, Portage (the famous USE flags).

I lean towards Tobias' suggestion: a dedicated field might be better.
The first issue that comes to my mind with package properties is "how do
we make the distinction between the "parameters" and other
non-user-facing properties?"

We need to be able to list the parameters/options of individual packages
as well as all options across all packages.  We could even offer
completion from the command line.

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parameterized packages

[Mark H Weaver]

Hi,

Tobias Geerinckx-Rice <me@tobias.gr> writes:

> Ludovic Courtès wrote:
>> While thinking about <https://issues.guix.info/issue/35671> and
>> looking
>> for ways to allow users to install just the locales they need right
>> from
>> ‘guix package’, I realized that “parameterized packages” are a
>> low-hanging fruit
>
> Wow.  Unexpected turn…
>
> I'm still thinking about this, so all this is just me doing it out
> loud:
>
>>   (package
>>     (inherit glibc-utf8-locales)
>>     (properties `((locales . ("zh_CN.utf8")))))
>>
>> and tadaam! we have a parameterized package.
>>
>> There’s a couple of gotchas:
>>
>>   • We’d need to store more info in manifest entries so that
>>     transformation options are preserved upon ‘guix upgrade’.
>>
>>   • We might need to expose the package parameters somehow, so
>> that if
>>     one types ‘--with-argument=foobar=baz’, they get a correct
>> error
>>     message saying that “foobar” is not a known parameter.
>
> Interesting idea to piggy-back on the properties field, and it might
> save some code (at least initially), but I don't see the overlap here.
>
> None of the current properties (superseded, upstream-name, *-variant,
> cpe-name, hidden?) make much sense to expose in this way; they're
> semimmutable facts about the package.

Also, at present, the current 'properties' field can be changed without
changing the derivation, and I think that's quite useful.  It's nice to
be able to do things like increase the build timeouts on a core package,
for the sake of a slow architecture, without forcing rebuilds of
everything on top of that package on other architectures.

So, I would prefer to see a different package field used for this.

     Thanks,
       Mark

Re: Parameterized packages

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 2011 bytes --]

Mark H Weaver <mhw@netris.org> writes:

>> Ludovic Courtès wrote:
>>> While thinking about <https://issues.guix.info/issue/35671> and
>>> looking
>>> for ways to allow users to install just the locales they need right
>>> from
>>> ‘guix package’, I realized that “parameterized packages” are a
>>> low-hanging fruit
>>
>> Wow.  Unexpected turn…
>>
>> I'm still thinking about this, so all this is just me doing it out
>> loud:
>>
>>>   (package
>>>     (inherit glibc-utf8-locales)
>>>     (properties `((locales . ("zh_CN.utf8")))))
>>>
>>> and tadaam! we have a parameterized package.
>>>
>>> There’s a couple of gotchas:
>>>
>>>   • We’d need to store more info in manifest entries so that
>>>     transformation options are preserved upon ‘guix upgrade’.
>>>
>>>   • We might need to expose the package parameters somehow, so
>>> that if
>>>     one types ‘--with-argument=foobar.z’, they get a correct
>>> error
>>>     message saying that “foobar” is not a known parameter.
>>
>> Interesting idea to piggy-back on the properties field, and it might
>> save some code (at least initially), but I don't see the overlap here.
>>
>> None of the current properties (superseded, upstream-name, *-variant,
>> cpe-name, hidden?) make much sense to expose in this way; they're
>> semimmutable facts about the package.
>
> Also, at present, the current 'properties' field can be changed without
> changing the derivation, and I think that's quite useful.  It's nice to
> be able to do things like increase the build timeouts on a core package,
> for the sake of a slow architecture, without forcing rebuilds of
> everything on top of that package on other architectures.
>
> So, I would prefer to see a different package field used for this.

I agree with Mark; using a custom package field seems better here
because the arguments change the derivation (don't they?), but
properties do not.  Maybe "argument-overrides"?

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Parameterized packages

[ison]

On Fri, May 17, 2019 at 02:15:29PM -0400, Mark H Weaver wrote:
> Also, at present, the current 'properties' field can be changed without
> changing the derivation, and I think that's quite useful.  It's nice to
> be able to do things like increase the build timeouts on a core package,
> for the sake of a slow architecture, without forcing rebuilds of
> everything on top of that package on other architectures.
> 
> So, I would prefer to see a different package field used for this.

Just a suggestion: Could the field allow extra arguments without throwing
an error, and also without causing a rebuild if the arguments aren't used?
If so then users could emulate _global_ USE flags in their system config by
simply mapping all packages through a function to modify the parameters.

Such an idea also seems to call for some standardization of parameters
across packages.

Re: Parameterized packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 4525 bytes --]

Hi!

Bringing this topic back to life now that I'm starting working on this.

1. Everyone seems to agree that we need a dedicated field in the package
   declaration.

   For now, 3 names were proposed:
   - parameters
   - options
   - argument-overrides

   I find "argument-overrides" a bit too verbose :p
   I'll go with "parameters" for now since it's the title of this thread.

2. As Tobias and Ison suggested, we need to centralize the declaration of
   parameters, so that we can list them and centralize metadata like the
   description & types.  Any suggestion on how to implement this in practice?
   Where do we store them?  How?

3. From Ludo's comment:

	> • We’d need to store more info in manifest entries so that
	> 	transformation options are preserved upon ‘guix upgrade’.
  > ...
	> • We might want to make the CLI option less verbose too, but how?

  We'd also need a way to specify the parameters in the manifest specs and from
  command line.
  Now we have "@" and ":" for versions and outputs respectively.  What about
  parentheses?  E.g.

    "foo@2.0(:locale fr_FR.UTF-8 :audio pulseaudio :gui X)"

  I think a plist like above would be less verbose than an alist.

  To apply parameters to the rest of the command line, we could also do:

    guix install --with-parameters="(:locale ...)" package-foo package-bar

4. From Ludo's comment:
   > • We might need to expose the package parameters somehow, so that if
   > 	one types ‘--with-argument=foobar=baz’, they get a correct error
   > 	message saying that “foobar” is not a known parameter.

   I think we should only display a warning so that we can emulate global USE
   flags as Ison suggested.

5. Global parameter storage

   When using "guix install" it can be convenient not to have to repeat the
   parameter specification.  So it'd be nice to store it somewhere.  What about
   a file in ~/.config/guix/parameters.scm with something like

   (parameters
     :locale fr_FR.UTF-8
     :audio pulseaudio
     ...)

   ?

6. Leverage build systems to automate parameterization when possible.

   Some build systems use well-known flags.  For instance, for the
   gnu-build-system if the user sets the parameter "(:audio pulseaudio)" we could
   automatically pass --enable-pulseaudio to the "configure" flags, even
   for packages that didn't specify their own well-known parameters.

   It would be nice if build systems had a way to report if a flag is supported
   or not.  Any clue?

   To implement this, we could do the following:

   (define-parameter audio
     "Some description"
     (types ...)
     (argument
       (lambda (pkg)
         (match (package-build-system pkg)
           (gnu-build-system
             (add-to-configure-flags "--enable-pulseaudio"))
           (cmake-build-system
             (...))))))

7. Dependency management.
   Also known as the USE flag nightmare among Gentoo users...
   This is where hell breaks loose! :p

   The problem: the user wants to specify a parameter to use globally where it
   applies, on all installed packages and all their inputs, recursively.

   For instance, use guile-2.2.4 instead of guile for all guile libraries, or
   use pulseaudio everywhere, including in dependencies that are not explicitly
   installed to the user profile.

   The obstacle: A package may require inputs with a specific parameter.
   For instance, BAR depends on a FOO package built with ":audio
   pulseaudio".  What happens if the user seta ":audio alsa" globally
   and installs BAR?  BAR needs to specify explicitly that its FOO input
   requires pulseaudio.  Otherwise BAR would fail to build.

   To specify that a package input depends on a specific parameter, we could
   extend the syntax of the (inputs ...) and (native-inputs ...) fields like so:

   (input
    `(("foo" ,foo "(:audio pulseaudio)")))

   A bigger problem is that the parameter compatibility issue is combinatorial:
   "Which parameter combination does BAR support?"  It's hard to know it in
   advance.  Any idea how to tackle this?



   The easy way around this is to not propagate the global parameters by
   default, but this seriously limits the usefulness of global parameters.

   Maybe a better workaround would be to automatically fall back to the default
   value of all parameters when a build fails.
   
Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parameterized packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 201 bytes --]

Any thought on this, anyone?

My previous post may be a bit overwhelming, let me know if you'd like me
to clarify (and maybe provide some examples).

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parameterized packages

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 1255 bytes --]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> Any thought on this, anyone?

Having used Gentoo for some years, I'm not sure a "global" parameter
system is desirable.  It could encourage enabling feature "foo", even
for packages that have experimental or broken support for "foo".

But I see how it could be useful to parameterize packages nevertheless.
I imagine adding a 'parameters' field to the package type, and write
definitions along these lines.

(package
  (parameters
   '((pulseaudio . #t)))
  (arguments
   `(#:configure-flags ,(match parameters
                         ((('pulseaudio . #t)
                          '("--enable-pulseaudio" "--enable-dbus"))
                         (_ '()))))
  (inputs
   `(("foo" ,foo)
     ,@(match parameters
         ((('pulseaudio . #t)
           `(("pulseaudio" ,pulseaudio))))
         (_ '()))))))

And perhaps a "with-parameters" procedure for toggling it:

(inputs
  `(("foo" ,foo))
    ("bar" ,(with-parameters ((pulseaudio . #f)) bar)))

We could have a similar procedure that recursively toggles supported
parameters of all given packages.

Actually we can abuse the 'properties' field for this purpose without
having to change anything.

Well, food for thought, thanks for driving this :-)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parameterized packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1836 bytes --]

Hi Marius!

Marius Bakke <mbakke@fastmail.com> writes:

> Having used Gentoo for some years, I'm not sure a "global" parameter
> system is desirable.  It could encourage enabling feature "foo", even
> for packages that have experimental or broken support for "foo".

That said, the recursive activation of parameters that you mention below
has the same "compatibility" drawback (that I mentioned in my previous
email) as global parameters.

More below:

> (inputs
>   `(("foo" ,foo))
>     ("bar" ,(with-parameters ((pulseaudio . #f)) bar)))
>
> We could have a similar procedure that recursively toggles supported
> parameters of all given packages.

Breaking example:

--8<---------------cut here---------------start------------->8---
(with-recursive-parameters ((pulseaudio .#f)) bar)
--8<---------------cut here---------------end--------------->8---

with `bar' breaking if one of its dependencies has pulseaudio disabled.

To be usable, we would need something to say "build bar and all its
inputs without pulseaudio, except for some given packages".

While this is OK with 1 parameter, it's quickly gets much more
complicated when packages have multiple parameters that maybe conflict
with one another.

Does that make sense?

Both from the packager and the user point of view, this is the
combinatorial problem of what sets of parameters are compatible.  (Same
problem on Gentoo)

I can't think of a way to fix this beside the packagers encoding
themselves each set that works.

> Actually we can abuse the 'properties' field for this purpose without
> having to change anything.

Can you detail what do you have in mind?  Earlier in this thread it was
mentioned that leveraging `properties' would not be a good idea.


Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parameterized packages

[Ludovic Courtès]

Hello!

The way I see it, we’re still toying with the idea and its pros and
cons—discussions about CLI syntax can come later.  ;-)

The added flexibility of package parameters is definitely nice, but
really, maintainability is a big concern.  The example Tobias gave (a
parameter to enable/disable X11 support) is interesting because it shows
that parameters can quickly become ubiquitous and get “out of control”,
from a maintenance viewpoint.

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> 7. Dependency management.
>    Also known as the USE flag nightmare among Gentoo users...
>    This is where hell breaks loose! :p
>
>    The problem: the user wants to specify a parameter to use globally where it
>    applies, on all installed packages and all their inputs, recursively.
>
>    For instance, use guile-2.2.4 instead of guile for all guile libraries, or
>    use pulseaudio everywhere, including in dependencies that are not explicitly
>    installed to the user profile.
>
>    The obstacle: A package may require inputs with a specific parameter.
>    For instance, BAR depends on a FOO package built with ":audio
>    pulseaudio".  What happens if the user seta ":audio alsa" globally
>    and installs BAR?  BAR needs to specify explicitly that its FOO input
>    requires pulseaudio.  Otherwise BAR would fail to build.
>
>    To specify that a package input depends on a specific parameter, we could
>    extend the syntax of the (inputs ...) and (native-inputs ...) fields like so:
>
>    (input
>     `(("foo" ,foo "(:audio pulseaudio)")))
>
>    A bigger problem is that the parameter compatibility issue is combinatorial:
>    "Which parameter combination does BAR support?"  It's hard to know it in
>    advance.  Any idea how to tackle this?

That’s another good example of a problem that would arise.  :-/
It doesn’t seem reasonable to me to add complex logic in (guix packages)
to deal with this issue; I would very much prefer to leave input
handling unchanged.

I’m worried about the maintenance cost of parameters.  Having the
feature is one thing, but being able to guarantee that the package
combinations we offer all work is another one.

We could have the feature and use it very seldom in Guix itself, but I’m
guessing that that’s not what you have in mind.

Also, for CI purposes, we would need a way to enumerate all the possible
combinations…

Apologies for spoiling the party :-), but I think these concerns need
all our attention!

Thanks,
Ludo’.

Re: Parameterized packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1918 bytes --]

Hi Ludo!

Ludovic Courtès <ludo@gnu.org> writes:

> The way I see it, we’re still toying with the idea and its pros and
> cons—discussions about CLI syntax can come later.  ;-)

Sure thing!

> The added flexibility of package parameters is definitely nice, but
> really, maintainability is a big concern.  The example Tobias gave (a
> parameter to enable/disable X11 support) is interesting because it shows
> that parameters can quickly become ubiquitous and get “out of control”,
> from a maintenance viewpoint.

What I understood from Tobias comment on the X11 support parameter is
that we need to store the parameter metadata centrally.  Why do you
think this suggests it will get "out of control"?


> That’s another good example of a problem that would arise.  :-/
> It doesn’t seem reasonable to me to add complex logic in (guix packages)
> to deal with this issue; I would very much prefer to leave input
> handling unchanged.
>
> I’m worried about the maintenance cost of parameters.  Having the
> feature is one thing, but being able to guarantee that the package
> combinations we offer all work is another one.
>
> We could have the feature and use it very seldom in Guix itself, but I’m
> guessing that that’s not what you have in mind.

Indeed, the ultimate goal would be to do things like "install a headless
system" and "use static libraries everywhere", etc.

> Also, for CI purposes, we would need a way to enumerate all the possible
> combinations…

Considering the number of outputs would explode as parameters get added,
it does not seem reasonable to try building everything.  Maybe an easy
option would be:

- Only build the default packages (i.e. using the default values of the
  parameters).
- If a non-default output is requested often enough (to be defined) then
  build it.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parameterized packages

[zimoun]

Hi Pierre,

I do not not if it related.

Currently, the option "--with-input" does not re-write the implicit
inputs. Probably for good reasons. :-)

For example, say I would like to have the version 3.4.3 of R (2017
old). This version is really old and had disappeared. Other said, it
is older than the introduction of "inferior"s and so "guix
time-machine" does not help (but that another story).
At this point, it is not matter the GCC version that it is used to
compile this R version.

Well, it is not too complicated:

  git show cbe1314a7e:gnu/packages/statistics.scm > /tmp/stuff/old-stats.scm
  # edit old-stats.scm to tweak the modules
  guix build -L /tmp r-minimal@3.4.3

Now, I would like to build the package r-rcpp at the version 1.0.3
which is introduced recently using the old R version 3.4.3. Well, the
naive idea is:

   guix build -L /tmp --with-inputs=r-minimal=r-minimal@3.4.3 r-rcpp

but because r-minimal is an implicit inputs, it is not re-written.

This behaviour is expected because the bag is really huge. To be
convinced, try: "guix graph -t bag r-rcpp | dot -Tps >
r-rcpp-bag.ps"... and wait... :-)


From a practical point of view, re-write any implicit inputs is
touchy.  And I do not know if the "parametrized package" is related or
not at all -- I do not see real difference between re-compiling with
this flags and re-compiling with this version -- but be able to
recompile a matrix of combinaison would be really cool!


All the best,
simon

Re: Parameterized packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 765 bytes --]

Hi Simon!

You are making a good point: there are different levels of
"accessibility" when it comes to inputs:

- The source.
- The explicit inputs.
- the implicit inputs.
- The build inputs.
- The build system inputs.
- Recursive inputs.
- ...?

Which one should we expose?  I don't know.  If we want the system to
have some set of properties, I guess only the resulting packages matther
and the build inputs don't.

Should we touch implicit inputs, we would have to parse all the
references and not just the explicit inputs like --with-input does.

> but be able to
> recompile a matrix of combinaison would be really cool!

Indeed.  But now I'm starting to wonder if it is really doable :p

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parameterized packages

[zimoun]

Hi Pierre,

On Wed, 15 Jan 2020 at 10:40, Pierre Neidhardt <mail@ambrevar.xyz> wrote:

> - The source.
> - The explicit inputs.
> - the implicit inputs.
> - The build inputs.
> - The build system inputs.
> - Recursive inputs.
> - ...?
>
> Which one should we expose?  I don't know.  If we want the system to
> have some set of properties, I guess only the resulting packages matther
> and the build inputs don't.

But the resulting packages depends on the build inputs.
So if the resulting packages matter, then the build inputs too.

Well, you already described the issue here [1] (point 7.) -- if I
understand correctly.

[1] https://lists.gnu.org/archive/html/guix-devel/2020-01/msg00026.html


> Should we touch implicit inputs, we would have to parse all the
> references and not just the explicit inputs like --with-input does.

My mind is not clear at all.

What is the final aim to have parametrized packages?
What does it mean "parametrized"?
Does it mean extend the transformation options as Ludo described [2].
Does it mean more?

[2] https://lists.gnu.org/archive/html/guix-devel/2019-05/msg00285.html


> > but be able to
> > recompile a matrix of combinaison would be really cool!
>
> Indeed.  But now I'm starting to wonder if it is really doable :p

What I was thinking of is to parametrize only the build systems.

For example, be able to rebuild all the packages with GCC-8.3, or to
install Python packages with Python 3.5 instead of the current default
Python 3.7.



Cheers,
simon

Re: Parameterized packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 412 bytes --]

Another thought: Gentoo has a rather complicated "Atom" syntax to convey
which parameter combination is acceptable or not.
I found it very hard to use at times.

With Guix, I believe we can do better.  We can add a
#:parameter-predicate field which takes the user parameters and returns
a boolean (pass or no pass).  This would give us maximal flexibility.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parameterized packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 385 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> That’s another good example of a problem that would arise.  :-/
> It doesn’t seem reasonable to me to add complex logic in (guix packages)
> to deal with this issue; I would very much prefer to leave input
> handling unchanged.

To be clear, what do you mean with "input handling"?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parameterized packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 625 bytes --]

zimoun <zimon.toutoune@gmail.com> writes:

> For example, be able to rebuild all the packages with GCC-8.3, or to
> install Python packages with Python 3.5 instead of the current default
> Python 3.7.

I think this would tackle a different issue.  The point of parameters is
to offer _composable_ and _package-specific_ build options to the user.
A build system cannot know what it means to build with Pulseaudio or
without Wayland.

Can't we already do what you are saying by the way?  By means of
settings some build-system-specific arguments from fold-package?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parameterized packages

[zimoun]

On Wed, 15 Jan 2020 at 12:51, Pierre Neidhardt <mail@ambrevar.xyz> wrote:
>
> zimoun <zimon.toutoune@gmail.com> writes:
>
> > For example, be able to rebuild all the packages with GCC-8.3, or to
> > install Python packages with Python 3.5 instead of the current default
> > Python 3.7.
>
> I think this would tackle a different issue.  The point of parameters is
> to offer _composable_ and _package-specific_ build options to the user.
> A build system cannot know what it means to build with Pulseaudio or
> without Wayland.

I do not see the difference between using specific locale [1] (for a
package that does not explicitly depend on glibc) and using a specific
version of the "compiler" used by the build system.

Maybe I miss the point...

[1] https://lists.gnu.org/archive/html/guix-devel/2019-05/msg00285.html


> Can't we already do what you are saying by the way?  By means of
> settings some build-system-specific arguments from fold-package?

Everything is doable with Guix. ;-)
However, it is not clear to me what is the best/easiest way to go.
For example, here [2] I give a try.

[2] https://lists.gnu.org/archive/html/help-guix/2020-01/msg00087.html


And what I was thinking is a mechanism to easily set some arguments to
the build-system; for example changing the compiler toolchain (say
replacing GCC by Clang/LLVM).

Well, as I said, I do not know if it is related to "parametrized
packages" because I am not sure to understand the final aim for these
"parametrized packages". :-)


All the best,
simon

Re: Parameterized packages

[ison]

On Wed, Jan 15, 2020 at 02:54:25PM +0100, zimoun wrote:
> Everything is doable with Guix. ;-)
> However, it is not clear to me what is the best/easiest way to go.
> For example, here [2] I give a try.
> 
> [2] https://lists.gnu.org/archive/html/help-guix/2020-01/msg00087.html
> 
> 
> And what I was thinking is a mechanism to easily set some arguments to
> the build-system; for example changing the compiler toolchain (say
> replacing GCC by Clang/LLVM).
> 
> Well, as I said, I do not know if it is related to "parametrized
> packages" because I am not sure to understand the final aim for these
> "parametrized packages". :-)

Maybe the current discussion is trying too hard to emulate Gentoo's USE
flags and dependency graph concept (perhaps its my fault for bringing up
global flags). But that feels like introducing "side effects", and maybe the
whole idea should be treated more "functionally" in Guix.

That is, simplify the problem to the mere concept of passing arguments to
functions and nothing more. Take the headless server example: some parameter
is passed to a package such as
'("-X")
That package would then be entirely responsible for what to do with them. If
the package decides not to pass the same parameters to its inputs then the
inputs are simply built without any parameters.

This would have the benefit of not breaking any existing packages regardless
of what parameters are sent to them.
And packages can be modified by maintainers to accept parameters and pass
them on to their inputs as the need arises over time (e.g. packages typically
installed by headless servers can be modified to pass the "-X" flag on to
certain inputs).

Additionally if someone wanted to take the risk and _force_ all inputs to
use the same flags then they can use a function similar to
"package-input-rewriting" (there could be a "package-use-flags-rewriting"),
so it becomes similar to how other types of transformations are currently
handled in Guix.

Ideally packages can still be made to allow all flags to "trickle down" to
their dependencies without input-rewriting simply by passing all parameters
into their inputs in the package definition, but the difference here is that
we don't force it. That's something that can be approached over time, or
not. The idea is to just let packages do it if they want to, and let the
package maintainers filter the list of flags before passing on to inputs if
they need to, and the whole thing becomes more predictable and functional.
This also avoids the problem of which types of inputs to modify (normal
inputs, native-inputs, etc) because then it's just up to the package
maintainer.

Re: Parameterized packages

[Ricardo Wurmus]

ison <ison@airmail.cc> writes:

> Maybe the current discussion is trying too hard to emulate Gentoo's USE
> flags and dependency graph concept (perhaps its my fault for bringing up
> global flags). But that feels like introducing "side effects", and maybe the
> whole idea should be treated more "functionally" in Guix.
>
> That is, simplify the problem to the mere concept of passing arguments to
> functions and nothing more. Take the headless server example: some parameter
> is passed to a package such as
> '("-X")
> That package would then be entirely responsible for what to do with them. If
> the package decides not to pass the same parameters to its inputs then the
> inputs are simply built without any parameters.

I think this is a sensible approach, though it would require agreement
or at least coordination among package contributors about what
parameters to use.  For example, one such parameter could be
“#:headless” to disable any graphical user interface or desktop feature,
but a case could just as well be made for “#:gui” or “#:graphical”…

Coordination hasn’t been a problem so far because we managed to agree on
build systems and utilities, but it seems to me that parameters to match
on span a much larger range — I feel reminded of the problem of package
tags or categories, which are a similarly large field that defies
agreement.

--
Ricardo

Re: Parameterized packages

[L p R n d n]

Hello!

Also followed the thread from a little far away. It's probably an
awfull idea (I don't even know if it's possible) but throwing in
different perspetives migh give birth to new ones so here I go.

Instead of thinking in terms of use flags or parameterized packages maybe
using something that I would call "modificators" might be an idea.
I see it as some kind of list of funtions modifying packages (If it's
clear?). The main differences would be that the package definitions
would be left untouched and that modificators' definitions would be
centralized keeping the complexity and the maintaining burden away.
Something like modificators maintainer might emerge then?

That could look like:

(define-modificator headless
                    `((xorg (modify-package xorg
                            ...))

or something replacing a package directly without introducing new syntax:

                      (xorg (package (inherit xorg)
                            ...))))

Then givin a list of packages, apply a modificator to it and get a
modified list of packages?

Hope it helps,

Have a nice day,

L  p R n  d n

Re: Parameterized packages

[zimoun]

On Thu, 16 Jan 2020 at 20:06, ison <ison@airmail.cc> wrote:
> On Wed, Jan 15, 2020 at 02:54:25PM +0100, zimoun wrote:

> > And what I was thinking is a mechanism to easily set some arguments to
> > the build-system; for example changing the compiler toolchain (say
> > replacing GCC by Clang/LLVM).
> >
> > Well, as I said, I do not know if it is related to "parametrized
> > packages" because I am not sure to understand the final aim for these
> > "parametrized packages". :-)

[...]

> That is, simplify the problem to the mere concept of passing arguments to
> functions and nothing more. Take the headless server example: some parameter
> is passed to a package such as
> '("-X")
> That package would then be entirely responsible for what to do with them. If
> the package decides not to pass the same parameters to its inputs then the
> inputs are simply built without any parameters.

I agree.

What I was trying to suggest and/or discuss is to see the build-system
as a function where the compiler toolset is one argument.

Let consider these "compilers" (associated with complete toolset for
"compiling"): OCaml@4.01, OCaml@4.07, OCaml@4.09, CPython@2.7,
CPython@3.5, CPython@3.6, PyPy (not yet in Guix), GCC@8, GCC@9,
Clang@8, Clang@9, etc.

Today, for one particular build system, the compiler is fixed. To use
another compiler than the default one, the trick is to have
'package-with-explicit' and each build-system has to provide
'package-with-<version>' where <version> is hard coded.

To be clear, the seed (bootstrap) fixes how the compilers OCaml@4.01,
OCaml@4.07, OCaml@4.09, CPython@2.7, CPython@3.5, CPython@3.6, PyPy
(not yet in Guix), GCC@8, GCC@9, Clang@8, Clang@9, etc are compiled.
Then, it is difficult to use them to compile a package with one of
them.

Do we end with 'package-with-ocaml4.01', 'package-with-ocaml4.07',
'package-with-ocaml4.09', 'package-with-python2',
'package-with-python3.5', 'package-with-python3.6',
'package-with-pypy', 'package-with-gcc8', 'package-with-gcc9', etc.?

Do the users have to write all these 'package-with-<version>'?
And if one wants to use CPython@3.5 and GCC@8, does they need to
define the package using:

(define-public my-foo
  (package-with-python3.5
       (package-with-gcc8 foo)))

?

I feel something is lacking.

And do not take me wrong, I am not suggesting that Guix should
maintain and ensure it works for all the combinations. The default is
already enough! :-)
And yes, it means potentially recompiling the world. Try to plot "guix
graph -t bag-emerged foo". ;-)


Why do I want that?

For example, a couple of software have been used to simulate complex
physical phenomena. Using the commit A -- which provides the seed A
and all compiling tools in the version X -- I get some results. Then
using the commit B -- which provided the seed B and all the compiling
tools in the version Y -- I get different results. Where does come
from the difference? From my couple of software which are not
reproducible? From round-off errors and floating point arithmetic
mess? Well, the most probable... but can I reduce the search space of
the difference? And I would like to be able to fix the seed to A, then
used the compiling tools in the version X and compare using this very
same seed A with the compiling tools in the version Y. Do the same
with seed B.

Well, it would ease comparison in the HPC world. :-)


It is not related to the "parametrized package" in the sense of flag
options. :-)

And I do not know if it make sense. What do you think?


All the best,
simon

Re: Parameterized packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1196 bytes --]

Hi ison!

ison <ison@airmail.cc> writes:

> Maybe the current discussion is trying too hard to emulate Gentoo's USE
> flags and dependency graph concept (perhaps its my fault for bringing up
> global flags). But that feels like introducing "side effects", and maybe the
> whole idea should be treated more "functionally" in Guix.
>
> That is, simplify the problem to the mere concept of passing arguments to
> functions and nothing more. Take the headless server example: some parameter
> is passed to a package such as
> '("-X")
> That package would then be entirely responsible for what to do with them. If
> the package decides not to pass the same parameters to its inputs then the
> inputs are simply built without any parameters.

Yes, I am starting to come to a similar conclusion.
I think this makes more sense in terms of maintainability.

What that in mind, maintenance would still get more complex than it is
today: a package with N parameters would have 2^N possible outputs.
It quickly gets hard to test them all.  That said, we can probably assume
that in most cases parameters won't conflict with each other.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parameterized packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1187 bytes --]

Hi Ricardo,

Ricardo Wurmus <rekado@elephly.net> writes:

> I think this is a sensible approach, though it would require agreement
> or at least coordination among package contributors about what
> parameters to use.  For example, one such parameter could be
> “#:headless” to disable any graphical user interface or desktop feature,
> but a case could just as well be made for “#:gui” or “#:graphical”…
>
> Coordination hasn’t been a problem so far because we managed to agree on
> build systems and utilities, but it seems to me that parameters to match
> on span a much larger range — I feel reminded of the problem of package
> tags or categories, which are a similarly large field that defies
> agreement.

I think Tobias suggestion is a good solution to this: make parameters
first class, global objects.  A package can only use globally declared
parameters.  This way we have a limited set of well-defined parameters
under control (understand: no duplicates, consistent naming).

As Tobias pointed out, global declaration of parameters allows us to
centralize their documentation and types.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parameterized packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1356 bytes --]

Hi Nicolò,

Yes, you are exactly right.

I didn't know Nix had it, that's great to hear!
Would you happen to know where those USE flags are implemented?

> Taking handbrake[1] recipe as an example: there's the useGtk flag that
> can be passed to the package definition, and that affects build
> options. Since in guix dependencies build inputs are specified inside
> the package definition, propagating those flags should be easy.  If we
> standardize on flags names (don't have "-X", "noX", "disable-x"...), it
> should be easy to incrementally add flags support to all package
> dependencies.  I imagine (with an invalid syntax probably) something
> like:
>
> (inputs
> `(,(unless (member build-flags 'no-x) ("libx11" libx11))
>   ; this package always need x, don't pass flags
>   ("input" ,input "out" (filter (lambda (x) (not (eq? x 'no-x))) build-flags))
>   ("other-input" ,input))) ; flags are passed implicitly here?

Some questions arise though:

- What about passing parameters to implicit dependencies, e.g. those of
  the build system?

- Consider the following case: With package A depending on B depending
  on C, to get parameter X on A we need to enable X on C.
  I guess the workaround is to include "C with parameter X" as an
  explicit input of A.  Hmm...

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parameterized packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 608 bytes --]

Hi!

L  p R n  d n    <guix@lprndn.info> writes:

> (define-modificator headless
>                     `((xorg (modify-package xorg
>                             ...))

Thanks for sharing, this is an interesting approach!  But it may raise a
major challenge.  The point of parameters is that they must compose
(e.g. headless + static-lib), so can we do that with the "modificator"
approach?

Say for instance that 2 parameters in particular are not compatible for
a given package, how to we handle that?

This probably needs more thoughts! :)

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parameterized packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1091 bytes --]

Hi Zimoun!

zimoun <zimon.toutoune@gmail.com> writes:

> Well, it would ease comparison in the HPC world. :-)
>
>
> It is not related to the "parametrized package" in the sense of flag
> options. :-)
>
> And I do not know if it make sense. What do you think?

I think you are making a lot of sense and yes, parameters should
probably be able to customize the builder as well.

But as I understand it, all the "configurable part" of the builder is
expose through the #:arguments field.  A simple example of this is
python: the python-build-system has an argument which allows the package
to specify whether we use python-2 or python-3.

In this case, it's trivial to use parameters to influence which compiler
the build system will use.

For gnu-build-system (with gcc, clang, etc.) we can probably do similar
things already by setting CC.

The solution to your problem in my opinion is simply to expose just the
right amount of options through #:arguments for all build systems.
Would that be satisfactory to you?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parameterized packages

[Ludovic Courtès]

Hi,

Nicolò Balzarotti <anothersms@gmail.com> skribis:

> Hello! I've not followed the discussion with too much attention, so
> forgive me if I'm plain wrong.
> But, if I get this right, to me it seems we want something similar to
> what nix is already doing.

To me, what Nix is doing amounts to:

  (define (make-me-a-package option1 option2)
    (package
      …))

I believe the extra bit that would be provided here is a bridge between
the API and the user interface.

Ludo’.

Re: Parameterized packages

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> To be clear, the seed (bootstrap) fixes how the compilers OCaml@4.01,
> OCaml@4.07, OCaml@4.09, CPython@2.7, CPython@3.5, CPython@3.6, PyPy
> (not yet in Guix), GCC@8, GCC@9, Clang@8, Clang@9, etc are compiled.
> Then, it is difficult to use them to compile a package with one of
> them.
>
> Do we end with 'package-with-ocaml4.01', 'package-with-ocaml4.07',
> 'package-with-ocaml4.09', 'package-with-python2',
> 'package-with-python3.5', 'package-with-python3.6',
> 'package-with-pypy', 'package-with-gcc8', 'package-with-gcc9', etc.?
>
> Do the users have to write all these 'package-with-<version>'?
> And if one wants to use CPython@3.5 and GCC@8, does they need to
> define the package using:
>
> (define-public my-foo
>   (package-with-python3.5
>        (package-with-gcc8 foo)))
>
> ?
>
> I feel something is lacking.

I agree.  ‘package-input-rewriting’ gives us almost what you want, with
the limitation that implicit inputs are ignored (which is a good thing
sometimes, and a problem in cases where you want to experiment with
toolchains, as you write).

What we’d need is a variant of ‘package-input-rewriting’ that somehow
walks through implicit dependencies.  It could do that by changing the
build system, I guess.

Thanks,
Ludo’.

Re: Parameterized packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 810 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> I agree.  ‘package-input-rewriting’ gives us almost what you want, with
> the limitation that implicit inputs are ignored (which is a good thing
> sometimes, and a problem in cases where you want to experiment with
> toolchains, as you write).
>
> What we’d need is a variant of ‘package-input-rewriting’ that somehow
> walks through implicit dependencies.  It could do that by changing the
> build system, I guess.

To clarify, this does not cover the extent of what "parameters" could
do, since parameters are not just about input rewriting.

What do you think of my other answer on using parameters to tweak the
`arguments' field?  (E.g. the "python" argument for the python-build-system.)

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parameterized packages

[zimoun]

Hi Ludo,

On Sun, 19 Jan 2020 at 21:34, Ludovic Courtès <ludo@gnu.org> wrote:

> > I feel something is lacking.
>
> I agree.  ‘package-input-rewriting’ gives us almost what you want, with
> the limitation that implicit inputs are ignored (which is a good thing
> sometimes, and a problem in cases where you want to experiment with
> toolchains, as you write).

Yes. And I do not know what should be exposed, i.e., at which level.

As you probably better know than me, in HPC world, tweaking the
toolchain and its options is an habit (bad or good, not the point :-))


> What we’d need is a variant of ‘package-input-rewriting’ that somehow
> walks through implicit dependencies.  It could do that by changing the
> build system, I guess.

It would be cool! :-)


Cheers,
simon

Re: Parameterized packages

[zimoun]

Hi Pierre,

On Fri, 17 Jan 2020 at 17:56, Pierre Neidhardt <mail@ambrevar.xyz> wrote:

> In this case, it's trivial to use parameters to influence which compiler
> the build system will use.

I am not sure that "trivial" is the correct word. ;-)


> For gnu-build-system (with gcc, clang, etc.) we can probably do similar
> things already by setting CC.

Yes, and about Rust, Ruby, Perl, etc.

I do think it is the right abstraction. :-)


> The solution to your problem in my opinion is simply to expose just the
> right amount of options through #:arguments for all build systems.
> Would that be satisfactory to you?

Maybe. I am not sure to see what do you mean by '#:arguments'.

If you mean something that does what does
`package-with-explicit-python' to rewrite the 'arguments' field, yes
it is something in this flavor. But generic enough.

For example compare `package-with-explicit-python' and
`package-with-explicit-ocaml'.

Well, something to tweak `lower'. And the question is: what is the
right abstraction?


All the best,
simon

Re: Parameterized packages

[zimoun]

On Mon, 20 Jan 2020 at 10:08, Pierre Neidhardt <mail@ambrevar.xyz> wrote:
>
> Ludovic Courtès <ludo@gnu.org> writes:
>
> > I agree.  ‘package-input-rewriting’ gives us almost what you want, with
> > the limitation that implicit inputs are ignored (which is a good thing
> > sometimes, and a problem in cases where you want to experiment with
> > toolchains, as you write).
> >
> > What we’d need is a variant of ‘package-input-rewriting’ that somehow
> > walks through implicit dependencies.  It could do that by changing the
> > build system, I guess.
>
> To clarify, this does not cover the extent of what "parameters" could
> do, since parameters are not just about input rewriting.

As we discussed here [1]:
<<
> Should we touch implicit inputs, we would have to parse all the
> references and not just the explicit inputs like --with-input does.

My mind is not clear at all.

What is the final aim to have parametrized packages?
What does it mean "parametrized"?
Does it mean extend the transformation options as Ludo described [2].
Does it mean more?
>>

Could you define what do you mean by "parameters"?
Sorry to be slow but I am not clear about what we are talking about
precisely. :-)

1. Because the initial Ludo's message was about glic-locale which is
somehow an input.
2. The thread spoke also about flags à la Gentoo USE flags.
3. Other messages were about set X or Wayland or no-X for this or that package.
4. Another were about change the toolchain of the build systems.
etc.

All are "parameters". :-)

The solution of 1. and 4. seems to be able to re-write all the implicit inputs.

The solution of 2. and 3. seems to write, as Ludo mentioned:

--8<---------------cut here---------------start------------->8---
(define (make-me-a-package option1 option2)
    (package
      …))
--8<---------------cut here---------------end--------------->8---


Well, could you clarify what do you have in mind about "parameters"?
Just to stay on the wavelength and keep focus. :-)


[1] https://lists.gnu.org/archive/html/guix-devel/2020-01/msg00274.html




Cheers,
simon

Re: Parameterized packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 6182 bytes --]

zimoun <zimon.toutoune@gmail.com> writes:

> What is the final aim to have parametrized packages?
> What does it mean "parametrized"?

Easy and composable customization of packages.

> Does it mean extend the transformation options as Ludo described [2].

I think you forgot this reference.  If you meant "like Ludo described in
the original post" I'd say yes, more or less that.

> Could you define what do you mean by "parameters"?
> Sorry to be slow but I am not clear about what we are talking about
> precisely. :-)

No problem.  "Parameters" is a term Ludo came up with, I believe.  We
can always change the naming later.

In my understanding, a parameter is a "first class, globally defined
object -- that comes with a description and a type -- which value can be
used as condition in a package to modify its output.  A package may rely
on multiple parameters to modify its output."

> 1. Because the initial Ludo's message was about glic-locale which is
> somehow an input.

Yes, but in my opinion the example just happened to be an input
modification, it does not have to be.  Gentoo USE flags don't only
change the dependencies, they also change the compilation options, etc.

> 2. The thread spoke also about flags à la Gentoo USE flags.

Indeed, they are essentially the same, but recent discussions are
leading towards limiting Guix parameters a bit more to avoid
a combinatorial explosion in the maintenance complexity.

> 3. Other messages were about set X or Wayland or no-X for this or that package.

Just like Gentoo "X" USE flag I think.

> 4. Another were about change the toolchain of the build systems.
> etc.

In my opinion, we don't need to change anything in the build system.

The point is that parameters can be used anywhere in the package
definition, and thus it can change #:arguments fields like #:python to
customize the python builder.

> All are "parameters". :-)
>
> The solution of 1. and 4. seems to be able to re-write all the implicit inputs.

As Ison mentioned, this might not be reasonable because it might just
break too often.  A better approach could be to only customize the
direct inputs (and build system options).

> The solution of 2. and 3. seems to write, as Ludo mentioned:
>
> --8<---------------cut here---------------start------------->8---
> (define (make-me-a-package option1 option2)
>     (package
>       …))
> --8<---------------cut here---------------end--------------->8---

The ellipsis is a bit vague here.  What is this trying to do?

> Well, could you clarify what do you have in mind about "parameters"?
> Just to stay on the wavelength and keep focus. :-)

OK, I'll try with one full example:

--8<---------------cut here---------------start------------->8---
(define-public you-get
  (package
    (name "you-get")
    (version "0.4.1355")
    (PARAMETERS VIDEO-PLAYER PYTHON-VERSION WITH-FFMPEG)
    (source (origin
              (method git-fetch)
              (uri (git-reference
                    (url "https://github.com/soimort/you-get.git")
                    (commit (string-append "v" version))))
              (file-name (git-file-name name version))
              (sha256
               (base32
                "0xq7z04hvw3b3npiahlpzhbxsjvam9n9dynplyrkn84dx6k9ajbj"))))
    (build-system python-build-system)
    (inputs
     `(("PLAYER" ,(DEREF-PARAM VIDEO-PLAYER))
       ,@(IF (DEREF-PARAM WITH-FFMPEG)
             ;; FOR MULTI-PART AND >=1080P VIDEOS
             `("FFMPEG" ,FFMPEG)
             '())))             
    (arguments
     `(#:PYTHON ,(DEREF-PARAM PYTHON-VERSION)
       #:phases
       (modify-phases %standard-phases
         ,(WHEN (DEREF-PARAM WITH-FFMPEG)
            (add-after 'unpack 'qualify-input-references
              ;; Explicitly invoke the input ffmpeg, instead of whichever one
              ;; happens to be in the user's $PATH at run time.
              (lambda* (#:key inputs #:allow-other-keys)
                (let ((ffmpeg (string-append (assoc-ref inputs "ffmpeg")
                                             "/bin/ffmpeg")))
                  (substitute* "src/you_get/processor/ffmpeg.py"
                    ;; Don't blindly replace all occurrences of ‘'ffmpeg'’: the
                    ;; same string is also used when sniffing ffmpeg's output.
                    (("(FFMPEG == |\\()'ffmpeg'" _ prefix)
                     (string-append prefix "'" ffmpeg "'")))
                  #t))))
         (ADD-AFTER 'UNPACK 'TWEAK-PLAYER-SETTINGS
           (LAMBDA* (#:KEY INPUTS #:ALLOW-OTHER-KEYS)
             (MATCH ,(DEREF-PARAM VIDEO-PLAYER)
               (VLC
                ;; DO SOMETHING WITH VLC.
                )
               (MPV
                ;; DO SOMETHING WITH MPV.
                )
               (_
                ;; ERROR OUT?
                )))))
       #:tests? #f))                    ; XXX some tests need Internet access
    (synopsis "Download videos, audio, or images from Web sites")
    (description
     "You-Get is a command-line utility to download media contents (videos,
audio, images) from the Web.  It can use either mpv or vlc for playback.")
    (home-page "https://you-get.org/")
    (license license:expat)))
--8<---------------cut here---------------end--------------->8---

In the above I've highlighted the changes in uppercase.

On line (PARAMETERS ...) I've declared which parameters I'm going to use
in my package declaration.  Those parameters must be defined globally
somewhere in Guix.

The point of declaring the parameters in advance is that it allows the
user to list all parameters used by a given package.

In the above example, parameters are used for various things:

- To configure the build system, i.e. here to set which python version
  is going to be used to compile the program.

- To make the FFmpeg input optional.

- To let the user choose which video player to use.  This is a popular
  USE flag on Gentoo (maybe with a different name).

Does that make more sense now?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parameterized packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 343 bytes --]

Would anyone know of a good example package for parameters (USE flags)?
By "good" I mean:

- Benefits of parameters are obvious
- With 3 or more parameters
- 2 parameters compose and 2 conflict.
- 1 parameter impacts the build system / options and 1 parameter impacts
  the inputs.

?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parameterized packages

[ison]

On Mon, Jan 20, 2020 at 07:57:30PM +0100, Pierre Neidhardt wrote:
> > The solution of 2. and 3. seems to write, as Ludo mentioned:
> >
> > --8<---------------cut here---------------start------------->8---
> > (define (make-me-a-package option1 option2)
> >     (package
> >       …))
> > --8<---------------cut here---------------end--------------->8---

Just a thought, but could this be achieved using Guile's parameterization
feature?
https://www.gnu.org/software/guile/manual/html_node/Parameters.html

Not sure which method will be best overall, but its something to consider
and might avoid the need to define a bunch of extra functions. So basically
all standard parameters could be defined globally using make-parameter, then
in the parameters field of the package it lists the parameters they want
which will pick up the default values, or be overridden by parameterization.

Although I wonder if maybe it's better to not use global defaults for the
parameters and just let packages set their own defaults?

Re: Parameterized packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1494 bytes --]

ison <ison@airmail.cc> writes:

> Just a thought, but could this be achieved using Guile's parameterization
> feature?
> https://www.gnu.org/software/guile/manual/html_node/Parameters.html
>
> Not sure which method will be best overall, but its something to consider
> and might avoid the need to define a bunch of extra functions. So basically
> all standard parameters could be defined globally using make-parameter, then
> in the parameters field of the package it lists the parameters they want
> which will pick up the default values, or be overridden by parameterization.
>
> Although I wonder if maybe it's better to not use global defaults for the
> parameters and just let packages set their own defaults?

Actually I don't know.  An important feature of package parameters / USE
flags is typing.  But Guile does not have much support for typing as far
as know.  So if we need to implement some sort of typing in Guile, what
we could do is create a Guile record with the following fields:

- value
- predicate

The value should not be accessible directly, instead we need some
function that will reference the value by first applying the predicate
on it and throw an error if it returns #f.  This way we can enforce some
run-time typing.  Not ideal but that could work.  Any better option?

Typed/Racket would be great here :)

Otherwise, we can ditch the typing and use Guile parameters as you suggested.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parameterized packages

[Ludovic Courtès]

Hi,

ison <ison@airmail.cc> skribis:

> Just a thought, but could this be achieved using Guile's parameterization
> feature?
> https://www.gnu.org/software/guile/manual/html_node/Parameters.html

I don’t think so: these “parameters” (I like to write “SRFI-39
parameters” to avoid confusion, which refers to
<https://srfi.schemers.org/srfi-39/srfi-39.html>) have little to do with
the kind of parameters discussed here.

Ludo’.

Build systems and implicit inputs

[Ludovic Courtès]

Hi!

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> But as I understand it, all the "configurable part" of the builder is
> expose through the #:arguments field.  A simple example of this is
> python: the python-build-system has an argument which allows the package
> to specify whether we use python-2 or python-3.
>
> In this case, it's trivial to use parameters to influence which compiler
> the build system will use.
>
> For gnu-build-system (with gcc, clang, etc.) we can probably do similar
> things already by setting CC.
>
> The solution to your problem in my opinion is simply to expose just the
> right amount of options through #:arguments for all build systems.
> Would that be satisfactory to you?

I think the issue of tweaking the build system and its implicit inputs
must be addressed separately.  We first need a good API to do that.
When we have it, it’ll be nice and easy to drive it via package
parameters.  :-)

Currently each build system has ad-hoc keyword parameters to customize
its implicit inputs: #:python for ‘python-build-system’, #:cmake for
‘cmake-build-system’, #:implicit-inputs? for ‘gnu-build-system’, etc.

For each of them, it would be quite easy to provide a procedure that
takes a list of implicit inputs and returns a <build-system>.  It may
not be all that convenient, and perhaps a bit too ad-hoc still.

Another option would be to make implicit inputs a field of
<build-system>.

Needs more thought!

Ludo’.

Re: Parameterized packages

[zimoun]

Hi,

Thank you for the explanations.

On Mon, 20 Jan 2020 at 19:57, Pierre Neidhardt <mail@ambrevar.xyz> wrote:

> > The solution of 2. and 3. seems to write, as Ludo mentioned:
> >
> > --8<---------------cut here---------------start------------->8---
> > (define (make-me-a-package option1 option2)
> >     (package
> >       …))
> > --8<---------------cut here---------------end--------------->8---
>
> The ellipsis is a bit vague here.  What is this trying to do?

What you wrote below. :-)


> --8<---------------cut here---------------start------------->8---
> (define-public you-get
>   (package
>     (name "you-get")
>     (version "0.4.1355")
>     (PARAMETERS VIDEO-PLAYER PYTHON-VERSION WITH-FFMPEG)
>     (source (origin
>               (method git-fetch)
>               (uri (git-reference
>                     (url "https://github.com/soimort/you-get.git")
>                     (commit (string-append "v" version))))
>               (file-name (git-file-name name version))
>               (sha256
>                (base32
>                 "0xq7z04hvw3b3npiahlpzhbxsjvam9n9dynplyrkn84dx6k9ajbj"))))
>     (build-system python-build-system)
>     (inputs
>      `(("PLAYER" ,(DEREF-PARAM VIDEO-PLAYER))
>        ,@(IF (DEREF-PARAM WITH-FFMPEG)
>              ;; FOR MULTI-PART AND >=1080P VIDEOS
>              `("FFMPEG" ,FFMPEG)
>              '())))
>     (arguments
>      `(#:PYTHON ,(DEREF-PARAM PYTHON-VERSION)
>        #:phases
>        (modify-phases %standard-phases
>          ,(WHEN (DEREF-PARAM WITH-FFMPEG)
>             (add-after 'unpack 'qualify-input-references
>               ;; Explicitly invoke the input ffmpeg, instead of whichever one
>               ;; happens to be in the user's $PATH at run time.
>               (lambda* (#:key inputs #:allow-other-keys)
>                 (let ((ffmpeg (string-append (assoc-ref inputs "ffmpeg")
>                                              "/bin/ffmpeg")))
>                   (substitute* "src/you_get/processor/ffmpeg.py"
>                     ;; Don't blindly replace all occurrences of ‘'ffmpeg'’: the
>                     ;; same string is also used when sniffing ffmpeg's output.
>                     (("(FFMPEG == |\\()'ffmpeg'" _ prefix)
>                      (string-append prefix "'" ffmpeg "'")))
>                   #t))))
>          (ADD-AFTER 'UNPACK 'TWEAK-PLAYER-SETTINGS
>            (LAMBDA* (#:KEY INPUTS #:ALLOW-OTHER-KEYS)
>              (MATCH ,(DEREF-PARAM VIDEO-PLAYER)
>                (VLC
>                 ;; DO SOMETHING WITH VLC.
>                 )
>                (MPV
>                 ;; DO SOMETHING WITH MPV.
>                 )
>                (_
>                 ;; ERROR OUT?
>                 )))))
>        #:tests? #f))                    ; XXX some tests need Internet access
>     (synopsis "Download videos, audio, or images from Web sites")
>     (description
>      "You-Get is a command-line utility to download media contents (videos,
> audio, images) from the Web.  It can use either mpv or vlc for playback.")
>     (home-page "https://you-get.org/")
>     (license license:expat)))
> --8<---------------cut here---------------end--------------->8---
>
> In the above I've highlighted the changes in uppercase.

Welcome in a big and unmaintainable mess! :-)


And my opinion is that you described is already possible (more or
less) using 'inherit'. Except the modification of the compiling
options (build-system).


> On line (PARAMETERS ...) I've declared which parameters I'm going to use
> in my package declaration.  Those parameters must be defined globally
> somewhere in Guix.

This will end up with a big and unmaintainable mess, IMHO.


> The point of declaring the parameters in advance is that it allows the
> user to list all parameters used by a given package.

I bet that the number of broken packages will increase.



> - To let the user choose which video player to use.  This is a popular
>   USE flag on Gentoo (maybe with a different name).

What I do not understand is: people who used Gentoo and especially USE
flag are saying that it ends with a big mess with broken packages.
Therefore, why does Guix want to reproduce the mess?


All the best,
simon

Re: Build systems and implicit inputs

[zimoun]

Hi Ludo,

On Tue, 21 Jan 2020 at 11:56, Ludovic Courtès <ludo@gnu.org> wrote:

> > The solution to your problem in my opinion is simply to expose just the
> > right amount of options through #:arguments for all build systems.
> > Would that be satisfactory to you?
>
> I think the issue of tweaking the build system and its implicit inputs
> must be addressed separately.  We first need a good API to do that.
> When we have it, it’ll be nice and easy to drive it via package
> parameters.  :-)

Now I have a better understanding about "package parameters", I agree
that it is 2 separate stories.


> Currently each build system has ad-hoc keyword parameters to customize
> its implicit inputs: #:python for ‘python-build-system’, #:cmake for
> ‘cmake-build-system’, #:implicit-inputs? for ‘gnu-build-system’, etc.
>
> For each of them, it would be quite easy to provide a procedure that
> takes a list of implicit inputs and returns a <build-system>.  It may
> not be all that convenient, and perhaps a bit too ad-hoc still.

I was inclined for this option. :-)


> Another option would be to make implicit inputs a field of
> <build-system>.

Not sure to understand...

> Needs more thought!

...so yes! :-)


Cheers,
simon

Re: Build systems and implicit inputs

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 949 bytes --]

zimoun <zimon.toutoune@gmail.com> writes:

> Hi Ludo,
>
> On Tue, 21 Jan 2020 at 11:56, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> > The solution to your problem in my opinion is simply to expose just the
>> > right amount of options through #:arguments for all build systems.
>> > Would that be satisfactory to you?
>>
>> I think the issue of tweaking the build system and its implicit inputs
>> must be addressed separately.  We first need a good API to do that.
>> When we have it, it’ll be nice and easy to drive it via package
>> parameters.  :-)
>
> Now I have a better understanding about "package parameters", I agree
> that it is 2 separate stories.

Hmm... but does it have to?  It seems to me that we would gain a lot in
keeping those parameters general enough and not separate the handling of
the build system from the rest.  It would be simpler and more powerful.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parameterized packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1946 bytes --]

zimoun <zimon.toutoune@gmail.com> writes:

>> > --8<---------------cut here---------------start------------->8---
>> > (define (make-me-a-package option1 option2)
>> >     (package
>> >       …))
>> > --8<---------------cut here---------------end--------------->8---
>>
>> The ellipsis is a bit vague here.  What is this trying to do?
>
> What you wrote below. :-)

?

> And my opinion is that you described is already possible (more or
> less) using 'inherit'. Except the modification of the compiling
> options (build-system).

The crucial difference is the current approach with inherit /does not
compose/.  This is where USE flags stand strong because you can tell
what happens when both flags X and Y are set, e.g.

--8<---------------cut here---------------start------------->8---
  (if (and X Y)
     ; Do something when we have both X and Y activated.
     ; Else maybe just X, or just Y, or none.
     )
--8<---------------cut here---------------end--------------->8---

>> The point of declaring the parameters in advance is that it allows the
>> user to list all parameters used by a given package.
>
> I bet that the number of broken packages will increase.

Probably.  But if we keep the same standard for the default values of
the parameters, we would keep the same level of quality.  It should be OK.

>> - To let the user choose which video player to use.  This is a popular
>>   USE flag on Gentoo (maybe with a different name).
>
> What I do not understand is: people who used Gentoo and especially USE
> flag are saying that it ends with a big mess with broken packages.
> Therefore, why does Guix want to reproduce the mess?

Because it is extremely powerful: in a simple declaration like "no-X +
pulseaudio + python3 + gtk2" you can build /your entire system/,
according to the flags you've set, in just one command.

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Build systems and implicit inputs

[zimoun]

Hi Pierre,

On Tue, 21 Jan 2020 at 14:07, Pierre Neidhardt <mail@ambrevar.xyz> wrote:
> zimoun <zimon.toutoune@gmail.com> writes:
> > On Tue, 21 Jan 2020 at 11:56, Ludovic Courtès <ludo@gnu.org> wrote:

> >> > The solution to your problem in my opinion is simply to expose just the
> >> > right amount of options through #:arguments for all build systems.
> >> > Would that be satisfactory to you?
> >>
> >> I think the issue of tweaking the build system and its implicit inputs
> >> must be addressed separately.  We first need a good API to do that.
> >> When we have it, it’ll be nice and easy to drive it via package
> >> parameters.  :-)
> >
> > Now I have a better understanding about "package parameters", I agree
> > that it is 2 separate stories.
>
> Hmm... but does it have to?  It seems to me that we would gain a lot in
> keeping those parameters general enough and not separate the handling of
> the build system from the rest.  It would be simpler and more powerful.

I do not have a strong opinion. From my understanding, how to pass
arguments to the build system is a story and how to pass arguments to
packages is another one. Roughly speaking, they do not refer to the
same record, to the same functions that digest them, etc.. And I am
not a fan of adding global variables here and there.

Well, I am almost sure that treat them together will end with a big mess. :-)

Cheers,
simon

Re: Parameterized packages

[zimoun]

On Tue, 21 Jan 2020 at 14:13, Pierre Neidhardt <mail@ambrevar.xyz> wrote:
>
> zimoun <zimon.toutoune@gmail.com> writes:
>
> >> > --8<---------------cut here---------------start------------->8---
> >> > (define (make-me-a-package option1 option2)
> >> >     (package
> >> >       …))
> >> > --8<---------------cut here---------------end--------------->8---
> >>
> >> The ellipsis is a bit vague here.  What is this trying to do?
> >
> > What you wrote below. :-)
>
> ?

--8<---------------cut here---------------start------------->8---
(define (make-me-a-package VIDEO-PLAYER PYTHON-VERSION WITH-FFMPEG)
   (package
        [all your "messy" code using stuff]
      ))

(define-public you-get-vlc
    (make-me-a-package 'vlc 'python #t))
--8<---------------cut here---------------start------------->8---



> > And my opinion is that you described is already possible (more or
> > less) using 'inherit'. Except the modification of the compiling
> > options (build-system).
>
> The crucial difference is the current approach with inherit /does not
> compose/.  This is where USE flags stand strong because you can tell
> what happens when both flags X and Y are set, e.g.

I do not understand what you do mean by "does not compose".


To me, a package is:
"./configure && make && make check && make install"
so I understand why tweak the flags used by "./configure", for example
change "--with-vlc=" from the default 0 to the tuned 1. Or use another
compiling toolchain.
I understand also these flags could require different inputs.

And again from my understanding, this is more or less cover by
'inherit'. Or some macrology should be improved in 'package/inherit'
or in 'records.scm', i.e., something more fine grained when
inheriting.


--8<---------------cut here---------------start------------->8---
(define (make-you-get VIDEO-PLAYER PYTHON-VERSION WITH-FFMPEG)
  (package
     (inherit you-get
        #:add-inputs
         `(("PLAYER" ,VIDEO-PLAYER))
           ,@(IF WITH-FFMPEG)
             ;; FOR MULTI-PART AND >=1080P VIDEOS
             `("FFMPEG" ,FFMPEG)
        #:replace-arguments ...
       #:add-phase ...
             '())))

(define-public you-get-vlc (make-you-get 'vlc))
--8<---------------cut here---------------end--------------->8---


Something like that. And everything is more controlled, i.e., no mess
with global parameters.

What you want to do is: add/remove/replace inputs/arguments so
#:add-inputs, #:remove-native-inputs, etc.

And this composes in the same way function composes. And yes, it
composes explicitly which can be annoying, But implicit composition
will end up to the Gentoo USE flag mess, IMHO.



> >> - To let the user choose which video player to use.  This is a popular
> >>   USE flag on Gentoo (maybe with a different name).
> >
> > What I do not understand is: people who used Gentoo and especially USE
> > flag are saying that it ends with a big mess with broken packages.
> > Therefore, why does Guix want to reproduce the mess?
>
> Because it is extremely powerful: in a simple declaration like "no-X +
> pulseaudio + python3 + gtk2" you can build /your entire system/,
> according to the flags you've set, in just one command.

Hum? but months later your system is broken... so I am not convinced
it is powerful. :-)

It is broken because composing complex system is an hard task. Typing
helps a bit to detect error at compile time (in this case at building
package time) but run-time errors will still remain and it will be
hard time to find them.

Cheers,
simon

Re: Parameterized packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 2898 bytes --]

zimoun <zimon.toutoune@gmail.com> writes:

> I do not understand what you do mean by "does not compose".
>
>
> To me, a package is:
> "./configure && make && make check && make install"
> so I understand why tweak the flags used by "./configure", for example
> change "--with-vlc=" from the default 0 to the tuned 1. Or use another
> compiling toolchain.
> I understand also these flags could require different inputs.

If we reuse your example, a package may very well have --with-vlc and
--with-mpv.

Those flags conflict, so the package definition must know about the
corresponding "package parameters" at the same time to raise the
appropriate error (or choose The Right Option automatically).

Instead, if you'd inherit you'd always overwrite the changes without the
user noticing.  This removes a lot of control.

> --8<---------------cut here---------------start------------->8---
> (define (make-you-get VIDEO-PLAYER PYTHON-VERSION WITH-FFMPEG)
>   (package
>      (inherit you-get
>         #:add-inputs
>          `(("PLAYER" ,VIDEO-PLAYER))
>            ,@(IF WITH-FFMPEG)
>              ;; FOR MULTI-PART AND >=1080P VIDEOS
>              `("FFMPEG" ,FFMPEG)
>         #:replace-arguments ...
>        #:add-phase ...
>              '())))
>
> (define-public you-get-vlc (make-you-get 'vlc))
> --8<---------------cut here---------------end--------------->8---
>
>
> Something like that. And everything is more controlled,

What you propose here is essentially the same as what I propose, the
difference is that you wrapped it around `make-you-get` instead of
declaring the parameters inside as a field.

> i.e., no mess with global parameters.

I think there is a misunderstanding, there is no "mess with global
parameters".

Tobias suggestion was to declare the parameters (name, doc and type)
globally, just like we declare record globally.

> What you want to do is: add/remove/replace inputs/arguments so
> #:add-inputs, #:remove-native-inputs, etc.

Not just that, that's the point: we want the possibility to modify
anything, in particular what's inside #:arguments.

> Hum? but months later your system is broken... so I am not convinced
> it is powerful. :-)
>
> It is broken because composing complex system is an hard task. Typing
> helps a bit to detect error at compile time (in this case at building
> package time) but run-time errors will still remain and it will be
> hard time to find them.

Why would it be broken after months?  Packagers do the work once and it
works.  Of course the package parameters will have to be tested on
update of the package definition.  From the user's perspective, the
worst case scenario is that a formerly support parameter goes
unsupported for a given package, in which case we show a warning.

What do you think?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parameterized packages

[zimoun]

Hi Pierre,

On Wed, 22 Jan 2020 at 10:54, Pierre Neidhardt <mail@ambrevar.xyz> wrote:
> zimoun <zimon.toutoune@gmail.com> writes:

> > To me, a package is:
> > "./configure && make && make check && make install"
> > so I understand why tweak the flags used by "./configure", for example
> > change "--with-vlc=" from the default 0 to the tuned 1. Or use another
> > compiling toolchain.
> > I understand also these flags could require different inputs.
>
> If we reuse your example, a package may very well have --with-vlc and
> --with-mpv.
>
> Those flags conflict, so the package definition must know about the
> corresponding "package parameters" at the same time to raise the
> appropriate error (or choose The Right Option automatically).

It is the job to './configure' to detect conflicting parameters and
not the job of the package definition, IMHO.
The well-known: crap in, crap out! :-)


> Instead, if you'd inherit you'd always overwrite the changes without the
> user noticing.  This removes a lot of control.

Yes, I agree with the current implementation. That's why I wrote: ;-)
<<
And again from my understanding, this is more or less cover by
'inherit'. Or some macrology should be improved in 'package/inherit'
or in 'records.scm', i.e., something more fine grained when
inheriting.
>>



> > --8<---------------cut here---------------start------------->8---
> > (define (make-you-get VIDEO-PLAYER PYTHON-VERSION WITH-FFMPEG)
> >   (package
> >      (inherit you-get
> >         #:add-inputs
> >          `(("PLAYER" ,VIDEO-PLAYER))
> >            ,@(IF WITH-FFMPEG)
> >              ;; FOR MULTI-PART AND >=1080P VIDEOS
> >              `("FFMPEG" ,FFMPEG)
> >         #:replace-arguments ...
> >        #:add-phase ...
> >              '())))
> >
> > (define-public you-get-vlc (make-you-get 'vlc))
> > --8<---------------cut here---------------end--------------->8---
> >
> >
> > Something like that. And everything is more controlled,
>
> What you propose here is essentially the same as what I propose, the
> difference is that you wrapped it around `make-you-get` instead of
> declaring the parameters inside as a field.

The big differences are:

 1. the base package is untouched
 2. the arguments are locally defined
 3. everything is explicit

So less surprises IMHO.


> > i.e., no mess with global parameters.
>
> I think there is a misunderstanding, there is no "mess with global
> parameters".

Where should be defined what you called VIDEO-PLAYER?
the one in ' (PARAMETERS VIDEO-PLAYER PYTHON-VERSION WITH-FFMPEG)'.

Because the symbol needs to be defined somewhere, right?


> Tobias suggestion was to declare the parameters (name, doc and type)
> globally, just like we declare record globally.

It will end with a big mess because there is almost as many
"parameters" as packages.

And 'parameter' means somehow 'symbol', isn't it?


> > What you want to do is: add/remove/replace inputs/arguments so
> > #:add-inputs, #:remove-native-inputs, etc.
>
> Not just that, that's the point: we want the possibility to modify
> anything, in particular what's inside #:arguments.

I am suggesting the same thing. :-)

But instead I am suggesting to pass the modifications 'arguments' via
inheritance mechanism.


> > Hum? but months later your system is broken... so I am not convinced
> > it is powerful. :-)
> >
> > It is broken because composing complex system is an hard task. Typing
> > helps a bit to detect error at compile time (in this case at building
> > package time) but run-time errors will still remain and it will be
> > hard time to find them.
>
> Why would it be broken after months?  Packagers do the work once and it

I do not know but it is what Gentoo users are saying.
Well, I know a awesome guy wearing psychadelic T-shirt and former
Gentoo user that explained me that IRL. ;-)


> works.  Of course the package parameters will have to be tested on
> update of the package definition.  From the user's perspective, the

This adds a lot of workload to packagers, IMHO.

> worst case scenario is that a formerly support parameter goes
> unsupported for a given package, in which case we show a warning.
>
> What do you think?

Well, I am wanting the same thing: be able to modify the 'arguments'
field but I am not convinced by the design you are proposing because I
have the feeling that with such design then mess is coming. :-)


Cheers,
simon

Re: Parameterized packages

[ison]

On Wed, Jan 22, 2020 at 01:23:26PM +0100, zimoun wrote:
> > > --8<---------------cut here---------------start------------->8---
> > > (define (make-you-get VIDEO-PLAYER PYTHON-VERSION WITH-FFMPEG)
> > >   (package
> > >      (inherit you-get
> > >         #:add-inputs
> > >          `(("PLAYER" ,VIDEO-PLAYER))
> > >            ,@(IF WITH-FFMPEG)
> > >              ;; FOR MULTI-PART AND >=1080P VIDEOS
> > >              `("FFMPEG" ,FFMPEG)
> > >         #:replace-arguments ...
> > >        #:add-phase ...
> > >              '())))
> > >
> > > (define-public you-get-vlc (make-you-get 'vlc))
> > > --8<---------------cut here---------------end--------------->8---
> > >
> > >
> > > Something like that. And everything is more controlled,
> >
> > What you propose here is essentially the same as what I propose, the
> > difference is that you wrapped it around `make-you-get` instead of
> > declaring the parameters inside as a field.
> 
> The big differences are:
> 
>  1. the base package is untouched
>  2. the arguments are locally defined
>  3. everything is explicit
> 
> So less surprises IMHO.

My understanding of the global definitions they're talking about is that
they would just be meta objects, not global preferences.
For example (maybe it won't look like this, but just a guess):
Instead of passing the arguments "mpv" and "3.7" into your function you'd
give the parameters:
'((video-player . "mpv")
  (python-version . "3.7"))
to the package.
video-player and python-version would be globally defined meta objects
that just give descriptions about what these parameters should be used for
and such (so users can read more information about package parameters when
looking up packages). But the actual values that get used by the package
are not global, they're passed into the package just like your function
example.

I hope I'm not missing something here, but this is how I was reading the
conversation.

I'm not sure with the function method how easy it would be to inform the
user of all available options. For example when running `guix search`
would it be able to programmatically collect all the options that a
package has and display them there?
It almost seems like achieving that would require a parameters field in
the package anyway.

Also it seems that whether or not there are global meta objects is a
separate issue than whether to use parameters or functions. It looks like
either method could use them or not. In the function approach you could
still just pass in an alist as above.

Re: Parameterized packages

[John Soo]

Hey all!

I’ve been following very roughly. I have a couple issues with parameterized packages. 

> On Jan 22, 2020, at 4:24 AM, zimoun <zimon.toutoune@gmail.com> wrote:
> 
> Well, I am wanting the same thing: be able to modify the 'arguments'
> field but I am not convinced by the design you are proposing because I
> have the feeling that with such design then mess is coming. :-)

The maintenance and complexity does worry me.

Introducing more complexity may also make it harder to hack on guix itself. I love the hack ability of guix. (I correct me if I’m wrong, but isn’t hack ability a core design goal of the project?)

The other worry is that command line flags are close to another way of coding package modifications in guile. Maybe it can be more scriptable but I do want the guile code itself to continue to be the nice and fun place it is to work on. I’d rather flags not be a replacement for scheme.

So to sum up: not only could command line parameters potentially make it harder to work on guix code it might disincentivize actually doing any hacking.  I think there could be very nice ways of implementing flags that improve guix overall. I jut want to keep in mind the design goals of the project. 

-  John

Re: Parameterized packages

[zimoun]

Hi,

On Fri, 24 Jan 2020 at 23:06, ison <ison@airmail.cc> wrote:

> My understanding of the global definitions they're talking about is that
> they would just be meta objects, not global preferences.
> For example (maybe it won't look like this, but just a guess):
> Instead of passing the arguments "mpv" and "3.7" into your function you'd
> give the parameters:
> '((video-player . "mpv")
>   (python-version . "3.7"))
> to the package.
> video-player and python-version would be globally defined meta objects
> that just give descriptions about what these parameters should be used for
> and such (so users can read more information about package parameters when
> looking up packages). But the actual values that get used by the package
> are not global, they're passed into the package just like your function
> example.

Maybe I misread something and/or I misunderstand other thing but the
symbols 'video-player' and 'python' need to be defined somewhere. And
with this proposal adding the field 'parameters', this somewhere is
outside the package. Therefore, I will end up to a big mess, IMHO.

Let consider the packages 'foo' and 'bar'. They depend both on say the
'video-player' package.
Well, the definition looks like:

--8<---------------cut here---------------start------------->8---
(package
 (name "foo")
 (...)
 (parameters video-player)
 (...))

(package
 (name "bar")
 (...)
 (parameters video-player)
 (...))
--8<---------------cut here---------------end--------------->8---

Now, the user wants to install the both packages but:
 -  they requires that "foo" depends on vlc without any X support, say
"vlc-no-x"
 -  and they require that "bar" depends on full "vlc"

So how to do?
Where is defined 'video-player'?
How to install the both packages with the same transaction?


> I'm not sure with the function method how easy it would be to inform the
> user of all available options. For example when running `guix search`
> would it be able to programmatically collect all the options that a
> package has and display them there?

To me, the "official" packages in master should stay "simple" with no
parameter. Otherwise, it is will hard -- if not impossible -- to
ensure that all combinations work; or the number of parameters will be
very limited.

However, Guix should provide more tune-able rewriting package
functions. And these modified packages should live in channels;
possibly official ones to be built by Berlin (substitutes).


> Also it seems that whether or not there are global meta objects is a
> separate issue than whether to use parameters or functions. It looks like
> either method could use them or not. In the function approach you could
> still just pass in an alist as above.

Yes.
But the big differences are:

 1. the base package is untouched
 2. the arguments are locally defined
 3. everything is explicit

So less surprises IMHO.


Well, I really do not know and I do not have a strong opinion: I just
fear that the 'parameters' field would add complexity which would lead
to unmaintenable packages.


All the best,
simon

Re: Parameterized packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1039 bytes --]

ison <ison@airmail.cc> writes:

> My understanding of the global definitions they're talking about is that
> they would just be meta objects, not global preferences.
> For example (maybe it won't look like this, but just a guess):
> Instead of passing the arguments "mpv" and "3.7" into your function you'd
> give the parameters:
> '((video-player . "mpv")
>   (python-version . "3.7"))
> to the package.
> video-player and python-version would be globally defined meta objects
> that just give descriptions about what these parameters should be used for
> and such (so users can read more information about package parameters when
> looking up packages). But the actual values that get used by the package
> are not global, they're passed into the package just like your function
> example.
>
> I hope I'm not missing something here, but this is how I was reading the
> conversation.

You are correct, this is precisely what I meant.
Does that make more sense, Simon?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parameterized packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 3400 bytes --]

zimoun <zimon.toutoune@gmail.com> writes:

> Maybe I misread something and/or I misunderstand other thing but the
> symbols 'video-player' and 'python' need to be defined somewhere. And
> with this proposal adding the field 'parameters', this somewhere is
> outside the package. Therefore, I will end up to a big mess, IMHO.

I think you misread indeed :p

> Let consider the packages 'foo' and 'bar'. They depend both on say the
> 'video-player' package.

No, there is no video-player package.  It's a "package parameter".

> Well, the definition looks like:
>
> --8<---------------cut here---------------start------------->8---
> (package
>  (name "foo")
>  (...)
>  (parameters video-player)
>  (...))
>
> (package
>  (name "bar")
>  (...)
>  (parameters video-player)
>  (...))
> --8<---------------cut here---------------end--------------->8---
>
> Now, the user wants to install the both packages but:
>  -  they requires that "foo" depends on vlc without any X support, say
> "vlc-no-x"
>  -  and they require that "bar" depends on full "vlc"
>
> So how to do?
> Where is defined 'video-player'?

I think there is a misunderstanding on the definition of "package
parameters".

I intend to make package parameters first class objects for the reason
Ison explained in the previous message.

To answer your previous example, you'd do something like this, with a
made-up syntax:

guix install "foo(:video-player vlc-no-x)" "bar(:video-player vlc)"


>> I'm not sure with the function method how easy it would be to inform the
>> user of all available options. For example when running `guix search`
>> would it be able to programmatically collect all the options that a
>> package has and display them there?
>
> To me, the "official" packages in master should stay "simple" with no
> parameter. Otherwise, it is will hard -- if not impossible -- to
> ensure that all combinations work; or the number of parameters will be
> very limited.

Guix (unlike Gentoo) is functional, so that good news is that we can
have multiple versions of the same packages in the store.  This
essentially fixes the clash issues with Gentoo.

>> Also it seems that whether or not there are global meta objects is a
>> separate issue than whether to use parameters or functions. It looks like
>> either method could use them or not. In the function approach you could
>> still just pass in an alist as above.
>
> Yes.
> But the big differences are:
>
>  1. the base package is untouched
>  2. the arguments are locally defined
>  3. everything is explicit
>
> So less surprises IMHO.

But you can't compose the parameters, so this ruins the main point of
package parameters.

> Well, I really do not know and I do not have a strong opinion: I just
> fear that the 'parameters' field would add complexity which would lead
> to unmaintenable packages.

As you said, I think it would simply add more "tunable" options to
packages, that's it.  The main burden for packages is to make sure all
tunables of a package work together, but there is no escape to this.
Nix / Gentoo have the same problem here.

And well, if some parameter combination does not work, it's not the end
of the world either, the user would not have this specific version of a
package with present Guix either.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parameterized packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 532 bytes --]

Hi John,

I believe the complexity will remain under control.

"Flags" are not replacing Scheme: they are Scheme.  Just like we have
support for multiple outputs now.

For newcomers, package parameters are not an issue since they are
completely optional.  You won't see them until you want them!

Regarding the command line: maybe we don't have to update it after all.
We could stick to manifests and scripts to leverage package parameters.

Thoughts on this, everyone?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parameterized packages

[zimoun]

Hi Pierre,

On Mon, 27 Jan 2020 at 11:13, Pierre Neidhardt <mail@ambrevar.xyz> wrote:
>
> zimoun <zimon.toutoune@gmail.com> writes:
>
> > Maybe I misread something and/or I misunderstand other thing but the
> > symbols 'video-player' and 'python' need to be defined somewhere. And
> > with this proposal adding the field 'parameters', this somewhere is
> > outside the package. Therefore, I will end up to a big mess, IMHO.
>
> I think you misread indeed :p

Maybe I misread. :-)
I am still convinced that it is not the correct design and instead be
able to partially override the package definition seems more
appropriate.

At the end, code is law. :-)


Cheers,
simon

Re: Parameterized packages

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 365 bytes --]

zimoun <zimon.toutoune@gmail.com> writes:

> Maybe I misread. :-)
> I am still convinced that it is not the correct design and instead be
> able to partially override the package definition seems more
> appropriate.

If you have another design in mind that supports parameter composition, please
do share :)

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Parameterized packages

[zimoun]

On Mon, 27 Jan 2020 at 12:51, Pierre Neidhardt <mail@ambrevar.xyz> wrote:
>
> zimoun <zimon.toutoune@gmail.com> writes:
>
> > Maybe I misread. :-)
> > I am still convinced that it is not the correct design and instead be
> > able to partially override the package definition seems more
> > appropriate.
>
> If you have another design in mind that supports parameter composition, please
> do share :)

Héhé, you are kidding, right? :-)

Maybe improving how to override a package... :-D
Maybe something similar to Nix... which claims composition. ;-)

https://nixos.org/nixos/nix-pills/override-design-pattern.html
https://nixos.org/nixos/nix-pills/nixpkgs-overriding-packages.html

Anyway.
I should misunderstand something and should miss a key point.
You could explain me IRL at Guix Days in the front of the beer you owe me. ;-)


Cheers,
simon