Package transformations

Package transformations

[Ludovic Courtès]

Hello!

I’ve pushed my “train hacks”, which are a continuation of refactoring
work I had done in December (ca. 64ec0e2):

  • ‘guix build’ has a new ‘--with-input’ option to rewrite the
    dependency graph so you can run:

      guix build guix --with-input=guile=guile-next

    and guess what it does.  :-)

    However, it only works for explicit inputs.

  • ‘guix package’ supports ‘--with-source’ and ‘--with-input’, with the
    caveat that such transformations are forgotten upon upgrades.

  • There’s a fairly generic package transformation framework in (guix
    scripts build).

  • The documentation of command-line build options is a bit improved,
    with subsections for each category of options.

Feedback welcome!

Ludo’.

Re: Package transformations

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1769 bytes --]

On Mon, 01 Feb 2016 10:18:01 +0100
ludo@gnu.org (Ludovic Courtès) wrote:

> Hello!
> 
> I’ve pushed my “train hacks”, which are a continuation of refactoring
> work I had done in December (ca. 64ec0e2):
> 
>   • ‘guix build’ has a new ‘--with-input’ option to rewrite the
>     dependency graph so you can run:
> 
>       guix build guix --with-input=guile=guile-next
> 
>     and guess what it does.  :-)
> 
>     However, it only works for explicit inputs.
> 
>   • ‘guix package’ supports ‘--with-source’ and ‘--with-input’, with the
>     caveat that such transformations are forgotten upon upgrades.
> 
>   • There’s a fairly generic package transformation framework in (guix
>     scripts build).
> 
>   • The documentation of command-line build options is a bit improved,
>     with subsections for each category of options.
> 
> Feedback welcome!
> 
> Ludo’.
> 

I looked over the commit but not deeply enough yet, would it be possible to
use some of the logic in this to fix the package-with-python2 issue?

what about extending guix package so you could do `guix package -i $(guix
build foo --with-input=bar=baz) --named=foo-baz`, an on-the-fly custom
package?

In terms of figuring out how to make it more lasting, it would need to be
expanded if someone wanted to add/remove an input in their GUIX_PACKAGE_PATH
and have it propagate through the packages without having to make a custom
version of each one. Or to use as a substitute like when libpng had its CVE.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Package transformations

[Ludovic Courtès]

Efraim Flashner <efraim@flashner.co.il> skribis:

> I looked over the commit but not deeply enough yet, would it be possible to
> use some of the logic in this to fix the package-with-python2 issue?

Nope!

> what about extending guix package so you could do `guix package -i $(guix
> build foo --with-input=bar=baz) --named=foo-baz`, an on-the-fly custom
> package?

It’s always been possible to do:

  guix package -i $(guix build --whatever foobar)

but it’s a bit lossy in that ‘guix package’ has to guess the name,
version, and outputs being used, and it cannot know metadata such as the
package’s search paths.

With yesterday’s change, one can directly run:

  guix package --with-input=foo=bar -i foobar

> In terms of figuring out how to make it more lasting, it would need to be
> expanded if someone wanted to add/remove an input in their GUIX_PACKAGE_PATH
> and have it propagate through the packages without having to make a custom
> version of each one. Or to use as a substitute like when libpng had its CVE.

To preserve these transformations across upgrades, we’d have to write
them in profile manifests.  This introduces versioning complications
(what if the transformation mentioned in the manifest no longer exists
when you upgrade, or what if its on-disk format has changed, etc.)

Honestly, I’m not sure it’s a good idea to go this far.  I’d rather have
people write package variants they care about in a Scheme file.

Thoughts?

Ludo’.

Re: Package transformations

[Christopher Allan Webber]

Ludovic Courtès writes:

> Hello!
>
> I’ve pushed my “train hacks”, which are a continuation of refactoring
> work I had done in December (ca. 64ec0e2):
>
>   • ‘guix build’ has a new ‘--with-input’ option to rewrite the
>     dependency graph so you can run:
>
>       guix build guix --with-input=guile=guile-next
>
>     and guess what it does.  :-)
>
>     However, it only works for explicit inputs.

Wow, I only now just saw this!

I was happy to have done the guile-next packaging before, but I'm much
happier now.  This is so cool!

Re: Package transformations

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> I’ve pushed my “train hacks”, which are a continuation of refactoring
> work I had done in December (ca. 64ec0e2):
>
>   • ‘guix build’ has a new ‘--with-input’ option to rewrite the
>     dependency graph so you can run:
>
>       guix build guix --with-input=guile=guile-next
>
>     and guess what it does.  :-)
>
>     However, it only works for explicit inputs.

What does it mean that it only works for *explicit* inputs?  Does this
mean that default inputs added by the build system (e.g. GCC) cannot be
replaced?

Looking at the code it seems that it is operating on labels.  In the
past I have found that labels for the same package object are not always
identical.  For example “htseq” in “bioinformatics.scm” names uses the
label “python-numpy” for the package object named “python2-numpy”.  If I
used “--with-input=python2-numpy=my/python2-numpy” it would skip the
input labelled “python-numpy” (but change others in the graph with the
label “python2-numpy”) even though the package object is the same.

Is my understanding correct?  If so, should we instead attempt to match
on the package name of the package object rather than the label?  Or
should we be a little more careful with label names now that they are
also used outside of build phases?

I really like this additional feature!  Thanks!

~~ Ricardo

Re: Package transformations

[Ludovic Courtès]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> I’ve pushed my “train hacks”, which are a continuation of refactoring
>> work I had done in December (ca. 64ec0e2):
>>
>>   • ‘guix build’ has a new ‘--with-input’ option to rewrite the
>>     dependency graph so you can run:
>>
>>       guix build guix --with-input=guile=guile-next
>>
>>     and guess what it does.  :-)
>>
>>     However, it only works for explicit inputs.
>
> What does it mean that it only works for *explicit* inputs?  Does this
> mean that default inputs added by the build system (e.g. GCC) cannot be
> replaced?

Exactly.

Eventually we should permit it as well but as you know, it’s currently
less convenient to fiddle with implicit inputs than with explicit inputs
(see ‘package-with-explicit-inputs’ and co.)  We should improve the API
in this regard.

> Looking at the code it seems that it is operating on labels.

No, ‘transform-package-inputs’ in (guix scripts build) matches packages
by name, not by label, for the reasons you gave.  Maybe the doc should
be clarified?

Thanks for your feedback!

Ludo’.