Optional runtime dependencies in Guix

Optional runtime dependencies in Guix

[Gammel Holte]

[-- Attachment #1: Type: text/plain, Size: 1271 bytes --]

Hello,

I'm glad to see all interesting ideas from Nix being implemented as a DSL
on top of Scheme, which makes an awesome combination.

That said, I would like to bring up an issue I've faced multiple times
while using Nix, hoping that Guix can adopt a better solution here:
optional runtime dependencies.

Nix doesn't have a good decoupling between packages and their optional
runtime dependencies. You can disable them, but this would lead to a
different package hash, and thus a different build (negating the use of
prebuilt binaries).

Therefore, the culture seems to have all default package builds with all
optional runtime dependencies on. This leads to situations such as
installing mutt, and getting python as a dependency (via mutt -> gpgme ->
glib -> python), which is quite ugly.

Installing many unneeded packages in your system is undesirable for many
reasons. It has bothered me when using Nix in small embedded systems due to
resource limitations.

I haven't come up with any solution that also preserves purity. Perhaps one
needs to draw a line between what is a package dependency and what isn't.
In my option, a glib build is the same in the presence or absence of
python, just like firefox is the same whether you have flash installed or
not.

Alex.

[-- Attachment #2: Type: text/html, Size: 1445 bytes --]

Re: Optional runtime dependencies in Guix

[Ludovic Courtès]

Hello,

Gammel Holte <gammel.holte@gmail.com> skribis:

> Nix doesn't have a good decoupling between packages and their optional
> runtime dependencies. You can disable them, but this would lead to a
> different package hash, and thus a different build (negating the use of
> prebuilt binaries).
>
> Therefore, the culture seems to have all default package builds with all
> optional runtime dependencies on. This leads to situations such as
> installing mutt, and getting python as a dependency (via mutt -> gpgme ->
> glib -> python), which is quite ugly.

That’s indeed undesirable.

As I just wrote to Taylan Ulrich, this is currently handled on a
case-by-case basis using multiple outputs (which I think Nixpkgs doesn’t
use a lot yet.)

For instance, GLib has a separate “bin” output for this very reason (see
<http://bugs.gnu.org/17853>.)  Git, as I wrote, has separate outputs for
git-svn and Tcl stuff.  Same for WordNet.  There are also separate
outputs for debugging symbols.

So I wouldn’t claim this is a solved problem, because it really gets
fixed when we discover a problematic case, and we certainly overlook
some of them.  Yet, that’s something I pay attention to, and I think we
must clearly look to address more of such issues.

WDYT?

Thanks,
Ludo’.

Re: Optional runtime dependencies in Guix

[Gammel Holte]

[-- Attachment #1: Type: text/plain, Size: 3312 bytes --]

Apologies for the really late reply. Sadly I was ill for the best part of
December.

So I wouldn’t claim this is a solved problem, because it really gets
> fixed when we discover a problematic case, and we certainly overlook
> some of them.  Yet, that’s something I pay attention to, and I think we
> must clearly look to address more of such issues.
>
> WDYT?
>

I am super excited about Guix and the system seems to be progressing very
quickly. Loads of commits everyday. However, the more I dig into Guix, the
more i see this as an urgent issue.

For example, consider samtools, a package I use daily and that was recently
committed to Guix:

http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/bioinformatics.scm#n139

It forces me to install python. In contrast, consider Arch AUR's package:

https://aur.archlinux.org/packages/samtools/

Python is listed an optional dependency. In my opinion, it is highly
undesirable that Guix doesn't have a mechanism to decouple packages from
optional dependencies. This scenario seems to be arising mostly in case of
interpreters that may be called by the program to execute some optional
stuff.

An extreme example of this is weechat:

http://lists.gnu.org/archive/html/guix-devel/2014-09/msg00229.html

Compare with:

https://www.archlinux.org/packages/extra/i686/weechat/

Guix version forces the user to install all interpreters for running
user-defined scripts to extend Weechat. These are quite many: lua, perl,
python, ruby, tcl (and guile).

I understand Guix/Nix philosophy and I adhere to it. But at some point we
need to draw the line between a dependency and dynamic linking. Otherwise,
installing a package may lead to many undesired dependencies getting
installed. This has many implications, including security ones.

Best wishes,
A.

On Sun, Nov 23, 2014 at 8:47 PM, Ludovic Courtès <ludo@gnu.org> wrote:

> Hello,
>
> Gammel Holte <gammel.holte@gmail.com> skribis:
>
> > Nix doesn't have a good decoupling between packages and their optional
> > runtime dependencies. You can disable them, but this would lead to a
> > different package hash, and thus a different build (negating the use of
> > prebuilt binaries).
> >
> > Therefore, the culture seems to have all default package builds with all
> > optional runtime dependencies on. This leads to situations such as
> > installing mutt, and getting python as a dependency (via mutt -> gpgme ->
> > glib -> python), which is quite ugly.
>
> That’s indeed undesirable.
>
> As I just wrote to Taylan Ulrich, this is currently handled on a
> case-by-case basis using multiple outputs (which I think Nixpkgs doesn’t
> use a lot yet.)
>
> For instance, GLib has a separate “bin” output for this very reason (see
> <http://bugs.gnu.org/17853>.)  Git, as I wrote, has separate outputs for
> git-svn and Tcl stuff.  Same for WordNet.  There are also separate
> outputs for debugging symbols.
>
> So I wouldn’t claim this is a solved problem, because it really gets
> fixed when we discover a problematic case, and we certainly overlook
> some of them.  Yet, that’s something I pay attention to, and I think we
> must clearly look to address more of such issues.
>
> WDYT?
>
> Thanks,
> Ludo’.
>

[-- Attachment #2: Type: text/html, Size: 4614 bytes --]

Re: Optional runtime dependencies in Guix

[Ludovic Courtès]

Gammel Holte <gammel.holte@gmail.com> skribis:

> For example, consider samtools, a package I use daily and that was recently
> committed to Guix:
>
> http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/bioinformatics.scm#n139
>
> It forces me to install python. In contrast, consider Arch AUR's package:
>
> https://aur.archlinux.org/packages/samtools/

From looking at the page above, it seems that it would be feasible to
simply move varfilter.py to a different output.  That way, users would
be able to install the default output (which doesn’t depend on Python),
or the “python” output.  Ricardo, WDYT?

> An extreme example of this is weechat:
>
> http://lists.gnu.org/archive/html/guix-devel/2014-09/msg00229.html
>
> Compare with:
>
> https://www.archlinux.org/packages/extra/i686/weechat/
>
> Guix version forces the user to install all interpreters for running
> user-defined scripts to extend Weechat. These are quite many: lua, perl,
> python, ruby, tcl (and guile).

Yes, I hadn’t noticed this and I agree this is problematic.

Kevin, any idea on how to split things?

As I wrote before, there’s no one-size-fits-all recipe to address the
problem, just a couple of usable patterns (basically separate outputs or
separate packages.)  So we need to address this mostly on a case-by-case
basis, and also probably clarify this in the packaging guidelines.

Thanks,
Ludo’.

Re: Optional runtime dependencies in Guix

[Ricardo Wurmus]

Ludovic Courtès writes:

> Gammel Holte <gammel.holte@gmail.com> skribis:
>
>> For example, consider samtools, a package I use daily and that was recently
>> committed to Guix:
>>
>> http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/bioinformatics.scm#n139
>>
>> It forces me to install python. In contrast, consider Arch AUR's package:
>>
>> https://aur.archlinux.org/packages/samtools/
>
> From looking at the page above, it seems that it would be feasible to
> simply move varfilter.py to a different output.  That way, users would
> be able to install the default output (which doesn’t depend on Python),
> or the “python” output.  Ricardo, WDYT?

I could try to exclude this from the default output and install just the
Python files in a separate "python" output.

Is there a way to specify inputs that are required by certain outputs
only?  Then I could make the new "python" output depend on the Python
package as an input.

Re: Optional runtime dependencies in Guix

[宋文武]

Ludovic Courtès <ludo@gnu.org> writes:

> Gammel Holte <gammel.holte@gmail.com> skribis:
>
>> For example, consider samtools, a package I use daily and that was recently
>> committed to Guix:
>>
>> http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/bioinformatics.scm#n139
>>
>> It forces me to install python. In contrast, consider Arch AUR's package:
>>
>> https://aur.archlinux.org/packages/samtools/
>
> From looking at the page above, it seems that it would be feasible to
> simply move varfilter.py to a different output.  That way, users would
> be able to install the default output (which doesn’t depend on Python),
> or the “python” output.  Ricardo, WDYT?
Move it to a different output should work, but the 'python' output
doesn't make much sense to me compare to 'doc', 'bin' and 'debug'.

Note that 'python' is not a build dependency of 'samtools', so we
can only patch '#!/usr/bin/env' but not 'python' for varfilter.py.
Then we should give recommends or suggests, user need it could
install python manually.
>
>> An extreme example of this is weechat:
>>
>> http://lists.gnu.org/archive/html/guix-devel/2014-09/msg00229.html
>>
>> Compare with:
>>
>> https://www.archlinux.org/packages/extra/i686/weechat/
>>
>> Guix version forces the user to install all interpreters for running
>> user-defined scripts to extend Weechat. These are quite many: lua, perl,
>> python, ruby, tcl (and guile).
>
> Yes, I hadn’t noticed this and I agree this is problematic.
>
> Kevin, any idea on how to split things?
This is total different, those plugins must live in $out/lib/plugins
to work (can't move to seperated outputs).
Keep in mind that interpreters are both build and runtime dependencies
of weechat, the nature way is making them optional when building:

  (define-public (%weechat #:key (python? #t)
                                 (guile?  #t)
                                ...
    (package
      (inputs
      `(("python" ,(if python? python #nil))
        ("guile"  ,(if guile?  guile  #nil))
    ...
       

  (define-public weechat (%weechat)) ; our default version

Then user can install the customized version with:
  $ guix package -e '((@ (gnu packages weechat) %weechat) #:python? #f)'
>
> As I wrote before, there’s no one-size-fits-all recipe to address the
> problem, just a couple of usable patterns (basically separate outputs or
> separate packages.)  So we need to address this mostly on a case-by-case
> basis, and also probably clarify this in the packaging guidelines.
>
> Thanks,
> Ludo’.

Re: Optional runtime dependencies in Guix

[Ludovic Courtès]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

> Ludovic Courtès writes:
>
>> Gammel Holte <gammel.holte@gmail.com> skribis:
>>
>>> For example, consider samtools, a package I use daily and that was recently
>>> committed to Guix:
>>>
>>> http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/bioinformatics.scm#n139
>>>
>>> It forces me to install python. In contrast, consider Arch AUR's package:
>>>
>>> https://aur.archlinux.org/packages/samtools/
>>
>> From looking at the page above, it seems that it would be feasible to
>> simply move varfilter.py to a different output.  That way, users would
>> be able to install the default output (which doesn’t depend on Python),
>> or the “python” output.  Ricardo, WDYT?
>
> I could try to exclude this from the default output and install just the
> Python files in a separate "python" output.

Yes.

> Is there a way to specify inputs that are required by certain outputs
> only?  Then I could make the new "python" output depend on the Python
> package as an input.

There’s no way to do that.  So the build process would still depend on
Python.  However, someone using substitutes who does not want the Python
dependency would end up downloading just the Python-less output.

Thanks,
Ludo’.

Re: Optional runtime dependencies in Guix

[Ludovic Courtès]

宋文武 <iyzsong@gmail.com> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Gammel Holte <gammel.holte@gmail.com> skribis:
>>
>>> For example, consider samtools, a package I use daily and that was recently
>>> committed to Guix:
>>>
>>> http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/bioinformatics.scm#n139
>>>
>>> It forces me to install python. In contrast, consider Arch AUR's package:
>>>
>>> https://aur.archlinux.org/packages/samtools/
>>
>> From looking at the page above, it seems that it would be feasible to
>> simply move varfilter.py to a different output.  That way, users would
>> be able to install the default output (which doesn’t depend on Python),
>> or the “python” output.  Ricardo, WDYT?
> Move it to a different output should work, but the 'python' output
> doesn't make much sense to me compare to 'doc', 'bin' and 'debug'.

Yeah, but I don’t see anything against using “python” as an output name.

> Note that 'python' is not a build dependency of 'samtools', so we
> can only patch '#!/usr/bin/env' but not 'python' for varfilter.py.
> Then we should give recommends or suggests, user need it could
> install python manually.

Right, that’s probably even simpler.

>>> An extreme example of this is weechat:
>>>
>>> http://lists.gnu.org/archive/html/guix-devel/2014-09/msg00229.html
>>>
>>> Compare with:
>>>
>>> https://www.archlinux.org/packages/extra/i686/weechat/
>>>
>>> Guix version forces the user to install all interpreters for running
>>> user-defined scripts to extend Weechat. These are quite many: lua, perl,
>>> python, ruby, tcl (and guile).
>>
>> Yes, I hadn’t noticed this and I agree this is problematic.
>>
>> Kevin, any idea on how to split things?
> This is total different, those plugins must live in $out/lib/plugins
> to work (can't move to seperated outputs).

OK.  This could be relaxed using an environment variable (say
LTDL_LIBRARY_PATH, if Weechat uses libltdl.)

> Keep in mind that interpreters are both build and runtime dependencies
> of weechat, the nature way is making them optional when building:
>
>   (define-public (%weechat #:key (python? #t)
>                                  (guile?  #t)
>                                 ...
>     (package
>       (inputs
>       `(("python" ,(if python? python #nil))
>         ("guile"  ,(if guile?  guile  #nil))
>     ...
>        
>
>   (define-public weechat (%weechat)) ; our default version
>
> Then user can install the customized version with:
>   $ guix package -e '((@ (gnu packages weechat) %weechat) #:python? #f)'

Yes, but it’s not very convenient.

To begin with, we could have a “weechat” package with a “reasonable”
option set:

  (define weechat
    (make-weechat "weechat"))

And possibly another variant with, say, all the options enabled:

  (define weechat-full
    (make-weechat "weechat-full" #:python? #t #:lua? #t))

This should satisfy most users and would be easily usable.  This is
already done for a few packages, notably Emacs and PETSc.  And then
demanding users can do as you suggest.

WDYT?


A long term possibility would be to officially support something like
Gentoo’s “USE” flags.  These would be declared as part of the package,
and the build process would take them into account somehow:

  (define weechat
    (package
      ...
      (inputs `(,@(if (package-option weechat 'guile)
                      `(("guile" ,guile))
                      '())
                ...))
      (arguments
        `(#:configure-flags ,(if (package-option weechat 'guile)
                                 '("--with-guile")
                                 '())))
      (options (list (option-specification
                       (name 'guile)
                       (description "Whether to support Guile plug-ins.")
                       (default #t))))))

And then:

  $ guix package --show-options weechat
  ...

  $ guix package -i weechat -o guile

How does that sound?

Thanks,
Ludo’.

Re: Optional runtime dependencies in Guix

[Andreas Enge]

On Mon, Jan 12, 2015 at 05:26:02PM +0100, Ludovic Courtès wrote:
> To begin with, we could have a “weechat” package with a “reasonable”
> option set:
>   (define weechat
>     (make-weechat "weechat"))
> 
> And possibly another variant with, say, all the options enabled:
>   (define weechat-full
>     (make-weechat "weechat-full" #:python? #t #:lua? #t))

So far, our policy has rather been to enable all possible inputs. I think
this should be the default with the name "weechat" unaltered. If need be,
one could add another package with fewer inputs under the name
"weechat-small" or similar.

What do others think? If there is consensus, we could formalise something
in the package naming section of the manual.

Apart from that, I do not see why having several scripting languages enabled
is a problem; in the end, it is quite likely that they are present anyway due
to one package or another (it is rather difficult to avoid perl and python
these days!). So my real preference would be to not have such "...-small"
packages except for outrageously big default packages (texlive comes to
mind here...).

> A long term possibility would be to officially support something like
> Gentoo’s “USE” flags.  These would be declared as part of the package,
> and the build process would take them into account somehow:

To me, this sounds like overkill to solve a problem that I am not
convinced exists.

Andreas

Re: Optional runtime dependencies in Guix

[Gammel Holte]

[-- Attachment #1: Type: text/plain, Size: 2112 bytes --]

On Mon, Jan 12, 2015 at 6:47 PM, Andreas Enge <andreas@enge.fr> wrote:

> On Mon, Jan 12, 2015 at 05:26:02PM +0100, Ludovic Courtès wrote:
> > To begin with, we could have a “weechat” package with a “reasonable”
> > option set:
> >   (define weechat
> >     (make-weechat "weechat"))
> >
> > And possibly another variant with, say, all the options enabled:
> >   (define weechat-full
> >     (make-weechat "weechat-full" #:python? #t #:lua? #t))
>
> So far, our policy has rather been to enable all possible inputs. I think
> this should be the default with the name "weechat" unaltered. If need be,
> one could add another package with fewer inputs under the name
> "weechat-small" or similar.
>
> What do others think? If there is consensus, we could formalise something
> in the package naming section of the manual.
>
> Apart from that, I do not see why having several scripting languages
> enabled
> is a problem; in the end, it is quite likely that they are present anyway
> due
> to one package or another (it is rather difficult to avoid perl and python
> these days!). So my real preference would be to not have such "...-small"
> packages except for outrageously big default packages (texlive comes to
> mind here...).
>

I disagree here. I have very functional Arch & Gentoo installs with no
scripting language other than Perl, which is a dependency of many GNU tools.

In particular I'm doing just fine without Python. Installing everything by
default is a bit suboptimal from a security point of view, especially if
you're adding loads of interpreters.

Also, if you're working on a constrained system, the fewer packages the
better.

I liked the solution of giving recommends or suggests for interpreters.


> > A long term possibility would be to officially support something like
> > Gentoo’s “USE” flags.  These would be declared as part of the package,
> > and the build process would take them into account somehow:
>
> To me, this sounds like overkill to solve a problem that I am not
> convinced exists.
>
> Andreas
>
>
>

[-- Attachment #2: Type: text/html, Size: 2871 bytes --]

Re: Optional runtime dependencies in Guix

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> On Mon, Jan 12, 2015 at 05:26:02PM +0100, Ludovic Courtès wrote:
>> To begin with, we could have a “weechat” package with a “reasonable”
>> option set:
>>   (define weechat
>>     (make-weechat "weechat"))
>> 
>> And possibly another variant with, say, all the options enabled:
>>   (define weechat-full
>>     (make-weechat "weechat-full" #:python? #t #:lua? #t))
>
> So far, our policy has rather been to enable all possible inputs. I think
> this should be the default with the name "weechat" unaltered. If need be,
> one could add another package with fewer inputs under the name
> "weechat-small" or similar.
>
> What do others think? If there is consensus, we could formalise something
> in the package naming section of the manual.

Actually yes, weechat/weechat-small (rather than weechat-full/weechat)
is a good choice, and it’s what we did for Emacs notably.  Sorry for the
confusion.

> Apart from that, I do not see why having several scripting languages enabled
> is a problem; in the end, it is quite likely that they are present anyway due
> to one package or another (it is rather difficult to avoid perl and python
> these days!). So my real preference would be to not have such "...-small"
> packages except for outrageously big default packages (texlive comes to
> mind here...).

Well, pulling Python, Ruby, Lua, and Guile just to get an IRC client
really seems overkill.  In general, Guix is not about space-efficiency,
but we still want to make it easy to avoid adding extra weight when it’s
clearly unnecessary.

>> A long term possibility would be to officially support something like
>> Gentoo’s “USE” flags.  These would be declared as part of the package,
>> and the build process would take them into account somehow:
>
> To me, this sounds like overkill to solve a problem that I am not
> convinced exists.

Well, less code is better, and it could be that the full/small package
variants are enough in many cases.  We’ll see.

Ludo’.

Re: Optional runtime dependencies in Guix

[Ludovic Courtès]

Gammel Holte <gammel.holte@gmail.com> skribis:

> I disagree here. I have very functional Arch & Gentoo installs with no
> scripting language other than Perl, which is a dependency of many GNU tools.
>
> In particular I'm doing just fine without Python. Installing everything by
> default is a bit suboptimal from a security point of view, especially if
> you're adding loads of interpreters.

Yeah, adding loads of unused code in general is not very good.

> Also, if you're working on a constrained system, the fewer packages the
> better.

Another example is the USB installation image, which we do not want to
take too much space.  While tweaking it some time ago to reduce its
size, I noticed a few packages that would stealthily pull in a bunch of
big packages that were not strictly needed.

Ludo’.