guix environment

guix environment

[Federico Beffa]

Hi,

I believe that "guix environment" does not consider all outputs
properly. As one example, when I execute:

guix environment libpeas

the $PATH doesn't include /gnu/store/...glib-2.42.1-bin/bin where
("glib:bin" ,glib "bin") is one of the native-inputs of the package.

Is this intentional? If yes, what is the reason?

Thanks,
Fede

Re: guix environment

[Ludovic Courtès]

Federico Beffa <beffa@ieee.org> skribis:

> I believe that "guix environment" does not consider all outputs
> properly. As one example, when I execute:
>
> guix environment libpeas
>
> the $PATH doesn't include /gnu/store/...glib-2.42.1-bin/bin where
> ("glib:bin" ,glib "bin") is one of the native-inputs of the package.
>
> Is this intentional?

I don’t think so.

On closer inspection, I see two issues:

  (define (packages->transitive-inputs packages)
    "Return a list of the transitive inputs for all PACKAGES."
    (define (transitive-inputs package)
      (filter-map (match-lambda
                   ((_ (? package? package)) package)
                   (_ #f))    ; <---- !
                  (bag-transitive-inputs
                   (package->bag package))))
    (delete-duplicates
     (append-map transitive-inputs packages)))

Here only inputs of the form ("foo" PKG) are considered; things like
("glib" ,glib "bin") are discarded.

  (define (for-each-search-path proc inputs derivations pure?)
    (let ((paths (map derivation->output-path derivations))) ; <-- !
      [...]

Above, ‘derivation->output-path’ considers only the “out” output,
ignoring others if they are needed.

I think these would need to be adjusted.  Any takers?  :-)

Ludo’.

Re: guix environment

[David Thompson]

Ludovic Courtès <ludo@gnu.org> writes:

> Any takers?  :-)

I can try, unless someone beats me to it.

-- 
David Thompson
Web Developer - Free Software Foundation - http://fsf.org
GPG Key: 0FF1D807
Support the FSF: https://fsf.org/donate

Re: guix environment

[David Thompson]

[-- Attachment #1: Type: text/plain, Size: 970 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> On closer inspection, I see two issues:
>
>   (define (packages->transitive-inputs packages)
>     "Return a list of the transitive inputs for all PACKAGES."
>     (define (transitive-inputs package)
>       (filter-map (match-lambda
>                    ((_ (? package? package)) package)
>                    (_ #f))    ; <---- !
>                   (bag-transitive-inputs
>                    (package->bag package))))
>     (delete-duplicates
>      (append-map transitive-inputs packages)))
>
> Here only inputs of the form ("foo" PKG) are considered; things like
> ("glib" ,glib "bin") are discarded.
>
>   (define (for-each-search-path proc inputs derivations pure?)
>     (let ((paths (map derivation->output-path derivations))) ; <-- !
>       [...]
>
> Above, ‘derivation->output-path’ considers only the “out” output,
> ignoring others if they are needed.

Here's a patch.  WDYT?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-guix-environment-Consider-all-package-outputs.patch --]
[-- Type: text/x-diff, Size: 1848 bytes --]

From 9609806fb78557d74cf5b3fb47802898ef9d1ecf Mon Sep 17 00:00:00 2001
From: David Thompson <dthompson2@worcester.edu>
Date: Thu, 29 Jan 2015 17:53:17 -0500
Subject: [PATCH] guix: environment: Consider all package outputs.

* guix/scripts/environment.scm (for-each-search-path): Iterate over all
  derivation output paths.
  (packages->transitive-inputs): Process inputs that specify an output, too.
---
 guix/scripts/environment.scm | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
index ffa3a09..bb2ce53 100644
--- a/guix/scripts/environment.scm
+++ b/guix/scripts/environment.scm
@@ -40,7 +40,12 @@
 Use the output paths of DERIVATIONS to build each search path.  When PURE? is
 #t, the existing search path value is ignored.  Otherwise, the existing search
 path value is appended."
-  (let ((paths (map derivation->output-path derivations)))
+  (let ((paths (append-map (lambda (drv)
+                             (map (match-lambda
+                                   ((_ . output)
+                                    (derivation-output-path output)))
+                                  (derivation-outputs drv)))
+                           derivations)))
     (for-each (match-lambda
                (($ <search-path-specification>
                    variable directories separator)
@@ -177,7 +182,9 @@ packages."
   "Return a list of the transitive inputs for all PACKAGES."
   (define (transitive-inputs package)
     (filter-map (match-lambda
-                 ((_ (? package? package)) package)
+                 ((or (_ (? package? package))
+                      (_ (? package? package) _))
+                  package)
                  (_ #f))
                 (bag-transitive-inputs
                  (package->bag package))))
-- 
2.1.4


[-- Attachment #3: Type: text/plain, Size: 136 bytes --]


-- 
David Thompson
Web Developer - Free Software Foundation - http://fsf.org
GPG Key: 0FF1D807
Support the FSF: https://fsf.org/donate

Re: guix environment

[Ludovic Courtès]

David Thompson <dthompson2@worcester.edu> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> On closer inspection, I see two issues:
>>
>>   (define (packages->transitive-inputs packages)
>>     "Return a list of the transitive inputs for all PACKAGES."
>>     (define (transitive-inputs package)
>>       (filter-map (match-lambda
>>                    ((_ (? package? package)) package)
>>                    (_ #f))    ; <---- !
>>                   (bag-transitive-inputs
>>                    (package->bag package))))
>>     (delete-duplicates
>>      (append-map transitive-inputs packages)))
>>
>> Here only inputs of the form ("foo" PKG) are considered; things like
>> ("glib" ,glib "bin") are discarded.
>>
>>   (define (for-each-search-path proc inputs derivations pure?)
>>     (let ((paths (map derivation->output-path derivations))) ; <-- !
>>       [...]
>>
>> Above, ‘derivation->output-path’ considers only the “out” output,
>> ignoring others if they are needed.
>
> Here's a patch.  WDYT?
>
>
> From 9609806fb78557d74cf5b3fb47802898ef9d1ecf Mon Sep 17 00:00:00 2001
> From: David Thompson <dthompson2@worcester.edu>
> Date: Thu, 29 Jan 2015 17:53:17 -0500
> Subject: [PATCH] guix: environment: Consider all package outputs.
>
> * guix/scripts/environment.scm (for-each-search-path): Iterate over all
>   derivation output paths.
>   (packages->transitive-inputs): Process inputs that specify an output, too.

[...]

> --- a/guix/scripts/environment.scm
> +++ b/guix/scripts/environment.scm
> @@ -40,7 +40,12 @@
>  Use the output paths of DERIVATIONS to build each search path.  When PURE? is
>  #t, the existing search path value is ignored.  Otherwise, the existing search
>  path value is appended."
> -  (let ((paths (map derivation->output-path derivations)))
> +  (let ((paths (append-map (lambda (drv)
> +                             (map (match-lambda
> +                                   ((_ . output)
> +                                    (derivation-output-path output)))
> +                                  (derivation-outputs drv)))
> +                           derivations)))
>      (for-each (match-lambda
>                 (($ <search-path-specification>
>                     variable directories separator)
> @@ -177,7 +182,9 @@ packages."
>    "Return a list of the transitive inputs for all PACKAGES."
>    (define (transitive-inputs package)
>      (filter-map (match-lambda
> -                 ((_ (? package? package)) package)
> +                 ((or (_ (? package? package))
> +                      (_ (? package? package) _))
> +                  package)
>                   (_ #f))

LGTM, please push!

There’s another problem, though.  When a dependency is a multiple-output
package, all its outputs are added to the environment, because
‘package->transitive-inputs’ discards the information of which output is
needed.

So for instance, both the ‘out’ and the ‘debug’ output of Coreutils end
up being downloaded and added to the environment, even though only ‘out’
is an input.

Now, the problem is that ‘build-derivations’ can only build *all* the
outputs of the given derivation.  This could be worked around either:

  1. by creating a “sink” derivation, for instance with
     ‘profile-derivation’, that could refer precisely to the output(s)
     needed; not ideal.

  2. by using (build-things (list "/the/output/path")) and resorting to
     ‘build-derivations’ only if the ‘build-things’ call did nothing
     (when passed a non-.drv store item, ‘build-things’ tries to
     substitute and does nothing if that fails.)

Thoughts?

Ludo’.

Re: guix environment

[David Thompson]

Ludovic Courtès <ludo@gnu.org> writes:

> LGTM, please push!

Pushed.

> There’s another problem, though.  When a dependency is a multiple-output
> package, all its outputs are added to the environment, because
> ‘package->transitive-inputs’ discards the information of which output is
> needed.

Yes, I had thought about this, but couldn't think of a way to preserve
that information throughout the program.

> So for instance, both the ‘out’ and the ‘debug’ output of Coreutils end
> up being downloaded and added to the environment, even though only ‘out’
> is an input.
>
> Now, the problem is that ‘build-derivations’ can only build *all* the
> outputs of the given derivation.  This could be worked around either:
>
>   1. by creating a “sink” derivation, for instance with
>      ‘profile-derivation’, that could refer precisely to the output(s)
>      needed; not ideal.
>
>   2. by using (build-things (list "/the/output/path")) and resorting to
>      ‘build-derivations’ only if the ‘build-things’ call did nothing
>      (when passed a non-.drv store item, ‘build-things’ tries to
>      substitute and does nothing if that fails.)
>
> Thoughts?

I don't like option #1.  I haven't fully grokked option #2, but I guess
that will be the next thing to try.

Thanks for the nudge in the right direction.

-- 
David Thompson
Web Developer - Free Software Foundation - http://fsf.org
GPG Key: 0FF1D807
Support the FSF: https://fsf.org/donate

Re: bug#19641: guix environment

[Ludovic Courtès]

AFAIK the problem that Federico reported is fixed by 4b7ad2e3, so I’m
closing this bug, and opening a new one for the other issue I raised.

Thanks,
Ludo’.

guix environment

[Pjotr Prins]

Is there a way guix could quickly print out the information it now gives after
a successful package install:

The following environment variable definitions may be needed:
   export PATH="/home/pjotrp/.guix-profile/bin:/home/pjotrp/.guix-profile/sbin"
   export PKG_CONFIG_PATH="/home/pjotrp/.guix-profile/lib/pkgconfig"
   export GUILE_LOAD_PATH="/home/pjotrp/.guix-profile/share/guile/site/2.0"
   export GUILE_LOAD_COMPILED_PATH="/home/pjotrp/.guix-profile/share/guile/site/2.0"
   export CPATH="/home/pjotrp/.guix-profile/include"
   export LIBRARY_PATH="/home/pjotrp/.guix-profile/lib"
   export ACLOCAL_PATH="/home/pjotrp/.guix-profile/share/aclocal"
   export LOCPATH="/home/pjotrp/.guix-profile/lib/locale"

If that can be done on the fly we could include it in shell scripts.

A possible automation.

Would it be OK to add some RUBY paths?

Pj.

Re: guix environment

[Ricardo Wurmus]

Pjotr Prins <pjotr.public12@thebird.nl> writes:

> Is there a way guix could quickly print out the information it now gives after
> a successful package install:
>
> The following environment variable definitions may be needed:
>    export PATH="/home/pjotrp/.guix-profile/bin:/home/pjotrp/.guix-profile/sbin"
>    export PKG_CONFIG_PATH="/home/pjotrp/.guix-profile/lib/pkgconfig"
>    export GUILE_LOAD_PATH="/home/pjotrp/.guix-profile/share/guile/site/2.0"
>    export GUILE_LOAD_COMPILED_PATH="/home/pjotrp/.guix-profile/share/guile/site/2.0"
>    export CPATH="/home/pjotrp/.guix-profile/include"
>    export LIBRARY_PATH="/home/pjotrp/.guix-profile/lib"
>    export ACLOCAL_PATH="/home/pjotrp/.guix-profile/share/aclocal"
>    export LOCPATH="/home/pjotrp/.guix-profile/lib/locale"

"guix package --search-paths" should do that.  The only obstacle I see
in evaluating this automatically in profiles is that same variables are
not merged automatically, e.g. PYTHONPATH for different versions of
Python.

> Would it be OK to add some RUBY paths?

This is done on the package level.  LV2 audio plugins, for example,
suggest the LV2_PATH variable to be set.  For Ruby this could be done
by the build system.

~~ Ricardo

Re: guix environment

[Thompson, David]

On Mon, Jun 15, 2015 at 2:55 AM, Pjotr Prins <pjotr.public12@thebird.nl> wrote:
>
> Would it be OK to add some RUBY paths?

What additional Ruby search paths are needed?  We already have $GEM_PATH.

- Dave

Re: guix environment

[Claes Wallin (韋嘉誠)]

[-- Attachment #1: Type: text/plain, Size: 1096 bytes --]

On 15-Jun-2015 8:56 am, "Pjotr Prins" <pjotr.public12@thebird.nl> wrote:
>
> Is there a way guix could quickly print out the information it now gives
after
> a successful package install:
>
> The following environment variable definitions may be needed:
>    export
PATH="/home/pjotrp/.guix-profile/bin:/home/pjotrp/.guix-profile/sbin"
>    export PKG_CONFIG_PATH="/home/pjotrp/.guix-profile/lib/pkgconfig"
>    export
GUILE_LOAD_PATH="/home/pjotrp/.guix-profile/share/guile/site/2.0"
>    export
GUILE_LOAD_COMPILED_PATH="/home/pjotrp/.guix-profile/share/guile/site/2.0"
>    export CPATH="/home/pjotrp/.guix-profile/include"
>    export LIBRARY_PATH="/home/pjotrp/.guix-profile/lib"
>    export ACLOCAL_PATH="/home/pjotrp/.guix-profile/share/aclocal"
>    export LOCPATH="/home/pjotrp/.guix-profile/lib/locale"
>
> If that can be done on the fly we could include it in shell scripts.
>
> A possible automation.
>
> Would it be OK to add some RUBY paths?

Some of this is already in ~/.guix-profile/etc/profile. Maybe there should
be more in there? Or are even all of the search paths in there?

[-- Attachment #2: Type: text/html, Size: 1442 bytes --]

Re: guix environment

[Pjotr Prins]

On Mon, Jun 15, 2015 at 09:10:15AM -0400, Thompson, David wrote:
> On Mon, Jun 15, 2015 at 2:55 AM, Pjotr Prins <pjotr.public12@thebird.nl> wrote:
> >
> > Would it be OK to add some RUBY paths?
> 
> What additional Ruby search paths are needed?  We already have $GEM_PATH.

$GEM_HOME
$GEM_SPEC_CACHE

as in

https://github.com/pjotrp/guix-notes/blob/master/scripts/ruby-guix-env
-- 

Re: guix environment

[Thompson, David]

On Mon, Jun 15, 2015 at 3:10 PM, Pjotr Prins <pjotr.public12@thebird.nl> wrote:
> On Mon, Jun 15, 2015 at 09:10:15AM -0400, Thompson, David wrote:
>> On Mon, Jun 15, 2015 at 2:55 AM, Pjotr Prins <pjotr.public12@thebird.nl> wrote:
>> >
>> > Would it be OK to add some RUBY paths?
>>
>> What additional Ruby search paths are needed?  We already have $GEM_PATH.
>
> $GEM_HOME

This one doesn't make sense because it specifies where gems are to be
installed.  Store items are immutable, so one cannot install gems into
them.

> $GEM_SPEC_CACHE

Is this a real search path or can it only point to a single cache
directory?  If the latter, it cannot be a native search path in a
package recipe.

- Dave

Re: guix environment

[Ludovic Courtès]

"Claes Wallin (韋嘉誠)" <gnu@clacke.user.lysator.liu.se> skribis:

> Some of this is already in ~/.guix-profile/etc/profile. Maybe there should
> be more in there? Or are even all of the search paths in there?

Indeed, everything is there.

One can run:

  GUIX_PROFILE=$HOME/.guix-profile . ~/.guix-profile/etc/profile

or

  eval `guix package --search-paths`

See <http://www.gnu.org/software/guix/manual/html_node/Invoking-guix-package.html>.

And nowadays one can also use --search-paths=suffix or
--search-paths=prefix, for more flexibility.

Ludo’.

Re: guix environment

[Claes Wallin (韋嘉誠)]

[-- Attachment #1: Type: text/plain, Size: 488 bytes --]

On Jun 15, 2015 10:11 PM, "Ludovic Courtès" <ludo@gnu.org> wrote:
> "Claes Wallin (韋嘉誠)" <gnu@clacke.user.lysator.liu.se> skribis:
>
> > Some of this is already in ~/.guix-profile/etc/profile. Maybe there
should
> > be more in there? Or are even all of the search paths in there?
>
> Indeed, everything is there.
>
> One can run:
>
>   GUIX_PROFILE=$HOME/.guix-profile . ~/.guix-profile/etc/profile

Oh, *right*, that's what that override is for. Thanks for the tip.

[-- Attachment #2: Type: text/html, Size: 718 bytes --]

Re: guix environment

[Pjotr Prins]

On Mon, Jun 15, 2015 at 03:22:55PM -0400, Thompson, David wrote:
> On Mon, Jun 15, 2015 at 3:10 PM, Pjotr Prins <pjotr.public12@thebird.nl> wrote:
> > On Mon, Jun 15, 2015 at 09:10:15AM -0400, Thompson, David wrote:
> >> On Mon, Jun 15, 2015 at 2:55 AM, Pjotr Prins <pjotr.public12@thebird.nl> wrote:
> >> >
> >> > Would it be OK to add some RUBY paths?
> >>
> >> What additional Ruby search paths are needed?  We already have $GEM_PATH.
> >
> > $GEM_HOME
> 
> This one doesn't make sense because it specifies where gems are to be
> installed.  Store items are immutable, so one cannot install gems into
> them.

We still provide the gem tool ;). If we set this value to something
sensible (relative to $HOME) people can still use gems. The current
default setting is not good because it does not provide isolation.

I realise this may not be a guix concern, but for guix adoption it is
good to think about these things.

> > $GEM_SPEC_CACHE
> 
> Is this a real search path or can it only point to a single cache
> directory?  If the latter, it cannot be a native search path in a
> package recipe.

Same reasoning. We want isolation between different Ruby
installations. That is what I do in that script - pick up the HASH and
create a path relative to $HOME.

If this is what every normal user needs to do, why can't we have guix
help out? Ultimately it is part of the Ruby environment we work in -
therefore it is the resposibility of the Ruby package. It does away
with needing RVM :)

We can ignore this point, but it means every user will have to work
around it. What is the point of that?

Pj.

Re: guix environment

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> On Mon, Jun 15, 2015 at 03:22:55PM -0400, Thompson, David wrote:
>> On Mon, Jun 15, 2015 at 3:10 PM, Pjotr Prins <pjotr.public12@thebird.nl> wrote:
>> > On Mon, Jun 15, 2015 at 09:10:15AM -0400, Thompson, David wrote:
>> >> On Mon, Jun 15, 2015 at 2:55 AM, Pjotr Prins <pjotr.public12@thebird.nl> wrote:
>> >> >
>> >> > Would it be OK to add some RUBY paths?
>> >>
>> >> What additional Ruby search paths are needed?  We already have $GEM_PATH.
>> >
>> > $GEM_HOME
>> 
>> This one doesn't make sense because it specifies where gems are to be
>> installed.  Store items are immutable, so one cannot install gems into
>> them.
>
> We still provide the gem tool ;). If we set this value to something
> sensible (relative to $HOME) people can still use gems. The current
> default setting is not good because it does not provide isolation.

I don’t know Ruby, but this seems to be a variable that the *user* must
set, regardless of whether they use Guix or not, no?

Ludo’.

Re: guix environment

[Pjotr Prins]

On Tue, Jun 16, 2015 at 09:59:46AM +0200, Ludovic Courtès wrote:
> >> > $GEM_HOME
> >> 
> >> This one doesn't make sense because it specifies where gems are to be
> >> installed.  Store items are immutable, so one cannot install gems into
> >> them.
> >
> > We still provide the gem tool ;). If we set this value to something
> > sensible (relative to $HOME) people can still use gems. The current
> > default setting is not good because it does not provide isolation.
> 
> I don’t know Ruby, but this seems to be a variable that the *user* must
> set, regardless of whether they use Guix or not, no?

The current default points inside the Guix store. Which is (and should
be) read-only

    - INSTALLATION DIRECTORY: /gnu/store/c13v73jxmj2nir2xjqaz5259zywsa9zi-ruby-2.1.6/lib/ruby/gems/2.1.0

On Debian it is

    - INSTALLATION DIRECTORY: /var/lib/gems/1.9.1

(so, by default, you need sudo to run gem)

My proposal is to have it user based and isolated, e.g.,

    - INSTALLATION DIRECTORY: $HOME/.gem/c13v73jxmj2nir2xjqaz5259zywsa9zi-ruby-2.1.6/2.1.0

Which is similar to what rvm does.

The latter is what a user normally will set (or something similar).

Question is whether it belongs with the Guix package or not. The
current default makes no sense, unless you want to prevent a user from
installing modules and confuse him ;). 

Think my solution similar to what we do with emacs.  Emacs allows for
installation of modules in $HOME.

Pj.

Re: guix environment

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> On Tue, Jun 16, 2015 at 09:59:46AM +0200, Ludovic Courtès wrote:
>> >> > $GEM_HOME
>> >> 
>> >> This one doesn't make sense because it specifies where gems are to be
>> >> installed.  Store items are immutable, so one cannot install gems into
>> >> them.
>> >
>> > We still provide the gem tool ;). If we set this value to something
>> > sensible (relative to $HOME) people can still use gems. The current
>> > default setting is not good because it does not provide isolation.
>> 
>> I don’t know Ruby, but this seems to be a variable that the *user* must
>> set, regardless of whether they use Guix or not, no?
>
> The current default points inside the Guix store. Which is (and should
> be) read-only
>
>     - INSTALLATION DIRECTORY: /gnu/store/c13v73jxmj2nir2xjqaz5259zywsa9zi-ruby-2.1.6/lib/ruby/gems/2.1.0
>
> On Debian it is
>
>     - INSTALLATION DIRECTORY: /var/lib/gems/1.9.1
>
> (so, by default, you need sudo to run gem)
>
> My proposal is to have it user based and isolated, e.g.,
>
>     - INSTALLATION DIRECTORY: $HOME/.gem/c13v73jxmj2nir2xjqaz5259zywsa9zi-ruby-2.1.6/2.1.0

I suppose this would require patching Ruby, but why not.

Another option would be to do like Debian, which is presumably what
upstream intended, and which boils down to configuring Ruby with
--localstatedir=/var or similar.

Thoughts?

Ludo’.

Re: guix environment

[Pjotr Prins]

On Tue, Jun 16, 2015 at 01:23:23PM +0200, Ludovic Courtès wrote:
> > On Debian it is
> >
> >     - INSTALLATION DIRECTORY: /var/lib/gems/1.9.1
> >
> > (so, by default, you need sudo to run gem)
> >
> > My proposal is to have it user based and isolated, e.g.,
> >
> >     - INSTALLATION DIRECTORY: $HOME/.gem/c13v73jxmj2nir2xjqaz5259zywsa9zi-ruby-2.1.6/2.1.0
> 
> I suppose this would require patching Ruby, but why not.
> 
> Another option would be to do like Debian, which is presumably what
> upstream intended, and which boils down to configuring Ruby with
> --localstatedir=/var or similar.

I prefer defaulting to $HOME. It is easier to teach a sysadmin
different then an end-user ;)

This way gems will work out-of-the-box.

Pj.