User-profile search paths should include system-profile directories

User-profile search paths should include system-profile directories

[Carlos Sánchez de La Lama]

Hi guys,

I have an interesting case here. I have guile installed in my system
profile, so that

/var/guix/profiles/system/profile/share/aclocal/guile.m4

is there. However, autoconf is installed in my *user* profile, so
ACLOCAL_PATH is augmented in ~/.guix-profile/etc/profile to include

~/.guix-profile/share/aclocal

But not the system-profile aclocal directory, which would be put into
ACLOCAL_PATH by /var/guix/profiles/system/profile/etc/profile if
autoconf was installed in the systme profile as well.

Is this the intended behaviour? I am wondering whether packages with
search paths should include both the user-profile directories and the
system-profile ones.

Thoughts?

BR

Carlos

-- 
'There is prodigious strength,' I answered him, 'in sorrow and despair.' 

Charles Dickens, "A Tale of Two Cities" (1859)

Re: User-profile search paths should include system-profile directories

[Ludovic Courtès]

Hello,

csanchezdll@gmail.com (Carlos Sánchez de La Lama) skribis:

> I have an interesting case here. I have guile installed in my system
> profile, so that
>
> /var/guix/profiles/system/profile/share/aclocal/guile.m4
>
> is there. However, autoconf is installed in my *user* profile, so
> ACLOCAL_PATH is augmented in ~/.guix-profile/etc/profile to include
>
> ~/.guix-profile/share/aclocal
>
> But not the system-profile aclocal directory, which would be put into
> ACLOCAL_PATH by /var/guix/profiles/system/profile/etc/profile if
> autoconf was installed in the systme profile as well.
>
> Is this the intended behaviour? I am wondering whether packages with
> search paths should include both the user-profile directories and the
> system-profile ones.

I think you’re right.  This was discussed at
<http://bugs.gnu.org/20255>, leading to a patch (for GuixSD).

However, we failed to build consensus around the approach of this patch,
so we did not apply it.  If you have ideas, please email
20255@debbugs.gnu.org.  :-)

One of the fruits of this discussion, though, is that you can do:

  guix package -p /run/current-system/profile -p ~/.guix-profile --search-paths

to get the combined search paths.

Ludo’.

Re: User-profile search paths should include system-profile directories

[Alex Kost]

Ludovic Courtès (2016-09-14 16:58 +0200) wrote:

> Hello,
>
> csanchezdll@gmail.com (Carlos Sánchez de La Lama) skribis:
>
>> I have an interesting case here. I have guile installed in my system
>> profile, so that
>>
>> /var/guix/profiles/system/profile/share/aclocal/guile.m4
>>
>> is there. However, autoconf is installed in my *user* profile, so
>> ACLOCAL_PATH is augmented in ~/.guix-profile/etc/profile to include
>>
>> ~/.guix-profile/share/aclocal
>>
>> But not the system-profile aclocal directory, which would be put into
>> ACLOCAL_PATH by /var/guix/profiles/system/profile/etc/profile if
>> autoconf was installed in the systme profile as well.
>>
>> Is this the intended behaviour? I am wondering whether packages with
>> search paths should include both the user-profile directories and the
>> system-profile ones.
>
> I think you’re right.  This was discussed at
> <http://bugs.gnu.org/20255>, leading to a patch (for GuixSD).
>
> However, we failed to build consensus around the approach of this patch,
> so we did not apply it.  If you have ideas, please email
> 20255@debbugs.gnu.org.  :-)

I think I was the one who prevents the consensus.  To make it clear, I'm
for the suggested solution, but only *after* giving a user a freedom to
avoid loading such a heavy command as "guix package --search-paths".  On
a "usual" GNU/Linux distro a user can edit /etc/profile, but on GuixSD
it is not possible currently.  That's why I think there should be
provided a possibility to override /etc/profile at first.

> One of the fruits of this discussion, though, is that you can do:
>
>   guix package -p /run/current-system/profile -p ~/.guix-profile --search-paths
>
> to get the combined search paths.

This is what I do to combine my profiles (I don't use ~/.guix-profile
but several other profiles instead).

-- 
Alex

Customizing /etc

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 2006 bytes --]

Alex Kost <alezost@gmail.com> skribis:

> Ludovic Courtès (2016-09-14 16:58 +0200) wrote:
>
>> Hello,
>>
>> csanchezdll@gmail.com (Carlos Sánchez de La Lama) skribis:
>>
>>> I have an interesting case here. I have guile installed in my system
>>> profile, so that
>>>
>>> /var/guix/profiles/system/profile/share/aclocal/guile.m4
>>>
>>> is there. However, autoconf is installed in my *user* profile, so
>>> ACLOCAL_PATH is augmented in ~/.guix-profile/etc/profile to include
>>>
>>> ~/.guix-profile/share/aclocal
>>>
>>> But not the system-profile aclocal directory, which would be put into
>>> ACLOCAL_PATH by /var/guix/profiles/system/profile/etc/profile if
>>> autoconf was installed in the systme profile as well.
>>>
>>> Is this the intended behaviour? I am wondering whether packages with
>>> search paths should include both the user-profile directories and the
>>> system-profile ones.
>>
>> I think you’re right.  This was discussed at
>> <http://bugs.gnu.org/20255>, leading to a patch (for GuixSD).
>>
>> However, we failed to build consensus around the approach of this patch,
>> so we did not apply it.  If you have ideas, please email
>> 20255@debbugs.gnu.org.  :-)
>
> I think I was the one who prevents the consensus.  To make it clear, I'm
> for the suggested solution, but only *after* giving a user a freedom to
> avoid loading such a heavy command as "guix package --search-paths".  On
> a "usual" GNU/Linux distro a user can edit /etc/profile, but on GuixSD
> it is not possible currently.  That's why I think there should be
> provided a possibility to override /etc/profile at first.

Indeed, thanks for the reminder!

In fact, we have this through ‘etc-service-type’, except that currently
/etc/profile is systematically added.

So an idea that comes to mind is to allow ‘etc-service-type’ to be
extended with procedures that would be able to filter or otherwise
change the /etc entries (similar to what we do for PAM):


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1144 bytes --]

diff --git a/gnu/services.scm b/gnu/services.scm
index 7e322c5..9397232 100644
--- a/gnu/services.scm
+++ b/gnu/services.scm
@@ -426,9 +426,13 @@ directory."
                 (extensions
                  (list
                   (service-extension activation-service-type
-                                     (lambda (files)
-                                       (let ((etc
-                                              (files->etc-directory files)))
+                                     (lambda (files+procs)
+                                       (let* ((proc (apply compose
+                                                           (filter procedure?
+                                                                   files+procs)))
+                                              (files (filter pair? files+procs))
+                                              (etc
+                                               (files->etc-directory (proc files))))
                                          #~(activate-etc #$etc))))
                   (service-extension system-service-type etc-entry)))
                 (compose concatenate)

[-- Attachment #3: Type: text/plain, Size: 460 bytes --]


In your config, you could have something like:

  (services (cons (simple-service 'rm-/etc/profile etc-service-type
                                  (const (lambda (files)
                                           (assoc-delete "profile" files))))
                  %base-services))

WDYT?

In fact I think we would need to have a more generic mechanism to hook
into ‘fold-services’, but I’m not sure what it should look like.

Ludo’.

Re: Customizing /etc

[Alex Kost]

Ludovic Courtès (2016-09-19 23:08 +0900) wrote:

> Alex Kost <alezost@gmail.com> skribis:
>
>> Ludovic Courtès (2016-09-14 16:58 +0200) wrote:
[...]
>>> However, we failed to build consensus around the approach of this patch,
>>> so we did not apply it.  If you have ideas, please email
>>> 20255@debbugs.gnu.org.  :-)
>>
>> I think I was the one who prevents the consensus.  To make it clear, I'm
>> for the suggested solution, but only *after* giving a user a freedom to
>> avoid loading such a heavy command as "guix package --search-paths".  On
>> a "usual" GNU/Linux distro a user can edit /etc/profile, but on GuixSD
>> it is not possible currently.  That's why I think there should be
>> provided a possibility to override /etc/profile at first.
>
> Indeed, thanks for the reminder!
>
> In fact, we have this through ‘etc-service-type’, except that currently
> /etc/profile is systematically added.
>
> So an idea that comes to mind is to allow ‘etc-service-type’ to be
> extended with procedures that would be able to filter or otherwise
> change the /etc entries (similar to what we do for PAM):
>
> diff --git a/gnu/services.scm b/gnu/services.scm
> index 7e322c5..9397232 100644
> --- a/gnu/services.scm
> +++ b/gnu/services.scm
> @@ -426,9 +426,13 @@ directory."
>                  (extensions
>                   (list
>                    (service-extension activation-service-type
> -                                     (lambda (files)
> -                                       (let ((etc
> -                                              (files->etc-directory files)))
> +                                     (lambda (files+procs)
> +                                       (let* ((proc (apply compose
> +                                                           (filter procedure?
> +                                                                   files+procs)))
> +                                              (files (filter pair? files+procs))
> +                                              (etc
> +                                               (files->etc-directory (proc files))))
>                                           #~(activate-etc #$etc))))
>                    (service-extension system-service-type etc-entry)))
>                  (compose concatenate)
>
>
> In your config, you could have something like:
>
>   (services (cons (simple-service 'rm-/etc/profile etc-service-type
>                                   (const (lambda (files)
>                                            (assoc-delete "profile" files))))
>                   %base-services))
>
> WDYT?

I think it would be great!  I didn't realize it can be implemented this
way.

> In fact I think we would need to have a more generic mechanism to hook
> into ‘fold-services’, but I’m not sure what it should look like.

Yeah, a more generic way would be better of course, but I think this
mixing of files and procedures is already good enough for now, as it
provides a freedom in customizing a system that we didn't have before.
Besides I will no longer object against the "search-paths" fix for the
bug 20255 :-)

-- 
Alex

Re: User-profile search paths should include system-profile directories

[Carlos Sánchez de La Lama]

Hi,

>> Is this the intended behaviour? I am wondering whether packages with
>> search paths should include both the user-profile directories and the
>> system-profile ones.
>
> I think you’re right.  This was discussed at
> <http://bugs.gnu.org/20255>, leading to a patch (for GuixSD).
>
> However, we failed to build consensus around the approach of this patch,
> so we did not apply it.  If you have ideas, please email
> 20255@debbugs.gnu.org.  :-)

what about something like:

- /etc/profile
  # [...]
  GUIX_PROFILES="/run/current-system/profile:$HOME/.guix-profile"
  . /run/current-system/profile/etc/profile 
  # [...]
  . "$HOME/.guix-profile/etc/profile"
  # [...]

- /run/current-system/profile/etc/profile
  profiles="${GUIX_PROFILES:=/gnu/store/<hashA>-profile}"
  export PATH="${profiles//:/\/bin}/bin ${profiles//:/\/sbin}/sbin"
  # [...]

  
- $HOME/.guix-profile/etc/profile
  profiles="${GUIX_PROFILES:=/gnu/store/<hashB>-profile}"
  export ACLOCAL_PATH="${profiles//:/\/share/aclocal}/share/aclocal"
  # [...]

That is, each profile adds all its search paths to all the profile roots
passed in the colon separated variable GUIX_PROFILES. If GUIX_PROFILES
is empty, it adds the search paths inside its own directory only (as
until now).

No need to expand recursively (no ":$OLD_VALUE" at then end of each
assignment) as we are generating the profile, user additions will come
later. Should be fast enough, and a user can do the (slower) guix
--search-paths in its own profile if he wants to.

If you think it's fine, I will prepare a patch.

Thoughts?

Carlos

Re: User-profile search paths should include system-profile directories

[Alex Kost]

Carlos Sánchez de La Lama (2016-09-20 09:39 +0200) wrote:

> Hi,
>
>>> Is this the intended behaviour? I am wondering whether packages with
>>> search paths should include both the user-profile directories and the
>>> system-profile ones.
>>
>> I think you’re right.  This was discussed at
>> <http://bugs.gnu.org/20255>, leading to a patch (for GuixSD).
>>
>> However, we failed to build consensus around the approach of this patch,
>> so we did not apply it.  If you have ideas, please email
>> 20255@debbugs.gnu.org.  :-)
>
> what about something like:
>
> - /etc/profile
>   # [...]
>   GUIX_PROFILES="/run/current-system/profile:$HOME/.guix-profile"
>   . /run/current-system/profile/etc/profile
>   # [...]
>   . "$HOME/.guix-profile/etc/profile"
>   # [...]
>
> - /run/current-system/profile/etc/profile
>   profiles="${GUIX_PROFILES:=/gnu/store/<hashA>-profile}"
>   export PATH="${profiles//:/\/bin}/bin ${profiles//:/\/sbin}/sbin"
>   # [...]
>
>
> - $HOME/.guix-profile/etc/profile
>   profiles="${GUIX_PROFILES:=/gnu/store/<hashB>-profile}"
>   export ACLOCAL_PATH="${profiles//:/\/share/aclocal}/share/aclocal"
>   # [...]
>
> That is, each profile adds all its search paths to all the profile roots
> passed in the colon separated variable GUIX_PROFILES. If GUIX_PROFILES
> is empty, it adds the search paths inside its own directory only (as
> until now).

I think it was mentioned somewhere in the bug discussion: this will not
work for some things.  For example, if a user has 'guile' in a system
profile, and several guile packages (but not 'guile' itself) in
~/.guix-profile, then GUILE_LOAD_PATH will include
"<system-profile>/share/guile/site/2.0" but not
"<user-profile>/share/guile/site/2.0".  That's why combining profiles
inside "guix package --search-paths" command looks like the only
solution.

-- 
Alex

Re: User-profile search paths should include system-profile directories

[Carlos Sánchez de La Lama]

Alex Kost <alezost@gmail.com> writes:

> Carlos Sánchez de La Lama (2016-09-20 09:39 +0200) wrote:
>
>> Hi,
>>
>>>> Is this the intended behaviour? I am wondering whether packages with
>>>> search paths should include both the user-profile directories and the
>>>> system-profile ones.
>>>
>>> I think you’re right.  This was discussed at
>>> <http://bugs.gnu.org/20255>, leading to a patch (for GuixSD).
>>>
>>> However, we failed to build consensus around the approach of this patch,
>>> so we did not apply it.  If you have ideas, please email
>>> 20255@debbugs.gnu.org.  :-)
>>
>> what about something like:
>>
>> - /etc/profile
>>   # [...]
>>   GUIX_PROFILES="/run/current-system/profile:$HOME/.guix-profile"
>>   . /run/current-system/profile/etc/profile
>>   # [...]
>>   . "$HOME/.guix-profile/etc/profile"
>>   # [...]
>>
>> - /run/current-system/profile/etc/profile
>>   profiles="${GUIX_PROFILES:=/gnu/store/<hashA>-profile}"
>>   export PATH="${profiles//:/\/bin}/bin ${profiles//:/\/sbin}/sbin"
>>   # [...]
>>
>>
>> - $HOME/.guix-profile/etc/profile
>>   profiles="${GUIX_PROFILES:=/gnu/store/<hashB>-profile}"
>>   export ACLOCAL_PATH="${profiles//:/\/share/aclocal}/share/aclocal"
>>   # [...]
>>
>> That is, each profile adds all its search paths to all the profile roots
>> passed in the colon separated variable GUIX_PROFILES. If GUIX_PROFILES
>> is empty, it adds the search paths inside its own directory only (as
>> until now).
>
> I think it was mentioned somewhere in the bug discussion: this will not
> work for some things.  For example, if a user has 'guile' in a system
> profile, and several guile packages (but not 'guile' itself) in
> ~/.guix-profile, then GUILE_LOAD_PATH will include
> "<system-profile>/share/guile/site/2.0" but not
> "<user-profile>/share/guile/site/2.0".  That's why combining profiles
> inside "guix package --search-paths" command looks like the only
> solution.

It will actually solve it, that is the case I was trying to fix in
fact. The code I show will include /share/guile/site/2.0 prefixed
by every root in the colon-separated list GUIX_PROFILES (note the
difference with current code with has GUIX_PROFILE, singular). If
GUIX_PROFILES is (as in the example):

GUIX_PROFILES="/run/current-system/profile:$HOME/.guix-profile"

And guile is in the system profile, the line

export GUILE_LOAD_PATH="${profiles//://share/guile/site/2.0:}/share/guile/site/2.0"

will produce:

export GUILE_LOAD_PATH=/run/current-system/profile/share/guile/site/2.0:$HOME/.guix-profile/share/guile/site/2.0

As desired. Same result if guile is only in user profile, or if it is on
both. Try the following in bash:

profiles="/run/current-system/profile:$HOME/.guix-profile:/my/third/profile"
echo "${profiles//://share/guile/site/2.0:}/share/guile/site/2.0"

BR

Carlos

Re: User-profile search paths should include system-profile directories

[Ludovic Courtès]

Hi,

csanchezdll@gmail.com (Carlos Sánchez de La Lama) skribis:

> Alex Kost <alezost@gmail.com> writes:

[...]

>>> what about something like:
>>>
>>> - /etc/profile
>>>   # [...]
>>>   GUIX_PROFILES="/run/current-system/profile:$HOME/.guix-profile"
>>>   . /run/current-system/profile/etc/profile
>>>   # [...]
>>>   . "$HOME/.guix-profile/etc/profile"
>>>   # [...]
>>>
>>> - /run/current-system/profile/etc/profile
>>>   profiles="${GUIX_PROFILES:=/gnu/store/<hashA>-profile}"
>>>   export PATH="${profiles//:/\/bin}/bin ${profiles//:/\/sbin}/sbin"
>>>   # [...]
>>>
>>>
>>> - $HOME/.guix-profile/etc/profile
>>>   profiles="${GUIX_PROFILES:=/gnu/store/<hashB>-profile}"
>>>   export ACLOCAL_PATH="${profiles//:/\/share/aclocal}/share/aclocal"
>>>   # [...]
>>>
>>> That is, each profile adds all its search paths to all the profile roots
>>> passed in the colon separated variable GUIX_PROFILES. If GUIX_PROFILES
>>> is empty, it adds the search paths inside its own directory only (as
>>> until now).
>>
>> I think it was mentioned somewhere in the bug discussion: this will not
>> work for some things.  For example, if a user has 'guile' in a system
>> profile, and several guile packages (but not 'guile' itself) in
>> ~/.guix-profile, then GUILE_LOAD_PATH will include
>> "<system-profile>/share/guile/site/2.0" but not
>> "<user-profile>/share/guile/site/2.0".  That's why combining profiles
>> inside "guix package --search-paths" command looks like the only
>> solution.
>
> It will actually solve it, that is the case I was trying to fix in
> fact. The code I show will include /share/guile/site/2.0 prefixed
> by every root in the colon-separated list GUIX_PROFILES (note the
> difference with current code with has GUIX_PROFILE, singular). If
> GUIX_PROFILES is (as in the example):
>
> GUIX_PROFILES="/run/current-system/profile:$HOME/.guix-profile"
>
> And guile is in the system profile, the line
>
> export GUILE_LOAD_PATH="${profiles//://share/guile/site/2.0:}/share/guile/site/2.0"
>
> will produce:
>
> export GUILE_LOAD_PATH=/run/current-system/profile/share/guile/site/2.0:$HOME/.guix-profile/share/guile/site/2.0

Unless I’m mistaken, I don’t think this would work.

In the example Alex gave, none of the two profiles specifies
GUILE_LOAD_PATH if taken individually.  This example is also in the
manual to illustrate the use of multiple -p flags with --search-paths
(info "(guix) Invoking guix package").

Thoughts?

Ludo’.

Re: User-profile search paths should include system-profile directories

[Carlos Sánchez de La Lama]

Hi Ludo, Alex,

> In the example Alex gave, none of the two profiles specifies
> GUILE_LOAD_PATH if taken individually.  This example is also in the
> manual to illustrate the use of multiple -p flags with --search-paths
> (info "(guix) Invoking guix package").

I think I understood now. I knew "guix --search-paths" only returned
those search paths which *do exist* in the current profile (thus the
need for multiple -p parameters), but I (wrongly) thought search paths
were added unconditionally to <profile_root>/etc/profile.

Making this work would need two things:
1) not stripping unexisting directories from search paths
2) constructing environment variables prepending every profile root to
   the search paths of each profile.

And my suggestion only solved point 2.

Now, my question is, why are unexisting search paths removed? This is
done in guix-search-paths.scm(evaluate-sarch-paths), by calling
search-path-as-list, which filters out unexisting directories. Why not
just add all of them? Profiles are immutable once created, so having a
search path pointing into a subexisting subdirectory should not do any
harm.

Am I missing something?

Carlos

Re: User-profile search paths should include system-profile directories

[Ludovic Courtès]

Hi Carlos,

csanchezdll@gmail.com (Carlos Sánchez de La Lama) skribis:

> Hi Ludo, Alex,
>
>> In the example Alex gave, none of the two profiles specifies
>> GUILE_LOAD_PATH if taken individually.  This example is also in the
>> manual to illustrate the use of multiple -p flags with --search-paths
>> (info "(guix) Invoking guix package").
>
> I think I understood now. I knew "guix --search-paths" only returned
> those search paths which *do exist* in the current profile (thus the
> need for multiple -p parameters), but I (wrongly) thought search paths
> were added unconditionally to <profile_root>/etc/profile.
>
> Making this work would need two things:
> 1) not stripping unexisting directories from search paths
> 2) constructing environment variables prepending every profile root to
>    the search paths of each profile.
>
> And my suggestion only solved point 2.
>
> Now, my question is, why are unexisting search paths removed?

I don’t think this is really the problem here.  The problem here is that
if a profile contains only, say, guile-json, there is no way Guix can
know about GUILE_LOAD_PATH because that variable is associated with
Guile itself, not with guile-json.

Does that make sense?

Thanks,
Ludo’.

Re: User-profile search paths should include system-profile directories

[Carlos Sánchez de La Lama]

Hi Ludo,

>> Making this work would need two things:
>> 1) not stripping unexisting directories from search paths
>> 2) constructing environment variables prepending every profile root to
>>    the search paths of each profile.
>>
>> Now, my question is, why are unexisting search paths removed?
>
> I don’t think this is really the problem here.  The problem here is that
> if a profile contains only, say, guile-json, there is no way Guix can
> know about GUILE_LOAD_PATH because that variable is associated with
> Guile itself, not with guile-json.

The idea is each profile environment unconditionally adds search paths
for all the profiles in use (which are passed when evaluating
<profile>/etc/profile in the GUIX_PROFILES variable). This was the
proposal in my first message (looks *similar* to an earlier post in the
bug discussion, but it is quite different).

In the case you suggest, the profile cointaining guile will define
GUILE_LOAD_PATH to point inside its own profile and *also* inside the
profile containing guile-json.

The profile that has guile-json will do nothing, as its installer
packages have no search paths.

This is the key:

export ACLOCAL_PATH="${profiles//:/\/share/aclocal:}/share/aclocal

So ACLOCAL_PATH will have an entry for <each profile>/share/aclocal,
regardless of that path existing or not. If path is not there, no harm
done. If it is (i.e. that profile has a package installing an aclocal
.m4 file) then aclocal will find it.

I will try to prepare a patch, so we can see (and might be there is some
problem I still fail to see).

BR

Carlos

Re: User-profile search paths should include system-profile directories

[Ludovic Courtès]

Howdy,

csanchezdll@gmail.com (Carlos Sánchez de La Lama) skribis:

>>> Making this work would need two things:
>>> 1) not stripping unexisting directories from search paths
>>> 2) constructing environment variables prepending every profile root to
>>>    the search paths of each profile.
>>>
>>> Now, my question is, why are unexisting search paths removed?
>>
>> I don’t think this is really the problem here.  The problem here is that
>> if a profile contains only, say, guile-json, there is no way Guix can
>> know about GUILE_LOAD_PATH because that variable is associated with
>> Guile itself, not with guile-json.
>
> The idea is each profile environment unconditionally adds search paths
> for all the profiles in use (which are passed when evaluating
> <profile>/etc/profile in the GUIX_PROFILES variable). This was the
> proposal in my first message (looks *similar* to an earlier post in the
> bug discussion, but it is quite different).
>
> In the case you suggest, the profile cointaining guile will define
> GUILE_LOAD_PATH to point inside its own profile and *also* inside the
> profile containing guile-json.

The problem is that, as things currently are, the profile that contains
only Guile has no reason to define GUILE_LOAD_PATH.  So I’m unsure if
this can work as is.  We’ll see!

Thanks for looking into it,
Ludo’.