Re: 'guix build [P]' followed by 'guix install /gnu/store/[...]' vs. 'guix install [P]'

Re: 'guix build [P]' followed by 'guix install /gnu/store/[...]' vs. 'guix install [P]'

[Thomas Schwinge]

Hi!

A few days ago, I had posted this to <help-guix@gnu.org>, but not yet
gotten any response -- I understand everyone's busy, of course ;-) -- and
this morning had a quick chat on Guix IRC (see below), but as that also
wasn't really conclusive, I'd like to re-post on <guix-devel@gnu.org>:

On 2024-11-03T18:10:32+0100, I wrote:
> I was under the impression that 'guix build [P]' followed by
> 'guix install /gnu/store/[...]' would produce equivalent results to
> 'guix install [P]' -- but evidently that's not generally the case?  With
> up-to-date Guix:
>
>     $ guix build gcc-toolchain@4.8.5
>     [...]
>     /gnu/store/zq67w51hf6vpk3s2nriwnl7658biq9dz-gcc-toolchain-4.8.5-debug
>     /gnu/store/lahbqdidl3ynasd0vzxz2i0dmgh0v16i-gcc-toolchain-4.8.5
>     /gnu/store/82i6qfdqspg43rkphw0hhafny76z5bbr-gcc-toolchain-4.8.5-static
>     $ guix install -p bi /gnu/store/lahbqdidl3ynasd0vzxz2i0dmgh0v16i-gcc-toolchain-4.8.5
>     [...]
>
> ..., where '/gnu/store/[...]-gcc-toolchain-4.8.5' is the main ("out")
> output, which should -- per my understanding -- correspond to directly
> 'guix install'ing:
>
>     $ guix install -p i gcc-toolchain@4.8.5
>     [...]
>
> But now compare the two installations:
>
>     $ diff -ru bi/ i/
>     diff -ru bi/etc/profile i/etc/profile
>     --- bi/etc/profile      1970-01-01 01:00:01.000000000 +0100
>     +++ i/etc/profile       1970-01-01 01:00:01.000000000 +0100
>     @@ -8,4 +8,10 @@
>      # When GUIX_PROFILE is undefined, the various environment variables refer
>      # to this specific profile generation.
>      
>     -export PATH="${GUIX_PROFILE:-/gnu/store/fh258i84wjshhaxnv4bb2qm6xipfxsnl-profile}/bin:${GUIX_PROFILE:-/gnu/store/fh258i84wjshhaxnv4bb2qm6xipfxsnl-profile}/sbin${PATH:+:}$PATH"
>     +export PATH="${GUIX_PROFILE:-/gnu/store/2vk4q0ffg4621pz4jd9pprscpm9dfiwf-profile}/bin:${GUIX_PROFILE:-/gnu/store/2vk4q0ffg4621pz4jd9pprscpm9dfiwf-profile}/sbin${PATH:+:}$PATH"
>     +export GUIX_LOCPATH="${GUIX_PROFILE:-/gnu/store/2vk4q0ffg4621pz4jd9pprscpm9dfiwf-profile}/lib/locale${GUIX_LOCPATH:+:}$GUIX_LOCPATH"
>     +export LIBRARY_PATH="${GUIX_PROFILE:-/gnu/store/2vk4q0ffg4621pz4jd9pprscpm9dfiwf-profile}/lib${LIBRARY_PATH:+:}$LIBRARY_PATH"
>     +export OBJCPLUS_INCLUDE_PATH="${GUIX_PROFILE:-/gnu/store/2vk4q0ffg4621pz4jd9pprscpm9dfiwf-profile}/include/c++:${GUIX_PROFILE:-/gnu/store/2vk4q0ffg4621pz4jd9pprscpm9dfiwf-profile}/include${OBJCPLUS_INCLUDE_PATH:+:}$OBJCPLUS_INCLUDE_PATH"
>     +export OBJC_INCLUDE_PATH="${GUIX_PROFILE:-/gnu/store/2vk4q0ffg4621pz4jd9pprscpm9dfiwf-profile}/include${OBJC_INCLUDE_PATH:+:}$OBJC_INCLUDE_PATH"
>     +export CPLUS_INCLUDE_PATH="${GUIX_PROFILE:-/gnu/store/2vk4q0ffg4621pz4jd9pprscpm9dfiwf-profile}/include/c++:${GUIX_PROFILE:-/gnu/store/2vk4q0ffg4621pz4jd9pprscpm9dfiwf-profile}/include${CPLUS_INCLUDE_PATH:+:}$CPLUS_INCLUDE_PATH"
>     +export C_INCLUDE_PATH="${GUIX_PROFILE:-/gnu/store/2vk4q0ffg4621pz4jd9pprscpm9dfiwf-profile}/include${C_INCLUDE_PATH:+:}$C_INCLUDE_PATH"
>     diff -ru bi/manifest i/manifest
>     --- bi/manifest 1970-01-01 01:00:01.000000000 +0100
>     +++ i/manifest  1970-01-01 01:00:01.000000000 +0100
>     @@ -9,4 +9,40 @@
>          (("gcc-toolchain"
>            "4.8.5"
>            "out"
>     -      "/gnu/store/lahbqdidl3ynasd0vzxz2i0dmgh0v16i-gcc-toolchain-4.8.5"))))
>     +      "/gnu/store/lahbqdidl3ynasd0vzxz2i0dmgh0v16i-gcc-toolchain-4.8.5"
>     +      (search-paths
>     +        (("C_INCLUDE_PATH" ("include") ":" directory #f)
>     +         ("CPLUS_INCLUDE_PATH"
>     +          ("include/c++" "include")
>     +          ":"
>     +          directory
>     +          #f)
>     +         ("OBJC_INCLUDE_PATH"
>     +          ("include")
>     +          ":"
>     +          directory
>     +          #f)
>     +         ("OBJCPLUS_INCLUDE_PATH"
>     +          ("include/c++" "include")
>     +          ":"
>     +          directory
>     +          #f)
>     +         ("LIBRARY_PATH" ("lib" "lib64") ":" directory #f)
>     +         ("GUIX_LOCPATH" ("lib/locale") ":" directory #f)
>     +         ("TZDIR" ("share/zoneinfo") #f directory #f)))
>     +      (properties
>     +        ((provenance
>     +           (repository
>     +             (version 0)
>     +             (url "https://git.savannah.gnu.org/git/guix.git")
>     +             (branch "master")
>     +             (commit
>     +               "8964dfdb84f7d21dbc89c217ca4f4546a15990af")
>     +             (name guix)
>     +             (introduction
>     +               (channel-introduction
>     +                 (version 0)
>     +                 (commit
>     +                   "9edb3f66fd807b096b48283debdcddccfea34bad")
>     +                 (signer
>     +                   "BBB0 2DDF 2CEA F6A8 0D1D  E643 A2A0 6DF2 A33A 54FA")))))))))))
>
> This means that the 'bi' installation isn't usable.
>
> Where is the error in (very likely) my thinking?

Same behavior, by the way, for more recent versions of 'gcc-toolchain',
including current 'gcc-toolchain', so it's not a problem specific to
ancient 'gcc-toolchain@4.8.5'.

Guix IRC, 2024-11-07:

    [...]
    11:15:47 <tschwinge> Anybody got any opinion on
      <https://lists.gnu.org/archive/html/help-guix/2024-11/msg00017.html>
      "'guix build [P]' followed by 'guix install /gnu/store/[...]'
      vs. 'guix install [P]'"?
    11:16:56 <Rutherther> tschwinge: hi. I mean I would expect that. The
      search paths are stored with the definition of the package. You
      don't know the package definition from the store path. So the
      information about search paths is lost. I am mainly quite surprised
      guix even allows guix install directly on gnu store paths
    [...]
    11:28:43 <tschwinge> Rutherther: Interesting, thanks for looking into
      this.
    11:29:32 <tschwinge> So, what then is the right way of doing 'guix
      install --system=i686-linux gcc-toolchain@4.8.5' on a x86_64-linux
      system?  ('guix install' doesn't support '--system=[...]'.)
    11:34:07 <futurile> I do that all the time, so it might be a bug.
    11:35:03 <yelninei> tschwinge: Does it work with guix shell (for a
      workaround)?
    [...]
    11:50:42 <tschwinge> futurile: What is "that" in your sentence?
    11:50:46 <tschwinge> yelninei: Yes, 'guix shell --system=i686-linux
      [...]' works as expected -- but I'll have to think how to fiddle
      that into my non-interactive workflow/build scripts.
    11:51:35 <tschwinge> ..., or should I "just" try to make
      '--system=[...]' work for 'guix install'/'guix package'?  ;-)
    11:52:00 <Rutherther> tschwinge: ideally you could package the build
      scripts so they just use correct dependencies. Or you an also give
      them shebang with guix shell to provide the dependencies
    [...]
    11:58:45 <yelninei> tschwinge: You can pass a command to run in the
      new shell after a -- which should also work non-interactively
    [...]

So, the workarounds aside, what is the expected behavior?


Further on:

> How do I, by the way, programmatically get from the 'guix build' list of
> (here: three) outputs to the main ("out") output?  Via
> '/gnu/store/*-gcc-toolchain-4.8.5.drv' (as produced by
> 'guix build --derivations [...]'), I suppose, but what's the standard
> way?  Or, is there even a way to instruct 'guix build' to only produce
> the main ("out") output, for example?
>
>
> All this came up in context of wanting to install '--system=i686-linux'
> packages on '--system=x86_64-linux', and I'm not able to just
> 'guix install --system=i686-linux gcc-toolchain@4.8.5' there.
>
> (Is there a fundamental reason for not allowing that, or just not yet
> implemented?)


Grüße
 Thomas

Re: 'guix build [P]' followed by 'guix install /gnu/store/[...]' vs. 'guix install [P]'

[Rutherther]

Hello,

> How do I, by the way, programmatically get from the 'guix build' list of
> (here: three) outputs to the main ("out") output?  Via

the outputs work something like this: when the package is being built
the builder scripts build everything. At the end, something
is copied to out output, something to debug output, and something to
static output.

So when you are building the package locally, you cannot just tell guix
build to produce only one of those outputs. Because the phases are
already coded in a way to produce all of them.

If you really wanted to not build them, you would have to change
the package definition itself. Specifically with gcc-toolchain it
would be quite easy as it's only sort of a wrapper package that
builds together gcc and libc. The static/debug outputs
are actually just copied outputs of those.

> (Is there a fundamental reason for not allowing that, or just not yet
> implemented?)

So yeah, there is. But sometimes substitutes will give you only
one output you request. (I am not sure what the condition is,
seems that for some build systems it's possible), this of course
applies only to packages that have substitutes and only if you use
command like guix shell / guix install.

Additionally, if you are only concerned about the size it takes up in
store, you can guix gc those paths right after the build.

Regards,
Rutherther

Re: 'guix build [P]' followed by 'guix install /gnu/store/[...]' vs. 'guix install [P]'

[Ludovic Courtès]

Hi, Thomas!

Thomas Schwinge <tschwinge@baylibre.com> skribis:

>>     $ guix install -p bi /gnu/store/lahbqdidl3ynasd0vzxz2i0dmgh0v16i-gcc-toolchain-4.8.5
>>     [...]
>>
>> ..., where '/gnu/store/[...]-gcc-toolchain-4.8.5' is the main ("out")
>> output, which should -- per my understanding -- correspond to directly
>> 'guix install'ing:
>>
>>     $ guix install -p i gcc-toolchain@4.8.5
>>     [...]
>>
>> But now compare the two installations:
>>
>>     $ diff -ru bi/ i/

[...]

>>     --- bi/manifest 1970-01-01 01:00:01.000000000 +0100
>>     +++ i/manifest  1970-01-01 01:00:01.000000000 +0100
>>     @@ -9,4 +9,40 @@
>>          (("gcc-toolchain"
>>            "4.8.5"
>>            "out"
>>     -      "/gnu/store/lahbqdidl3ynasd0vzxz2i0dmgh0v16i-gcc-toolchain-4.8.5"))))
>>     +      "/gnu/store/lahbqdidl3ynasd0vzxz2i0dmgh0v16i-gcc-toolchain-4.8.5"
>>     +      (search-paths
>>     +        (("C_INCLUDE_PATH" ("include") ":" directory #f)
>>     +         ("CPLUS_INCLUDE_PATH"
>>     +          ("include/c++" "include")
>>     +          ":"
>>     +          directory
>>     +          #f)

[...]

>> This means that the 'bi' installation isn't usable.

This may sound surprising but it’s expected.

The reason is that when you run:

  guix install gcc-toolchain

‘gcc-toolchain’ is a live package with metadata that Guix uses when it
builds the profile, in particular data about search paths.

However, when you run:

  guix package -i /gnu/store/…

then all Guix sees is an inert store item with no associated metadata.
This is why it ends up creating a profile without search path info.

This is one of the reasons why I would recommend against that second
method.  It might be useful as a last resort but should be avoided as
much as possible.

(For development, I’d also recommend ‘guix shell’ over ‘guix install’!)

HTH,
Ludo’.

Re: 'guix build [P]' followed by 'guix install /gnu/store/[...]' vs. 'guix install [P]'

[Thomas Schwinge]

Hi Ludo!

On 2024-11-10T00:31:12+0100, Ludovic Courtès <ludo@gnu.org> wrote:
> Thomas Schwinge <tschwinge@baylibre.com> skribis:
>
>>>     $ guix install -p bi /gnu/store/lahbqdidl3ynasd0vzxz2i0dmgh0v16i-gcc-toolchain-4.8.5
>>>     [...]
>>>
>>> ..., where '/gnu/store/[...]-gcc-toolchain-4.8.5' is the main ("out")
>>> output, which should -- per my understanding -- correspond to directly
>>> 'guix install'ing:
>>>
>>>     $ guix install -p i gcc-toolchain@4.8.5
>>>     [...]
>>>
>>> But now compare the two installations:
>>>
>>>     $ diff -ru bi/ i/
>
> [...]
>
>>>     --- bi/manifest 1970-01-01 01:00:01.000000000 +0100
>>>     +++ i/manifest  1970-01-01 01:00:01.000000000 +0100
>>>     @@ -9,4 +9,40 @@
>>>          (("gcc-toolchain"
>>>            "4.8.5"
>>>            "out"
>>>     -      "/gnu/store/lahbqdidl3ynasd0vzxz2i0dmgh0v16i-gcc-toolchain-4.8.5"))))
>>>     +      "/gnu/store/lahbqdidl3ynasd0vzxz2i0dmgh0v16i-gcc-toolchain-4.8.5"
>>>     +      (search-paths
>>>     +        (("C_INCLUDE_PATH" ("include") ":" directory #f)
>>>     +         ("CPLUS_INCLUDE_PATH"
>>>     +          ("include/c++" "include")
>>>     +          ":"
>>>     +          directory
>>>     +          #f)
>
> [...]
>
>>> This means that the 'bi' installation isn't usable.
>
> This may sound surprising but it’s expected.
>
> The reason is that when you run:
>
>   guix install gcc-toolchain
>
> ‘gcc-toolchain’ is a live package with metadata that Guix uses when it
> builds the profile, in particular data about search paths.
>
> However, when you run:
>
>   guix package -i /gnu/store/…
>
> then all Guix sees is an inert store item with no associated metadata.
> This is why it ends up creating a profile without search path info.

Hmm, I see.  That also matches what Rutherther had written on IRC.
However, conceptually, I don't understand the rationale for doing it this
way.  Isn't this Guix use of 'C_INCLUDE_PATH' etc. intimately specific to
the way how GCC has to be built/used in the Guix context?  Therefore, I'd
expect such setup code to be an artifact (meta data) of 'guix build',
that is, written into '/gnu/store/[...]-gcc-toolchain-4.8.5/etc/profile'
at 'guix build' time, instead of into the '[Guix profile]/etc/profile'
created by 'guix install', and 'guix install' would then just use that
file (like, add to the '[Guix profile]/etc/profile' a shell
'source /gnu/store/[...]-gcc-toolchain-4.8.5/etc/profile').

But, of course, I'm still very new with learning Guix concepts and
implementation, so...  ;-|

> This is one of the reasons why I would recommend against that second
> method.  It might be useful as a last resort but should be avoided as
> much as possible.

Hmm.  But is it really a good idea then at all, to offer this way of
'guix install'ing packages, if it can't be expected to work reliably?


What is then the proper Guix way to achieve the following:

| All this came up in context of wanting to install '--system=i686-linux'
| packages on '--system=x86_64-linux', and I'm not able to just
| 'guix install --system=i686-linux gcc-toolchain@4.8.5' there.
|
| (Is there a fundamental reason for not allowing that, or just not yet
| implemented?)


> (For development, I’d also recommend ‘guix shell’ over ‘guix install’!)

That's also what yelninei suggested on IRC.  I'm aware of 'guix shell',
and 'guix shell --system=i686-linux [...]' does work as expected, but
that's not feasible as an interactive shell if you'd like to use Guix GCC
to build upstream GCC: see <https://debbugs.gnu.org/72669>
"gcc-toolchain environment variables", for example.  I could have wrapper
'gcc' etc. scripts that internally do:

    guix shell --system=i686-linux gcc-toolchain@4.8.5 -- gcc "$@"

..., but I'm not sure about the overhead of all those hundreds of
'guix shell [...]' invocations?


I'm happy to continue the discussion; hopefully with my GCC/toolchain
experience I can eventually make a useful contribution to Guix here?
(Unless, of course, I get convinced that the status quo is already the
right way...)  ;-)


Grüße
 Thomas

Re: 'guix build [P]' followed by 'guix install /gnu/store/[...]' vs. 'guix install [P]'

[Ludovic Courtès]

Hello!

Thomas Schwinge <tschwinge@baylibre.com> skribis:

>> The reason is that when you run:
>>
>>   guix install gcc-toolchain
>>
>> ‘gcc-toolchain’ is a live package with metadata that Guix uses when it
>> builds the profile, in particular data about search paths.
>>
>> However, when you run:
>>
>>   guix package -i /gnu/store/…
>>
>> then all Guix sees is an inert store item with no associated metadata.
>> This is why it ends up creating a profile without search path info.
>
> Hmm, I see.  That also matches what Rutherther had written on IRC.
> However, conceptually, I don't understand the rationale for doing it this
> way.  Isn't this Guix use of 'C_INCLUDE_PATH' etc. intimately specific to
> the way how GCC has to be built/used in the Guix context?  Therefore, I'd
> expect such setup code to be an artifact (meta data) of 'guix build',
> that is, written into '/gnu/store/[...]-gcc-toolchain-4.8.5/etc/profile'
> at 'guix build' time, instead of into the '[Guix profile]/etc/profile'
> created by 'guix install', and 'guix install' would then just use that
> file (like, add to the '[Guix profile]/etc/profile' a shell
> 'source /gnu/store/[...]-gcc-toolchain-4.8.5/etc/profile').

Yeah, I understand this can be confusing; it has to do with metadata
only existing in package definitions and not in the build output in
/gnu/store.

>> This is one of the reasons why I would recommend against that second
>> method.  It might be useful as a last resort but should be avoided as
>> much as possible.
>
> Hmm.  But is it really a good idea then at all, to offer this way of
> 'guix install'ing packages, if it can't be expected to work reliably?

It can be helpful in “rescue mode” so to speak, but honestly, I haven’t
used it in years and I thought nobody knew about it.  :-)

Maybe we should document it less prominently and/or add warnings against
it, at least.

> What is then the proper Guix way to achieve the following:
>
> | All this came up in context of wanting to install '--system=i686-linux'
> | packages on '--system=x86_64-linux', and I'm not able to just
> | 'guix install --system=i686-linux gcc-toolchain@4.8.5' there.
> |
> | (Is there a fundamental reason for not allowing that, or just not yet
> | implemented?)
>
>
>> (For development, I’d also recommend ‘guix shell’ over ‘guix install’!)
>
> That's also what yelninei suggested on IRC.  I'm aware of 'guix shell',
> and 'guix shell --system=i686-linux [...]' does work as expected, but
> that's not feasible as an interactive shell if you'd like to use Guix GCC
> to build upstream GCC: see <https://debbugs.gnu.org/72669>
> "gcc-toolchain environment variables", for example.  I could have wrapper
> 'gcc' etc. scripts that internally do:
>
>     guix shell --system=i686-linux gcc-toolchain@4.8.5 -- gcc "$@"
>
> ..., but I'm not sure about the overhead of all those hundreds of
> 'guix shell [...]' invocations?

‘guix shell’ maintains a cache, which means that subsequent invocations
(cache hit) runs in ~100 ms:

--8<---------------cut here---------------start------------->8---
$ time guix shell -s i686-linux gcc-toolchain@4.8.5 -- gcc --version
gcc (GCC) 4.8.5
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


real    0m0.124s
user    0m0.131s
sys     0m0.025s
--8<---------------cut here---------------end--------------->8---

The overhead is small, but it might still be noticeable if you’re going
to run it hundreds of times.

Ideally, you would set up the entire development environment for GCC
with just a single ‘guix shell’ invocation and then go from here.  I
haven’t looked at the bug report you mention above and how that prevents
it from being practical, though.

HTH,
Ludo’.