bug#63267: gcc-toolchain is missing libstdc++.so

bug#63267: gcc-toolchain is missing libstdc++.so

[John Kehayias via Bug reports for GNU Guix]

Hi Christopher,

On Thu, May 04, 2023 at 11:05 AM, Christopher Rodriguez wrote:
>
> Sorry for the spam; Resending this without the bugs address, but with
> the issue's address.
>
> Christopher Rodriguez <yewscion@gmail.com> writes:
>
>>
>> Hello All,
>>
>> I noticed today that libstdc++.so.1 (and some others), which used to be
>> part of gcc:lib, are not included in any of the outputs of the
>> superceding `gcc-toolchain` package.
>>
>> Is there another method for getting these needed shared libraries in a
>> guix system at this point? It's entirely possible I'm missing something.
>>
>> I am CCing guix-devel@gnu.org per podiki[m]'s request.
>>
>> Thanks!

Thanks for opening this and cc'ing; this has come up with some
frequency on IRC, especially recently. In discussing there today, the
current reasoning is that usually one will just call g++ which knows
how to find libstdc++. So, gcc-toolchain does not include gcc:lib as
part of what it makes available.

I think what we (and when this comes up, others) are getting at are
some edge cases or different use cases where one wants to directly get
at libstdc++. Previously it was more direct to use gcc:lib; of course
one still can in code and/or cli with the proper call. For example,
guix build -e "(@@ (gnu packages gcc) gcc)" will download/build/show
the lib output of the (hidden) gcc package. Though I'm not sure how to
select just the lib output here.

My use case currently is in the FHS container where a binary wants to
find some libraries directly. Previously one would include the gcc:lib
package output in the guix shell call. Now some of those libraries can
be found elsewhere, like libgccjit, but libstdc++ seems to be the
trickier one. Open to other suggestions/workarounds, or thoughts on if
it is worthwhile to include gcc:lib in the gcc-toolchain package (or
make a gcc-toolchain:lib output?).

Thanks all!
John

bug#63267: gcc-toolchain is missing libstdc++.so

[John Kehayias via Bug reports for GNU Guix]

Hi again,

On Thu, May 04, 2023 at 11:19 AM, John Kehayias wrote:

> Thanks for opening this and cc'ing; this has come up with some
> frequency on IRC, especially recently. In discussing there today, the
> current reasoning is that usually one will just call g++ which knows
> how to find libstdc++. So, gcc-toolchain does not include gcc:lib as
> part of what it makes available.
>

I tried locally just adding gcc:lib as an input for gcc-toolchain and
that does the trick. And since it is just a union-build, very quick to
try out :)

guix size reports an increase in gcc-toolchain as 0.1 MiB with gcc:lib
included.

> I think what we (and when this comes up, others) are getting at are
> some edge cases or different use cases where one wants to directly get
> at libstdc++. Previously it was more direct to use gcc:lib; of course
> one still can in code and/or cli with the proper call. For example,
> guix build -e "(@@ (gnu packages gcc) gcc)" will download/build/show
> the lib output of the (hidden) gcc package. Though I'm not sure how to
> select just the lib output here.
>
> My use case currently is in the FHS container where a binary wants to
> find some libraries directly. Previously one would include the gcc:lib
> package output in the guix shell call. Now some of those libraries can
> be found elsewhere, like libgccjit, but libstdc++ seems to be the
> trickier one. Open to other suggestions/workarounds, or thoughts on if
> it is worthwhile to include gcc:lib in the gcc-toolchain package (or
> make a gcc-toolchain:lib output?).
>

I tried with my local gcc-toolchain modification and this gets me what
I wanted.

On that note, I forgot to bring up the problem I had with using
make-libstdc++: it does not seem to build the same libstdc++ as
included in the gcc package. The doc string for that procedure notes
that this is meant to be used when using non-gcc toolchains, but we
also have the libstdc++ variable which seems to suggest that
(make-libstdc++ gcc) should be the same library as in gcc.

I'm not sure the difference in looking at the package definitions, but
I don't really know this stuff. Here's an example of the difference I
was finding:

I was running something and it complained that

--8<---------------cut here---------------start------------->8---
<some-binary> symbol lookup error: <some-binary>: undefined symbol: _ZNSt18condition_variableD1Ev, version GLIBCXX_3.4.11
--8<---------------cut here---------------end--------------->8---

Indeed, looking at the libstdc++ I used via (or could have used
libstdc++ here directly since I used the default gcc):

--8<---------------cut here---------------start------------->8---
guix shell -e "(begin (use-modules (gnu packages gcc)) (make-libstdc++ gcc))"
--8<---------------cut here---------------end--------------->8---

I see

--8<---------------cut here---------------start------------->8---
$strings /gnu/store/6897bpw5858bdng744ddqw8rrqjb4frr-libstdc++-11.3.0/lib/libstdc++.so | grep "_ZNSt18condition_variableD1Ev"
--8<---------------cut here---------------end--------------->8---

while for gcc:lib it is defined

--8<---------------cut here---------------start------------->8---
$ strings /gnu/store/l684qgqlrqkbsh8jffp9d8ag6vrpcwgs-gcc-11.3.0-lib/lib/libstdc++.so | grep "_ZNSt18condition_variableD1Ev"
_ZNSt18condition_variableD1Ev
--8<---------------cut here---------------end--------------->8---

and using that libstdc++ does not result in that error.

Is make-libstdc++ not meant to be used/mixed with e.g. gcc-toolchain?
Is this expected that it is a different library produced or is this a
bug?

Thanks!
John

Re: gcc-toolchain is missing libstdc++.so

[John Kehayias]

Hi all,

> I have similar use cases of FHS containers to run binaries (primarily
> games). I recently ran into the issue of gcc:lib going away and no
> output from a visible package providing libstdc++. My current
> workaround was to implement a replacement for specifications->manifest
> that could handle packages and '(package "output") pairs in addition
> to strings, so that I could include `(,(@@ (gnu packages gcc) gcc)
> "lib") in place of "gcc:lib". Internally it resolves package strings
> to packages with specification->package, then passes the package and
> optional output specifier to package->manifest-entry. But I digress a
> little...

Nice little hack Kaelyn, would you mind sharing somewhere? I wonder if
this should be something we should have more easily anyway.

On Thu, May 04, 2023 at 12:14 PM, Katherine Cox-Buday wrote:

> On 5/4/23 11:33 AM, Kaelyn wrote:
>
>> Regarding solutions, I would prefer to have libstdc++ in it's own
>> package or output rather than bundled into gcc-toolchain:out; it
>> feels messy and against the grain of isolating programs in
>> containers if I have to make the gcc and g++ compilers available in
>> the container in order to run a program that needs libstdc++.
>
> +1. I recently ran into this as well and went looking for it.
>
> I think a good reason to give libstdc++ its own output is that this
> question continually gets asked.

That sounds reasonable to me as well. I would think the make-libstdc++
procedure would work for this, but as I detailed in my other message,
I'm not sure why it seems to be missing symbols. We would have just
what we need there and could just expose some public package versions
through that or leave it similar to how it is and document (so it is
more of an advanced or edge case scenario and not have more people
going that way when what they really want is the actual gcc-toolchain
package).

John

Re: gcc-toolchain is missing libstdc++.so

[Kaelyn]

[-- Attachment #1: Type: text/plain, Size: 1547 bytes --]

------- Original Message -------
On Thursday, May 4th, 2023 at 9:50 PM, John Kehayias <john.kehayias@protonmail.com> wrote:

> > I have similar use cases of FHS containers to run binaries (primarily
> > games). I recently ran into the issue of gcc:lib going away and no
> > output from a visible package providing libstdc++. My current
> > workaround was to implement a replacement for specifications->manifest
> > that could handle packages and '(package "output") pairs in addition
> > to strings, so that I could include `(,(@@ (gnu packages gcc) gcc)
> > "lib") in place of "gcc:lib". Internally it resolves package strings
> > to packages with specification->package, then passes the package and
> > optional output specifier to package->manifest-entry. But I digress a
> > little...
> 
> 
> Nice little hack Kaelyn, would you mind sharing somewhere? I wonder if
> this should be something we should have more easily anyway.

I wasn't sure the best place to share it, so I've attached my "run" script for running the binary download of PolyMC in a container. It is both a shell script and a guix package manifest, and is the one place so far I've worked around the removal of gcc:lib. The main program-specific bits are what CMD defaults to and which packages need to be included (most of the various shares are to get things like hardware 3D, pulseaudio, and dbus working and aren't all always needed). It also contains the original manifest commented-out for comparison. Hope it can be of help to folks!

Cheers,
Kaelyn

[-- Attachment #2: run --]
[-- Type: application/octet-stream, Size: 1873 bytes --]

#!/bin/sh
: ${CMD:=./PolyMC}
if [ $# -gt 0 ] ; then
    CMD="$@"
fi

test -d home/ || mkdir home
set -ex

exec guix shell -C -F -N -E ^DISPLAY --share="$PWD/home=$HOME" --share="$PWD" \
     --expose=/dev/snd --expose=/etc/asound.conf --expose=/etc/machine-id \
     --expose=/dev/dri --expose=/dev/input --expose=/sys/class/input \
     --expose=/sys/dev --expose=/sys/devices --expose=/var/run/dbus \
     --share=/dev/shm --share=$HOME/.config/pulse --share=$HOME/.Xauthority \
     --share=/run/user/$UID/pulse --share=/run/user/$UID/pulse=/var/run/pulse \
     -m "$0" --pure -- env LD_LIBRARY_PATH=/lib ${CMD}
!#

(use-modules (gnu))

;; (specifications->manifest
;;  '("bash" ; always required
;;    "coreutils" ; always required
;;    "glibc" ; always required

;;    "gcc:lib"
;;    "qtbase@5.15"
;;    "qtcharts@5.15"
;;    "zlib"
;;    "glfw"
;;    "nss-certs"
;;    "openal"
;;    "openjdk@18"
;;    "pulseaudio"
;;    "unrar"
;;    "unzip"
;;    "zip"))

(manifest
 (map (lambda (spec)
        (apply package->manifest-entry
               (cond
                ((pair? spec) (let ((pkg (car spec))
                                    (output (cadr spec)))
                                (list
                                 (if (string? pkg)
                                     (specification->package pkg)
                                     pkg)
                                 output)))
                ((string? spec) (list (specification->package spec)))
                (else (list spec)))))
      `("bash" ; always required
        "coreutils" ; always required
        "glibc" ; always required

        (,(@@ (gnu packages gcc) gcc) "lib")
        "qtbase@5.15"
        "qtcharts@5.15"
        "zlib"
        "glfw"
        "nss-certs"
        "openal"
        "openjdk@18"
        "pulseaudio"
        "unrar"
        "unzip"
        "zip")))

Re: gcc-toolchain is missing libstdc++.so

[John Kehayias]

Hi Kaelyn,

On Thu, May 04, 2023 at 11:45 PM, Kaelyn wrote:

> ------- Original Message -------
> I wasn't sure the best place to share it, so I've attached my "run"
> script for running the binary download of PolyMC in a container. It is
> both a shell script and a guix package manifest, and is the one place
> so far I've worked around the removal of gcc:lib. The main
> program-specific bits are what CMD defaults to and which packages need
> to be included (most of the various shares are to get things like
> hardware 3D, pulseaudio, and dbus working and aren't all always
> needed). It also contains the original manifest commented-out for
> comparison. Hope it can be of help to folks!
>

Thanks, that's a nice little hack! Just something very minor I
noticed, but you don't need to specify glibc directly for the -F (FHS)
option in guix shell, as it will automatically include the (modified)
glibc.

This topic came up again on IRC today and GNUtoo had the correct cli
invocation for getting at gcc:lib. I thought I had tried something
similar but must have missed something, or else didn't notice that
this will only work for guix shell, as guix build doesn't take outputs
in the package list:

--8<---------------cut here---------------start------------->8---
guix shell  -e $'(list (@@ (gnu packages gcc) gcc) "lib")'
--8<---------------cut here---------------end--------------->8---

Thanks to both of you I have some options for workarounds currently,
but based on how this topic keeps coming up I still think we should
have a more straightforward option.

John

Re: gcc-toolchain is missing libstdc++.so

[Kaelyn]

------- Original Message -------
On Friday, May 5th, 2023 at 8:59 PM, John Kehayias <john.kehayias@protonmail.com> wrote:


> 
> 
> Hi Kaelyn,
> 
> On Thu, May 04, 2023 at 11:45 PM, Kaelyn wrote:
> 
> > ------- Original Message -------
> > I wasn't sure the best place to share it, so I've attached my "run"
> > script for running the binary download of PolyMC in a container. It is
> > both a shell script and a guix package manifest, and is the one place
> > so far I've worked around the removal of gcc:lib. The main
> > program-specific bits are what CMD defaults to and which packages need
> > to be included (most of the various shares are to get things like
> > hardware 3D, pulseaudio, and dbus working and aren't all always
> > needed). It also contains the original manifest commented-out for
> > comparison. Hope it can be of help to folks!
> 
> 
> Thanks, that's a nice little hack! Just something very minor I
> noticed, but you don't need to specify glibc directly for the -F (FHS)
> option in guix shell, as it will automatically include the (modified)
> glibc.

Thanks! A small side note: I have glibc in there mainly so ldd is available for debugging problems or getting new binaries working (I think the comment with it is a remnant of an older version of the manifest from before "-F" was added to "guix shell").

Cheers,
Kaelyn

bug#63267: gcc-toolchain is missing libstdc++.so

[Josselin Poiret via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 615 bytes --]

Hi Kaelyn,

Kaelyn via Bug reports for GNU Guix <bug-guix@gnu.org> writes:

> Thanks! A small side note: I have glibc in there mainly so ldd is available for debugging problems or getting new binaries working (I think the comment with it is a remnant of an older version of the manifest from before "-F" was added to "guix shell").

Small note: `ldd` is only a wrapper around setting
`LD_TRACE_LOADED_OBJECTS=1`, so you don't really need to pull in all of
glibc just for this.  There's also LD_DEBUG with possible values
explained by LD_DEBUG=help, which I use quite often.

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

Re: bug#63267: gcc-toolchain is missing libstdc++.so

[Simon Tournier]

Hi,

I am proposing patch#63393 [1] which adds the output lib to
gcc-toolchain.  Well, quoting the comment:

      ;; The main raison d'être of this "meta-package" is (1) to conveniently
      ;; install everything that we need, and (2) to make sure ld-wrapper comes
      ;; before Binutils' ld in the user's profile.

I think, this package gcc-toolchain should be the visible package and
battery included.

WDYT?

1: https://issues.guix.gnu.org/issue/63393

Cheers,
simon

bug#63267: gcc-toolchain is missing libstdc++.so

[John Kehayias via Bug reports for GNU Guix]

Hi everyone,

Apologies for the long delay on this.

On Tue, May 09, 2023 at 07:07 PM, Simon Tournier wrote:

> Hi,
>
> I am proposing patch#63393 [1] which adds the output lib to
> gcc-toolchain.  Well, quoting the comment:
>
>       ;; The main raison d'être of this "meta-package" is (1) to conveniently
>       ;; install everything that we need, and (2) to make sure ld-wrapper comes
>       ;; before Binutils' ld in the user's profile.
>
> I think, this package gcc-toolchain should be the visible package and
> battery included.
>
> WDYT?
>
> 1: https://issues.guix.gnu.org/issue/63393
>
> Cheers,
> simon

I've just pushed, as b47ae1ecc43baaf726701ab2d2f810ecfaa75428, the
patches from that issue (the first from simon, the second my simplified
one just adding gcc:lib to gcc-toolchain).

This should fix the original bug here, so I am closing. However, it was
raised here and in 63393 about alternatives in how we use gcc-toolchain
outputs. As well as the issue I ran into about make-libstdc++.

So, if anyone would like to change anything from the new status quo,
please open a new issue. At least now we are working from a better
default I would say, with gcc-toolchain including the libraries from
gcc:lib.

Thanks all,
John

Re: bug#63267: gcc-toolchain is missing libstdc++.so

[Simon Tournier]

Hi,

On mer., 17 avril 2024 at 05:21, John Kehayias via Bug reports for GNU Guix <bug-guix@gnu.org> wrote:

> I've just pushed, as b47ae1ecc43baaf726701ab2d2f810ecfaa75428,

Cool!  Thank you for crossing the finish line.

Cheers,
simon