gold linker and collect2: fatal error: cannot find 'ld'

gold linker and collect2: fatal error: cannot find 'ld'

[Pjotr Prins]

Hi all,

I am facing some issues with the gold linker. Dlang's ldc compiler
recently defaulted to gold as a linker. Ironically if I build ldc
itself with binutils-gold it does not find the ld binary. If I don't add
binutils-gold ldc builds fine but the tests fail to find the gold
linker. 

Nice one.

When I build ldc without tests (successfully) the created compiler
environment requires gold. At this point ldc won't honour the --linker
switch where you can provide an alternative (i think this is an ldc
bug - no matter because I think using gold is a great idea). Anyway,
pulling in binutils-gold compiles my software fine if you set
LD_LIBRARY_PATH to LIBRARY_PATH.  This is because the latter is not
honoured by gold.

Note that gold has a compatibility layer for ld, but it does not pick
up environment variables (on purpose).

Unfortunately the runtime of compiled software fails because the rpath is
not updated either. So that requires adding in the RPATH explicitly on
the ldc command line. When I set the RPATH the runtime is fine. 

I am posting this here so others can benefit when they run into
similar issues later. Maybe the solution is to create an ld-wrapper
style gold-wrapper for gold.

Wdyt?

Also I need to somehow resolve the ldc tests running on gold. Can we
pull in standard ld and gold at the same time? binutils-gold suggests
not.

Pj.

Re: gold linker and collect2: fatal error: cannot find 'ld'

[Pjotr Prins]

On Sun, Mar 17, 2019 at 05:06:16PM +0100, Pjotr Prins wrote:
> Unfortunately the runtime of compiled software fails because the rpath is
> not updated either. So that requires adding in the RPATH explicitly on
> the ldc command line. When I set the RPATH the runtime is fine. 

Reading up on the ld-wrapper code - that is exactly what happens. A
gold-wrapper can do same.

I think, like with rustc, I need to wrap the ldc compiler build with
ld-wrapper and the ldc tests with gold-wrapper (yet to be written).

gold does not honour LIBRARY_PATH so that means we'll have to use
LD_LIBRARY_PATH or pass the lib path(s) on the command line. 

Does this make sense? If it does we can make an attempt.

Pj.

Re: gold linker and collect2: fatal error: cannot find 'ld'

[Pjotr Prins]

No one suggestions? Should we disable gold?

On Mon, Mar 18, 2019 at 04:28:49PM +0100, Pjotr Prins wrote:
> On Sun, Mar 17, 2019 at 05:06:16PM +0100, Pjotr Prins wrote:
> > Unfortunately the runtime of compiled software fails because the rpath is
> > not updated either. So that requires adding in the RPATH explicitly on
> > the ldc command line. When I set the RPATH the runtime is fine. 
> 
> Reading up on the ld-wrapper code - that is exactly what happens. A
> gold-wrapper can do same.
> 
> I think, like with rustc, I need to wrap the ldc compiler build with
> ld-wrapper and the ldc tests with gold-wrapper (yet to be written).
> 
> gold does not honour LIBRARY_PATH so that means we'll have to use
> LD_LIBRARY_PATH or pass the lib path(s) on the command line. 
> 
> Does this make sense? If it does we can make an attempt.
> 
> Pj.
> 

Re: gold linker and collect2: fatal error: cannot find 'ld'

[Ludovic Courtès]

Hi Pjotr,

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> On Sun, Mar 17, 2019 at 05:06:16PM +0100, Pjotr Prins wrote:
>> Unfortunately the runtime of compiled software fails because the rpath is
>> not updated either. So that requires adding in the RPATH explicitly on
>> the ldc command line. When I set the RPATH the runtime is fine. 
>
> Reading up on the ld-wrapper code - that is exactly what happens. A
> gold-wrapper can do same.
>
> I think, like with rustc, I need to wrap the ldc compiler build with
> ld-wrapper and the ldc tests with gold-wrapper (yet to be written).

Indeed.  We’d just need to add a parameter to ‘make-ld-wrapper’ so we
can specify the executable name, which would be “gold” instead of “ld”
in this case.

Would you like to give it a try?

> gold does not honour LIBRARY_PATH so that means we'll have to use
> LD_LIBRARY_PATH or pass the lib path(s) on the command line. 

LIBRARY_PATH is honored by gcc, which “converts” it into a list of -L
flags for the linker, so no worries here.

HTH, and sorry for the delay!

Ludo’.

Re: gold linker and collect2: fatal error: cannot find 'ld'

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 422 bytes --]

I can give this a try.

> Indeed.  We’d just need to add a parameter to ‘make-ld-wrapper’ so we
> can specify the executable name, which would be “gold” instead of “ld”
> in this case.

Just to be clear:

1. Extend make-ld-wrapper to accept a "linker" parameter.
2. Extend the GNU build system to accept a "linker" argument.

Would this be enough?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: gold linker and collect2: fatal error: cannot find 'ld'

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 275 bytes --]

I'm on it.

Changing make-ld-wrapper recompiles everything, most of it with only one
core so my computer has been hard at work for more than 12 hours
now... :)

I'll report back as soon as I have ldc working.

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: gold linker and collect2: fatal error: cannot find 'ld'

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1589 bytes --]

Alright, it finished in a bit less than 24 hours (on a Ryzen 5 with 12
threads).

It failed with

--8<---------------cut here---------------start------------->8---
...
phobos-ldc-v1.14.0/win64.mak
patching file std/socket.d
Hunk #1 FAILED at 448.
1 out of 2 hunks FAILED -- saving rejects to file std/socket.d.rej
source is under 'phobos-ldc-v1.14.0'
applying '/gnu/store/72h2qglkyvhcajiwwh2km7k46zrb7yry-ldc-disable-phobos-tests.patch'...
Backtrace:
           3 (primitive-load "/gnu/store/c6m8yq9rx0hlab7wbjl7523xhpk…")
In ice-9/eval.scm:
    619:8  2 (_ #(#<directory (guile-user) 5cf140> "phobos-ldc-v1.…"))
In srfi/srfi-1.scm:
    640:9  1 (for-each #<procedure apply-patch (a)> ("/gnu/store/72…"))
In guix/build/utils.scm:
    616:6  0 (invoke _ . _)

guix/build/utils.scm:616:6: In procedure invoke:
Throw to key `srfi-34' with args `(#<condition &invoke-error [program: "/gnu/store/frpnp3d1ggybwmfhpy7hqimmrndndxa9-patch-2.7.6/bin/patch" arguments: ("--force" "--no-backup-if-mismatch" "-p1" "--input" "/gnu/store/72h2qglkyvhcajiwwh2km7k46zrb7yry-ldc-disable-phobos-tests.patch") exit-status: 1 term-signal: #f stop-signal: #f] 56dd00>)'.
--8<---------------cut here---------------end--------------->8---

Looks like the patch is out of date.  Did it every work for you, Pjotr?

I've tried fixing it but somehow Guix is recompiling everything again.
I'll let it run for a bit, but I can't proceed if it takes 24 hours to
iterate, so I'd like to understand why it rebuilds the world every
time.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: gold linker and collect2: fatal error: cannot find 'ld'

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 3047 bytes --]

So it seems that the following change to make-ld-wrapper triggers a
rebuild of the world on every "guix build".  Any idea why?  Ludo?

--8<---------------cut here---------------start------------->8---
@@ -537,14 +537,15 @@ included.")
                           binutils
                           (guile (canonical-package guile-2.2))
                           (bash (canonical-package bash))
-                          (guile-for-build guile))
-  "Return a package called NAME that contains a wrapper for the 'ld' program
-of BINUTILS, which adds '-rpath' flags to the actual 'ld' command line.  The
-wrapper uses GUILE and BASH.
+                          (guile-for-build guile)
+                          (linker-name "ld"))
+  "Return a package called NAME that contains a wrapper for the linker program,
+('ld' of BINUTILS by default), which adds '-rpath' flags to the actual linker
+command line.  The wrapper uses GUILE and BASH.
 
 TARGET must be a one-argument procedure that, given a system type, returns a
 cross-compilation target triplet or #f.  When the result is not #f, make a
-wrapper for the cross-linker for that target, called 'TARGET-ld'."
+wrapper for the cross-linker for that target, called 'TARGET-LINKER-NAME'."
   ;; Note: #:system->target-triplet is a procedure so that the evaluation of
   ;; its result can be delayed until the 'arguments' field is evaluated, thus
   ;; in a context where '%current-system' is accurate.
@@ -569,8 +570,8 @@ wrapper for the cross-linker for that target, called 'TARGET-ld'."
                      (let* ((out (assoc-ref %outputs "out"))
                             (bin (string-append out "/bin"))
                             (ld  ,(if target
-                                      `(string-append bin "/" ,target "-ld")
-                                      '(string-append bin "/ld")))
+                                      `(string-append bin "/" ,target "-" ,linker-name)
+                                      `(string-append bin "/" ,linker-name)))
                             (go  (string-append ld ".go")))
 
                        (setvbuf (current-output-port) _IOLBF)
@@ -593,8 +594,8 @@ wrapper for the cross-linker for that target, called 'TARGET-ld'."
                           (string-append (assoc-ref %build-inputs "binutils")
                                          ,(if target
                                               (string-append "/bin/"
-                                                             target "-ld")
-                                              "/bin/ld"))))
+                                                             target "-" linker-name)
+                                              (string-append "/bin/" linker-name)))))
                        (chmod ld #o555)
                        (compile-file ld #:output-file go)
                        #t)))))

--8<---------------cut here---------------end--------------->8---

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: gold linker and collect2: fatal error: cannot find 'ld'

[Ludovic Courtès]

Hi Pierre,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> So it seems that the following change to make-ld-wrapper triggers a
> rebuild of the world on every "guix build".  Any idea why?  Ludo?

“Everything” depends on ‘ld-wrapper’ (see commencement.scm), so you have
to come up with changes that do not modify the default ‘ld-wrapper’.

>                              (ld  ,(if target
> -                                      `(string-append bin "/" ,target "-ld")
> -                                      '(string-append bin "/ld")))
> +                                      `(string-append bin "/" ,target "-" ,linker-name)
> +                                      `(string-append bin "/" ,linker-name)))

In these two cases, you’re changing the generated code from:

  (string-append bin "/" … "-ld")
  (string-append bin "/ld")

to:

  (string-append bin "/" … "-" "ld")
  (string-append bin "/" "ld")

To avoid a rebuild, you’ll have to special-case “ld” such that the
generated code is unchanged.

HTH!

Ludo’.

Re: gold linker and collect2: fatal error: cannot find 'ld'

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 83 bytes --]

That makes perfect sense, thanks!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: gold linker and collect2: fatal error: cannot find 'ld'

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1352 bytes --]

Alright, this works now.

I'm using Pjotr's patch
https://gitlab.com/genenetwork/guix/commits/ldc-1.14-20190222/gnu/packages/dlang.scm
to build ldc 1.14, but I hit the opposite issue of what was initially
described in this post:

- It does not seem to build against gold, it seems to use the regular
  binutils.
- Tests fail because they can't find 'ld'.

The first failing test is:
--8<---------------cut here---------------start------------->8---
         1552/1587 Test #1552: druntime-test-aa ...................................................................***Failed    0.49 sec
         make[1]: Entering directory '/tmp/guix-build-ldc-1.14.0.drv-0/ldc-1.14.0/runtime/druntime/test/aa'
         /tmp/guix-build-ldc-1.14.0.drv-0/build/bin/ldmd2   -w -I../../src -I../../import -Isrc -defaultlib= -debuglib= -dip1000 -L/tmp/guix-build-ldc-1.14.0.drv-0/build/lib/libdruntime-ldc.a -O -release -of/tmp/guix-build-ldc-1.14.0.drv-0/build/runtime/druntime-test-aa/test_aa src/test_aa.d
         collect2: fatal error: cannot find 'ld'
         compilation terminated.
--8<---------------cut here---------------end--------------->8---

I don't really understand how it can fail to find 'ld' considering it's
supposed to be building with binutils.

Am I missing something?  Pjotr?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: gold linker and collect2: fatal error: cannot find 'ld'

[Pjotr Prins]

The tests are looking for gold. That *is* the problem. You can build
ldc with ld, but the ldc compiler is looking for gold. When you make
gold available at test time it will work.

Pj.

On Sat, May 25, 2019 at 06:35:37PM +0200, Pierre Neidhardt wrote:
> Alright, this works now.
> 
> I'm using Pjotr's patch
> https://gitlab.com/genenetwork/guix/commits/ldc-1.14-20190222/gnu/packages/dlang.scm
> to build ldc 1.14, but I hit the opposite issue of what was initially
> described in this post:
> 
> - It does not seem to build against gold, it seems to use the regular
>   binutils.
> - Tests fail because they can't find 'ld'.
> 
> The first failing test is:
> --8<---------------cut here---------------start------------->8---
>          1552/1587 Test #1552: druntime-test-aa ...................................................................***Failed    0.49 sec
>          make[1]: Entering directory '/tmp/guix-build-ldc-1.14.0.drv-0/ldc-1.14.0/runtime/druntime/test/aa'
>          /tmp/guix-build-ldc-1.14.0.drv-0/build/bin/ldmd2   -w -I../../src -I../../import -Isrc -defaultlib= -debuglib= -dip1000 -L/tmp/guix-build-ldc-1.14.0.drv-0/build/lib/libdruntime-ldc.a -O -release -of/tmp/guix-build-ldc-1.14.0.drv-0/build/runtime/druntime-test-aa/test_aa src/test_aa.d
>          collect2: fatal error: cannot find 'ld'
>          compilation terminated.
> --8<---------------cut here---------------end--------------->8---
> 
> I don't really understand how it can fail to find 'ld' considering it's
> supposed to be building with binutils.
> 
> Am I missing something?  Pjotr?
> 
> -- 
> Pierre Neidhardt
> https://ambrevar.xyz/