Defining a toolchain with musl libc

Defining a toolchain with musl libc

[David Arroyo]

[-- Attachment #1: Type: text/plain, Size: 3916 bytes --]

Hello,

I want to rebuild the packages in the `skarnet` module to be statically
linked with musl libc. I don't mind if my C/C++ toolchain itself does
not use musl libc, so initially, I tried simply modifying the packages
and their direct dependencies, like so:

	(define (compose . fns)
	  (let ((call (lambda (f x) (f x))))
	    (lambda (p) (fold-right call p fns))))
	
	(define (static-link p)
	  (package/inherit p
	    (arguments
	      (substitute-keyword-arguments (package-arguments p)
	        ((#:configure-flags flags ''())
	        `(cons* "LDFLAGS=-static" "--disable-shared" ,flags))))))
	
	(define replace-glibc
	  (package-input-rewriting `((,glibc . ,musl)) #:deep? #t))
	
	(define (append-name suffix)
	  (lambda (p)
	    (package/inherit p
	      (name (string-append p suffix)))))
	
	(define package-with-musl
	  (package-mapping
	    (compose static-link replace-glibc) #:deep? #f))
	
	(define hello-musl    (package-with-musl hello))
	(define execline-musl (package-with-musl execline))

Then I tested with 

	$ guix build -L /path/to/module -e '(@ (module path) hello-musl)'

This worked with simple packages like `hello`, but other packages
like `execline` still ended up with dynamically linked binaries:

	$ out=$(guix build -L $(pwd) -e '(@ (aqwari packages skaware) execline-musl)')
	$ file $out/bin/fdmove
	/gnu/store/82r5ilk3jir7cssmhc8bi87qqj46p29h-execline-2.8.1.0/bin/fdmove: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33/lib/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, not stripped
	
	$ guix gc --requisites $out
	/gnu/store/720rj90bch716isd8z7lcwrnvz28ap4y-bash-static-5.1.8
	/gnu/store/hxxbjrm4iw6aj203zli6hj4ds1l27s3k-skalibs-2.11.0.0
	/gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33
	/gnu/store/4y5m9lb8k3qkb1y9m02sw9w9a6hacd16-bash-minimal-5.1.8
	/gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib
	/gnu/store/82r5ilk3jir7cssmhc8bi87qqj46p29h-execline-2.8.1.0

I don't know how gcc and glibc ended up in there. I tried reading the
code, but I am not familiar enough with it to follow it yet. I think
they may get added as implicit inputs by gnu-build-system, and I might
not be using the package-input-rewriting transform correctly.

On my second try, I tried to create a GCC+musl toolchain and use that
I tried to create a toolchain like so:

	(define my-toolchain (make-gcc-toolchain gcc musl))

But this failed with the error:

	guix build: error: reference to invalid output 'static' of derivation '/gnu/store/jmp8rccdwnrnl9x947xp7a75kmgdhylp-musl-1.2.2.drv'

This is because the musl package does not have a "static" output, which
the package builder in `make-gcc-toolchain` tries to access. To work
around that issue, I added empty outputs to the musl package:

	(define musl-fix-outputs
	  (package
	    (inherit musl)
	    (outputs '("out" "static" "debug"))
	    (arguments
	      (list
	        #:phases
	          #~(modify-phases %standard-phases
	              (delete 'check)
	              (add-after 'build 'ensure-outputs
	                (lambda _
	                  (mkdir-p #$output:static)
	                  (mkdir-p #$output:debug))))))))
	
	(define my-toolchain
	  (make-gcc-toolchain gcc musl-fix-outputs))

Then I tried to build the `hello` package:

	(package-with-c-toolchain hello my-toolchain)

This rebuilds the musl package, and gets to the 'patch-source-shebangs
phase of the gcc rebuild, where it fails with the error:

	error: in phase 'patch-source-shebangs': uncaught exception:
	encoding-error "scm_to_stringn" "cannot convert narrow string to output locale" 84 #f #f

I've attached the full error trace to this email. I am not sure what to
make of this error, or whether it is a red herring or not. Does anyone
see what I'm doing wrong? Or does anyone have an example of building a
non-trivial package against musl?

Thanks,
David

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: guix-build.log --]
[-- Type: text/x-log; name="guix-build.log", Size: 2349 bytes --]

patch-shebang: ./gcc/testsuite/gcc.target/i386/vperm.pl: changing `/usr/bin/perl' to `/gnu/store/hy6abswwv4d89zp464fw52z65fkzr7h5-perl-5.34.0/bin/perl'
patch-shebang: ./gcc/testsuite/gcc.test-framework/gen_directive_tests: changing `/bin/bash' to `/gnu/store/4y5m9lb8k3qkb1y9m02sw9w9a6hacd16-bash-minimal-5.1.8/bin/bash'
error: in phase 'patch-source-shebangs': uncaught exception:
encoding-error "scm_to_stringn" "cannot convert narrow string to output locale" 84 #f #f 
phase `patch-source-shebangs' failed after 5.1 seconds
Backtrace:
          11 (primitive-load "/gnu/store/jg8zzfaiv938lxi1xyvhc5wz63p?")
In guix/build/gnu-build-system.scm:
    906:2 10 (gnu-build #:source _ #:outputs _ #:inputs _ #:phases . #)
In ice-9/boot-9.scm:
  1752:10  9 (with-exception-handler _ _ #:unwind? _ # _)
In srfi/srfi-1.scm:
    634:9  8 (for-each #<procedure 7fffeef967e0 at guix/build/gnu-b?> ?)
In ice-9/boot-9.scm:
  1752:10  7 (with-exception-handler _ _ #:unwind? _ # _)
In guix/build/gnu-build-system.scm:
   927:23  6 (_)
In srfi/srfi-1.scm:
    634:9  5 (for-each #<procedure 7fffeffdc840 at guix/build/utils?> ?)
In guix/build/utils.scm:
    268:8  4 (call-with-ascii-input-file "./gcc/testsuite/gdc.test/?" ?)
  1094:34  3 (_ #<input: ./gcc/testsuite/gdc.test/compilable/test135?>)
In unknown file:
           2 (regexp-exec #<regexp 7fffeefc1f80> "/opt/dmd/?????/rd?" ?)
In ice-9/boot-9.scm:
  1685:16  1 (raise-exception _ #:continuable? _)
  1685:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
Throw to key `encoding-error' with args `("scm_to_stringn" "cannot convert narrow string to output locale" 84 #f #f)'.
builder for `/gnu/store/vnysv6hbb8zwyy8jawy1ayd2nfv38l4y-gcc-musl-1.2.2-10.3.0.drv' failed with exit code 1
build of /gnu/store/vnysv6hbb8zwyy8jawy1ayd2nfv38l4y-gcc-musl-1.2.2-10.3.0.drv failed
View build log at '/var/log/guix/drvs/vn/ysv6hbb8zwyy8jawy1ayd2nfv38l4y-gcc-musl-1.2.2-10.3.0.drv.bz2'.
cannot build derivation `/gnu/store/mkm0w0mjsf55fh9c6nil2c690k256xim-gcc-musl-1.2.2-toolchain-10.3.0.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/isnkvn568b50wvlbbimf5b5q6fqz96df-hello-musl-static-2.12.drv': 1 dependencies couldn't be built
guix build: error: build of `/gnu/store/isnkvn568b50wvlbbimf5b5q6fqz96df-hello-musl-static-2.12.drv' failed

Re: Defining a toolchain with musl libc

[David Arroyo]

On Mon, Apr 11, 2022, at 22:34, David Arroyo wrote:
> 	error: in phase 'patch-source-shebangs': uncaught exception:
> 	encoding-error "scm_to_stringn" "cannot convert narrow string to 
> output locale" 84 #f #f

Looking further into this error, errno 84 is EILSEQ, "illegal byte
sequence". The error is raised during the 'patch-source-shebangs' phase,
while patching  ./gcc/testsuite/gdc.test/compilable/test13512.d  This
file has the shebang

	#!/opt/dmd/ÐÒÏÂÙ/rdmd

This in turn seems to be tripping up this line in guix/build/utils.scm:1094 :

                 (and=> (and line (regexp-exec shebang-rx line))

I'm not sure why I don't hit this error when building gcc-toolchain.
Perhaps there is different locale information in the build environment
without glibc.

David

Re: Defining a toolchain with musl libc

[zimoun]

Hi,

This sounds cool!


On Mon, 11 Apr 2022 at 22:34, "David Arroyo" <david@aqwari.net> wrote:

> 	$ out=$(guix build -L $(pwd) -e '(@ (aqwari packages skaware) execline-musl)')
> 	$ file $out/bin/fdmove
> 	/gnu/store/82r5ilk3jir7cssmhc8bi87qqj46p29h-execline-2.8.1.0/bin/fdmove: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33/lib/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, not stripped
> 	
> 	$ guix gc --requisites $out
> 	/gnu/store/720rj90bch716isd8z7lcwrnvz28ap4y-bash-static-5.1.8
> 	/gnu/store/hxxbjrm4iw6aj203zli6hj4ds1l27s3k-skalibs-2.11.0.0
> 	/gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33
> 	/gnu/store/4y5m9lb8k3qkb1y9m02sw9w9a6hacd16-bash-minimal-5.1.8
> 	/gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib
> 	/gnu/store/82r5ilk3jir7cssmhc8bi87qqj46p29h-execline-2.8.1.0
>
> I don't know how gcc and glibc ended up in there. I tried reading the
> code, but I am not familiar enough with it to follow it yet. I think
> they may get added as implicit inputs by gnu-build-system, and I might
> not be using the package-input-rewriting transform correctly.

Maybe “guix graph --path” could help.  Well, maybe skalibs and/or
execline retain a reference to bash or bash-minimal which probably drags
glibc and gcc.

But that does not explain the “dynamically linked“ above.


> On my second try, I tried to create a GCC+musl toolchain and use that
> I tried to create a toolchain like so:
>
> 	(define my-toolchain (make-gcc-toolchain gcc musl))

[...]

> 	(package-with-c-toolchain hello my-toolchain)
>
> This rebuilds the musl package, and gets to the 'patch-source-shebangs
> phase of the gcc rebuild, where it fails with the error:

[...]

> Throw to key `encoding-error' with args `("scm_to_stringn" "cannot convert narrow string to output locale" 84 #f #f)'.
> builder for `/gnu/store/vnysv6hbb8zwyy8jawy1ayd2nfv38l4y-gcc-musl-1.2.2-10.3.0.drv' failed with exit code 1
> build of /gnu/store/vnysv6hbb8zwyy8jawy1ayd2nfv38l4y-gcc-musl-1.2.2-10.3.0.drv failed
> View build log at '/var/log/guix/drvs/vn/ysv6hbb8zwyy8jawy1ayd2nfv38l4y-gcc-musl-1.2.2-10.3.0.drv.bz2'.

Well, I do not know if GCC needs some locales from Glibc to build.  It
is rebuilt because,

--8<---------------cut here---------------start------------->8---
(define* (make-gcc-toolchain gcc
[...]
  (let ((gcc (if libc (make-gcc-libc gcc libc) gcc))
--8<---------------cut here---------------end--------------->8---

where ’make-gcc-libc’ is defined in (gnu packages base).  Therefore, it
is probably not straightforward to use ’make-gcc-toolchain’ with musl as
libc.

Note that usually musl comes from a wrapper around gcc, named
’musl-gcc’.  It is specifically disabled by the musl package:

       #:configure-flags
       (list "--disable-gcc-wrapper")))

I do not know why.  Maybe because at the time of commit
ce728f70e5ef8783a28652e382c2c9f61c7b4c06, it was not necessary or maybe
because the license of musl is incompatible with the one of GCC.

Well, I would take this path: rebuild a variant of musl with the wrapper
and try to make a C toolchain using this wrapper.

Maybe the package musl-cross could be interesting.


Cheers,
simon

Re: Defining a toolchain with musl libc

[David Arroyo]

Hi Simon, thanks for your reply!

I must confess that I started this thread with an XY problem; my problem was with execline binaries segfaulting after I performed a pivot_root operation. I don't fully recall why, but I suspected the segfault occurred  when execv() tried to load and/or use a dynamic library. I was trying to verify this theory by statically linking execline and trying to reproduce the crash.

But since then there some updates to the glibc package and I cannot reproduce the issue anymore. I could try to go back with the time-machine and reproduce again but it's not urgent for me anymore.

On Mon, May 9, 2022, at 13:09, zimoun wrote:
> On Mon, 11 Apr 2022 at 22:34, "David Arroyo" <david@aqwari.net> wrote:
> Maybe “guix graph --path” could help.  Well, maybe skalibs and/or
> execline retain a reference to bash or bash-minimal which probably drags
> glibc and gcc.

That is a nifty tool, I didn't know about the `--path` option. Unfortunately it didn't seem to work here, I tried the following from the root of my channel repo:

	$ guix graph --path --type=bag-emerged -L $(pwd) -e '(@ (aqwari packages skaware) execline-musl-static)' glibc
	guix graph: error: no path from 'execline-musl-static@2.8.1.0' to 'glibc@2.33'

However, if I render the full `--type=bag-emerged` graph, massive as it is, glibc is clearly present and referenced by dozens of other nodes. There are too many edges for me to trace each one, but it looks like the path you predicted from execline -> bash-minimal -> glibc is there.

> Note that usually musl comes from a wrapper around gcc, named
> ’musl-gcc’.  It is specifically disabled by the musl package:
>
>        #:configure-flags
>        (list "--disable-gcc-wrapper")))
>
> I do not know why.  Maybe because at the time of commit
> ce728f70e5ef8783a28652e382c2c9f61c7b4c06, it was not necessary or maybe
> because the license of musl is incompatible with the one of GCC.

Newer versions of GCC actually have an `-mmusl` flag and allow for target
triplets include `-musl-`, which appear to accomplish something similar. In fact, in a separate project this G-expression worked well for compiling a single C file with musl, statically:

https://git.sr.ht/~droyo/guix-channel/tree/640fe1b4eabe9a3aaa4b532c0ed71e7abb74dae8/item/aqwari/namespace.scm#L288-299

> Well, I would take this path: rebuild a variant of musl with the wrapper
> and try to make a C toolchain using this wrapper.

This sounds like a reasonable approach. The only potential issue I see is this option in mus-gcc's spec file:

	*link:
	-dynamic-linker /lib/ld-musl-x86_64.so.1 -nostdlib %{shared:-shared} %{static:-static} %{rdynamic:-export-dynamic}

but I'm not producing dynamically linked binaries anyway. While this is not urgent for me anymore, I'm still interested and I will probably try this in the next few weeks.

> Maybe the package musl-cross could be interesting.

I will take a look. Initially I was put off because that package hard-codes its own gcc, linux headers, and binutils versions rather than tracking the common packages. But it could be good inspiration.

Thanks again!
David

Re: Defining a toolchain with musl libc

[zimoun]

Hi,

On Mon, 09 May 2022 at 19:54, "David Arroyo" <david@aqwari.net> wrote:

> That is a nifty tool, I didn't know about the `--path`
> option. Unfortunately it didn't seem to work here, I tried the
> following from the root of my channel repo:
>
> 	$ guix graph --path --type=bag-emerged -L $(pwd) -e  glibc
> 	guix graph: error: no path from 'execline-musl-static@2.8.1.0' to 'glibc@2.33'

Is ’glibc’ listed by “guix size -e '(@ (aqwari packages skaware) execline-musl-static)'”?


>> Note that usually musl comes from a wrapper around gcc, named
>> ’musl-gcc’.  It is specifically disabled by the musl package:
>>
>>        #:configure-flags
>>        (list "--disable-gcc-wrapper")))
>>
>> I do not know why.  Maybe because at the time of commit
>> ce728f70e5ef8783a28652e382c2c9f61c7b4c06, it was not necessary or maybe
>> because the license of musl is incompatible with the one of GCC.
>
> Newer versions of GCC actually have an `-mmusl` flag and allow for
> target

Maybe, try the compile the package using this flag and passing the
package musl as native-inputs.


> triplets include `-musl-`, which appear to accomplish something
> similar. In fact, in a separate project this G-expression worked well
> for compiling a single C file with musl, statically: 
>
> https://git.sr.ht/~droyo/guix-channel/tree/640fe1b4eabe9a3aaa4b532c0ed71e7abb74dae8/item/aqwari/namespace.scm#L288-299

Cool!

>
>> Well, I would take this path: rebuild a variant of musl with the wrapper
>> and try to make a C toolchain using this wrapper.
>
> This sounds like a reasonable approach. The only potential issue I see is this option in mus-gcc's spec file:
>
> 	*link:
> 	-dynamic-linker /lib/ld-musl-x86_64.so.1 -nostdlib %{shared:-shared} %{static:-static} %{rdynamic:-export-dynamic}
>
> but I'm not producing dynamically linked binaries anyway. While this
> is not urgent for me anymore, I'm still interested and I will probably
> try this in the next few weeks.

Well, from my understanding, we do not have a good story for using the
package transformation ’with-c-toolchain’ with another compiler than GCC
and another C library than Glibc.

And then, another story of static.  For instance, are you able to
produce static binary using the regular C toolchain.

Exploring 2 problem spaces in the same time makes hard to tackle each
individually since they are orthogonal, at first.  IMHO.



Cheers,
simon