Bootstrappable bitcoin release builds with Guix

Bootstrappable bitcoin release builds with Guix

[Carl Dong]

Hi all,

I've been on a quest to use Guix for Bitcoin Core's reproducible builds as I
believe that Guix's focus on bootstrappability, and Guile's simplicity and
flexibility are very desirable qualities in building an auditable, secure, and
reliable build process. My pull request (very short thanks to Guix's
infrastructure) can be found here:
https://github.com/bitcoin/bitcoin/pull/15277/files

I've submitted patches for the Guix bitcoin-core package to make it
reproducible, which seems to work fine. However, for easier acceptance into the
bitcoin core process, I need to produce tarballs like the ones we have on our
servers today: https://bitcoincore.org/bin/bitcoin-core-0.17.1/

For some context, we have a "mini-guix" of sorts seen in our "depends tree"
here: https://github.com/bitcoin/bitcoin/tree/master/depends. This builds all
the dependencies for bitcoin just the way we want them, in preparation for
getting linked into bitcoin itself.

My current approach for the build process is to produce a Guix container in
which I execute a build of our "depends tree" followed by a build of bitcoin
itself. See the Guix manifest and scripts here:
https://github.com/bitcoin/bitcoin/pull/15277/files

However, there were three hiccups that I had to hack my way around:

1. libstdc++ would not link statically even with "-static-libstdc++". The hack
   was to remove the .la file under $LIBRARY_PATH.

2. Upon inspection of the binaries produced at the end of this process, they all
   had rpaths. The hack was to use patchelf --remove-rpath on them.

3. Upon inspection of the binaries produced at the end of this process, their
   interpreters all had a `/gnu/store/blahblah-glibc-2.28' prefix. The hack was
   to use patchelf --set-interpreter on them.

My questions are:

1. Is there a way to avoid the hacks that I listed above? I understand that it
   might mean writing custom gcc packages and I'm 100% okay with that.

2. Is there an easier way of achieving the same thing?

Thank you all in advance for helping with this, and I hope that we'll see
boostrappable Guix release builds of bitcoin very soon!

Cheers,
Carl Dong
contact@carldong.me
"I fight for the users"

Re: Bootstrappable bitcoin release builds with Guix

[Ludovic Courtès]

Hi Carl,

Carl Dong <contact@carldong.me> skribis:

> I've been on a quest to use Guix for Bitcoin Core's reproducible builds as I
> believe that Guix's focus on bootstrappability, and Guile's simplicity and
> flexibility are very desirable qualities in building an auditable, secure, and
> reliable build process. My pull request (very short thanks to Guix's
> infrastructure) can be found here:
> https://github.com/bitcoin/bitcoin/pull/15277/files

I’m late to the party, but this is awesome!  The goals you’re pursuing
are in line with those we’re after in Guix, Reproducible Builds, and
Bootstrappable, so it’s great to see that Guix can help others here.

> I've submitted patches for the Guix bitcoin-core package to make it
> reproducible, which seems to work fine. However, for easier acceptance into the
> bitcoin core process, I need to produce tarballs like the ones we have on our
> servers today: https://bitcoincore.org/bin/bitcoin-core-0.17.1/
>
> For some context, we have a "mini-guix" of sorts seen in our "depends tree"
> here: https://github.com/bitcoin/bitcoin/tree/master/depends. This builds all
> the dependencies for bitcoin just the way we want them, in preparation for
> getting linked into bitcoin itself.
>
> My current approach for the build process is to produce a Guix container in
> which I execute a build of our "depends tree" followed by a build of bitcoin
> itself. See the Guix manifest and scripts here:
> https://github.com/bitcoin/bitcoin/pull/15277/files

OK.  You could store the output of ‘guix describe -f channels’ in
addition to the manifest, and then do ‘guix pull -C’ of this file.  That
way, you’d be pinning a specific package set.

> However, there were three hiccups that I had to hack my way around:
>
> 1. libstdc++ would not link statically even with "-static-libstdc++". The hack
>    was to remove the .la file under $LIBRARY_PATH.

Weird.  Could you send a small test case for this to bug-guix@gnu.org?

> 2. Upon inspection of the binaries produced at the end of this process, they all
>    had rpaths. The hack was to use patchelf --remove-rpath on them.

Yes, ‘ld-wrapper’ and our ‘gcc’ packages add those on purpose; they’re
required for dynamically-linked binaries.  But you’re producing
statically-linked executables in the end, right?

> 3. Upon inspection of the binaries produced at the end of this process, their
>    interpreters all had a `/gnu/store/blahblah-glibc-2.28' prefix. The hack was
>    to use patchelf --set-interpreter on them.

To use /lib/ld-linux-x86-64.so.2 instead?

You can do that, but there’s a risk: this is assuming that the loader
and libc on the user’s machine are compatible with those you built
against.

> My questions are:
>
> 1. Is there a way to avoid the hacks that I listed above? I understand that it
>    might mean writing custom gcc packages and I'm 100% okay with that.
>
> 2. Is there an easier way of achieving the same thing?

I think the difficulty here lies in producing non-self-contained binary
tarballs, paradoxically.  :-)

By that I mean that the tarball at
<https://bitcoincore.org/bin/bitcoin-core-0.17.1/bitcoin-0.17.1-x86_64-linux-gnu.tar.gz>
makes assumptions on the user’s machine:

--8<---------------cut here---------------start------------->8---
$ file /tmp/bitcoin-0.17.1/bin/bitcoin-cli 
/tmp/bitcoin-0.17.1/bin/bitcoin-cli: ELF 64-bit LSB pie executable x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=aced8cd0e28adb7a9e0c15f2e794aae7150d348e, stripped
$ objdump -x /tmp/bitcoin-0.17.1/bin/bitcoin-cli |grep NEED
  NEEDED               librt.so.1
  NEEDED               libm.so.6
  NEEDED               libgcc_s.so.1
  NEEDED               libpthread.so.0
  NEEDED               libc.so.6
  NEEDED               ld-linux-x86-64.so.2
  VERNEED              0x0000000000002850
  VERNEEDNUM           0x0000000000000006
--8<---------------cut here---------------end--------------->8---

The binary assumes that /lib64/ld-linux-x86-64.so.2 exists, that
libgcc_s.so & co. will be found in the “standard locations”, etc.  These
assumptions do not hold on a standalone Guix system or on NixOS or
Gobolinux.  They are probably reasonable for FHS distros, though.

Can you achieve this by simply taking the ouptut of “guix build
bitcoin-core-alt” (where ‘bitcoin-core-alt’ is a variant that statically
links against libstdc++ etc.), patching the ELF interpreter, and then
copying it in a tarball?

You’d also get cross-compilation by adding ‘--target’.

Thanks,
Ludo’.

Re: Bootstrappable bitcoin release builds with Guix

[Carl Dong]

Hi Ludovic!

> OK. You could store the output of ‘guix describe -f channels’ in
> addition to the manifest, and then do ‘guix pull -C’ of this file. That
> way, you’d be pinning a specific package set.

Ah... things like this make me more fall in love with Guix everyday.

> > 1.  libstdc++ would not link statically even with "-static-libstdc++". The hack
> >     was to remove the .la file under $LIBRARY_PATH.
> >
>
> Weird. Could you send a small test case for this tobug-guix@gnu.org?

I realized afterwards that this is actually another problem. The real problem is that configure insists on adding `-lstdc++` to `config.status` and `libtool` even when `-static-libstdc++` is in `LDFLAGS`. I will do some more digging but it seems to only happen in Guix environments. I would appreciate any insight.

> > 2.  Upon inspection of the binaries produced at the end of this process, they all
> >     had rpaths. The hack was to use patchelf --remove-rpath on them.
> >
>
> Yes, ‘ld-wrapper’ and our ‘gcc’ packages add those on purpose; they’re
> required for dynamically-linked binaries. But you’re producing
> statically-linked executables in the end, right?

We're producing a executable that's dynamically linked in the end, we perform a few checks to make sure that only the libraries we want to dynamically link to are dynamically linked to: https://github.com/bitcoin/bitcoin/blob/40a720acb8472f6e8afdf8b8d1897f35d58daf1f/contrib/devtools/symbol-check.py#L57. I have resorted to modifying the default gcc package to omit the `GNU_USER_TARGET_LIB_SPEC` substitute which is how the rpaths sneak in. I suspect that using a specfile would have also worked?

> > 3.  Upon inspection of the binaries produced at the end of this process, their
> >     interpreters all had a `/gnu/store/blahblah-glibc-2.28' prefix. The hack was
> >     to use patchelf --set-interpreter on them.
> >
>
> To use /lib/ld-linux-x86-64.so.2 instead?
>
> You can do that, but there’s a risk: this is assuming that the loader
> and libc on the user’s machine are compatible with those you built
> against.

Right. This is why we check our libc version is one that is reasonably old to work on old systems: https://github.com/bitcoin/bitcoin/blob/40a720acb8472f6e8afdf8b8d1897f35d58daf1f/contrib/devtools/symbol-check.py#L38. I'm not familiar with how the loader might be incompatible?

> Can you achieve this by simply taking the ouptut of “guix build
> bitcoin-core-alt” (where ‘bitcoin-core-alt’ is a variant that statically
> links against libstdc++ etc.), patching the ELF interpreter, and then
> copying it in a tarball?
>
> You’d also get cross-compilation by adding ‘--target’.

Yes, I think you're right that that would be a little easier. I might transition to this method once I'm ready, and I don't think patching the ELF interpreter is that bad.

Cheers,
Carl Dong
contact@carldong.me
"I fight for the users"

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, April 17, 2019 4:58 PM, Ludovic Courtès <ludo@gnu.org> wrote:

> Hi Carl,
>
> Carl Dong contact@carldong.me skribis:
>
> > I've been on a quest to use Guix for Bitcoin Core's reproducible builds as I
> > believe that Guix's focus on bootstrappability, and Guile's simplicity and
> > flexibility are very desirable qualities in building an auditable, secure, and
> > reliable build process. My pull request (very short thanks to Guix's
> > infrastructure) can be found here:
> > https://github.com/bitcoin/bitcoin/pull/15277/files
>
> I’m late to the party, but this is awesome! The goals you’re pursuing
> are in line with those we’re after in Guix, Reproducible Builds, and
> Bootstrappable, so it’s great to see that Guix can help others here.
>
> > I've submitted patches for the Guix bitcoin-core package to make it
> > reproducible, which seems to work fine. However, for easier acceptance into the
> > bitcoin core process, I need to produce tarballs like the ones we have on our
> > servers today: https://bitcoincore.org/bin/bitcoin-core-0.17.1/
> > For some context, we have a "mini-guix" of sorts seen in our "depends tree"
> > here: https://github.com/bitcoin/bitcoin/tree/master/depends. This builds all
> > the dependencies for bitcoin just the way we want them, in preparation for
> > getting linked into bitcoin itself.
> > My current approach for the build process is to produce a Guix container in
> > which I execute a build of our "depends tree" followed by a build of bitcoin
> > itself. See the Guix manifest and scripts here:
> > https://github.com/bitcoin/bitcoin/pull/15277/files
>
> OK. You could store the output of ‘guix describe -f channels’ in
> addition to the manifest, and then do ‘guix pull -C’ of this file. That
> way, you’d be pinning a specific package set.
>
> > However, there were three hiccups that I had to hack my way around:
> >
> > 1.  libstdc++ would not link statically even with "-static-libstdc++". The hack
> >     was to remove the .la file under $LIBRARY_PATH.
> >
>
> Weird. Could you send a small test case for this tobug-guix@gnu.org?
>
> > 2.  Upon inspection of the binaries produced at the end of this process, they all
> >     had rpaths. The hack was to use patchelf --remove-rpath on them.
> >
>
> Yes, ‘ld-wrapper’ and our ‘gcc’ packages add those on purpose; they’re
> required for dynamically-linked binaries. But you’re producing
> statically-linked executables in the end, right?
>
> > 3.  Upon inspection of the binaries produced at the end of this process, their
> >     interpreters all had a `/gnu/store/blahblah-glibc-2.28' prefix. The hack was
> >     to use patchelf --set-interpreter on them.
> >
>
> To use /lib/ld-linux-x86-64.so.2 instead?
>
> You can do that, but there’s a risk: this is assuming that the loader
> and libc on the user’s machine are compatible with those you built
> against.
>
> > My questions are:
> >
> > 1.  Is there a way to avoid the hacks that I listed above? I understand that it
> >     might mean writing custom gcc packages and I'm 100% okay with that.
> >
> > 2.  Is there an easier way of achieving the same thing?
> >
>
> I think the difficulty here lies in producing non-self-contained binary
> tarballs, paradoxically. :-)
>
> By that I mean that the tarball at
> https://bitcoincore.org/bin/bitcoin-core-0.17.1/bitcoin-0.17.1-x86_64-linux-gnu.tar.gz
> makes assumptions on the user’s machine:
>
> --8<---------------cut here---------------start------------->8---
> $ file /tmp/bitcoin-0.17.1/bin/bitcoin-cli
> /tmp/bitcoin-0.17.1/bin/bitcoin-cli: ELF 64-bit LSB pie executable x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=aced8cd0e28adb7a9e0c15f2e794aae7150d348e, stripped
> $ objdump -x /tmp/bitcoin-0.17.1/bin/bitcoin-cli |grep NEED
> NEEDED librt.so.1
> NEEDED libm.so.6
> NEEDED libgcc_s.so.1
> NEEDED libpthread.so.0
> NEEDED libc.so.6
> NEEDED ld-linux-x86-64.so.2
> VERNEED 0x0000000000002850
> VERNEEDNUM 0x0000000000000006
> --8<---------------cut here---------------end--------------->8---
>
> The binary assumes that /lib64/ld-linux-x86-64.so.2 exists, that
> libgcc_s.so & co. will be found in the “standard locations”, etc. These
> assumptions do not hold on a standalone Guix system or on NixOS or
> Gobolinux. They are probably reasonable for FHS distros, though.
>
> Can you achieve this by simply taking the ouptut of “guix build
> bitcoin-core-alt” (where ‘bitcoin-core-alt’ is a variant that statically
> links against libstdc++ etc.), patching the ELF interpreter, and then
> copying it in a tarball?
>
> You’d also get cross-compilation by adding ‘--target’.
>
> Thanks,
> Ludo’.

Re: Bootstrappable bitcoin release builds with Guix

[Ludovic Courtès]

Hello Carl!

Carl Dong <contact@carldong.me> skribis:

>> > 1.  libstdc++ would not link statically even with "-static-libstdc++". The hack
>> >     was to remove the .la file under $LIBRARY_PATH.
>> >
>>
>> Weird. Could you send a small test case for this tobug-guix@gnu.org?
>
> I realized afterwards that this is actually another problem. The real problem is that configure insists on adding `-lstdc++` to `config.status` and `libtool` even when `-static-libstdc++` is in `LDFLAGS`. I will do some more digging but it seems to only happen in Guix environments. I would appreciate any insight.

Weird.  You did check ‘gcc -dumpspecs’ already, right?

>> > 2.  Upon inspection of the binaries produced at the end of this process, they all
>> >     had rpaths. The hack was to use patchelf --remove-rpath on them.
>> >
>>
>> Yes, ‘ld-wrapper’ and our ‘gcc’ packages add those on purpose; they’re
>> required for dynamically-linked binaries. But you’re producing
>> statically-linked executables in the end, right?
>
> We're producing a executable that's dynamically linked in the end, we perform a few checks to make sure that only the libraries we want to dynamically link to are dynamically linked to: https://github.com/bitcoin/bitcoin/blob/40a720acb8472f6e8afdf8b8d1897f35d58daf1f/contrib/devtools/symbol-check.py#L57. I have resorted to modifying the default gcc package to omit the `GNU_USER_TARGET_LIB_SPEC` substitute which is how the rpaths sneak in. I suspect that using a specfile would have also worked?

Yes I think so.

>> > 3.  Upon inspection of the binaries produced at the end of this process, their
>> >     interpreters all had a `/gnu/store/blahblah-glibc-2.28' prefix. The hack was
>> >     to use patchelf --set-interpreter on them.
>> >
>>
>> To use /lib/ld-linux-x86-64.so.2 instead?
>>
>> You can do that, but there’s a risk: this is assuming that the loader
>> and libc on the user’s machine are compatible with those you built
>> against.
>
> Right. This is why we check our libc version is one that is reasonably old to work on old systems: https://github.com/bitcoin/bitcoin/blob/40a720acb8472f6e8afdf8b8d1897f35d58daf1f/contrib/devtools/symbol-check.py#L38. I'm not familiar with how the loader might be incompatible?

I don’t have any specific example in mind and the risk is probably low
given that it’s a stable piece of software.  The loader does know and
use libc internals, so there could be problems in theory.

Thanks,
Ludo’.