title: New Supported Platform: powerpc64le-linux
date: 2021-04-08 00:00
author: Chris Marusich and Léo Le Bouter
tags: porting, powerpc64le, bootstrapping, cross-compilation, reproducibility
---

It is a pleasure to announce that support for powerpc64le-linux
(PowerISA v.2.07 and later) has now been
[merged](https://issues.guix.gnu.org/47182) to the master branch of
GNU Guix!

This means that GNU Guix can be used immediately on this platform from
a [from a Git
checkout](https://guix.gnu.org/manual/en/html_node/Building-from-Git.html).
Starting with the next release (Guix v1.2.1), you will also be able to
[download a copy of Guix pre-built for
powerpc64le-linux](https://guix.gnu.org/manual/en/html_node/Binary-Installation.html#Binary-Installation).
Regardless of how you get it, you can run the new powerpc64le-linux
port of GNU Guix on top of any existing powerpc64le GNU/Linux
distribution.

This new platform is available as a "technology preview".  This means
that although it is supported,
[substitutes](https://guix.gnu.org/manual/en/html_node/Substitutes.html)
are not yet available from the build farm, and some packages may fail
to build.  Although powerpc64le-linux support is nascent, the Guix
community is actively working on improving it, and this is a great
time to [get
involved](https://guix.gnu.org/manual/en/html_node/Contributing.html)!

### Why Is This Important?

This is important because it means that GNU Guix now works on the
[Talos II and Talos II Lite
mainboards](https://www.fsf.org/news/talos-ii-mainboard-and-talos-ii-lite-mainboard-now-fsf-certified-to-respect-your-freedom),
which use [IBM POWER9
processors](https://wiki.raptorcs.com/wiki/POWER9).  This is a modern,
performant hardware platform that has recently received [Respects Your
Freedom (RYF) certification](https://ryf.fsf.org/) from the FSF.  It
can run without any non-free code, all the way down to its bootloader
and firmware.  In other words, it's a freedom-friendly platform that
aligns well with GNU Guix's commitment to software freedom.

How is this any different from existing RYF hardware, you might ask?
One reason is performance.  The existing RYF
[laptops](https://ryf.fsf.org/products?category=1&vendor=All&sort_by=created&sort_order=DESC),
[mainboards](https://ryf.fsf.org/products?category=5&vendor=All&sort_by=created&sort_order=DESC),
and
[workstations](https://ryf.fsf.org/products?category=30&vendor=All&sort_by=created&sort_order=DESC)
can only really be used with Intel Core Duo or AMD Opteron processors.
Those processors were released over 15 years ago.  Since then,
processor performance has increased drastically.  People should not
have to choose between performance and freedom, but for many years
that is exactly what we were forced to do.  However, the Talos II and
Talos II Lite have changed this: the free software community now has
an RYF-certified option that [can compete with the performance of
modern Intel and AMD
systems](https://www.phoronix.com/scan.php?page=article&item=power9-threadripper-core9&num=1).

Although the performance of POWER9 processors is competitive with
modern Intel and AMD processors, the real advantage of the Talos II
and Talos II Lite is that they were designed from the start to respect
your freedom.  Modern processors from [both Intel and AMD include back
doors](https://www.fsf.org/blogs/sysadmin/the-management-engine-an-attack-on-computer-users-freedom)
over which you are given no control.  Even though the back doors can
be removed [in some cases with significant
effort](https://www.fsf.org/news/libreboot-x200-laptop-now-fsf-certified-to-respect-your-freedom),
this is an obstacle that nobody should have to overcome just to
control their own computer.  No such obstacles exist when using the
Talos II or Talos II Lite.  In fact, although
[Intel](https://arstechnica.com/gadgets/2020/10/in-a-first-researchers-extract-secret-key-used-to-encrypt-intel-cpu-code/)
and
[AMD](https://www.extremetech.com/computing/292722-amds-secure-processor-firmware-is-now-explorable-thanks-to-new-tool)
both go out of their way to keep you from understanding what is going
on in your own computer, Raptor Computing Systems releases [all of the
software and firmware used on the
board](https://git.raptorcs.com/git/) as free software.  They even
include circuit diagrams when they ship you the machine.

Compared to the existing options, the RYF Talos II and Talos II Lite
are a breath of fresh air that the free software community really
deserves.  Raptor Computing Systems' commitment to software freedom
and owner control is an inspiring reminder that it **is** possible to
ship a great product that respects the freedom of your customers.  And
going forward, the future looks bright for the open, royalty-free
Power ISA, [which is now a Linux Foundation
project](https://www.linuxfoundation.org/press-release/2019/08/the-linux-foundation-announces-new-open-hardware-technologies-and-collaboration/)
(see also: [the same announcement from The OpenPOWER
Foundation](https://openpowerfoundation.org/the-next-step-in-the-openpower-foundation-journey/).

### Bootstrapping powerpc64le-linux: A Journey

To build software, you need software.  How can one port Guix to a
platform before support for that platform exists?  This is a
[bootstrapping
problem](https://guix.gnu.org/manual/en/html_node/Bootstrapping.html).

In Guix, all software for a given platform (e.g., powerpc64le-linux)
is built starting from a small set of "bootstrap binaries".  These are
binaries of Guile, GCC, Binutils, libc, and a few other packages,
pre-built for the relevant platform.  It is intended that the
bootstrap binaries are the only pieces of software in the entire
package collection that Guix cannot build from source.  In practice,
[additional bootstrap roots are
possible](https://lists.gnu.org/archive/html/guix-devel/2015-02/msg00814.html),
but introducing them in Guix is highly discouraged, and our community
[actively](https://guix.gnu.org/en/blog/2019/guix-reduces-bootstrap-seed-by-50/)
[works](https://guix.gnu.org/en/blog/2020/guix-further-reduces-bootstrap-seed-to-25/)
to [reduce](https://guix.gnu.org/en/blog/2018/bootstrapping-rust/) our
overall bootstrap footprint.  There is one set of bootstrap binaries
for each platform that Guix supports.

This means that to port Guix to a new platform, you must first build
the bootstrap binaries for that platform.  In theory, you can do this
in many ways.  For example, you might try to manually compile them on
an existing system.  However, Guix has [package
definitions](https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/make-bootstrap.scm?id=5d8c2c00d60196c46a32b68c618ccbe2b3aa48f4)
that you can use to build them - using Guix, of course!

Commonly, the first step in [porting Guix to a new
platform](https://guix.gnu.org/manual/en/html_node/Porting.html) is to
use Guix to cross-compile the bootstrap binaries for that new platform
from a platform on which Guix is already supported. This can be done
by running a command like the following on a system where Guix is
already installed:

```scheme
guix build --target=powerpc64le-linux-gnu bootstrap-tarballs
```

This is the route that we took when building the powerpc64le-linux
bootstrap binaries, as described in commit
[8a1118a](https://git.savannah.gnu.org/cgit/guix.git/commit/?id=8a1118a96c9ae128302c3d435ae77cb3dd693aea).
You might wonder why the target above is "powerpc64le-linux-gnu" even
though the new Guix platform is called "powerpc64le-linux".  This is
because "powerpc64le-linux-gnu" is a GNU
[triplet](https://wiki.osdev.org/Target_Triplet) identifying the new
platform, but "powerpc64le-linux" is the name of a "system" (i.e., a
platform) in Guix.  Guix contains code that converts between the two
as needed (see `nix-system->gnu-triplet` and `gnu-triplet->nix-system`
in
[`guix/utils.scm`](https://git.savannah.gnu.org/cgit/guix.git/tree/guix/utils.scm?id=83991a34d5c1d4985e54dd029a81412277ad062a).
When cross-compiling, you only need to specify the GNU triplet.

Note that before you can even do this, you must first update the
`glibc-dynamic-linker` and `system->linux-architecture` procedures in
Guix's code, as described in
[Porting](https://guix.gnu.org/manual/en/html_node/Porting.html).  In
addition, the versions of packages in Guix that make up the GNU
toolchain (gcc, glibc, etc.) must already support the target platform.
This pre-existing toolchain support needs to be good enough so that
Guix can (1) build, on some already-supported platform, a
cross-compilation toolchain for the target platform, (2) use, on the
already-supported platform, the cross-compilation toolchain to
cross-compile the bootstrap binaries for the target platform, and (3)
use, on the target platform, the bootstrap binaries to natively build
the rest of the Guix package collection.  The above [`guix
build`](https://guix.gnu.org/manual/en/html_node/Invoking-guix-build.html#Invoking-guix-build)
command takes care of steps (1) and (2) automatically.

Step (3) is a little more involved.  Once the bootstrap binaries for
the target platform have been built, they must be published online for
anyone to download.  After that, Guix's code must be updated so that
(a) it recognizes the "system" name (e.g., "powerpc64le-linux") that
will be used to identify the new platform and (b) it fetches the new
platform's bootstrap binaries from the right location.  After all that
is done, you just have to try building things and see what breaks.
For example, you can run `./pre-inst-env guix build hello` from your
Git checkout to try building GNU Hello.

The actual bootstrap binaries for powerpc64le-linux are stored on the
[alpha.gnu.org FTP
server](https://alpha.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/).
Chris Marusich built these bootstrap binaries in an x86_64-linux Guix
System VM which was running on hardware owned by Léo Le Bouter.  Chris
then signed the binaries and provided them to Ludovic Courtès, who in
turn verified their authenticity, signed them, and [uploaded them to
alpha.gnu.org](https://debbugs.gnu.org/cgi/bugreport.cgi?bug=41669#125).
After that, we updated the code to use the newly published bootstrap
binaries in commit
[8a1118a](https://git.savannah.gnu.org/cgit/guix.git/commit/?id=8a1118a96c9ae128302c3d435ae77cb3dd693aea).
Once all that was done, we could begin bootstrapping the rest of the
system - or trying to, at least.

There were many stumbling blocks.  For example, to resolve some test
failures, we had to update the code in Guix that enables it to make
[certain](https://git.savannah.gnu.org/cgit/guix.git/commit/?id=b57de27d0331198c9cafb09a1cf8a5fa4f691e36)
[syscalls](https://git.savannah.gnu.org/cgit/guix.git/commit/?id=c29bfbfc78ccd9e5c10d38faf3d269eafed12854)
from scheme.  In another example, we had to [patch GCC so that it
looks for the 64-bit libraries in
/lib](https://debbugs.gnu.org/cgi/bugreport.cgi?bug=46253), rather
than /lib64, since that is where Guix puts its 64-bit libraries by
convention.  In addition, some packages required in order to build
Guix failed to build, so we had to debug those build failures, too.

For a list of all the changes, see [the patch
series](https://debbugs.gnu.org/cgi/bugreport.cgi?bug=47182) or the
actual commits, which are:

```
$ git log --oneline --no-decorate 8a1118a96c9ae128302c3d435ae77cb3dd693aea..65c46e79e0495fe4d32f6f2725d7233fff10fd70
65c46e79e04 gnu: sed: Make it build on SELinux-enabled kernels.
93f21e1a35e utils: Fix target-64bit? on powerpc64le-linux.
8d9aece8c49 ci: %cross-targets: Add powerpc64le-linux-gnu.
c29bfbfc78c syscalls: Fix RNDADDTOENTCNT on powerpc64le-linux.
b57de27d033 syscalls: Fix clone on powerpc64le-linux.
a16eb6c5f97 Add powerpc64le-linux as a supported Guix architecture.
b50f4268035 gnu: libelf: Fix compilation for powerpc64le-linux.
1a0f4013d33 gnu: texlive-latex-base: Fix compilation on powerpc64le*.
e9938dc8f0e gnu: texlive-bin: Fix compilation on powerpc64le*.
69b3907adf6 gnu: guile-avahi: Fix compilation on powerpc64le-linux.
4cc2d2aa599 gnu: bdb-4.8: Fix configure on powerpc64le-linux.
be4b1cf53bd gnu: binutils-final: Support more Power architectures.
060478c32c9 gnu: binutils-final: Provide bash for binary on powerpc-linux.
b2135b5d576 gnu: gcc-boot0: Enable 128-bit long double for POWER9.
6e98e9ca923 gnu: glibc: Fix ldd path on powerpc*.
cac88b28b83 gnu: gcc-4.7: On powerpc64le, fix /lib64 references.
fc7cf0c1ecc utils: Add target-powerpc? procedure.
```

In the end, through the combined efforts of multiple people, we slowly
worked through the issues until we reached a point where we could do
all of the following things successfully:

- Build Guix manually on a [Debian GNU/Linux
  ppc64el](https://wiki.debian.org/ppc64el) machine (this is Debian's
  name for a system using the powerpc64le-linux-gnu triplet), and
  verify that its `make check` tests passed.
- Build GNU Hello using Guix and run it.
- Run [`guix
  pull`](https://guix.gnu.org/manual/en/html_node/Invoking-guix-pull.html#Invoking-guix-pull)
  to build and install the most recent version of Guix, with
  powerpc64le-linux support.
- Build a release binary tarball for powerpc64le-linux via: `make
  guix-binary.powerpc64le-linux.tar.xz`
- Use that binary to install a version of Guix that could build/run
  GNU Hello and run `guix pull` successfully.

This was an exciting moment!  But there was still more work to be
done.

Originally, we did this work on the
[wip-ppc64le](https://git.savannah.gnu.org/cgit/guix.git/log/?h=wip-ppc64le)
branch, with the intent of merging it into core-updates.  By
convention, the "core-updates" branch in Guix is [where changes are
made if they cause too many
rebuilds](https://guix.gnu.org/manual/en/html_node/Submitting-Patches.html).
Since we were updating package definitions so deep in the dependency
graph of the package collection, we assumed it wouldn't be possible to
avoid rebuilding the world.  For this reason, we had based the
wip-ppc64le branch on core-updates.

However, Efraim Flashner proved us wrong!  He created a separate
branch, wip-ppc64le-for-master, where he adjusted some of the
wip-ppc64le commits to avoid rebuilding the world on other platforms.
Thanks to his work, we were able to merge the changes directly to
master!  This meant that we would be able to include it in the next
release (Guix v.1.2.1).

In short, the initial porting work is done, and it is now possible for
anyone to easily try out Guix on this new platform.  Because `guix
pull` works, too, it is also easy to iterate on what we have and work
towards improving support for the platform.  It took a lot of
cooperation and effort to get this far, but there are multiple people
actively contributing to this port in the Guix community who want to
see it succeed.  We hope you will join us in exploring the limits of
this exciting new freedom-friendly platform!

### Other Porting Challenges

Very early in the porting process, there were some other problems that
stymied our work.

First, we actually thought we would try to port to powerpc64-linux
(big-endian).  However, this did not prove to be any easier than the
little-endian port.  In addition, other distributions (e.g.,
[Debian](https://www.debian.org/ports/) and
[Fedora](https://fedoraproject.org/wiki/Architectures)) have recently
dropped their big-endian powerpc64 ports, so the little-endian variant
is more likely to be tested and supported in the community.  For these
reasons, we decided to focus our efforts on the little-endian variant,
and so far we haven't looked back.

In both the big-endian and little-endian case, we were saddened to
discover that the bootstrap binaries are not entirely reproducible.
This fact is documented in [bug
41669](https://debbugs.gnu.org/cgi/bugreport.cgi?bug=41669), along
with our extensive investigations.

In short, if you build the bootstrap binaries on two separate machines
without using any substitutes, you will find that the derivation which
cross-compiles %gcc-static (the bootstrap GCC, version 5.5.0) produces
different output on the two systems.  However, if you build
%gcc-static twice on the same system, it builds reproducibly.  This
suggests that something in the transitive closure of inputs of
%gcc-static is perhaps contributing to its non-reproducibility.  There
is an interesting graph [toward the end of the bug
report](https://debbugs.gnu.org/cgi/bugreport.cgi?bug=41669#137),
shown below:

![Differing
Derivations](https://debbugs.gnu.org/cgi/bugreport.cgi?att=1;msg=137;filename=nonreproducible-drvs-small.png;bug=41669)

This graph shows the derivations that produce differing outputs across
two Guix System machines, when everything is built without
substitutes.  It starts from the derivation that cross-compiles
%gcc-static for powerpc64-linux-gnu (from x86_64-linux) using Guix at
commit 1ced8379c7641788fa607b19b7a66d18f045362b.  Then, it walks the
graph of derivation inputs, recording only those derivations which
produce differing output on the two different machines.  If the
non-reproducibility (across systems) of %gcc-static is caused by a
non-reproducible input, then it is probably caused by one or more of
the derivations shown in this graph.

At some point, you have to cut your losses and move on.  After months
of investigation without resolving the reproducibility issue, we
finally decided to move forward with the bootstrap binaries produced
earlier.  If necessary, we can always go back and try to fix this
issue.  However, it seemed more important to get started with the
bootstrapping work.

Anyone who is interested in solving this problem is welcome to comment
on the bug report and help us to figure out the mystery.  We are very
interested in solving it, but at the moment we are more focused on
building the rest of the Guix package collection on the
powerpc64le-linux platform using the existing bootstrap binaries.

### Next Steps

It is now possible to install Guix on a powerpc64le-linux system and
use it to build some useful software - in particular, Guix itself.  So
Guix is now "self-hosted" on this platform, which gives us a
comfortable place to begin further work.

The following tasks still need to be done.  Anyone can help, so please
get in touch if you want to contribute!

- Solve [the GCC bootstrap binary reproducibility
  issue](https://debbugs.gnu.org/cgi/bugreport.cgi?bug=41669)
  described above.
- Get [Guix
  System](https://guix.gnu.org/manual/en/html_node/System-Installation.html)
  to work on powerpc64le-linux.
- Get CI infrastructure to work
  ([Cuirass](https://guix.gnu.org/en/cuirass/) (see also: [Cuirass in
  the Guix
  manual](https://guix.gnu.org/manual/en/html_node/Continuous-Integration.html),
  [guix-build-coordinator](https://git.cbaines.net/guix/build-coordinator/)
  (see also: [Guix Build Coordinator in the Guix
  manual](https://guix.gnu.org/manual/en/html_node/Guix-Services.html),
  [substitutes](https://guix.gnu.org/manual/en/html_node/Substitutes.html),
  etc.)
- Try to build your favorite packages using Guix, [report
  problems](https://guix.gnu.org/manual/en/html_node/Tracking-Bugs-and-Patches.html),
  [try to fix
  them](https://guix.gnu.org/manual/en/html_node/Contributing.html),
  and [ask for help](https://guix.gnu.org/en/help/) if you're feeling
  stuck or not sure how to start.
- Try building rust, and if it works, judiciously re-introduce the
  librsvg dependency for powerpc64le-linux in gtk+ and gtk+-2, since
  [it is currently
  missing](https://git.savannah.gnu.org/cgit/guix.git/commit/?id=5d2863dfe4613d5091e61800fcd5a48922c8ce4e).
- Upgrade the default GCC to 8 on core-updates, try to build guix
  (e.g., `./pre-inst-env guix build guix`), and report/fix whatever
  issues occur.  We want to upgrade GCC to 8 because, on the
  core-updates branch, glibc has been upgraded from 2.31 to 2.32.
  Unfortunately, on powerpc64le-linux, upgrading glibc from 2.31 to
  2.32 without also upgrading the default GCC (it's currently 7.5.0)
  causes a lot of problems.  Right now, we believe the best path
  forward is probably just to upgrade to GCC 8 on core-updates.
- Merge core-updates to master after that.

#### About GNU Guix

[GNU Guix](https://guix.gnu.org) is a transactional package manager and
an advanced distribution of the GNU system that [respects user
freedom](https://www.gnu.org/distros/free-system-distribution-guidelines.html).
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, and AArch64 machines.

In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection.  When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management.  Guix is highly customizable
and hackable through [Guile](https://www.gnu.org/software/guile)
programming interfaces and extensions to the
[Scheme](http://schemers.org) language.