* [bug#61765] custom toolchain blog post
2023-02-27 12:50 ` Ludovic Courtès
@ 2023-02-27 15:35 ` Mitchell Schmeisser via Guix-patches
2023-03-11 12:11 ` Ludovic Courtès
0 siblings, 1 reply; 8+ messages in thread
From: Mitchell Schmeisser via Guix-patches @ 2023-02-27 15:35 UTC (permalink / raw)
To: Ludovic Courtès
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: multipart/mixed; boundary="=-=-=", Size: 61545 bytes --]
--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Hello Ludo,
Thank you for your feedback!
> One thing that=E2=80=99s not clear to me: with this in place, can you do =
=E2=80=9Cguix
> build --target=3Darm-zephyr-eabi hello=E2=80=9D, for instance? If not, w=
hat=E2=80=99s
> missing to support it?
You cannot. I do not know how to describe it in a succinct way but
suffice to say the applications need to know they are targeting Zephyr
when they are written. The application will include a `prj.conf` which
is analogous to a custom defconfig in the Linux kernel.
Either in this file, or somewhere in the CMakeLists.txt a `BOARD`
variable is set to specify a specific board definition.
These board definitions contain information about the architecture and
the CMake scripts themselves pick the toolchain.
It's not that it's impossible to implement something like `guix build
--target=3Darm-zephyr-eabi k64f-hello-world` but the k64f-hello-world
would be written in such a way that the target is implicit in the
package.
The way I envision the `--target/system` flags being used in this
context is `guix build --target=3Darm-linux-gnueabihf k64f-hello-world`
which will still produce the correct firmware but will allow the
firmware to be staged to another machine which will be responsible for
the final deployment over some transport.
> I wonder if it would be worth mentioning
> <https://guix.gnu.org/manual/en/html_node/Platforms.html> too, and how
> one would go about adding a module. WDYT?
I considered trying to add Zephyr platforms but I'm not sure it's worth
the effort.
In addition to the patch to the website I attached another post(in org)
which describes how I integrated this toolchain into the Guix
infrastructure to allow defining firmware packages.
Maybe there will be additional information in there which can help you
understand where I'm going with all of this.
There will be a part 3 (and possibly more) about how to practically use
this stuff in a real project.
- Mitchell
--=-=-=
Content-Type: text/plain
Content-Disposition: attachment;
filename=0001-website-Add-toolchain-blog-post.patch
Content-Description: patch
From 0920ec7d951354c94c3da277d58e54b587522622 Mon Sep 17 00:00:00 2001
From: Mitchell Schmeisser <mitchellschmeisser@librem.one>
Date: Mon, 27 Feb 2023 10:20:32 -0500
Subject: [PATCH] website: Add toolchain blog post
website/blog/custom-toolchains-with-guix.md: New file
---
website/posts/custom-toolchains-with-guix.md | 576 +++++++++++++++++++
1 file changed, 576 insertions(+)
create mode 100644 website/posts/custom-toolchains-with-guix.md
diff --git a/website/posts/custom-toolchains-with-guix.md b/website/posts/custom-toolchains-with-guix.md
new file mode 100644
index 0000000..3bd38be
--- /dev/null
+++ b/website/posts/custom-toolchains-with-guix.md
@@ -0,0 +1,576 @@
+title: Building Toolchains with Guix
+author: Mitchell Schmeisser <mitchellschmeisser@librem.one>
+date: 2023-02-24 12:00
+tags: Software Development, Embedded, Zephyr, Scheme API
+---
+
+In order to deploy embedded software using Guix we first need to teach Guix
+how to build it. Since Guix bootstraps everything this means we must teach Guix
+how to build our toolchain.
+
+The [Zephyr Project](https://zephyrproject.org) uses its own fork of GCC with custom configs for
+the architectures supported by the project.
+
+This is implemented as a Guix Channel.
+All code is available at [here](https://github.com/paperclip4465/guix-zephyr).
+
+# About ZephyrRTOS
+
+ZephyrRTOS is a Real Time Operating System from the Linux Foundation.
+It aims to provide a common environment which can target even the most
+resource constrained devices.
+
+Zephyr introduces a module system which allows third parties to share code
+in a uniform way. Zephyr uses CMake to perform _physical component composition_
+of these modules. It searches the filesystem and generates scripts which
+the toolchain will use to successfully combine those components into a
+firmware image.
+
+The fact that Zephyr provides this mechanism is one reason I chose to
+target it in the first place.
+
+This separation of modules in an embedded context is a really great thing.
+It brings many of the advantages that it brings to the Linux world such as
+code re-use, smaller binaries, more efficient cache/RAM usage, etc.
+It also allows us to work as independent groups and compose
+contributions from many teams.
+
+It also brings all of the complexity. Suddenly most of the problems
+that plague traditional deployment now apply to our embedded
+system. The fact that the libraries are statically linked at compile
+time instead of dynamically at runtime is simply an implementation detail.
+I say most because everything is statically linked so there is no runtime
+component discovery that needs to be accounted for.
+
+
+# Anatomy of a Toolchain
+
+Toolchains are responsible for taking high level descriptions of programs
+and lowering them down to a series of equivalent machine instructions.
+This process involves more than just a compiler. The compiler uses the `binutils`
+to manipulate it's internal representation down to a given architecture.
+It also needs the use of the C standard library as well as a few other libraries
+needed for some compiler optimizations.
+
+The C library provides the interface to the underlying kernel. System calls like `write`
+and `read` are provided by Glibc on most Linux distributions.
+
+In embedded systems smaller implementations like Redhat's newlib and
+newlib-nano are used.
+
+# Bootstrapping a Toolchain
+
+In order to compile GCC we need a C library that's been compiled for
+our target architecture. How can we cross compile our C library if we
+need our C library to build a cross compiler? The solution is to build
+a simpler compiler that doesn't require the C library to function.
+It will not be capable of as many optimizations and it will be very slow,
+however it will be able to build the C libraries as well as the complete version
+of GCC.
+
+In order to build the simpler compiler we need to compile the Binutils to
+work with our target architecture.
+The `binutils` can be bootstrapped with our host GCC and have no target dependencies.
+
+[For more information read this.](https://crosstool-ng.github.io/docs/toolchain-construction/)
+
+Doesn't sound so bad right? It isn't... in theory.
+However internet forums since time immemorial have been
+littered with the laments of those who came before.
+From incorrect versions of ISL to the wrong C library being linked
+or the host linker being used, etc.
+The one commonality between all of these issues is the environment.
+Building GCC is difficult because isolating build environments is hard.
+
+In fact as of =v0.14.2= the zephyr SDK repository took down the build
+instructions and posted a sign that read
+"Building this is too complicated, don't worry about it."
+(I'm paraphrasing, but
+[not by much](https://github.com/zephyrproject-rtos/sdk-ng/tree/v0.14.2#build-process).)
+
+We will neatly side step all of these problems and not
+risk destroying or polluting our host system with garbage
+by using Guix to manage our environments for us.
+
+Our toolchain only requires the first pass compiler because
+newlib(-nano) is statically linked and introduced to the toolchain
+by normal package composition.
+
+
+# Defining the Packages
+
+All of the base packages are defined in `zephyr/packages/zephyr.scm`.
+Zephyr modules (coming soon!) are defined in `zephyr/packages/zephyr-xyz.scm`,
+following the pattern of other module systems implemented by Guix.
+
+## Binutils
+
+First thing we need to build is the `arm-zephyr-eabi` binutils.
+This is very easy in Guix.
+
+```scheme
+(define-public arm-zephyr-eabi-binutils
+ (let ((xbinutils (cross-binutils "arm-zephyr-eabi")))
+ (package (inherit xbinutils)
+ (name "arm-zephyr-eabi-binutils")
+ (version "2.38")
+ (source
+ (origin (method git-fetch)
+ (uri (git-reference
+ (url "https://github.com/zephyrproject-rtos/binutils-gdb")
+ (commit "6a1be1a6a571957fea8b130e4ca2dcc65e753469")))
+ (file-name (git-file-name name version))
+ (sha256 (base32 "0ylnl48jj5jk3jrmvfx5zf8byvwg7g7my7jwwyqw3a95qcyh0isr"))))
+ (arguments
+ `(#:tests? #f
+ ,@(substitute-keyword-arguments (package-arguments xbinutils)
+ ((#:configure-flags flags)
+ `(cons "--program-prefix=arm-zephyr-eabi-" ,flags)))))
+ (native-inputs
+ (append
+ (list texinfo
+ bison
+ flex
+ gmp
+ dejagnu)
+ (package-native-inputs xbinutils)))
+ (home-page "https://zephyrproject.org")
+ (synopsis "binutils for zephyr RTOS"))))
+```
+
+The function `cross-binutils` returns a package which has been
+configured for the given gnu triplet. We simply inherit that package
+and replace the source.
+The zephyr build system expects the binutils to be prefixed with
+`arm-zephyr-eabi-` which is accomplished by adding another flag to the
+`#:configure-flags` argument.
+
+We can test our package definition using the =-L= flag with `guix build`
+to add our packages.
+
+```sh
+guix build -L guix-zephyr zephyr-binutils
+
+/gnu/store/...-zephyr-binutils-2.38
+```
+
+This directory contains the results of `make install`.
+
+## GCC sans libc
+
+This one is a bit more involved. Don't be afraid!
+This version of GCC wants ISL version 0.15. It's easy enough
+to make that happen. Inherit the current version of ISL and swap
+out the source and update the version. For most packages the build process doesn't
+change that much between versions.
+
+```scheme
+(define-public isl-0.15
+ (package
+ (inherit isl)
+ (version "0.15")
+ (source (origin
+ (method url-fetch)
+ (uri (list (string-append "mirror://sourceforge/libisl/isl-"
+ version ".tar.gz")))
+ (sha256
+ (base32
+ "11vrpznpdh7w8jp4wm4i8zqhzq2h7nix71xfdddp8xnzhz26gyq2"))))))
+
+```
+
+Like the binutils, there is a function for creating cross-gcc packages.
+This one accepts keywords specifying which binutils and libc to use.
+If libc isn't given (like here), gcc is configured with many options disabled
+to facilitate being built without libc. Therefore we need to add the extra options
+we want (I got them from the SDK configuration scripts on the
+[sdk github](https://github.com/zephyrproject-rtos/sdk-ng) as well as the
+commits to use for each of the tools).
+
+
+```scheme
+(define-public gcc-arm-zephyr-eabi-12
+ (let ((xgcc (cross-gcc "arm-zephyr-eabi"
+ #:xbinutils zephyr-binutils)))
+ (package
+ (inherit xgcc)
+ (version "12.1.0")
+ (source (origin (method git-fetch)
+ (uri (git-reference
+ (url "https://github.com/zephyrproject-rtos/gcc")
+ (commit "0218469df050c33479a1d5be3e5239ac0eb351bf")))
+ (file-name (git-file-name (package-name xgcc) version))
+ (sha256
+ (base32 "1s409qmidlvzaw1ns6jaanigh3azcxisjplzwn7j2n3s33b76zjk"))
+ (patches
+ (search-patches "gcc-12-cross-environment-variables.patch"
+ "gcc-cross-gxx-include-dir.patch"))))
+ (native-inputs
+ (modify-inputs (package-native-inputs xgcc)
+ ;; Get rid of stock ISL
+ (delete "isl")
+ ;; Add additional dependencies that xgcc doesn't have
+ ;; including our special ISL
+ (prepend flex
+ perl
+ python-3
+ gmp
+ isl-0.15
+ texinfo
+ python
+ mpc
+ mpfr
+ zlib)))
+ (arguments
+ (substitute-keyword-arguments (package-arguments xgcc)
+ ((#:phases phases)
+ `(modify-phases ,phases
+ (add-after 'unpack 'fix-genmultilib
+ (lambda _
+ (substitute# "gcc/genmultilib"
+ (("#!/bin/sh") (string-append "#!" (which "sh"))))
+ #t))
+
+ (add-after 'set-paths 'augment-CPLUS_INCLUDE_PATH
+ (lambda# (#:key inputs #:allow-other-keys)
+ (let ((gcc (assoc-ref inputs "gcc")))
+ ;; Remove the default compiler from CPLUS_INCLUDE_PATH to
+ ;; prevent header conflict with the GCC from native-inputs.
+ (setenv "CPLUS_INCLUDE_PATH"
+ (string-join
+ (delete (string-append gcc "/include/c++")
+ (string-split (getenv "CPLUS_INCLUDE_PATH")
+ #\:))
+ ":"))
+ (format #t
+ "environment variable `CPLUS_INCLUDE_PATH' changed to `a`%"
+ (getenv "CPLUS_INCLUDE_PATH"))
+ #t)))))
+
+ ((#:configure-flags flags)
+ ;; The configure flags are largely identical to the flags used by the
+ ;; "GCC ARM embedded" project.
+ `(append (list "--enable-multilib"
+ "--with-newlib"
+ "--with-multilib-list=rmprofile"
+ "--with-host-libstdcxx=-static-libgcc -Wl,-Bstatic,-lstdc++,-Bdynamic -lm"
+ "--enable-plugins"
+ "--disable-decimal-float"
+ "--disable-libffi"
+ "--disable-libgomp"
+ "--disable-libmudflap"
+ "--disable-libquadmath"
+ "--disable-libssp"
+ "--disable-libstdcxx-pch"
+ "--disable-nls"
+ "--disable-shared"
+ "--disable-threads"
+ "--disable-tls"
+ "--with-gnu-ld"
+ "--with-gnu-as"
+ "--enable-initfini-array")
+ (delete "--disable-multilib" ,flags)))))
+ (native-search-paths
+ (list (search-path-specification
+ (variable "CROSS_C_INCLUDE_PATH")
+ (files '("arm-zephyr-eabi/include")))
+ (search-path-specification
+ (variable "CROSS_CPLUS_INCLUDE_PATH")
+ (files '("arm-zephyr-eabi/include"
+ "arm-zephyr-eabi/c++"
+ "arm-zephyr-eabi/c++/arm-zephyr-eabi")))
+ (search-path-specification
+ (variable "CROSS_LIBRARY_PATH")
+ (files '("arm-zephyr-eabi/lib")))))
+ (home-page "https://zephyrproject.org")
+ (synopsis "GCC for zephyr RTOS"))))
+```
+
+This GCC can be built like so.
+
+```sh
+guix build -L guix-zephyr gcc-cross-sans-libc-arm-zephyr-eabi
+
+/gnu/store/...-gcc-cross-sans-libc-arm-zephyr-eabi-12.1.0-lib
+/gnu/store/...-gcc-cross-sans-libc-arm-zephyr-eabi-12.1.0
+
+```
+Great! We now have our stage-1 compiler.
+
+## Newlib(-nano)
+
+The newlib package package is quite straight forward (relatively).
+It is mostly adding in the relevent configuration flags and patching
+the files the `patch-shebangs` phase missed.
+
+```scheme
+(define-public zephyr-newlib
+ (package
+ (name "zephyr-newlib")
+ (version "3.3")
+ (source (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url "https://github.com/zephyrproject-rtos/newlib-cygwin")
+ (commit "4e150303bcc1e44f4d90f3489a4417433980d5ff")))
+ (sha256
+ (base32 "08qwjpj5jhpc3p7a5mbl7n6z7rav5yqlydqanm6nny42qpa8kxij"))))
+ (build-system gnu-build-system)
+ (arguments
+ `(#:out-of-source? #t
+ #:configure-flags '("--target=arm-zephyr-eabi"
+ "--enable-newlib-io-long-long"
+ "--enable-newlib-io-float"
+ "--enable-newlib-io-c99-formats"
+ "--enable-newlib-retargetable-locking"
+ "--enable-newlib-lite-exit"
+ "--enable-newlib-multithread"
+ "--enable-newlib-register-fini"
+ "--enable-newlib-extra-sections"
+ "--disable-newlib-wide-orient"
+ "--disable-newlib-fseek-optimization"
+ "--disable-newlib-supplied-syscalls"
+ "--disable-newlib-target-optspace"
+ "--disable-nls")
+ #:phases
+ (modify-phases %standard-phases
+ (add-after 'unpack 'fix-references-to-/bin/sh
+ (lambda _
+ (substitute# '("libgloss/arm/cpu-init/Makefile.in"
+ "libgloss/arm/Makefile.in"
+ "libgloss/libnosys/Makefile.in"
+ "libgloss/Makefile.in")
+ (("/bin/sh") (which "sh")))
+ #t)))))
+ (native-inputs
+ `(("xbinutils" ,zephyr-binutils)
+ ("xgcc" ,gcc-arm-zephyr-eabi-12)
+ ("texinfo" ,texinfo)))
+ (home-page "https://www.sourceware.org/newlib/")
+ (synopsis "C library for use on embedded systems")
+ (description "Newlib is a C library intended for use on embedded
+systems. It is a conglomeration of several library parts that are easily
+usable on embedded products.")
+ (license (license:non-copyleft
+ "https://www.sourceware.org/newlib/COPYING.NEWLIB"))))
+```
+
+And the build.
+
+```sh :exports both
+$ guix build -L guix-zephyr zephyr-newlib
+
+/gnu/store/...-zephyr-newlib-3.3
+```
+
+## Complete Toolchain
+
+_Mostly_ complete. libstdc++ does not build because
+`arm-zephyr-eabi` is not `arm-none-eabi` so a dynamic link check is
+performed/failed. I cannot figure out how crosstool-ng handles this.
+
+Now that we've got the individual tools it's time to create our complete toolchain.
+For this we need to do some package transformations.
+Because these transformations are going to have to be done for every combination of
+binutils/gcc/newlib it is best to create a function which we can reuse for every version
+of the SDK.
+
+```scheme :exports code
+ (define (arm-zephyr-eabi-toolchain xgcc newlib version)
+ "Produce a cross-compiler zephyr toolchain package with the compiler XGCC and the C
+ library variant NEWLIB."
+ (let ((newlib-with-xgcc (package (inherit newlib)
+ (native-inputs
+ (alist-replace "xgcc" (list xgcc)
+ (package-native-inputs newlib))))))
+ (package
+ (name (string-append "arm-zephyr-eabi"
+ (if (string=? (package-name newlib-with-xgcc)
+ "newlib-nano")
+ "-nano" "")
+ "-toolchain"))
+ (version version)
+ (source #f)
+ (build-system trivial-build-system)
+ (arguments
+ '(#:modules ((guix build union)
+ (guix build utils))
+ #:builder
+ (begin
+ (use-modules (ice-9 match)
+ (guix build union)
+ (guix build utils))
+ (let ((out (assoc-ref %outputs "out")))
+ (mkdir-p out)
+ (match %build-inputs
+ (((names . directories) ...)
+ (union-build (string-append out "/arm-zephyr-eabi")
+ directories)
+ #t))))))
+ (inputs
+ `(("binutils" ,zephyr-binutils)
+ ("gcc" ,xgcc)
+ ("newlib" ,newlib-with-xgcc)))
+ (synopsis "Complete GCC tool chain for ARM zephyrRTOS development")
+ (description "This package provides a complete GCC tool chain for ARM
+ bare metal development with zephyr rtos. This includes the GCC arm-zephyr-eabi cross compiler
+ and newlib (or newlib-nano) as the C library. The supported programming
+ language is C.")
+ (home-page (package-home-page xgcc))
+ (license (package-license xgcc)))))
+```
+
+This function creates a special package which consists of the toolchain
+in a special directory hierarchy, i.e `arm-zephyr-eabi/`.
+Our complete toolchain definition looks like this.
+
+```scheme
+(define-public arm-zephyr-eabi-toolchain-0.15.0
+ (arm-zephyr-eabi-toolchain
+ gcc-arm-zephyr-eabi-12
+ zephyr-newlib
+ "0.15.0"))
+```
+
+To build:
+
+```sh
+guix build -L guix-zephyr arm-zephyr-eabi-toolchain
+/gnu/store/...-arm-zephyr-eabi-toolchain-0.15.0
+```
+
+# Integrating with Zephyr Build System
+
+Zephyr uses CMake as its build system. It contains numerous CMake files in both the so-called `ZEPHYR_BASE`,
+the zephyr source code repository, as well as a handful in the SDK which help select the correct toolchain
+for a given board.
+
+There are standard locations the build system will look for the SDK. We are not using any of them.
+Our SDK lives in the store, immutable forever.
+According to [[https://docs.zephyrproject.org/latest/develop/west/without-west.html][this]] the variable `ZEPHYR_SDK_INSTALL_DIR` needs to point to our custom spot.
+
+We also need to grab the cmake files from the
+[repository](https://github.com/zephyrproject-rtos/sdk-ng)
+and create a file, `sdk_version`, which
+contains the version string `ZEPHYR_BASE` uses to find a compatible SDK.
+
+Along with the SDK proper we need to include a number of
+python packages required by the build system.
+
+```scheme
+(define-public zephyr-sdk
+ (package
+ (name "zephyr-sdk")
+ (version "0.15.0")
+ (home-page "https://zephyrproject.org")
+ (source (origin (method git-fetch)
+ (uri (git-reference
+ (url "https://github.com/zephyrproject-rtos/sdk-ng")
+ (commit "v0.15.0")))
+ (file-name (git-file-name name version))
+ (sha256 (base32 "04gsvh20y820dkv5lrwppbj7w3wdqvd8hcanm8hl4wi907lwlmwi"))))
+ (build-system trivial-build-system)
+ (arguments
+ `(#:modules ((guix build union)
+ (guix build utils))
+ #:builder
+ (begin
+ (use-modules (guix build union)
+ (ice-9 match)
+ (guix build utils))
+ (let# ((out (assoc-ref %outputs "out"))
+ (cmake-scripts (string-append (assoc-ref %build-inputs "source")
+ "/cmake"))
+ (sdk-out (string-append out "/zephyr-sdk-0.15.0")))
+ (mkdir-p out)
+
+ (match (assoc-remove! %build-inputs "source")
+ (((names . directories) ...)
+ (union-build sdk-out directories)))
+
+ (copy-recursively cmake-scripts
+ (string-append sdk-out "/cmake"))
+
+ (with-directory-excursion sdk-out
+ (call-with-output-file "sdk_version"
+ (lambda (p)
+ (format p "0.15.0")))
+ #t)))))
+ (propagated-inputs
+ (list
+ arm-zephyr-eabi-toolchain-0.15.0
+ zephyr-binutils
+ dtc
+ python-3
+ python-pyelftools
+ python-pykwalify
+ python-pyyaml
+ python-packaging))
+ (native-search-paths
+ (list (search-path-specification
+ (variable "ZEPHYR_SDK_INSTALL_DIR")
+ (files '("")))))
+ (synopsis "SDK for zephyrRTOS")
+ (description "zephyr-sdk contains bundles a complete gcc toolchain as well
+as host tools like dtc, openocd, qemu, and required python packages.")
+ (license license:apsl2)))
+```
+
+# Testing
+
+In order to test we will need an environment with the SDK installed.
+We can take advantage of `guix shell` to avoid installing test packages into
+our home environment. This way if it causes problems we can just exit the shell
+and try again.
+
+```sh
+guix shell -L guix-zephyr zephyr-sdk cmake ninja git
+```
+
+`ZEPHYR_BASE` can be cloned into a temporary workspace to test our toolchain functionality.
+(For now. Eventually we will need to create a package for `zephyr-base` that
+our guix zephyr-build-system can use.)
+
+```sh
+mkdir /tmp/zephyr-project
+cd /tmp/zephyr-project
+git clone https://github.com/zephyrproject-rtos/zephyr
+export ZEPHYR_BASE=/tmp/zephyr-project/zephyr
+```
+
+In order to build for the test board (k64f in this case) we need to get a hold of the vendor
+Hardware Abstraction Layers and CMSIS.
+(These will also need to become guix packages to allow the build system to compose modules).
+
+```sh
+git clone https://github.com/zephyrproject-rtos/hal_nxp &&
+git clone https://github.com/zephyrproject-rtos/cmsis
+```
+
+To inform the build system about this module we pass it in with `-DZEPHYR_MODULES=` which is
+a semicolon separated list of paths containing a module.yml file.
+
+To build the hello world sample we use the following incantation.
+```sh
+cmake -Bbuild $ZEPHYR_BASE/samples/hello_world \
+ -GNinja \
+ -DBOARD=frdm_k64f \
+ -DBUILD_VERSION=3.1.0 \
+ -DZEPHYR_MODULES="/tmp/zephyr-project/hal_nxp;/tmp/zephyr-project/cmsis" \
+ && ninja -Cbuild
+```
+
+If everything is set up correctly we will end up with a =./build=
+directory with all our build artifacts. The SDK is installed correctly!
+
+# Conclusion
+
+A customized cross toolchain is one of the most difficult pieces of
+software to build. Using Guix, we do not need to be afraid of the
+complexity! We can fiddle with settings, swap out components, and do
+the most brain dead things to our environments without a care in the
+world. Just exit the environment and it's like it never happened at
+all.
+
+It highlights one of my favorite aspects of Guix, every package is a
+working reference design for you to modify and learn from.
\ No newline at end of file
--
2.39.1
--=-=-=
Content-Type: text/plain
Content-Disposition: attachment; filename=custom-build-systems-with-guix.org
Content-Description: build-systems
#+title: Custom Build Systems with Guix
#+author: Mitchell Schmeisser <mitchellschmeisser@librem.one>
All code is available [[https://github.com/paperclip4465/guix-zephyr][here]].
* Overview
Before software can be deployed, it needs to be built.
In Guix each package must specify a so-called ~build-system~ which
knows how to bring a package from its inputs to deployment.
Build systems are responsible for setting up the environment and performing
build actions with in that environment. The most ubiquitous of these is the
[[https://www.gnu.org/software/automake/manual/html_node/GNU-Build-System.html][~gnu-build-system~]]. Packages using this build system are build using the usual
~./configure && make && make install~ process.
Any package can alter its build system by removing some steps or
adding extra ones. This is extremely common and almost every package
makes some adjustment to the build process. A ~build-system~ in Guix
hides away some of the common configuration choices. For example there is
no need to specify ~make~ or ~gcc~ as native inputs when using the ~gnu-build-system~
because they are added implicitly when our package is gets lowered into a /bag/.
* Anatomy of a Guix Build System
The job of a build system is to compile our /packages/ into /bags/.
Bags are a lower level representation of a package without all the bells and whistles[fn:1],
they are then used by the build daemon to create an isolated environment suitable to
run our /build phases/ in.
Below is how guix defines a build system.
It's surprisingly simple with just three items, two of which are for human consumption.
#+BEGIN_SRC scheme
(define-record-type* <build-system> build-system make-build-system
build-system?
(name build-system-name) ; symbol
(description build-system-description) ; short description
(lower build-system-lower)) ; args ... -> bags
#+END_SRC
The last field ~lower~ is a function which takes the list of arguments
given in the ~(package ... (arguments ...))~ field.
The keyword arguments that we are allowed to supply when writing the
package.
** Code Strata
Guix builds are implemented in two parts.
1. Code that compiles =packages->derivations=.
Derivations are the language the Guix Daemon speaks.
They describe everything about how to /derive/ our package
from the inputs to the environment and all the code on
how to drive the build tools.
2. Code the builder actually runs in the isolation to produce the package from it's inputs.
The code which produces the above derivations is run on the dirty,
unreproducible user environment.
Code that runs in the host environment /stages/ code which will run in isolation.
This is where G-Expressions really shine.
They provide the syntax to describe this relationship.
** Build Phases
All programs are built more or less the same way.
1. Unpack the source code.
Whether it's tarball or a version controlled repository we must
copy it's source tree into our build environment.
2. Patch the shebangs.
Many projects contain scripts written to aid the build process.
In linux executable scripts can contain a so-called /shebang/
which contains an absolute path to the program which is meant to
interpret it. E.g. ~#!/bin/sh~. Most distrobutions provide a
POSIX compliant shell interpreter at this location. Not on ours.
Our ~sh~ is yet another component in the store so all of these files must
be patched to point to the new location which is only known at
build time. We cannot rely on our =PATH= to store this information
because the kernel does not respect such things.
3. Configure the build.
Whether it's the autotools or CMake or whatever, if you're relying
on tools from the host system, you have a step which enables the
host system to tell you where to find those tools.
4. Patch the generated shebangs.
Sometimes the configure phases creates scripts which run during the build phase,
these often contain references to =/bin/sh= and so must be patched as well.
5. Build!
That's right folks we are off to the races and the program is building.
Usually this takes the form of a call to ~make~ with a handful of
flags and last minute definitions, but there are other more /exotic/[fn:5] methods.
6. Test.
Now that we have our software built we should test it.
Not all packages have tests but what can do you?
8. Install.
Now that we've verified everything works as expected it's time to run
~make install~ or equivalent. This phase copies our build artifacts into
their final resting place in the store.
7. Validate the Installation.
Here we can our outputs and make sure they don't contain any component
paths which are not specified by the package inputs. That would lead
to incomplete deployment and would be "bad".
There are more steps but they are not universally applicable.
Of course no generic model such as this captures all the chaos of the
world and every package has exceptions to it.
Guix implements /build phases/ as a list of lambdas.
As such our package can modify this list and add/delete/replace
lambdas as they require. It is so common Guix provides a syntax
for manipulating build phases, ~modify-phases~.
A build system contains a default set of phases called ~%standard-phases~.
Every build system starts with the ~gnu-build-system~ ~%standard-phases~
and modifies it to their needs. It is then provided to the applications
using that build system.
The ~cmake-build-system~ is nearly identical to the ~gnu-build-system~
except two phases.
#+BEGIN_SRC scheme
;;
;; Excerpt from `guix/build/cmake-build-system.scm`
;;
(define %standard-phases
;; Everything is as with the GNU Build System except for the `configure'
;; and 'check' phases.
(modify-phases gnu:%standard-phases
(delete 'bootstrap)
(replace 'check check)
(replace 'configure configure)))
#+END_SRC
* The Zephyr Build System
Zephyr uses ~cmake~ to perform /physical component composition/.
It searches the filesystem and generates scripts which the toolchain will
use to successfully combine those components into a firmware image.
The fact that Zephyr provides this mechanism is one reason I chose to
target zephyr in the first place.
This separation of projects in an embedded context is a really great thing.
It brings many of the advantages that it brings to the Linux world such as
code re-use, smaller binaries, more efficient cache/RAM usage, etc.
It also allows us to work as independent groups and compose
contributions from many teams.
It also brings all of the complexity. Suddenly all of the problems
that plague traditional deployment now apply to our embedded
system. The fact that the libraries are statically linked at compile
time instead of dynamically at runtime is simply an implementation detail.
We now have dependencies which we must track and compose in the correct environments.
The Zephyr Project offers a tool to help manage this new complexity, [[https://docs.zephyrproject.org/latest/develop/west/index.html][The West Meta-tool]]!
To call it a "meta-tool" is an abuse of language. It's not even meta
enough to bootstrap itself and relies on [[https://docs.zephyrproject.org/latest/develop/getting_started/index.html][other package managers]] to get
started.
In fact, it isn't even suitable for managing multiple embedded projects as a composition!
The project recommends [[https://docs.zephyrproject.org/latest/build/sysbuild/index.html][Yet Another Tool]] for this very common use case.
In fact ~west~ does not really bring anything new to the table and we
can replace it in the same way we can replace every other package
manager.[fn:4] *PUTTING SOFTWARE ON THE SYSTEM IS THE JOB OF THE(1)
PACKAGE MANAGER!*.
Let's see what it takes to actually provide a meta-tool and bootstrap
these complex embedded systems from the ground up in a flexible, composable,
and /reproducible/ way.
** Why not use ~cmake-build-system~?
A fair question! Zephyr's CMake scripts require additional information
about the build to be provided at configure time. Most tedius for
zephyr-packages is the =ZEPHYR_MODULES= variable which must be formatted and
passed to CMake on the command line[fn:6]
** Host Side
Our job at this level is to take packages described using the ~package~ syntax and
lower it into a derivation that the daemon can understand.
Here is the derivation for building hello world for the =frdm_k64f=, hashes removed for brevity.
The ~package~ syntax provides a human friendly veneer over this garbage.
#+BEGIN_SRC
Derive
([("debug","/gnu/store/...-zephyr-hello-world-newlib-frdm-k64f-3.1.0-0.zephyr--debug","","")
,("out","/gnu/store/...-zephyr-hello-world-newlib-frdm-k64f-3.1.0-0.zephyr-","","")]
,[("/gnu/store/...-newlib-nano-3.3.drv",["out"])
,("/gnu/store/...-hal-nxp-3.1.0-0.708c958.drv",["out"])
,("/gnu/store/...-make-4.3.drv",["out"])
,("/gnu/store/...-findutils-4.8.0.drv",["out"])
,("/gnu/store/...-grep-3.6.drv",["out"])
,("/gnu/store/...-sed-4.8.drv",["out"])
,("/gnu/store/...-ld-wrapper-0.drv",["out"])
,("/gnu/store/...-bash-minimal-5.1.8.drv",["out"])
,("/gnu/store/...-hal-cmsis-5.8.0-0.093de61.drv",["out"])
,("/gnu/store/...-gawk-5.1.0.drv",["out"])
,("/gnu/store/...-gzip-1.10.drv",["out"])
,("/gnu/store/...-python-six-1.16.0.drv",["out"])
,("/gnu/store/...-zephyr-3.1.0-0.zephyr--checkout.drv",["out"])
,("/gnu/store/...-linux-libre-headers-5.10.35.drv",["out"])
,("/gnu/store/...-python-3.9.9.drv",["out"])
,("/gnu/store/...-diffutils-3.8.drv",["out"])
,("/gnu/store/...-arm-zephyr-eabi-nano-toolchain-12.1.0.drv",["out"])
,("/gnu/store/...-python-pyelftools-0.28.drv",["out"])
,("/gnu/store/...-guile-3.0.7.drv",["out"])
,("/gnu/store/...-python-dateutil-2.8.2.drv",["out"])
,("/gnu/store/...-patch-2.7.6.drv",["out"])
,("/gnu/store/...-gcc-10.3.0.drv",["out"])
,("/gnu/store/...-bzip2-1.0.8.drv",["out"])
,("/gnu/store/...-dtc-1.6.1.drv",["out"])
,("/gnu/store/...-gcc-cross-sans-libc-arm-zephyr-eabi-12.1.0.drv",["out"])
,("/gnu/store/...-cmake-minimal-3.21.4.drv",["out"])
,("/gnu/store/...-python-pyyaml-6.0.drv",["out"])
,("/gnu/store/...-python-packaging-21.3.drv",["out"])
,("/gnu/store/...-arm-zephyr-eabi-binutils-2.38.drv",["out"])
,("/gnu/store/...-glibc-2.33.drv",["out","static"])
,("/gnu/store/...-qemu-7.2.0.drv",["out"])
,("/gnu/store/...-ninja-1.10.2.drv",["out"])
,("/gnu/store/...-tar-1.34.drv",["out"])
,("/gnu/store/...-xz-5.2.5.drv",["out"])
,("/gnu/store/...-binutils-2.37.drv",["out"])
,("/gnu/store/...-python-pykwalify-1.7.0.drv",["out"])
,("/gnu/store/...-zephyr-3.1.0-0.zephyr-.drv",["out"])
,("/gnu/store/...-glibc-utf8-locales-2.33.drv",["out"])
,("/gnu/store/...-gdb-arm-zephyr-eabi-12.1.drv",["out"])
,("/gnu/store/...-module-import-compiled.drv",["out"])
,("/gnu/store/...-file-5.39.drv",["out"])
,("/gnu/store/...-python-pyparsing-3.0.6.drv",["out"])
,("/gnu/store/...-python-docopt-0.6.2.drv",["out"])
,("/gnu/store/...-arm-zephyr-eabi-sdk-0.15.0.drv",["out"])
,("/gnu/store/...-coreutils-8.32.drv",["out"])]
,["/gnu/store/...-zephyr-hello-world-newlib-frdm-k64f-3.1.0-0.zephyr--builder","/gnu/store/...-module-import"]
,"x86_64-linux","/gnu/store/...-guile-3.0.7/bin/guile",["--no-auto-compile"
,"-L","/gnu/store/...-module-import"
,"-C","/gnu/store/...-module-import-compiled"
,"/gnu/store/...-zephyr-hello-world-newlib-frdm-k64f-3.1.0-0.zephyr--builder"]
,[("debug","/gnu/store/...-zephyr-hello-world-newlib-frdm-k64f-3.1.0-0.zephyr--debug")
,("out","/gnu/store/...-zephyr-hello-world-newlib-frdm-k64f-3.1.0-0.zephyr-")])
#+END_SRC
*** Lowering packages to bags
We must provide the build system with a function which the following lambda form.
#+BEGIN_SRC scheme
(lambda* (name #:key #:allow-other-keys) ...)
#+END_SRC
This means it takes one required argument ~name~ and any amount of keys.
Individual procedures can specify keys they are interested in such as
~inputs~ or ~outputs~.
Which keys are ultimately supported is defined by our ~lower~ function
and our /build phases/.
#+BEGIN_SRC scheme
;; Use module-ref instead of referencing the variables directly
;; to avoid circular dependencies.
(define %zephyr-build-system-modules
`((mfs build zephyr-build-system)
,@%cmake-build-system-modules))
(define default-zephyr-base
(module-ref (resolve-interface '(mfs packages zephyr))
'zephyr))
(define default-zephyr-sdk
(module-ref (resolve-interface '(mfs packages zephyr))
'arm-zephyr-eabi-sdk))
(define default-ninja
(module-ref (resolve-interface '(gnu packages ninja))
'ninja))
(define default-cmake
(module-ref (resolve-interface '(gnu packages cmake))
'cmake-minimal))
(define* (lower name
#:key source inputs native-inputs outputs system target
(zephyr default-zephyr-base)
(sdk default-zephyr-sdk)
(ninja default-ninja)
(cmake default-cmake)
#:allow-other-keys
#:rest arguments)
"Return a bag for NAME."
(define private-keywords `(#:zephyr #:inputs #:native-inputs #:target))
(bag
(name name)
(system system)
(target target)
(build-inputs `(,@(if source `(("source" ,source)) '())
,@`(("cmake" ,cmake))
,@`(("zephyr-sdk" ,sdk))
,@`(("zephyr" ,zephyr))
,@`(("ninja" ,ninja))
,@native-inputs
,@(standard-packages)))
;; Inputs need to be available at build time
;; since everything is statically linked.
(host-inputs inputs)
(outputs outputs)
(build zephyr-build)
(arguments (strip-keyword-arguments private-keywords arguments))))
#+END_SRC
Here our ~lower~ function provides default values for the packages
every zephyr package needs, the SDK, cmake, and zephyr-base and adds
them to the build-inputs.
Notice we also strip out some keywords which do not get passed to the
build function because they get included as part of the broader
abstractions the build system provides.
*** Lowering Bags to Derivations
Here is the definition for the actual build procedure. There is a lot
of abstract fuckery going on here so do not worry if you don't understand it.
#+BEGIN_SRC scheme
(define* (zephyr-build name inputs
#:key guile source
board
(outputs '("out")) (configure-flags ''())
(search-paths '())
(make-flags ''())
(out-of-source? #t)
(tests? #f)
(test-target "test")
(parallel-build? #t) (parallel-tests? #t)
(validate-runpath? #f)
(patch-shebangs? #t)
(phases '%standard-phases)
(system (%current-system))
(substitutable? #t)
(imported-modules %zephyr-build-system-modules)
;; The modules referenced here contain code
;; which will be staged in the build environment with us.
;; Our build gexp down below will only be able to access this code
;; and we must be careful not to reference anything else.
(modules '((mfs build zephyr-build-system)
(guix build utils))))
"Build SOURCE using CMAKE, and with INPUTS. This assumes that SOURCE
provides a 'CMakeLists.txt' file as its build system."
;; This is the build gexp. It handles staging values from our host
;; system into code that our build system can run.
(define build
(with-imported-modules imported-modules
#~(begin
(use-modules #$@(sexp->gexp modules))
#$(with-build-variables inputs outputs
#~(zephyr-build #:source #+source
#:system #$system
#:outputs %outputs
#:inputs %build-inputs
#:board #$board
#:search-paths '#$(sexp->gexp
(map search-path-specification->sexp
search-paths))
#:phases #$(if (pair? phases)
(sexp->gexp phases)
phases)
#:configure-flags #$(if (pair? configure-flags)
(sexp->gexp configure-flags)
configure-flags)
#:make-flags #$make-flags
#:out-of-source? #$out-of-source?
#:tests? #$tests?
#:test-target #$test-target
#:parallel-build? #$parallel-build?
#:parallel-tests? #$parallel-tests?
#:validate-runpath? #$validate-runpath?
#:patch-shebangs? #$patch-shebangs?
#:strip-binaries? #f)))))
;; look we even have monads like the cool kids!
(mlet %store-monad ((guile (package->derivation (or guile (default-guile))
system #:graft? #f)))
(gexp->derivation name build
#:system system
#:target #f
#:graft? #f
#:substitutable? substitutable?
#:guile-for-build guile)))
#+END_SRC
Finally we define our build system which the package definitions can reference.
#+BEGIN_SRC scheme
(define zephyr-build-system
(build-system
(name 'zephyr)
(description "The standard Zephyr build system")
(lower lower)))
#+END_SRC
Easy right?
** Build Side
The build side is not as complex as you might initially expect.
Our build system is almost exactly the same as the CMake build system except our configure phase
passes different values to CMake. Our job is much easier.
*** Locating Modules
Zephyr CMake requires the zephyr /modules/ which are needed for the
build to be supplied on the command line.
Unfortunately for us the [[https://docs.zephyrproject.org/latest/develop/application/index.html#important-build-vars][documentation]] is wrong and the =ZEPHYR_MODULES=
is not honored. Thus we must implement some other solution for
locating modules until they fix that.
*Input Scanning* - Lucky for us we are keeping detailed information
about our dependencies. It is a simple matter to write a file tree
walker which collects all the zephyr modules in our inputs.
#+BEGIN_SRC scheme
(define* (find-zephyr-modules directories)
"Return the list of directories containing zephyr/module.yml found
under DIRECTORY, recursively. Return the empty list if DIRECTORY is
not accessible."
(define (module-directory file)
(dirname (dirname file)))
(define (enter? name stat result)
;; Skip version control directories.
(not (member (basename name) '(".git" ".svn" "CVS"))))
(define (leaf name stat result)
;; Add module root directory to results
(if (and (string= "module.yml" (basename name))
(string= "zephyr" (basename (dirname name))))
(cons (module-directory name) result)
result))
(define (down name stat result) result)
(define (up name stat result) result)
(define (skip name stat result) result)
(define (find-modules directory)
(file-system-fold enter? leaf down up skip error
'() (canonicalize-path directory)))
(append-map find-modules directories))
(define (zephyr-modules-cmake-argument modules)
"Return a proper CMake list from MODULES, a list of filepaths"
(format #f "-DZEPHYR_MODULES='~{~a~^;~}'" modules))
#+END_SRC
Here are two functions. The first one ~find-zephyr-modules~ walks a directory tree and
returns a list of every module it finds. The second one is just for
syntactic convenience when writing the CMake invokation.
From here we just need to provide alternate implementations of ~configure~ and ~install~[fn:7]
#+BEGIN_SRC scheme
(define* (configure #:key outputs (configure-flags '())
inputs (out-of-source? #t)
build-type
#:allow-other-keys)
"Configure the given package."
(let* ((out (assoc-ref outputs "out"))
(abs-srcdir (getcwd))
(srcdir (if out-of-source?
(string-append "../" (basename abs-srcdir))
".")))
(format #t "source directory: ~s (relative from build: ~s)~%"
abs-srcdir srcdir)
(when out-of-source?
(mkdir "../build")
(chdir "../build"))
(format #t "build directory: ~s~%" (getcwd))
;; this is required because zephyr tries to optimize
;; future calls to the build scripts by keep a cache.
(setenv "XDG_CACHE_HOME" (getcwd))
(let ((args `(,srcdir
,@(if build-type
(list (string-append "-DCMAKE_BUILD_TYPE="
build-type))
'())
;; enable verbose output from builds
"-DCMAKE_VERBOSE_MAKEFILE=ON"
,(zephyr-modules-cmake-argument
(find-zephyr-modules (map cdr inputs)))
,@configure-flags)))
(format #t "running 'cmake' with arguments ~s~%" args)
(apply invoke "cmake" args))))
(define* (install #:key outputs #:allow-other-keys)
(let* ((out (string-append (assoc-ref outputs "out") "/firmware"))
(dbg (string-append (assoc-ref outputs "debug") "/share/zephyr")))
(mkdir-p out)
(mkdir-p dbg)
(copy-file "zephyr/.config" (string-append dbg "/config"))
(copy-file "zephyr/zephyr.map" (string-append dbg "/zephyr.map"))
(copy-file "zephyr/zephyr.elf" (string-append out "/zephyr.elf"))
(copy-file "zephyr/zephyr.bin" (string-append out "/zephyr.bin"))))
;; Define new standard-phases
(define %standard-phases
(modify-phases cmake:%standard-phases
(replace 'configure configure)
(replace 'install install)))
;; Call cmake build with our new phases
(define* (zephyr-build #:key inputs (phases %standard-phases)
#:allow-other-keys #:rest args)
(apply cmake:cmake-build
#:inputs inputs #:phases phases args))
#+END_SRC
One thing to note is the "debug" output. This exists so we don't
retain references to our build environment and make the file system
closure huge. If you put all of the build outputs in the same store
path the deployment closure grows from 2MB[fn:8] to 833MB.
* Defining Zephyr Packages
Now that we have a proper build system, it's time to define some packages!
** Zephyr Base
Zephyr base contains the Zephyr source code.
It is equivalent (in my mind anyway) to the linux kernel in that packages
which target the linux kernel do not need to specify as much in their definitions.
The selection of operating system actually comes from the toolchain.
When we build linux packages the way we specify this is with the [[https://www.gnu.org/software/autoconf/manual/autoconf-2.65/html_node/Specifying-Target-Triplets.html][gnu triplet]].
When we select the ~arm-linux-gnueabihf~ we are specifying our operating system.
It is the same for Zephyr. When we build for zephyr we use the ~arm-zephyr-eabi~
toolchain. However, unlike linux applications, zephyr applications are
embedded firmware images and are generally statically linked.
Thus this package just consists of it's source code and is not compiled directly.
#+BEGIN_SRC scheme
(define-public zephyr
(let ((version "3.1.0")
(commit "zephyr-v3.1.0"))
(package
(name "zephyr")
(version version)
(home-page "https://zephyrproject.org")
(source (origin (method git-fetch)
(uri (git-reference
(url "https://github.com/zephyrproject-rtos/zephyr")
(commit commit)))
(file-name (git-file-name name version))
(sha256
(base32 "1yl5y9757xc3l037k3g1dynispv6j5zqfnzrhsqh9cx4qzd485lx"))
(patches
;; this patch makes this package work in a symlinked profile
(search-patches "zephyr-3.1-linker-gen-abs-path.patch"))))
(build-system copy-build-system)
(arguments
`(#:install-plan
'(("." "zephyr-workspace/zephyr"))
#:phases
(modify-phases %standard-phases
(add-after 'unpack 'patch-cmake-scripts
(lambda* _
(format #t "~a~&" (getcwd))
;; Some cmake scripts assume the presence of a
;; git repository in the source directory.
;; We will just hard-code that information now
(substitute* "CMakeLists.txt"
(("if\\(DEFINED BUILD_VERSION\\)" all)
(format #f "set(BUILD_VERSION \"~a-~a\")~&~a"
,version ,commit all))))))))
(propagated-inputs
(list python-3
python-pyelftools
python-pykwalify
python-pyyaml
python-packaging))
(native-search-paths
(list (search-path-specification
(variable "ZEPHYR_BASE")
(files '("zephyr-workspace/zephyr")))))
(synopsis "Source code for zephyr rtos")
(description "Zephyr rtos source code.")
(license license:apsl2))))
(define-public zephyr-3.2.0-rc3
(package (inherit zephyr)
(version "3.2.0-rc3")
(source (origin (method git-fetch)
(uri (git-reference
(url "https://github.com/zephyrproject-rtos/zephyr")
(commit "v3.2.0-rc3")))
(file-name (git-file-name "zephyr" version))
(sha256
(base32 "06ksd9zj4j19jq0zg3lms13jx0gxzjc41433zgb91cnd2cqmn5cb"))
(patches
(search-patches "zephyr-3.1-linker-gen-abs-path.patch"))))))
#+END_SRC
Here we use the ~copy-build-system~ which takes a list of source destination pairs.
In our case we just copy everything to the output directory, but not before patching some files
to accomodate our special environment.
While developing this I wanted to test some toolchain/board features
on the latest release of Zephyr. I included an example of that package
definition to show how we can easily accomodate side by side package
variants and experiment without breaking anything.[fn:9]
** Hello world
It's finally time to define some firmware!
Zephyr packages some examples in ~$ZEPHYR_BASE/samples~ including a
basic hello world. The k64 development board is already supported
by Zephyr so building the example is trivial.
In order to actually target the k64 we need to two modules, the NXP
hardware abstraction layer, and CMSIS.
Looking at ~$ZEPHYR_BASE/west.yml~ we can see the repositories
and commits which contain these modules. This is how West does
dependency management.
Defining these packages is not so bad.
*NOT SHOWN*: I made a ~zephyr-module-build-system~ as well which is
just the ~copy-build-system~ that mimics the default zephyr workspace layout
as provided by west. This way we do not need to provide the same install-plan for
every module we ever package. However as I use it more I think it doesn't really
provide much over just using the copy-build system.
#+BEGIN_SRC scheme
(define-public hal-cmsis
(package
(name "hal-cmsis")
(version "5.8.0")
(home-page "https://developer.arm.com/tools-and-software/embedded/cmsis")
(source (origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/zephyrproject-rtos/cmsis")
(commit "093de61c2a7d12dc9253daf8692f61f793a9254a")))
(file-name (git-file-name name version))
(sha256
(base32 "0f7cipnwllna7iknsnz273jkvrly16yr6wm4y2018i6njpqh67wi"))))
(build-system zephyr-module-build-system)
(arguments `(#:workspace-path "/modules/hal/cmsis"))
(synopsis "Zephyr module providing the Common Microcontroller
Software Interface Standard")
(description "Zephyr module providing the Common Microcontroller
Software Interface Standard")
(license license:apsl2)))
(define-public hal-nxp
(package
(name "hal-nxp")
(version "3.1.0")
(home-page "https://nxp.com")
(source (origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/zephyrproject-rtos/hal_nxp")
(commit "708c95825b0d5279620935a1356299fff5dfbc6e")))
(file-name (git-file-name name version))
(sha256
(base32 "1c0i26bpk6cyhr1q4af183jclfmxshv4d15i7k3cz7brzb12m8q1"))))
(build-system zephyr-module-build-system)
(arguments `(#:workspace-path "/modules/hal/nxp"))
(native-search-paths
(list (search-path-specification
(variable "ZEPHYR_MODULES")
(files `(,(string-append %zephyr-workspace-name module-path)))
(separator ";"))))
(synopsis "Zephyr module for NXP Hardware Abstraction Layer")
(description "Provides sources for NXP HAL zephyr module")
(license license:bsd-3)))
#+END_SRC
With these two modules defined we can write ~zephyr-hello-world-frdm-k64f~.
#+BEGIN_SRC scheme
(define-public zephyr-hello-world-frdm-k64f
(package
(name "zephyr-hello-world-frdm-k64f")
(version (package-version zephyr))
(home-page "https://zephyrproject.org")
(source (file-append (package-source zephyr)
"/samples/hello_world"))
(build-system zephyr-build-system)
(arguments
'(#:configure-flags '("-DBOARD=frdm_k64f")))
(outputs '("out" "debug"))
(inputs
(list hal-cmsis
hal-nxp))
(synopsis "Hello world example from Zephyr Project")
(description "Sample package for zephyr project")
(license license:apsl2)))
#+END_SRC
How elegant is that?
*** Building
Our above definition can be build using the following.
When testing package definitions I use the ~-L~ flag to point to the local
repository to load the new packages. I will be omitting that flag as
if I had ~guix pull~ed successfully from my channel.
#+BEGIN_SRC sh :exports both
guix build zephyr-hello-world-frdm-k64f
#+END_SRC
#+RESULTS:
| /gnu/store/...-zephyr-hello-world-frdm-k64f-3.1.0-debug |
| /gnu/store/...-zephyr-hello-world-frdm-k64f-3.1.0 |
This actually doesn't fully test our toolchain. The hello world
example, unless prodded not to, will use a zephyr provided
minimal implementation of the C library and will not link against
newlib.
#+BEGIN_SRC scheme
(define-public zephyr-hello-world-newlib-frdm-k64f
(package
(inherit zephyr-hello-world-frdm-k64f)
(name "zephyr-hello-world-newlib-frdm-k64f")
(arguments
(substitute-keyword-arguments (package-arguments zephyr-hello-world-frdm-k64f)
((#:configure-flags flags)
`(append
'("-DCONFIG_MINIMAL_LIBC=n"
"-DCONFIG_NEWLIB_LIBC=y")
,flags))))))
#+END_SRC
#+BEGIN_SRC sh :exports both
guix build zephyr-hello-world-newlib-frdm-k64f
#+END_SRC
#+RESULTS:
| /gnu/store/...-zephyr-hello-world-newlib-frdm-k64f-3.1.0-debug |
| /gnu/store/...-zephyr-hello-world-newlib-frdm-k64f-3.1.0 |
Woohoo!
* Further Musings
One thing I learned while going through the pains of getting this
working is that even though the components are "modular" there is
still a lot rigid interdependencies, especially on the zephyr base.
Just having two versions of Zephyr in the code base made component
composition fragile[fn:2].
Modules rely on specific features from the kernel
This is hidden from developers normally by west automagically walking
the `west.yml` of all of the declared dependencies recursively to
discover the graph.
* Footnotes
[fn:2] To be fair: 3.2.0 was a large and backwards breaking release.
[fn:9] In my opinion this is the most important finding of this research.
The ability to easily/safely experiment with a working reference design is extremely valuable.
[fn:8]2MB because we are putting the elf file in there which contains lots of extra information. The
binary itself is ~170KB.
[fn:7] The build scripts generated by zephyr do not contain an "install" target in the traditional sense.
[fn:6] I am going to implement a patch so that the zephyr cmake scripts can work correctly from
an environment variable. This will make component discovery translate well to development environments.
More on this later.
[fn:5] Read: bad
[fn:4] West is the way it is for a reason. It is very practical to design the package
manager in this way because it enables Windows users to access the
build environment. A more in-depth discussion on the material conditions which
lead to this or that design decision of the West tool is beyond the scope of this
document.
[fn:1] Makes sense since we are implementing them here.
--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Ludovic Court=C3=A8s <ludo@gnu.org> writes:
> Hello Mitchell,
>
> Mitchell Schmeisser <mitchellschmeisser@librem.one> skribis:
>
>> Here is a rough draft of my toolchain post.
>> I hope it can make an interesting contribution to the Guix literature.
>
> Yay, definitely!
>
>>>From 4f6c43091ffd67cdbc5f041e496f61bc8a06070e Mon Sep 17 00:00:00 2001
>> From: Mitchell Schmeisser <mitchellschmeisser@librem.one>
>> Date: Fri, 24 Feb 2023 13:02:05 -0500
>> Subject: [PATCH] website: Add custom toolchain blog post
>>
>> * website/posts/custom-toolchains-with-guix.md: New file.
>
> This looks great to me! It=E2=80=99s useful insight for anyone who might=
want
> to add a cross-compilation target to Guix or simply learn how this is
> implemented.
>
>> +++ b/website/posts/custom-toolchains-with-guix.md
>> @@ -0,0 +1,557 @@
>> +# Table of Contents
>> +
>> +1. [Overview](#org2633a51)
>> +2. [Anatomy of a toolchain](#orgc440e9e)
>> +3. [Bootstrapping a Toolchain](#orgd42b6c3)
>> +4. [Defining the Packages](#org55042c5)
>> + 1. [Binutils](#org67da1ec)
>> + 2. [GCC sans libc](#org82d6f83)
>> + 3. [Newlib(-nano)](#orgf6bafbc)
>> + 4. [Complete toolchain](#org052f2a2)
>> +5. [Integrating with Zephyr Build System](#orgc3f87f4)
>> + 1. [Testing](#org9f3c314)
>> +
>> +All code is available at [guix-zephyr](https://github.com/paperclip4465=
/guix-zephyr) channel.
>> +
>> +
>> +<a id=3D"org2633a51"></a>
>
> You can remove the table of contents and all the HTML snippets that
> pandoc added=E2=80=94I don=E2=80=99t think that works as expected with Ha=
unt/CommonMark.
>
>> +# Overview
>
> You can drop the heading.
>
>> +In order to deploy embedded software using Guix we first need to teach =
Guix
>> +how to build it. Since Guix bootstraps everything this means we must te=
ach Guix
>> +how to build our toolchain.
>> +
>> +The [Zephyr Project](https://zephyrproject.org) uses its own fork of GC=
C with custom configs for
>> +the architectures supported by the project.
>
> We want blog posts to be widely accessible so I would recommend
> providing more context in the intro. In particular, I=E2=80=99d suggest:
>
> 1. Mentioning that Guix supports cross-compilation (not everyone is
> aware of that), with a link to
> <https://guix.gnu.org/en/manual/en/html_node/Cross_002dCompilation.h=
tml>.
>
> 2. Adding a couple of sentences saying what Zephyr is (I didn=E2=80=99t=
know
> about it before :-)).
>
> 3. Saying a few words as to why Zephyr uses a GCC fork.
>
> 4. Clearly stating that you added support for cross-compilation to
> Zephyr with Guix in a channel, linking to said channel, and that
> this is what the remainder of the post will describe.
>
> You can check out posts at <https://guix.gnu.org/blog> for inspiration.
>
>> +# Anatomy of a toolchain
>> +
>> +Toolchains are responsible for taking high level descriptions of progra=
ms
>> +and lowering them down to a series of equivalent machine instructions.
>> +This process involves more than just a compiler. The compiler uses the =
`binutils`
>> +to manipulate it's internal representation down to a given architecture.
>> +It also needs the use of the C standard library as well as a few other =
libraries
>> +needed for some compiler optimizations.
>> +
>> +The C library provides the interface to the underlying kernel. System c=
alls like `write`
>> +and `read` are provided by `Glibc` on most Linux distributions.
>>
>> +In embedded systems smaller implementations like `newlib` and `newlib-n=
ano` are used.
>
> I=E2=80=99d suggest not using fixed-width font (backquotes) for proper no=
uns;
> you can write =E2=80=9Cglibc=E2=80=9D or =E2=80=9Cthe GNU C Library (glib=
c)=E2=80=9D, =E2=80=9CBinutils=E2=80=9D or =E2=80=9Cthe
> GNU Binary Utilities (Binutils)=E2=80=9D, etc.
>
>> +First thing we need to build is the `arm-zephyr-eabi` binutils.
>> +This is very easy in Guix.
>> +
>> + (define-module (zephyr packages zephyr)
>> + #:use-module (guix packages))
>> +
>> + (define-public arm-zephyr-eabi-binutils
>> + (let ((xbinutils (cross-binutils "arm-zephyr-eabi")))
>> + (package
>> + (inherit xbinutils)
>> + (name "arm-zephyr-eabi-binutils")
>> + (version "2.38")
>> + (source
>> + (origin (method git-fetch)
>> + (uri (git-reference
>> + (url "https://github.com/zephyrproject-rtos/binutils-gdb")
>> + (commit "6a1be1a6a571957fea8b130e4ca2dcc65e753469")))
>> + (file-name (git-file-name name version))
>> + (sha256 (base32 "0ylnl48jj5jk3jrmvfx5zf8byvwg7g7my7jwwyqw3a95qcy=
h0isr"))))
>> + (arguments
>> + `(#:tests? #f
>> + ,@(substitute-keyword-arguments (package-arguments xbinutils)
>> + ((#:configure-flags flags)
>> + `(cons "--program-prefix=3Darm-zephyr-eabi-" ,flags)))))
>> + (native-inputs
>> + (append
>> + (list texinfo
>> + bison
>> + flex
>> + gmp
>> + dejagnu)
>> + (package-native-inputs xbinutils)))
>> + (home-page "https://zephyrproject.org")
>> + (synopsis "binutils for zephyr RTOS"))))
>
> Code snippets should be written like this, without leading indentation:
>
> ```scheme
> (define =E2=80=A6)
> ```
>
> This will enable syntax highlighting.
>
>> +We can test our package definition using the `-L` flag with `guix build`
>> +to add our packages.
>> +
>> + guix build -L guix-zephyr zephyr-binutils
>> +
>> + /gnu/store/...-zephyr-binutils-2.38
>
> Likewise:
>
> ```
> guix build foo
> ```
>
>> +# Integrating with Zephyr Build System
>> +
>> +Zephyr uses CMake as it's build system. It contains numerous CMake
>
> s/it's/its/
>
> One thing that=E2=80=99s not clear to me: with this in place, can you do =
=E2=80=9Cguix
> build --target=3Darm-zephyr-eabi hello=E2=80=9D, for instance? If not, w=
hat=E2=80=99s
> missing to support it?
>
> It would be great if you could finish with a short conclusion stating,
> for instance, the key takeaway message, lessons learned, and/or your
> thoughts on how this could benefit others in the broader community.
>
> I wonder if it would be worth mentioning
> <https://guix.gnu.org/manual/en/html_node/Platforms.html> too, and how
> one would go about adding a module. WDYT?
>
> Could you send an updated patch?
>
> Thanks for contributing this article!
>
> Ludo=E2=80=99.
--=-=-=--
^ permalink raw reply related [flat|nested] 8+ messages in thread