From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id AAZSB7wlO2QxGgAASxT56A (envelope-from ) for ; Sun, 16 Apr 2023 00:31:24 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id +JVWB7wlO2SyGAAAauVa8A (envelope-from ) for ; Sun, 16 Apr 2023 00:31:24 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 892FC3DC81 for ; Sun, 16 Apr 2023 00:31:23 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pnoQD-0007sn-ER; Sat, 15 Apr 2023 18:31:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pnoQA-0007sP-Qk for guix-patches@gnu.org; Sat, 15 Apr 2023 18:31:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pnoQA-0001jv-Ie for guix-patches@gnu.org; Sat, 15 Apr 2023 18:31:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pnoQA-0007uv-4g for guix-patches@gnu.org; Sat, 15 Apr 2023 18:31:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#62356] [PAtCH guix-artwork] website: posts: Add Dissecting Guix, Part 3: G-Expressions. References: <20230321205749.4974-1-paren@disroot.org> In-Reply-To: <20230321205749.4974-1-paren@disroot.org> Resent-From: "(" Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 15 Apr 2023 22:31:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 62356 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 62356@debbugs.gnu.org Cc: "\(" , =?UTF-8?Q?Th=C3=A9o?= Maxime Tyburn , Simon Tournier Received: via spool by 62356-submit@debbugs.gnu.org id=B62356.168159781130307 (code B ref 62356); Sat, 15 Apr 2023 22:31:02 +0000 Received: (at 62356) by debbugs.gnu.org; 15 Apr 2023 22:30:11 +0000 Received: from localhost ([127.0.0.1]:50264 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pnoPJ-0007sg-89 for submit@debbugs.gnu.org; Sat, 15 Apr 2023 18:30:11 -0400 Received: from knopi.disroot.org ([178.21.23.139]:37738) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pnoPF-0007sO-25 for 62356@debbugs.gnu.org; Sat, 15 Apr 2023 18:30:07 -0400 Received: from localhost (localhost [127.0.0.1]) by disroot.org (Postfix) with ESMTP id B6C6240289; Sun, 16 Apr 2023 00:30:03 +0200 (CEST) X-Virus-Scanned: SPAM Filter at disroot.org Received: from knopi.disroot.org ([127.0.0.1]) by localhost (disroot.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id n2FEgVuWtGSo; Sun, 16 Apr 2023 00:30:00 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=disroot.org; s=mail; t=1681597800; bh=/3lI2URPkwMyegmAM2GqzkZo0Blx45YvQEZ07QGQ0hw=; h=From:To:Cc:Subject:Date; b=QbGWztsiu3ByTJNxE9j83ldB2GuaNQbEzX5euRy6UisekpEKcLMiTDuWvcS0Mo0xi KsDadxUr0AIbmD6icA5O0qLtcITIsoRj8PTAhfV8SL+NCQLI+J4Kq0+V9zKeVGnVdJ ICLH0oh2anwkGYSLwDaue+VZXunr9NnMGidY6yKZFMELotVPifHzXSljkBj5Cv0idK yEaDi8RBdKcUktiJFJ4MpO9rnGij11suzSoyzBY9KzYSFdaF4vC//zYJ6xE9t1/yVG Gk5VSNSyszMnqdBBLYk/fg1NHXtFqSLwEQuNtP42NllVuzZVtPtmqkbnREHlhzisaY C0mSjKwedvp4w== Date: Sat, 15 Apr 2023 23:29:54 +0100 Message-Id: <20230415222954.567-1-paren@disroot.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: "\(" X-ACL-Warn: , "\( via Guix-patches" From: "\( via Guix-patches" via Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN ARC-Seal: i=1; s=key1; d=yhetil.org; t=1681597883; a=rsa-sha256; cv=none; b=NdhKGduvwVuXmlWekZ+qWW3KLQ72bgYNYiVy6gaguom7PAchDJecEtdfnNSo7GQgzy5eui VoAZUVxz/+KAk1HcKYdFiAqvul8/CTdvJyPWvzY8cu5ZVlWlJuf9J1ADxHdKtceK4QdjEN 1aYWtyBrWetcBfGJdZlToUhdFEMVa35G2fRKZ4ufvCxxx2R2iFLvQUy5HmhRwFrUmvNQdD 0QR4cxx8A9MfsQGtqe1+g48SO3rgsXtXp9IMpVc1ox4w3yJbJNX2O6Wte4prQczBxrq+NK c3WRRryvPhuoV/1qXDcty1MlQUC3XXts8fAvRjakW6jwxZ7mhMqGsKf2EOboyw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=disroot.org header.s=mail header.b=QbGWztsi; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1681597883; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=/2GPRT0zkMwvJSsp9A/U+ocwwYurYnIxJBRvgzow58M=; b=enaoR2h4iemvcup/T+mHIyn+T1DiLMREVPjyO4KssUQu8EhKlofPfw06AsH1N1T6jQilg4 sHbdlueMFj0Fk71AS5ZjzULliTO2qFki4pgPMgBQTnFupDYD0L2BlsWqgqfjggg+4VhuE0 HtUQGRMBsxt9nZK5e9wxehRiKqZYMNPXlyECwcQH/cguLDcnD0MB+7IP7FWbNSdj6WxVLD b69ufOZ/+4/BZYqMbzmhVS11CcZetJORcc0Bcnxuy+1YF6aKpYK0Dq0TuFDrjKpjw/x8C+ uWY2LZE1OkC4/D9QKit9ZcdalKZnZQDmLd1VtSgNexp7G6sHdr+tEZtJRXS0OA== X-Migadu-Scanner: scn1.migadu.com X-Migadu-Spam-Score: -0.72 X-Spam-Score: -0.72 X-Migadu-Queue-Id: 892FC3DC81 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=disroot.org header.s=mail header.b=QbGWztsi; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org X-TUID: 2ffAW44qEBm9 * website/posts/dissecting-guix-3-gexps.md: New blog post. --- website/posts/dissecting-guix-3-gexps.md | 735 +++++++++++++++++++++++ 1 file changed, 735 insertions(+) create mode 100644 website/posts/dissecting-guix-3-gexps.md diff --git a/website/posts/dissecting-guix-3-gexps.md b/website/posts/dissecting-guix-3-gexps.md new file mode 100644 index 0000000..cd56243 --- /dev/null +++ b/website/posts/dissecting-guix-3-gexps.md @@ -0,0 +1,735 @@ +title: Dissecting Guix, Part 3: G-Expressions +date: TBC +author: ( +tags: Dissecting Guix, Functional package management, Programming interfaces, Scheme API +--- +Welcome back to [Dissecting Guix](https://guix.gnu.org/en/blog/tags/dissecting-guix)! +Last time, we discussed [monads](https://guix.gnu.org/en/blog/2023/dissecting-guix-part-2-the-store-monad), +the functional programming idiom used by Guix to thread a store connection +through a series of store-related operations. + +Today, we'll be talking about a concept rather more specific to Guix: +_g-expressions_. Being an implementation of the Scheme language, Guile is built +around [_s-expressions_](https://en.wikipedia.org/wiki/S-expression), which can +represent, as the saying goes, _code as data_, thanks to the simple structure of +Scheme forms. + +As Guix's package recipes are written in Scheme, it naturally needs some way to +represent code that is to be run only when the package is built. Additionally, +there needs to be some way to reference dependencies and retrieve output paths; +otherwise, you wouldn't be able to, for instance, create a phase to install a +file in the output directory. + +So, how do we implement this "deferred" code? Well, initially Guix used plain +old s-expressions for this purpose. + +# Once Upon a Time + +Let's say we want to create a store item that's just a symlink to the +`bin/irssi` file of the `irssi` package. How would we do that with an +s-expression? Well, the s-expression itself, which we call the _builder_, is +fairly simple: + +```scheme +(define sexp-builder + `(let* ((out (assoc-ref %outputs "out")) + (irssi (assoc-ref %build-inputs "irssi")) + (bin/irssi (string-append irssi "/bin/irssi"))) + (symlink bin/irssi out))) +``` + +If you aren't familliar with the "quoting" syntax used to create s-expressions, +I strongly recommend that you read the excellent Scheme Primer; specifically, +section 7, [_Lists and +"cons"_](https://spritely.institute/static/papers/scheme-primer.html#scheme-lists-and-cons) +and section 11, [_On the extensibility of Scheme (and Lisps in +general)_](https://spritely.institute/static/papers/scheme-primer.html#scheme-extensibility) + +The `%outputs` and `%build-inputs` variables are bound within builder scripts to +_association lists_, which are lists of pairs that act like key/value stores, +for instance: + +```scheme +'(("foo" . "bar") + ("floob" . "blarb") + ("fvoolag" . "bvarlag")) +``` + +To retrieve values from association lists, which are often referred to as +_alists_, we use the `assoc-ref` procedure: + +```scheme +(assoc-ref '(("boing" . "bouncy") + ("floing" . "flouncy")) + "boing") +⇒ "bouncy" +``` + +`%outputs`, as the name might suggest, maps derivation output names to the paths +of their respective store items, the default output being `out`, and +`%build-inputs` maps inputs labels to their store items. + +The builder is the easy part; we now need to turn it into a derivation and tell +it what `"irssi"` actually refers to. For this, we use the +`build-expression->derivation` procedure from `(guix derivations)`: + +```scheme +(use-modules (guix derivations) + (guix packages) + (guix store) + (gnu packages guile) + (gnu packages irc)) + +(with-store store + (let ((guile-3.0-drv (package-derivation store guile-3.0)) + (irssi-drv (package-derivation store irssi))) + (build-expression->derivation store "irssi-symlink" sexp-builder + #:guile-for-build guile-3.0-drv + #:inputs `(("irssi" ,irssi-drv))))) +⇒ # /gnu/store/…-irssi-symlink …> +``` + +There are several things to note here: + +- The inputs _must_ all be derivations, so we need to first convert the packages + using `package-derivation`. +- We need to explicitly set `#:guile-for-build`; there's no default value. +- The `build-expression->derivation` and `package-derivation` procedures are + _not_ monadic, so we need to explicitly pass them the store connection. + +The shortcomings of using s-expressions in this way are numerous: we have to +convert everything to a derivation before using it, and _inputs are not an +inherent aspect of the builder_. G-expressions were designed to overcome these +issues. + +# Premortem Examination + +A g-expression is fundamentally a record of type ``, which is, naturally, +defined in `(guix gexp)`. The two most important fields of this record type, +out of a total of five, are `proc` and `references`; the former is a procedure +that returns the equivalent s-expression, the latter a list containing +everything from the "outside world" that's used by the g-expression. + +When we want to turn the g-expression into something that we can actually run as +code, we combine these two fields by first building any g-expression inputs that +can become derivations (leaving alone those that cannot), and then passing the +built `references` as the arguments of `proc`. + +Here's an example g-expression that is essentially equivalent to our +`sexp-builder`: + +```scheme +(use-modules (guix gexp)) + +(define gexp-builder + #~(symlink #$(file-append irssi "/bin/irssi") + #$output)) +``` + +`gexp-builder` is far more concise than `sexp-builder`; let's examine the syntax +and the `` object we've created. To make a g-expression, we use the `#~` +syntax, equivalent to the `gexp` macro, rather than the `quasiquote` backtick +used to create s-expressions. + +When we want to embed values from outside as references, we use `#$`, or +`ungexp`, which is, in appearance if not function, equivalent to `unquote` +(`,`). `ungexp` can accept any of four reference types: + +- S-expressions (strings, lists, etc), which will be embedded literally. +- Other g-expressions, embedded literally. +- Expressions returning any sort of object that can be lowered into a + derivation, such as ``, embedding that object's `out` store item; if + the expression is specifically a symbol bound to a buildable object, you can + optionally follow it with a colon and an alternative output name, so + `package:lib` is permitted, but `(get-package):lib` isn't. +- The symbol `output`, embedding an output path. Like symbols bound to + buildable objects, this can be followed by a colon and the output name that + should be used rather than the default `out`. + +All these reference types will be represented by `` records in the +`references` field, except for the last kind, which will become `` +records. To give an example of each type of reference (with the return value +output formatted for easier reading): + +```scheme +(use-modules (gnu packages glib)) + +#~(list #$"foobar" ;s-expression + #$#~(string-append "foo" "bar") ;g-expression + #$(file-append irssi "/bin/irssi") ;buildable object (expression) + #$glib:bin ;buildable object (symbol) + #$output:out) ;output +⇒ # + #:out> + # "/bin/irssi">:out> + #:bin> + #) …> +``` + +Note the use of `file-append` in both the previous example and `gexp-builder`; +this procedure produces a `` object that builds its first argument +and is embedded as the concatenation of the first argument's output path and the +second argument, which should be a string. For instance, +`(file-append irssi "/bin/irssi")` builds `irssi` and expands to +`/gnu/store/…-irssi/bin/irssi`, rather than the `/gnu/store/…-irssi` that the +package alone would be embedded as. + +So, now that we have a g-expression, how do we turn it into a derivation? This +process is known as _lowering_; it entails the use of the aptly-named +`lower-gexp` monadic procedure to combine `proc` and `references` and produce a +`` record, which acts as a sort of intermediate representation +between g-expressions and derivations. We can piece apart this lowered form to +get a sense of what the final derivation's builder script would look like: + +```scheme +(define lowered-gexp-builder + (with-store store + (run-with-store store + (lower-gexp gexp-builder)))) + +(lowered-gexp-sexp lowered-gexp-builder) +⇒ (symlink + "/gnu/store/…-irssi-1.4.3/bin/irssi" + ((@ (guile) getenv) "out")) +``` + +And there you have it: a s-expression compiled from a g-expression, ready to be +written into a builder script file in the store. So, how exactly do you turn +this into said derivation? + +Well, it turns out that there isn't an interface for turning lowered +g-expressions into derivations, only one for turning regular g-expressions into +derivations that first uses `lower-gexp`, then implements the aforementioned +conversion internally, rather than outsourcing it to some other procedure, so +that's what we'll use. + +Unsurprisingly, that procedure is called `gexp->derivation`, and unlike its +s-expression equivalent, it's monadic. (`build-expression->derivation` and +other deprecated procedures were in Guix since before the monads system +existed.) + +```scheme +(with-store store + (run-with-store store + (gexp->derivation "irssi-symlink" gexp-builder))) +⇒ # /gnu/store/…-irssi-symlink …> +``` + +Finally, we have a g-expression-based equivalent to the derivation we earlier +created with `build-expression->derivation`! Here's the code we used for the +s-expression version in full: + +```scheme +(define sexp-builder + `(let* ((out (assoc-ref %outputs "out")) + (irssi (assoc-ref %build-inputs "irssi")) + (bin/irssi (string-append irssi "/bin/irssi"))) + (symlink bin/irssi out))) + +(with-store store + (let ((guile-3.0-drv (package-derivation store guile-3.0)) + (irssi-drv (package-derivation store irssi))) + (build-expression->derivation store "irssi-symlink" sexp-builder + #:guile-for-build guile-3.0-drv + #:inputs `(("irssi" ,irssi-drv))))) +``` + +And here's the g-expression equivalent: + +```scheme +(define gexp-builder + #~(symlink #$(file-append irssi "/bin/irssi") + #$output)) + +(with-store store + (run-with-store store + (gexp->derivation "irssi-symlink" gexp-builder))) +``` + +That's a lot of complexity abstracted away! For more complex packages and +services, especially, g-expressions are a lifesaver; you can refer to the output +paths of inputs just as easily as you would a string constant. You do, however, +have to watch out for situations where `ungexp-native`, written as `#+`, would +be preferable over regular `ungexp`, and that's something we'll discuss later. + +A brief digression before we continue: if you'd like to look inside a `` +record, but you'd rather not build anything, you can use the +`gexp->approximate-sexp` procedure, which replaces all references with dummy +values: + +```scheme +(gexp->approximate-sexp gexp-builder) +⇒ (symlink (*approximate*) (*approximate*)) +``` + +# The Lowerable-Object Hardware Shop + +We've seen two examples already of records we can turn into derivations, which +are generally referred to as _lowerable objects_ or _file-like objects_: + +- ``, a Guix package. +- ``, which wraps another lowerable object and appends a string to + the embedded output path when `ungexp`ed. + +There are many more available to us. Recall from the previous post, +[_The Store Monad_](https://guix.gnu.org/en/blog/2023/dissecting-guix-part-2-the-store-monad), +that Guix provides the two monadic procedures `text-file` and `interned-file`, +which can be used, respectively, to put arbitrary text or files from the +filesystem in the store, returning the path to the created item. + +This doesn't work so well with g-expressions, though; you'd have to wrap each +`ungexp`ed use of either of them with +`(with-store store (run-with-store store …))`, which would be quite tedious. +Thankfully, `(guix gexp)` provides the `plain-file` and `local-file` procedures, +which return equivalent lowerable objects. This code example builds a directory +containing symlinks to files greeting the world: + +```scheme +(use-modules (guix monads) + (ice-9 ftw) + (ice-9 textual-ports)) + +(define (build-derivation monadic-drv) + (with-store store + (run-with-store store + (mlet* %store-monad ((drv monadic-drv)) + (mbegin %store-monad + ;; BUILT-DERIVATIONS is the monadic version of BUILD-DERIVATIONS. + (built-derivations (list drv)) + (return (derivation-output-path + (assoc-ref (derivation-outputs drv) "out")))))))) + +(define world-greeting-output + (build-derivation + (gexp->derivation "world-greeting" + #~(begin + (mkdir #$output) + (symlink #$(plain-file "hi-world" + "Hi, world!") + (string-append #$output "/hi")) + (symlink #$(plain-file "hello-world" + "Hello, world!") + (string-append #$output "/hello")) + (symlink #$(plain-file "greetings-world" + "Greetings, world!") + (string-append #$output "/greetings")))))) + +;; We turn the list into multiple values using (APPLY VALUES …). +(apply values + (map (lambda (file-path) + (let* ((path (string-append world-greeting-output "/" file-path)) + (contents (call-with-input-file path get-string-all))) + (list path contents))) + ;; SCANDIR from (ICE-9 FTW) returns the list of all files in a + ;; directory (including ``.'' and ``..'', so we remove them with the + ;; second argument, SELECT?, which specifies a predicate). + (scandir world-greeting-output + (lambda (path) + (not (or (string=? path ".") + (string=? path ".."))))))) +⇒ ("/gnu/store/…-world-greeting/greetings" "Greetings, world!") +⇒ ("/gnu/store/…-world-greeting/hello" "Hello, world!") +⇒ ("/gnu/store/…-world-greeting/hi" "Hi, world!") +``` + +Note that we define a procedure for building the output; we will need to build +more derivations in a very similar fashion later, so it helps to have this to +reuse instead of copying the code in `world-greeting-output`. + +There are many other useful lowerable objects available as part of the gexp +library. These include `computed-file`, which accepts a gexp that builds +the output file, `program-file`, which creates an executable Scheme script in +the store using a g-expression, and `mixed-text-file`, which allows you to, +well, mix text and lowerable objects; it creates a file from the concatenation +of a sequence of strings and file-likes. The +[G-Expressions](https://guix.gnu.org/manual/en/html_node/G_002dExpressions.html) +manual page has more details. + +So, you may be wondering, at this point: there's so many lowerable objects +included with the g-expression library, surely there must be a way to define +more? Naturally, there is; this is Scheme, after all! We simply need to +acquaint ourselves with the `define-gexp-compiler` macro. + +The most basic usage of `define-gexp-compiler` essentially creates a procedure +that takes as arguments a record to lower, the host system, and the target +system, and returns a derivation or store item as a monadic value in +`%store-monad`. + +Let's try implementing a lowerable object representing a file that greets the +world. First, we'll define the record type: + +```scheme +(use-modules (srfi srfi-9)) + +(define-record-type + (greeting-file greeting) + greeting? + (greeting greeting-file-greeting)) +``` + +Now we use `define-gexp-compiler` like so; note how we can use `lower-object` +to compile down any sort of lowerable object into the equivalent store item or +derivation; essentially, `lower-object` is just the procedure for applying the +right gexp-compiler to an object: + +```scheme +(use-modules (ice-9 i18n)) + +(define-gexp-compiler (greeting-file-compiler + (greeting-file ) + system target) + (lower-object + (let ((greeting (greeting-file-greeting greeting-file))) + (plain-file (string-append greeting "-greeting") + (string-append (string-locale-titlecase greeting) ", world!"))))) +``` + +Let's try it out now. Here's how we could rewrite our greetings directory +example from before using ``: + +```scheme +(define world-greeting-2-output + (build-derivation + (gexp->derivation "world-greeting-2" + #~(begin + (mkdir #$output) + (symlink #$(greeting-file "hi") + (string-append #$output "/hi")) + (symlink #$(greeting-file "hello") + (string-append #$output "/hello")) + (symlink #$(greeting-file "greetings") + (string-append #$output "/greetings")))))) + +(apply values + (map (lambda (file-path) + (let* ((path (string-append world-greeting-2-output + "/" file-path)) + (contents (call-with-input-file path get-string-all))) + (list path contents))) + (scandir world-greeting-2-output + (lambda (path) + (not (or (string=? path ".") + (string=? path ".."))))))) +⇒ ("/gnu/store/…-world-greeting-2/greetings" "Greetings, world!") +⇒ ("/gnu/store/…-world-greeting-2/hello" "Hello, world!") +⇒ ("/gnu/store/…-world-greeting-2/hi" "Hi, world!") +``` + +Now, this is probably not worth a whole new gexp-compiler. How about something +a bit more complex? Sharp-eyed readers who are trying all this in the REPL may +have noticed the following output when they used `define-gexp-compiler` +(formatted for ease of reading): + +```scheme +⇒ #< + type: #> + lower: # + expand: #> +``` + +Now, the purpose of `type` and `lower` is self-explanatory, but what's this +`expand` procedure here? Well, if you recall `file-append`, you may realise +that the text produced by a gexp-compiler for embedding into a g-expression +doesn't necessarily have to be the exact output path of the produced derivation. + +There turns out to be another way to write a `define-gexp-compiler` form that +allows you to specify _both_ the lowering procedure, which produces the +derivation or store item, and the expanding procedure, which produces the text. + +Let's try making another new lowerable object; this one will let us build a +Guile package and expand to the path to its module directory. Here's our +record: + +```scheme +(define-record-type + (module-directory package) + module-directory? + (package module-directory-package)) +``` + +Here's how we define both a compiler and expander for our new record: + +```scheme +(use-modules (gnu packages guile) + (guix utils)) + +(define lookup-expander (@@ (guix gexp) lookup-expander)) + +(define-gexp-compiler module-directory-compiler + compiler => (lambda (obj system target) + (let ((package (module-directory-package obj))) + (lower-object package system #:target target))) + expander => (lambda (obj drv output) + (let* ((package (module-directory-package obj)) + (expander (or (lookup-expander package) + (lookup-expander drv))) + (out (expander package drv output)) + (guile (or (lookup-package-input package "guile") + guile-3.0)) + (version (version-major+minor + (package-version guile)))) + (string-append out "/share/guile/site/" version)))) +``` + +Let's try this out now: + +```scheme +(use-modules (gnu packages guile-xyz)) + +(define module-directory-output/guile-webutils + (build-derivation + (gexp->derivation "module-directory-output" + #~(symlink #$(module-directory guile-webutils) #$output)))) + +(readlink module-directory-output/guile-webutils) +⇒ "/gnu/store/…-guile-webutils-0.1-1.d309d65/share/guile/site/3.0" + +(scandir module-directory-output/guile-webutils) +⇒ ("." ".." "webutils") + +(define module-directory-output/guile2.2-webutils + (build-derivation + (gexp->derivation "module-directory-output" + #~(symlink #$(module-directory guile2.2-webutils) #$output)))) + +(readlink module-directory-output/guile2.2-webutils) +⇒ "/gnu/store/…-guile-webutils-0.1-1.d309d65/share/guile/site/2.2" + +(scandir module-directory-output/guile2.2-webutils) +⇒ ("." ".." "webutils") +``` + +Who knows why you'd want to do this, but it certainly works! We've looked at +why we need g-expressions, how they work, and how to extend them, and we've now +only got two more advanced features to cover: cross-build support, and modules. + +# Importing External Modules + +Let's try using one of the helpful procedures from the `(guix build utils)` +module in a g-expression. + +```scheme +(define simple-directory-output + (build-derivation + (gexp->derivation "simple-directory" + #~(begin + (use-modules (guix build utils)) + (mkdir-p (string-append #$output "/a/rather/simple/directory")))))) +``` + +Looks fine, right? We've even got a `use-modules` in th-- + +```Scheme +ERROR: + 1. &store-protocol-error: + message: "build of `/gnu/store/…-simple-directory.drv' failed" + status: 100 +``` + +OUTRAGEOUS. Fortunately, there's an explanation to be found in the Guix build +log directory, `/var/log/guix/drvs`; locate the file using the first two +characters of the store hash as the subdirectory, and the rest as the file name, +and remember to use `zcat` or `zless`, as the logs are gzipped: + +```scheme +Backtrace: + 9 (primitive-load "/gnu/store/…") +In ice-9/eval.scm: + 721:20 8 (primitive-eval (begin (use-modules (guix build #)) (?))) +In ice-9/psyntax.scm: + 1230:36 7 (expand-top-sequence ((begin (use-modules (guix ?)) #)) ?) + 1090:25 6 (parse _ (("placeholder" placeholder)) ((top) #(# # ?)) ?) + 1222:19 5 (parse _ (("placeholder" placeholder)) ((top) #(# # ?)) ?) + 259:10 4 (parse _ (("placeholder" placeholder)) (()) _ c&e (eval) ?) +In ice-9/boot-9.scm: + 3927:20 3 (process-use-modules _) + 222:17 2 (map1 (((guix build utils)))) + 3928:31 1 (_ ((guix build utils))) + 3329:6 0 (resolve-interface (guix build utils) #:select _ #:hide ?) + +ice-9/boot-9.scm:3329:6: In procedure resolve-interface: +no code for module (guix build utils) +``` + +It turns out `use-modules` can't actually find `(guix build utils)` at all. +There's no typo; it's just that to ensure the build is isolated, Guix builds +`module-import` and `module-importe-compiled` directories, and sets the +_Guile module path_ within the build environment to contain said directories, +along with those containing the Guile standard library modules. + +So, what to do? Turns out one of the fields in `` is `modules`, which, +funnily enough, contains the names of the modules which will be used to build +the aforementioned directories. To add to this field, we use the +`with-imported-modules` macro. (`gexp->derivation` _does_ provide a `modules` +parameter, but `with-imported-modules` lets you add the required modules +directly to the g-expression value, rather than later on.) + +```scheme +(define simple-directory-output + (build-derivation + (gexp->derivation "simple-directory" + (with-imported-modules '((guix build utils)) + #~(begin + (use-modules (guix build utils)) + (mkdir-p (string-append #$output "/a/rather/simple/directory"))))))) + +simple-directory-output +⇒ "/gnu/store/…-simple-directory" +``` + +It works, yay. It's worth noting that while passing just the list of modules to +`with-imported-modules` works in this case, this is only because +`(guix build utils)` has no dependencies on other Guix modules. Were we to try +adding, say, `(guix build emacs-build-system)`, we'd need to use the +`source-module-closure` procedure to add its dependencies to the list: + +```scheme +(use-modules (guix modules)) + +(source-module-closure '((guix build emacs-build-system))) +⇒ ((guix build emacs-build-system) + (guix build gnu-build-system) + (guix build utils) + (guix build gremlin) + (guix elf) + (guix build emacs-utils)) +``` + +Here's another scenario: what if we want to use a module not from Guix or Guile +but a third-party library? In this example, we'll use [guile-json +](https://github.com/aconchillo/guile-json), a library for converting between +S-expressions and [JavaScript Object Notation](https://json.org). + +We can't just `with-imported-modules` its modules, since it's not part of Guix, +so `` provides another field for this purpose: `extensions`. Each of +these extensions is a lowerable object that produces a Guile package directory; +so usually a package. Let's try it out using the `guile-json-4` package to +produce a JSON file from a Scheme value within a g-expression. + +```scheme +(define helpful-guide-output + (build-derivation + (gexp->derivation "json-file" + (with-extensions (list guile-json-4) + #~(begin + (use-modules (json)) + (mkdir #$output) + (call-with-output-file (string-append #$output "/helpful-guide.json") + (lambda (port) + (scm->json '((truth . "Guix is the best!") + (lies . "Guix isn't the best!")) + port)))))))) + +(call-with-input-file + (string-append helpful-guide-output "/helpful-guide.json") + get-string-all) +⇒ "{\"truth\":\"Guix is the best!\",\"lies\":\"Guix isn't the best!\"}" +``` + +Amen to that, `helpful-guide.json`. Before we continue on to cross-compilation, +there's one last feature of `with-imported-modules` you should note. We can +add modules to a g-expression by name, but we can also create entirely new ones +using lowerable objects, such as in this pattern, which is used in several +places in the Guix source code to make an appropriately-configured +`(guix config)` module available: + +```scheme +(with-imported-modules `(((guix config) => ,(make-config.scm)) + …) + …) +``` + +In case you're wondering, `make-config.scm` is found in `(guix self)` and +returns a lowerable object that compiles to a version of the `(guix config)` +module, which contains constants usually substituted into the source code at +compile time. + +# Native `ungexp` + +There is another piece of syntax we can use with g-expressions, and it's called +`ungexp-native`. This helps us distinguish between native inputs and regular +host-built inputs in cross-compilation situations. We'll cover +cross-compilation in detail at a later date, but the gist of it is that it +allows you to compile a derivation for one architecture X, the target, using a +machine of architecture Y, the host, and Guix has excellent support for it. + +If we cross-compile a g-expression G that _non-natively_ `ungexp`s L1, a +lowerable object, from architecture Y to architecture X, both G and L1 will be +compiled for architecture X. However, if G _natively_ `ungexp`s L1, G will be +compiled for X and L1 for Y. + +Essentially, we use `ungexp-native` in situations where there would be no +difference between compiling on different architectures (for instance, if `L1` +were a `plain-file`), or where using L1 built for X would actually _break_ G +(for instance, if `L1` corresponds to a compiled executable that needs to be run +during the build; the executable would fail to run on Y if it was built for X.) + +The `ungexp-native` macro naturally has a corresponding reader syntax, `#+`, and +there's also `ungexp-native-splicing`, which is written as `#+@`. These two +pieces of syntax are used in the same way as their regular counterparts. + +# Conclusion + +What have we learned in this post? To summarise: + ++ G-expressions are essentially abstractions on top of s-expressions used in + Guix to stage code, often for execution within a build environment or a + Shepherd service script. ++ Much like you can `unquote` external values within a `quasiquote` form, you + can `ungexp` external values to access them within a `gexp` form. The key + difference is that you may use not only s-expressions with `ungexp`, but other + g-expressions and lowerable objects too. ++ When a lowerable object is used with `ungexp`, the g-expression ultimately + receives the path to the object's store item (or whatever string the lowerable + object's expander produces), rather than the object itself. ++ A lowerable object is any record that has a "g-expression compiler" defined + for it using the `define-gexp-compiler` macro. G-expression compilers always + contain a `compiler` procedure, which converts an appropriate record into a + derivation, and sometimes an `expander` procedure, which produces the string + that is to be expanded to within g-expressions when the object is `ungexp`ed. ++ G-expressions record the list of modules available in their environment, which + you may expand using `with-imported-modules` to add Guix modules, and + `with-extensions` to add modules from third-party Guile packages. ++ `ungexp-native` may be used within g-expressions to compile lowerable objects + for the host rather than the target system in cross-compilation scenarios. + +Mastering g-expressions is essential to understanding Guix's inner workings, so +the aim of this blog post is to be as thorough as possible. However, if you +still find yourself with questions, please don't hesitate to stop by at the IRC +channel `#guix:libera.chat` and mailing list `help-guix@gnu.org`; we'll be glad +to assist you! + +Also note that due to the centrality of g-expressions to Guix, there exist a +plethora of alternative resources on this topic; here are some which you may +find useful: + ++ Arun Isaac's + [post](https://www.systemreboot.net/post/deploy-scripts-using-g-expressions) + on using g-expressions with `guix deploy`. ++ Marius Bakke's + ["Guix Drops" post](https://gexp.no/blog/guix-drops-part-3-g-expressions.html) + which explains g-expressions in a more "top-down" way. ++ This 2020 + [FOSDEM talk](https://archive.fosdem.org/2020/schedule/event/gexpressionsguile/) + by Christopher Marusich on the uses of g-expressions. ++ And, of course, the one and only original + [g-expression paper](https://hal.inria.fr/hal-01580582v1) by Ludovic Courtès, + the original author of Guix. + +#### About GNU Guix + +[GNU Guix](https://guix.gnu.org) is a transactional package manager and +an advanced distribution of the GNU system that [respects user +freedom](https://www.gnu.org/distros/free-system-distribution-guidelines.html). +Guix can be used on top of any system running the Hurd or the Linux +kernel, or it can be used as a standalone operating system distribution +for i686, x86_64, ARMv7, AArch64 and POWER9 machines. + +In addition to standard package management features, Guix supports +transactional upgrades and roll-backs, unprivileged package management, +per-user profiles, and garbage collection. When used as a standalone +GNU/Linux distribution, Guix offers a declarative, stateless approach to +operating system configuration management. Guix is highly customizable +and hackable through [Guile](https://www.gnu.org/software/guile) +programming interfaces and extensions to the +[Scheme](http://schemers.org) language. base-commit: 9d8f36a722d33f75e2e081a8d8f04cf20c4d3511 -- 2.39.2