[bug#62356] [PATCH guix-artwork] website: posts: Add Dissecting Guix, Part 3: G-Expressions.

[bug#62356] [PATCH guix-artwork] website: posts: Add Dissecting Guix, Part 3: G-Expressions.

[( via Guix-patches via]

* website/posts/dissecting-guix-3-gexps.md: New blog post.
---
Heya Guix,

Here's the third post in the Dissecting Guix series; this one aims to demystify
g-expressions ;)

  -- (

 website/posts/dissecting-guix-3-gexps.md | 673 +++++++++++++++++++++++
 1 file changed, 673 insertions(+)
 create mode 100644 website/posts/dissecting-guix-3-gexps.md

diff --git a/website/posts/dissecting-guix-3-gexps.md b/website/posts/dissecting-guix-3-gexps.md
new file mode 100644
index 0000000..32f5d51
--- /dev/null
+++ b/website/posts/dissecting-guix-3-gexps.md
@@ -0,0 +1,673 @@
+title: Dissecting Guix, Part 3: G-Expressions
+date: TBC
+author: (
+tags: Dissecting Guix, Functional package management, Programming interfaces, Scheme API
+---
+Welcome back to [Dissecting Guix](https://guix.gnu.org/en/blog/tags/dissecting-guix)!
+Last time, we discussed [monads](https://guix.gnu.org/en/blog/2023/dissecting-guix-part-2-the-store-monad),
+the functional programming idiom used by Guix to thread a store connection
+through a series of store-related operations.
+
+Today, we'll be talking about a concept rather more specific to Guix:
+_g-expressions_.  Being an implementation of the Scheme language, Guile is built
+around [_s-expressions_](https://en.wikipedia.org/wiki/S-expression), which can
+represent, as the saying goes, _code as data_, thanks to the simple structure of
+Scheme forms.
+
+As Guix's package recipes are written in Scheme, it naturally needs some way to
+represent code that is to be run only when the package is built.  Additionally,
+there needs to be some way to reference dependencies and retrieve output paths;
+otherwise, you wouldn't be able to, for instance, create a phase to install a
+file in the output directory.
+
+So, how do we implement this "deferred" code?  Well, initially Guix used plain
+old s-expressions for this purpose.
+
+# Once Upon a Time
+
+Let's say we want to create a store item that's just a symlink to the
+`bin/irssi` file of the `irssi` package.  How would we do that with an
+s-expression?  Well, the s-expression itself, which we call the _builder_, is
+fairly simple:
+
+```scheme
+(define sexp-builder
+  `(let* ((out (assoc-ref %outputs "out"))
+          (irssi (assoc-ref %build-inputs "irssi"))
+          (bin/irssi (string-append irssi "/bin/irssi")))
+     (symlink bin/irssi out)))
+```
+
+If you aren't familliar with the "quoting" syntax used to create s-expressions,
+I strongly recommend that you read the excellent Scheme Primer; specifically,
+section 7, [_Lists and
+"cons"_](https://spritely.institute/static/papers/scheme-primer.html#scheme-lists-and-cons)
+and section 11, [_On the extensibility of Scheme (and Lisps in
+general)_](https://spritely.institute/static/papers/scheme-primer.html#scheme-extensibility)
+
+The `%outputs` and `%build-inputs` variables are bound within builder scripts to
+_association lists_, which are lists of pairs that act like key/value stores,
+for instance:
+
+```scheme
+'(("foo" . "bar")
+  ("floob" . "blarb")
+  ("fvoolag" . "bvarlag"))
+```
+
+To retrieve values from association lists, which are often referred to as
+_alists_, we use the `assoc-ref` procedure:
+
+```scheme
+(assoc-ref '(("boing" . "bouncy")
+             ("floing" . "flouncy"))
+           "boing")
+⇒ "bouncy"
+```
+
+`%outputs`, as the name might suggest, maps derivation output names to the paths
+of their respective store items, the default output being `out`, and
+`%build-inputs` maps inputs labels to their store items.
+
+The builder is the easy part; we now need to turn it into a derivation and tell
+it what `"irssi"` actually refers to.  For this, we use the
+`build-expression->derivation` procedure from `(guix derivations)`:
+
+```scheme
+(use-modules (guix derivations)
+             (guix packages)
+             (guix store)
+             (gnu packages guile)
+             (gnu packages irc))
+
+(with-store store
+  (let ((guile-3.0-drv (package-derivation store guile-3.0))
+        (irssi-drv (package-derivation store irssi)))
+    (build-expression->derivation store "irssi-symlink" sexp-builder
+      #:guile-for-build guile-3.0-drv
+      #:inputs `(("irssi" ,irssi-drv)))))
+⇒ #<derivation /gnu/store/…-irssi-symlink.drv => /gnu/store/…-irssi-symlink …>
+```
+
+There are several things to note here:
+
+- The inputs _must_ all be derivations, so we need to first convert the packages
+  using `package-derivation`.
+- We need to explicitly set `#:guile-for-build`; there's no default value.
+- The `build-expression->derivation` and `package-derivation` procedures are
+  _not_ monadic, so we need to explicitly pass them the store connection.
+
+The shortcomings of using s-expressions in this way are numerous: we have to
+convert everything to a derivation before using it, and _inputs are not an
+inherent aspect of the builder_.  G-expressions were designed to overcome these
+issues.
+
+# Premortem Examination
+
+A gexp is fundumentally a record of type `<gexp>`, which is, naturally, defined
+in `(guix gexp)`.  The two most important fields of this record type, out of a
+total of five, are `proc` and `references`; the former is a procedure that
+returns the equivalent sexp, the latter a list containing everything from the
+"outside world" that's used by the gexp.
+
+When we want to turn the gexp into something that we can actually run as code,
+we combine these two fields by first building any gexp inputs that can become
+derivations (leaving alone those that cannot, such as and then passing the built
+`references` as the arguments of `proc`.
+
+Here's an example gexp that is essentially equivalent to our `sexp-builder`:
+
+```scheme
+(use-modules (guix gexp))
+
+(define gexp-builder
+  #~(symlink #$(file-append irssi "/bin/irssi")
+             #$output))
+```
+
+`gexp-builder` is far more concise than `sexp-builder`; let's examine the syntax
+and the `<gexp>` object we've created.  To make a gexp, we use the `#~` syntax,
+equivalent to the `gexp` macro, rather than the `quasiquote` backtick used to
+create sexps.
+
+When we want to embed values from outside as references, we use `#$`, or
+`ungexp`, which is, in appearance if not function, equivalent to `unquote`
+(`,`).  `ungexp` can accept any of four reference types:
+
+- Sexps (strings, lists, etc), which will be embedded literally.
+- Other gexps, embedded literally.
+- Expressions returning any sort of object that can be lowered into a
+  derivation, such as `<package>`, embedding that object's `out` store item; if
+  the expression is specifically a symbol bound to a buildable object, you can
+  optionally follow it with a colon and an alternative output name, so
+  `package:lib` is permitted, but `(get-package):lib` isn't.
+- The symbol `output`, embedding an output path.  Like symbols bound to
+  buildable objects, this can be followed by a colon and the output name that
+  should be used rather than the default `out`.
+
+All these reference types will be represented by `<gexp-input>` records in the
+`references` field, except for the last kind, which will become `<gexp-output>`
+records.  To give an example of each type of reference (with the return value
+output formatted for easier reading):
+
+```scheme
+(use-modules (gnu packages glib))
+
+#~(list #$"foobar"                         ;s-expression
+        #$#~(string-append "foo" "bar")    ;g-expression
+        #$(file-append irssi "/bin/irssi") ;buildable object (expression)
+        #$glib:bin                         ;buildable object (symbol)
+        #$output:out)                      ;output
+⇒ #<gexp (list #<gexp-input "foobar":out>
+               #<gexp-input #<gexp (string-append "foo" "bar") …>:out>
+               #<gexp-input #<file-append #<package irssi@1.4.3 …> "/bin/irssi">:out>
+               #<gexp-input #<package glib@2.70.2 …>:bin>
+               #<gexp-output out>) …>
+```
+
+Note the use of `file-append` in both the previous example and `gexp-builder`;
+this procedure produces a `<file-append>` object that builds its first argument
+and is embedded as the concatenation of the first argument's output path and the
+second argument, which should be a string.  For instance,
+`(file-append irssi "/bin/irssi")` builds `irssi` and expands to
+`/gnu/store/…-irssi/bin/irssi`, rather than the `/gnu/store/…-irssi` that the
+package alone would be embedded as.
+
+So, now that we have a gexp, how do we turn it into a derivation?  This process
+is known as _lowering_; it entails the use of the aptly-named `lower-gexp`
+monadic procedure to combine `proc` and `references` and produce a
+`<lowered-gexp>` record, which acts as a sort of intermediate representation
+between gexps and derivations.  We can piece apart this lowered form to get a
+sense of what the final derivation's builder script would look like:
+
+```scheme
+(define lowered-gexp-builder
+  (with-store store
+    (run-with-store store
+      (lower-gexp gexp-builder))))
+
+(lowered-gexp-sexp lowered-gexp-builder)
+⇒ (symlink
+   "/gnu/store/…-irssi-1.4.3/bin/irssi"
+   ((@ (guile) getenv) "out"))
+```
+
+And there you have it: a s-expression compiled from a g-expression, ready to be
+written into a builder script file in the store.  So, how exactly do you turn
+this into said derivation?
+
+Well, it turns out that there isn't an interface for turning lowered gexps into
+derivations, only one for turning regular gexps into derivations that first uses
+`lower-gexp`, then implements the aforementioned conversion internally, rather
+than outsourcing it to some other procedure, so that's what we'll use.
+
+Unsurprisingly, that procedure is called `gexp->derivation`, and unlike its sexp
+equivalent, it's monadic.  (`build-expression->derivation` and other deprecated
+procedures were in Guix since before the monads system existed.)
+
+```scheme
+(with-store store
+  (run-with-store store
+    (gexp->derivation "irssi-symlink" gexp-builder)))
+⇒ #<derivation /gnu/store/…-irssi-symlink.drv => /gnu/store/…-irssi-symlink …>
+```
+
+Finally, we have a gexp-based equivalent to the derivation we earlier created
+with `build-expression->derivation`!  Here's the code we used for the sexp
+version in full:
+
+```scheme
+(define sexp-builder
+  `(let* ((out (assoc-ref %outputs "out"))
+          (irssi (assoc-ref %build-inputs "irssi"))
+          (bin/irssi (string-append irssi "/bin/irssi")))
+     (symlink bin/irssi out)))
+
+(with-store store
+  (let ((guile-3.0-drv (package-derivation store guile-3.0))
+        (irssi-drv (package-derivation store irssi)))
+    (build-expression->derivation store "irssi-symlink" sexp-builder
+      #:guile-for-build guile-3.0-drv
+      #:inputs `(("irssi" ,irssi-drv)))))
+```
+
+And here's the gexp equivalent:
+
+```scheme
+(define gexp-builder
+  #~(symlink #$(file-append irssi "/bin/irssi")
+             #$output))
+
+(with-store store
+  (run-with-store store
+    (gexp->derivation "irssi-symlink" gexp-builder)))
+```
+
+That's a lot of complexity abstracted away!  For more complex packages and
+services, especially, gexps are a lifesaver; you can refer to the output paths
+of inputs just as easily as you would a string constant.  You do, however, have
+to watch out for situations where `ungexp-native`, written as `#+`, would be
+preferable over regular `ungexp`, and that's something we'll discuss later.
+
+A brief digression before we continue: if you'd like to look inside a `<gexp>`
+record, but you'd rather not build anything, you can use the
+`gexp->approximate-sexp` procedure, which replaces all references with dummy
+values:
+
+```scheme
+(gexp->approximate-sexp gexp-builder)
+⇒ (symlink (*approximate*) (*approximate*))
+```
+
+# The Lowerable-Object Hardware Shop
+
+We've seen two examples already of records we can turn into derivations, which
+are generally referred to as _lowerable objects_ or _file-like objects_:
+
+- `<package>`, a Guix package.
+- `<file-append>`, which wraps another lowerable object and appends a string to
+  the embedded output path when ungexped.
+
+There are many more available to us.  Recall from the previous post,
+[_The Store Monad_](https://guix.gnu.org/en/blog/2023/dissecting-guix-part-2-the-store-monad),
+that Guix provides the two monadic procedures `text-file` and `interned-file`,
+which can be used, respectively, to put arbitrary text or files from the
+filesystem in the store, returning the path to the created item.
+
+This doesn't work so well with gexps, though; you'd have to wrap each ungexped
+use of either of them with `(with-store store (run-with-store store …))`, which
+would be quite tedious.  Thankfully, `(guix gexp)` provides the `plain-file` and
+`local-file` procedures, which return equivalent lowerable objects.  This code
+example builds a directory containing symlinks to files greeting the world:
+
+```scheme
+(use-modules (guix monads)
+             (ice-9 ftw)
+             (ice-9 textual-ports))
+
+(define (build-derivation monadic-drv)
+  (with-store store
+    (run-with-store store
+      (mlet* %store-monad ((drv monadic-drv))
+        (mbegin %store-monad
+          ;; BUILT-DERIVATIONS is the monadic version of BUILD-DERIVATIONS.
+          (built-derivations (list drv))
+          (return (derivation-output-path
+                   (assoc-ref (derivation-outputs drv) "out"))))))))
+                   
+(define world-greeting-output
+  (build-derivation
+   (gexp->derivation "world-greeting"
+     #~(begin
+         (mkdir #$output)
+         (symlink #$(plain-file "hi-world"
+                      "Hi, world!")
+                  (string-append #$output "/hi"))
+         (symlink #$(plain-file "hello-world"
+                      "Hello, world!")
+                  (string-append #$output "/hello"))
+         (symlink #$(plain-file "greetings-world"
+                      "Greetings, world!")
+                  (string-append #$output "/greetings"))))))
+
+;; We turn the list into multiple values using (APPLY VALUES …).
+(apply values
+       (map (lambda (file-path)
+              (let* ((path (string-append world-greeting-output "/" file-path))
+                     (contents (call-with-input-file path get-string-all)))
+                (list path contents)))
+            ;; SCANDIR from (ICE-9 FTW) returns the list of all files in a
+            ;; directory (including ``.'' and ``..'', so we remove them with the
+            ;; second argument, SELECT?, which specifies a predicate).
+            (scandir world-greeting-output
+                     (lambda (path)
+                       (not (or (string=? path ".")
+                                (string=? path "..")))))))
+⇒ ("/gnu/store/…-world-greeting/greetings" "Greetings, world!")
+⇒ ("/gnu/store/…-world-greeting/hello" "Hello, world!")
+⇒ ("/gnu/store/…-world-greeting/hi" "Hi, world!")
+```
+
+Note that we define a procedure for building the output; we will need to build
+more derivations in a very similar fashion later, so it helps to have this to
+reuse instead of copying the code in `world-greeting-output`.
+
+There are many other useful lowerable objects available as part of the gexp
+library.  These include `computed-file`, which accepts a gexp that builds
+the output file, `program-file`, which creates an executable Scheme script in
+the store using a gexp, and `mixed-text-file`, which allows you to, well, mix
+text and lowerable objects; it creates a file from the concatenation of a
+sequence of strings and file-likes.  The
+[G-Expressions](https://guix.gnu.org/manual/en/html_node/G_002dExpressions.html)
+manual page has more details.
+
+So, you may be wondering, at this point: there's so many lowerable objects
+included with the gexps library, surely there must be a way to define more?
+Naturally, there is; this is Scheme, after all!  We simply need to acquaint
+ourselves with the `define-gexp-compiler` macro.
+
+The most basic usage of `define-gexp-compiler` essentially creates a procedure
+that takes as arguments a record to lower, the host system, and the target
+system, and returns a derivation or store item as a monadic value in
+`%store-monad`.
+
+Let's try implementing a lowerable object representing a file that greets the
+world.  First, we'll define the record type:
+
+```scheme
+(use-modules (srfi srfi-9))
+
+(define-record-type <greeting-file>
+  (greeting-file greeting)
+  greeting?
+  (greeting greeting-file-greeting))
+```
+
+Now we use `define-gexp-compiler` like so; note how we can use `lower-object`
+to compile down any sort of lowerable object into the equivalent store item or
+derivation; essentially, `lower-object` is just the procedure for applying the
+right gexp compiler to an object:
+
+```scheme
+(use-modules (ice-9 i18n))
+
+(define-gexp-compiler (greeting-file-compiler
+                       (greeting-file <greeting-file>)
+                       system target)
+  (lower-object
+   (let ((greeting (greeting-file-greeting greeting-file)))
+     (plain-file (string-append greeting "-greeting")
+       (string-append (string-locale-titlecase greeting) ", world!")))))
+```
+
+Let's try it out now.  Here's how we could rewrite our greetings directory
+example from before using `<greeting-file>`:
+
+```scheme
+(define world-greeting-2-output
+  (build-derivation
+   (gexp->derivation "world-greeting-2"
+     #~(begin
+         (mkdir #$output)
+         (symlink #$(greeting-file "hi")
+                  (string-append #$output "/hi"))
+         (symlink #$(greeting-file "hello")
+                  (string-append #$output "/hello"))
+         (symlink #$(greeting-file "greetings")
+                  (string-append #$output "/greetings"))))))
+
+(apply values
+       (map (lambda (file-path)
+              (let* ((path (string-append world-greeting-2-output
+                                          "/" file-path))
+                     (contents (call-with-input-file path get-string-all)))
+                (list path contents)))
+            (scandir world-greeting-2-output
+                     (lambda (path)
+                       (not (or (string=? path ".")
+                                (string=? path "..")))))))
+⇒ ("/gnu/store/…-world-greeting-2/greetings" "Greetings, world!")
+⇒ ("/gnu/store/…-world-greeting-2/hello" "Hello, world!")
+⇒ ("/gnu/store/…-world-greeting-2/hi" "Hi, world!")
+```
+
+Now, this is probably not worth a whole new gexp compiler.  How about something
+a bit more complex?  Sharp-eyed readers who are trying all this in the REPL may
+have noticed the following output when they used `define-gexp-compiler`
+(formatted for ease of reading):
+
+```scheme
+⇒ #<<gexp-compiler>
+    type: #<record-type <greeting-file>>
+    lower: #<procedure … (greeting-file system target)>
+    expand: #<procedure default-expander (thing obj output)>>
+```
+
+Now, the purpose of `type` and `lower` is self-explanatory, but what's this
+`expand` procedure here?  Well, if you recall `file-append`, you may realise
+that the text produced by a gexp compiler for embedding into a gexp doesn't
+necessarily have to be the exact output path of the produced derivation.
+
+There turns out to be another way to write a `define-gexp-compiler` form that
+allows you to specify _both_ the lowering procedure, which produces the
+derivation or store item, and the expanding procedure, which produces the text.
+
+Let's make another record; this one will let us build a store item containing a
+`bin` directory with multiple scripts inside, and expand to the full path to
+that script.
+
+```scheme
+(define-record-type <script-directory>
+  (script-directory scripts)
+  script-directory?
+  (scripts script-directory-scripts))
+```
+
+Here's how we define both a compiler and expander for our new record:
+
+```scheme
+(define-gexp-compiler script-directory-compiler <script-directory>
+  compiler => (lambda (obj system target)
+                (gexp->derivation "script-directory"
+                  #~(let ((bindir (string-append #$output "/bin")))
+                      (mkdir #$output)
+                      (mkdir bindir)
+                      (for-each
+                       (lambda (pair)
+                         (let* ((name (car pair))
+                                (path (cdr pair))
+                                (bin-path (string-append bindir "/" name)))
+                           (symlink path bin-path)))
+                       #$(script-directory-scripts obj)))))
+  expander => (lambda (obj drv output)
+                (string-append output "/bin/" (script-directory-name obj))))
+```
+
+Let's try this out now:
+
+```scheme
+(use-modules (gnu packages vim))
+
+(define script-directory-output
+  (build-derivation
+   (lower-object
+    (script-directory
+     #~'(("irc" . #$(file-append irssi "/bin/irssi"))
+         ("editor" . #$(file-append neovim "/bin/nvim")))))))
+
+(scandir (string-append script-directory-output "/bin"))
+⇒ ("." ".." "editor" "irc")
+```
+
+Who knows why you'd want to do this, but it certainly works!  We've looked at
+why we need gexps, how they work, and how to extend them, and we've now only got
+two more advanced features to cover: cross-build support, and modules.
+
+# Importing External Modules
+
+Let's try using one of the helpful procedures from the `(guix build utils)`
+module in a gexp.
+
+```scheme
+(define silly-directory-output
+  (build-derivation
+   (gexp->derivation "silly-directory"
+     #~(begin
+         (use-modules (guix build utils))
+         (mkdir-p (string-append #$output "/what/a/silly/directory"))))))
+```
+
+Looks fine, right?  We've even got a `use-modules` in th--
+
+```Scheme
+ERROR:
+  1. &store-protocol-error:
+      message: "build of `/gnu/store/…-silly-directory.drv' failed"
+      status: 100
+```
+
+OUTRAGEOUS.  Fortunately, there's an explanation to be found in the Guix build
+log directory, `/var/log/guix/drvs`; locate the file using the first two
+characters of the store hash as the subdirectory, and the rest as the file name,
+and remember to use `zcat` or `zless`, as the logs are gzipped:
+
+```scheme
+Backtrace:
+           9 (primitive-load "/gnu/store/…")
+In ice-9/eval.scm:
+   721:20  8 (primitive-eval (begin (use-modules (guix build #)) (?)))
+In ice-9/psyntax.scm:
+  1230:36  7 (expand-top-sequence ((begin (use-modules (guix ?)) #)) ?)
+  1090:25  6 (parse _ (("placeholder" placeholder)) ((top) #(# # ?)) ?)
+  1222:19  5 (parse _ (("placeholder" placeholder)) ((top) #(# # ?)) ?)
+   259:10  4 (parse _ (("placeholder" placeholder)) (()) _ c&e (eval) ?)
+In ice-9/boot-9.scm:
+  3927:20  3 (process-use-modules _)
+   222:17  2 (map1 (((guix build utils))))
+  3928:31  1 (_ ((guix build utils)))
+   3329:6  0 (resolve-interface (guix build utils) #:select _ #:hide ?)
+
+ice-9/boot-9.scm:3329:6: In procedure resolve-interface:
+no code for module (guix build utils)
+```
+
+It turns out `use-modules` can't actually find `(guix build utils)` at all.
+There's no typo; it's just that to ensure the build is isolated, Guix builds
+`module-import` and `module-importe-compiled` directories, and sets the
+_Guile module path_ within the build environment to contain said directories,
+along with those containing the Guile standard library modules.
+
+So, what to do?  Turns out one of the fields in `<gexp>` is `modules`, which,
+funnily enough, contains the names of the modules which will be used to build
+the aforementioned directories.  To add to this field, we use the
+`with-imported-modules` macro.  (`gexp->derivation` _does_ provide a `modules`
+parameter, but `with-imported-modules` lets you add the required modules
+directly to the gexp value, rather than later on.)
+
+```scheme
+(define silly-directory-output
+  (build-derivation
+   (gexp->derivation "silly-directory"
+     (with-imported-modules '((guix build utils))
+       #~(begin
+           (use-modules (guix build utils))
+           (mkdir-p (string-append #$output "/what/a/silly/directory")))))))
+           
+silly-directory-output
+⇒ "/gnu/store/…-silly-directory"
+```
+
+It works, yay.  It's worth noting that while passing just the list of modules to
+`with-imported-modules` works in this case, this is only because
+`(guix build utils)` has no dependencies on other Guix modules.  Were we to try
+adding, say, `(guix build emacs-build-system)`, we'd need to use the
+`source-module-closure` procedure to add its dependencies to the list:
+
+```scheme
+(source-module-closure '((guix build emacs-build-system)))
+⇒ ((guix build emacs-build-system)
+   (guix build gnu-build-system)
+   (guix build utils)
+   (guix build gremlin)
+   (guix elf)
+   (guix build emacs-utils))
+```
+
+Here's another scenario: what if we want to use a module not from Guix or Guile
+but a third-party library?  In this example, we'll use [guile-json
+](https://github.com/aconchillo/guile-json), a library for converting between
+S-expressions and [JavaScript Object Notation](https://json.org).
+
+We can't just `with-imported-modules` its modules, since it's not part of Guix,
+so `<gexp>` provides another field for this purpose: `extensions`.  Each of
+these extensions is a lowerable object that produces a Guile package directory;
+so usually a package.  Let's try it out.
+
+```scheme
+(use-modules (gnu packages guile))
+
+(define helpful-guide-output
+  (build-derivation
+   (gexp->derivation "json-file"
+     (with-extensions (list guile-json-4)
+       #~(begin
+           (use-modules (json))
+           (mkdir #$output)
+           (call-with-output-file (string-append #$output "/helpful-guide.json")
+             (lambda (port)
+               (scm->json '((truth . "Guix is the best!")
+                            (lies . "Guix isn't the best!"))
+                          port))))))))
+
+(call-with-input-file
+    (string-append helpful-guide-output "/helpful-guide.json")
+  get-string-all)
+⇒ "{\"truth\":\"Guix is the best!\",\"lies\":\"Guix isn't the best!\"}"
+```
+
+Amen to that, `helpful-guide.json`.  Before we continue on to cross-compilation,
+there's one last feature of `with-imported-modules` you should note.  We can
+add modules to a gexp by name, but we can also create entirely new ones with
+lowerable objects, like this pattern, which is used in several places in the
+Guix source code:
+
+```scheme
+(with-imported-modules `(((guix config) => ,(make-config.scm))
+                         …)
+  …)
+```
+
+In case you're wondering, `make-config.scm` is found in `(guix self)` and
+returns a lowerable object that compiles to a version of the `(guix config)`
+module, which contains constants usually substituted into the source code at
+compile time.
+
+# Native Ungexp
+
+There is another piece of syntax we can use with gexps, and it's called
+`ungexp-native`.  This helps us distinguish between native inputs and regular
+host-built inputs in cross-compilation situations.  We'll cover
+cross-compilation in detail at a later date, but the gist of it is that it
+allows you to compile a derivation for one architecture X, the target, using a
+machine of architecture Y, the host, and Guix has excellent support for it.
+
+If we cross-compile a gexp G that _non-natively_ ungexps L1, a lowerable object,
+from architecture Y to architecture X, both G and L1 will be compiled for
+architecture X.  However, if G _natively_ ungexps L1, G will be compiled for X
+and L1 for Y.
+
+Essentially, we use `ungexp-native` in situations where there would be no
+difference between compiling on different architectures (for instance, if `L1`
+were a `plain-file`), or where using L1 built for X would actually _break_ G
+(for instance, if `L1` corresponds to a compiled executable that needs to be run
+during the build; the executable would fail to run on Y if it was built for X.)
+
+The `ungexp-native` macro naturally has a corresponding reader syntax, `#+`, and
+there's also `ungexp-native-splicing`, which is written as `#+@`.  These two
+pieces of syntax are used in the same way as their regular counterparts.
+
+# Conclusion
+
+Mastering gexps is essential to understanding Guix's inner workings, so the aim
+of this blog post is to be as thorough as possible.  However, if you still find
+yourself with questions, please don't hesitate to stop by at the IRC channel
+`#guix:libera.chat` and mailing list `help-guix@gnu.org`; we'll be glad to
+assist you!
+
+#### About GNU Guix
+
+[GNU Guix](https://guix.gnu.org) is a transactional package manager and
+an advanced distribution of the GNU system that [respects user
+freedom](https://www.gnu.org/distros/free-system-distribution-guidelines.html).
+Guix can be used on top of any system running the Hurd or the Linux
+kernel, or it can be used as a standalone operating system distribution
+for i686, x86_64, ARMv7, AArch64 and POWER9 machines.
+
+In addition to standard package management features, Guix supports
+transactional upgrades and roll-backs, unprivileged package management,
+per-user profiles, and garbage collection.  When used as a standalone
+GNU/Linux distribution, Guix offers a declarative, stateless approach to
+operating system configuration management.  Guix is highly customizable
+and hackable through [Guile](https://www.gnu.org/software/guile)
+programming interfaces and extensions to the
+[Scheme](http://schemers.org) language.

base-commit: a81137f5c5eedc0c327e345285d40e84c68ed10b
-- 
2.39.1

[bug#62356] [PATCH guix-artwork] website: posts: Add Dissecting Guix, Part 3: G-Expressions.

[Simon Tournier]

Hi,

Cool!  Thanks, it’s very helpful.

Minor comments.

On mar., 21 mars 2023 at 20:57, "\( via Guix-patches" via <guix-patches@gnu.org> wrote:

[...]

> +The shortcomings of using s-expressions in this way are numerous: we have to
> +convert everything to a derivation before using it, and _inputs are not an
> +inherent aspect of the builder_.  G-expressions were designed to overcome these
> +issues.

Here I would link to the paper introducing G-expressions,

    https://hal.inria.fr/hal-01580582v1


> +# Premortem Examination

[...]

> +Here's an example gexp that is essentially equivalent to our `sexp-builder`:
> +
> +```scheme
> +(use-modules (guix gexp))
> +
> +(define gexp-builder
> +  #~(symlink #$(file-append irssi "/bin/irssi")
> +             #$output))
> +```
> +
> +`gexp-builder` is far more concise than `sexp-builder`; let's examine the syntax
> +and the `<gexp>` object we've created.  To make a gexp, we use the `#~` syntax,
> +equivalent to the `gexp` macro, rather than the `quasiquote` backtick used to
> +create sexps.
> +
> +When we want to embed values from outside as references, we use `#$`, or
> +`ungexp`, which is, in appearance if not function, equivalent to `unquote`
> +(`,`).  `ungexp` can accept any of four reference types:

Well, maybe it is a bit stretching and is probably not natural at all
but I would try to introduce some unquote in sexp-builder.  I think it
would help to see the parallel between S-exp and G-exp; well how G-exp
extend S-exp as you explained in the introduction.

> 
> +- Sexps (strings, lists, etc), which will be embedded literally.

From a stylistic point of view, I would write ’S-expressions’ in plain
and not S-exps or sexps…

> +- Other gexps, embedded literally.

…Similarly for G-expression.  Both over all the post.  Except when it
refers to code as ’gexp-builder’.


> +That's a lot of complexity abstracted away!  For more complex packages and
> +services, especially, gexps are a lifesaver; you can refer to the output paths
> +of inputs just as easily as you would a string constant.  You do, however, have
> +to watch out for situations where `ungexp-native`, written as `#+`, would be
> +preferable over regular `ungexp`, and that's something we'll discuss later.

Before the brief digression, I would do another. ;-)  Mention ,build and
,lower from “guix repl”.

> +A brief digression before we continue: if you'd like to look inside a `<gexp>`

[...]

> +# The Lowerable-Object Hardware Shop

[...]

> +There are many other useful lowerable objects available as part of the gexp
> +library.  These include `computed-file`, which accepts a gexp that builds
> +the output file, `program-file`, which creates an executable Scheme script in
> +the store using a gexp, and `mixed-text-file`, which allows you to, well, mix
> +text and lowerable objects; it creates a file from the concatenation of a
> +sequence of strings and file-likes.  The
> +[G-Expressions](https://guix.gnu.org/manual/en/html_node/G_002dExpressions.html)
> +manual page has more details.

Maybe, I would start another section here; or split with 2 subsections.

> +So, you may be wondering, at this point: there's so many lowerable objects
> +included with the gexps library, surely there must be a way to define more?
> +Naturally, there is; this is Scheme, after all!  We simply need to acquaint
> +ourselves with the `define-gexp-compiler` macro.

[...]

> +Let's try this out now:
> +
> +```scheme
> +(use-modules (gnu packages vim))
> +
> +(define script-directory-output
> +  (build-derivation
> +   (lower-object
> +    (script-directory
> +     #~'(("irc" . #$(file-append irssi "/bin/irssi"))
       ---^

Hum, maybe #~' needs an explanation.  Well, using G-expressions, I am
missing why Schemers are complaining about Haskell syntax. ;-)

> +         ("editor" . #$(file-append neovim "/bin/nvim")))))))
> +
> +(scandir (string-append script-directory-output "/bin"))
> +⇒ ("." ".." "editor" "irc")
> +```
> +
> +Who knows why you'd want to do this, but it certainly works!  We've looked at
> +why we need gexps, how they work, and how to extend them, and we've now only got
> +two more advanced features to cover: cross-build support, and modules.

Here, I would link to another introduction of G-expression,

    https://archive.fosdem.org/2020/schedule/event/gexpressionsguile/

or maybe in the Conclusion section.

> +# Importing External Modules

[...]

> +```scheme
> +(define silly-directory-output

Maybe instead of ’silly’, I would pick another name as ’simple’ or
’empty’ or ’trivial’ or ’not-serious’ or else. :-)

And similarly for snippets from above.


> +# Conclusion
> +
> +Mastering gexps is essential to understanding Guix's inner workings, so the aim
> +of this blog post is to be as thorough as possible.  However, if you still find
> +yourself with questions, please don't hesitate to stop by at the IRC channel
> +`#guix:libera.chat` and mailing list `help-guix@gnu.org`; we'll be glad to
> +assist you!

Maybe, you could link to Arun’s or Marius’s posts; for the ones I am
aware of. :-)

https://www.systemreboot.net/post/deploy-scripts-using-g-expressions
https://gexp.no/blog/guix-drops-part-3-g-expressions.html


Cheers,
simon

[bug#62356] [PATCH guix-artwork] website: posts: Add Dissecting Guix, Part 3: G-Expressions.

[( via Guix-patches via]

Hi,

Thanks for the review! :D

Simon Tournier <zimon.toutoune@gmail.com> writes:
>> +The shortcomings of using s-expressions in this way are numerous: we have to
>> +convert everything to a derivation before using it, and _inputs are not an
>> +inherent aspect of the builder_.  G-expressions were designed to overcome these
>> +issues.
>
> Here I would link to the paper introducing G-expressions,
>
>     https://hal.inria.fr/hal-01580582v1

Good idea.  I'll do that :)

> Well, maybe it is a bit stretching and is probably not natural at all
> but I would try to introduce some unquote in sexp-builder.  I think it
> would help to see the parallel between S-exp and G-exp; well how G-exp
> extend S-exp as you explained in the introduction.

I'll try, but no guarantees I'll be able to make that make sense.

> From a stylistic point of view, I would write ’S-expressions’ in plain
> and not S-exps or sexps…
>
> …Similarly for G-expression.  Both over all the post.  Except when it
> refers to code as ’gexp-builder’.

Okay.

> Before the brief digression, I would do another. ;-)  Mention ,build and
> ,lower from “guix repl”.

,LOWER is mentioned in part 1
<https://guix.gnu.org/en/blog/2023/dissecting-guix-part-1-derivations/>;
I should have mentioned ,BUILD there too, but it's too late now, and
I don't think such an explanation fits a post meant to explain how gexps
work.

> Hum, maybe #~' needs an explanation.  Well, using G-expressions, I am
> missing why Schemers are complaining about Haskell syntax. ;-)

Heh :)  (I think it's more to do with Haskell's complexity than ease of
reading.)  I'll try to add a short note there.

> Here, I would link to another introduction of G-expression,
>
>     https://archive.fosdem.org/2020/schedule/event/gexpressionsguile/
>
> or maybe in the Conclusion section.

Yeah, I'll put the other references in the Conclusion.

> Maybe instead of ’silly’, I would pick another name as ’simple’ or
> ’empty’ or ’trivial’ or ’not-serious’ or else. :-)
>
> And similarly for snippets from above.

Okay.

> Maybe, you could link to Arun’s or Marius’s posts; for the ones I am
> aware of. :-)
>
> https://www.systemreboot.net/post/deploy-scripts-using-g-expressions
> https://gexp.no/blog/guix-drops-part-3-g-expressions.html

Yup, and the FOSDEM talk in the same place.

[bug#62356] [PAtCH guix-artwork] website: posts: Add Dissecting Guix, Part 3: G-Expressions.

[( via Guix-patches via]

* website/posts/dissecting-guix-3-gexps.md: New blog post.
---
 website/posts/dissecting-guix-3-gexps.md | 735 +++++++++++++++++++++++
 1 file changed, 735 insertions(+)
 create mode 100644 website/posts/dissecting-guix-3-gexps.md

diff --git a/website/posts/dissecting-guix-3-gexps.md b/website/posts/dissecting-guix-3-gexps.md
new file mode 100644
index 0000000..cd56243
--- /dev/null
+++ b/website/posts/dissecting-guix-3-gexps.md
@@ -0,0 +1,735 @@
+title: Dissecting Guix, Part 3: G-Expressions
+date: TBC
+author: (
+tags: Dissecting Guix, Functional package management, Programming interfaces, Scheme API
+---
+Welcome back to [Dissecting Guix](https://guix.gnu.org/en/blog/tags/dissecting-guix)!
+Last time, we discussed [monads](https://guix.gnu.org/en/blog/2023/dissecting-guix-part-2-the-store-monad),
+the functional programming idiom used by Guix to thread a store connection
+through a series of store-related operations.
+
+Today, we'll be talking about a concept rather more specific to Guix:
+_g-expressions_.  Being an implementation of the Scheme language, Guile is built
+around [_s-expressions_](https://en.wikipedia.org/wiki/S-expression), which can
+represent, as the saying goes, _code as data_, thanks to the simple structure of
+Scheme forms.
+
+As Guix's package recipes are written in Scheme, it naturally needs some way to
+represent code that is to be run only when the package is built.  Additionally,
+there needs to be some way to reference dependencies and retrieve output paths;
+otherwise, you wouldn't be able to, for instance, create a phase to install a
+file in the output directory.
+
+So, how do we implement this "deferred" code?  Well, initially Guix used plain
+old s-expressions for this purpose.
+
+# Once Upon a Time
+
+Let's say we want to create a store item that's just a symlink to the
+`bin/irssi` file of the `irssi` package.  How would we do that with an
+s-expression?  Well, the s-expression itself, which we call the _builder_, is
+fairly simple:
+
+```scheme
+(define sexp-builder
+  `(let* ((out (assoc-ref %outputs "out"))
+          (irssi (assoc-ref %build-inputs "irssi"))
+          (bin/irssi (string-append irssi "/bin/irssi")))
+     (symlink bin/irssi out)))
+```
+
+If you aren't familliar with the "quoting" syntax used to create s-expressions,
+I strongly recommend that you read the excellent Scheme Primer; specifically,
+section 7, [_Lists and
+"cons"_](https://spritely.institute/static/papers/scheme-primer.html#scheme-lists-and-cons)
+and section 11, [_On the extensibility of Scheme (and Lisps in
+general)_](https://spritely.institute/static/papers/scheme-primer.html#scheme-extensibility)
+
+The `%outputs` and `%build-inputs` variables are bound within builder scripts to
+_association lists_, which are lists of pairs that act like key/value stores,
+for instance:
+
+```scheme
+'(("foo" . "bar")
+  ("floob" . "blarb")
+  ("fvoolag" . "bvarlag"))
+```
+
+To retrieve values from association lists, which are often referred to as
+_alists_, we use the `assoc-ref` procedure:
+
+```scheme
+(assoc-ref '(("boing" . "bouncy")
+             ("floing" . "flouncy"))
+           "boing")
+⇒ "bouncy"
+```
+
+`%outputs`, as the name might suggest, maps derivation output names to the paths
+of their respective store items, the default output being `out`, and
+`%build-inputs` maps inputs labels to their store items.
+
+The builder is the easy part; we now need to turn it into a derivation and tell
+it what `"irssi"` actually refers to.  For this, we use the
+`build-expression->derivation` procedure from `(guix derivations)`:
+
+```scheme
+(use-modules (guix derivations)
+             (guix packages)
+             (guix store)
+             (gnu packages guile)
+             (gnu packages irc))
+
+(with-store store
+  (let ((guile-3.0-drv (package-derivation store guile-3.0))
+        (irssi-drv (package-derivation store irssi)))
+    (build-expression->derivation store "irssi-symlink" sexp-builder
+      #:guile-for-build guile-3.0-drv
+      #:inputs `(("irssi" ,irssi-drv)))))
+⇒ #<derivation /gnu/store/…-irssi-symlink.drv => /gnu/store/…-irssi-symlink …>
+```
+
+There are several things to note here:
+
+- The inputs _must_ all be derivations, so we need to first convert the packages
+  using `package-derivation`.
+- We need to explicitly set `#:guile-for-build`; there's no default value.
+- The `build-expression->derivation` and `package-derivation` procedures are
+  _not_ monadic, so we need to explicitly pass them the store connection.
+
+The shortcomings of using s-expressions in this way are numerous: we have to
+convert everything to a derivation before using it, and _inputs are not an
+inherent aspect of the builder_.  G-expressions were designed to overcome these
+issues.
+
+# Premortem Examination
+
+A g-expression is fundamentally a record of type `<gexp>`, which is, naturally,
+defined in `(guix gexp)`.  The two most important fields of this record type,
+out of a total of five, are `proc` and `references`; the former is a procedure
+that returns the equivalent s-expression, the latter a list containing
+everything from the "outside world" that's used by the g-expression.
+
+When we want to turn the g-expression into something that we can actually run as
+code, we combine these two fields by first building any g-expression inputs that
+can become derivations (leaving alone those that cannot), and then passing the
+built `references` as the arguments of `proc`.
+
+Here's an example g-expression that is essentially equivalent to our
+`sexp-builder`:
+
+```scheme
+(use-modules (guix gexp))
+
+(define gexp-builder
+  #~(symlink #$(file-append irssi "/bin/irssi")
+             #$output))
+```
+
+`gexp-builder` is far more concise than `sexp-builder`; let's examine the syntax
+and the `<gexp>` object we've created.  To make a g-expression, we use the `#~`
+syntax, equivalent to the `gexp` macro, rather than the `quasiquote` backtick
+used to create s-expressions.
+
+When we want to embed values from outside as references, we use `#$`, or
+`ungexp`, which is, in appearance if not function, equivalent to `unquote`
+(`,`).  `ungexp` can accept any of four reference types:
+
+- S-expressions (strings, lists, etc), which will be embedded literally.
+- Other g-expressions, embedded literally.
+- Expressions returning any sort of object that can be lowered into a
+  derivation, such as `<package>`, embedding that object's `out` store item; if
+  the expression is specifically a symbol bound to a buildable object, you can
+  optionally follow it with a colon and an alternative output name, so
+  `package:lib` is permitted, but `(get-package):lib` isn't.
+- The symbol `output`, embedding an output path.  Like symbols bound to
+  buildable objects, this can be followed by a colon and the output name that
+  should be used rather than the default `out`.
+
+All these reference types will be represented by `<gexp-input>` records in the
+`references` field, except for the last kind, which will become `<gexp-output>`
+records.  To give an example of each type of reference (with the return value
+output formatted for easier reading):
+
+```scheme
+(use-modules (gnu packages glib))
+
+#~(list #$"foobar"                         ;s-expression
+        #$#~(string-append "foo" "bar")    ;g-expression
+        #$(file-append irssi "/bin/irssi") ;buildable object (expression)
+        #$glib:bin                         ;buildable object (symbol)
+        #$output:out)                      ;output
+⇒ #<gexp (list #<gexp-input "foobar":out>
+               #<gexp-input #<gexp (string-append "foo" "bar") …>:out>
+               #<gexp-input #<file-append #<package irssi@1.4.3 …> "/bin/irssi">:out>
+               #<gexp-input #<package glib@2.70.2 …>:bin>
+               #<gexp-output out>) …>
+```
+
+Note the use of `file-append` in both the previous example and `gexp-builder`;
+this procedure produces a `<file-append>` object that builds its first argument
+and is embedded as the concatenation of the first argument's output path and the
+second argument, which should be a string.  For instance,
+`(file-append irssi "/bin/irssi")` builds `irssi` and expands to
+`/gnu/store/…-irssi/bin/irssi`, rather than the `/gnu/store/…-irssi` that the
+package alone would be embedded as.
+
+So, now that we have a g-expression, how do we turn it into a derivation?  This
+process is known as _lowering_; it entails the use of the aptly-named
+`lower-gexp` monadic procedure to combine `proc` and `references` and produce a
+`<lowered-gexp>` record, which acts as a sort of intermediate representation
+between g-expressions and derivations.  We can piece apart this lowered form to
+get a sense of what the final derivation's builder script would look like:
+
+```scheme
+(define lowered-gexp-builder
+  (with-store store
+    (run-with-store store
+      (lower-gexp gexp-builder))))
+
+(lowered-gexp-sexp lowered-gexp-builder)
+⇒ (symlink
+   "/gnu/store/…-irssi-1.4.3/bin/irssi"
+   ((@ (guile) getenv) "out"))
+```
+
+And there you have it: a s-expression compiled from a g-expression, ready to be
+written into a builder script file in the store.  So, how exactly do you turn
+this into said derivation?
+
+Well, it turns out that there isn't an interface for turning lowered
+g-expressions into derivations, only one for turning regular g-expressions into
+derivations that first uses `lower-gexp`, then implements the aforementioned
+conversion internally, rather than outsourcing it to some other procedure, so
+that's what we'll use.
+
+Unsurprisingly, that procedure is called `gexp->derivation`, and unlike its
+s-expression equivalent, it's monadic.  (`build-expression->derivation` and
+other deprecated procedures were in Guix since before the monads system
+existed.)
+
+```scheme
+(with-store store
+  (run-with-store store
+    (gexp->derivation "irssi-symlink" gexp-builder)))
+⇒ #<derivation /gnu/store/…-irssi-symlink.drv => /gnu/store/…-irssi-symlink …>
+```
+
+Finally, we have a g-expression-based equivalent to the derivation we earlier
+created with `build-expression->derivation`!  Here's the code we used for the
+s-expression version in full:
+
+```scheme
+(define sexp-builder
+  `(let* ((out (assoc-ref %outputs "out"))
+          (irssi (assoc-ref %build-inputs "irssi"))
+          (bin/irssi (string-append irssi "/bin/irssi")))
+     (symlink bin/irssi out)))
+
+(with-store store
+  (let ((guile-3.0-drv (package-derivation store guile-3.0))
+        (irssi-drv (package-derivation store irssi)))
+    (build-expression->derivation store "irssi-symlink" sexp-builder
+      #:guile-for-build guile-3.0-drv
+      #:inputs `(("irssi" ,irssi-drv)))))
+```
+
+And here's the g-expression equivalent:
+
+```scheme
+(define gexp-builder
+  #~(symlink #$(file-append irssi "/bin/irssi")
+             #$output))
+
+(with-store store
+  (run-with-store store
+    (gexp->derivation "irssi-symlink" gexp-builder)))
+```
+
+That's a lot of complexity abstracted away!  For more complex packages and
+services, especially, g-expressions are a lifesaver; you can refer to the output
+paths of inputs just as easily as you would a string constant.  You do, however,
+have to watch out for situations where `ungexp-native`, written as `#+`, would
+be preferable over regular `ungexp`, and that's something we'll discuss later.
+
+A brief digression before we continue: if you'd like to look inside a `<gexp>`
+record, but you'd rather not build anything, you can use the
+`gexp->approximate-sexp` procedure, which replaces all references with dummy
+values:
+
+```scheme
+(gexp->approximate-sexp gexp-builder)
+⇒ (symlink (*approximate*) (*approximate*))
+```
+
+# The Lowerable-Object Hardware Shop
+
+We've seen two examples already of records we can turn into derivations, which
+are generally referred to as _lowerable objects_ or _file-like objects_:
+
+- `<package>`, a Guix package.
+- `<file-append>`, which wraps another lowerable object and appends a string to
+  the embedded output path when `ungexp`ed.
+
+There are many more available to us.  Recall from the previous post,
+[_The Store Monad_](https://guix.gnu.org/en/blog/2023/dissecting-guix-part-2-the-store-monad),
+that Guix provides the two monadic procedures `text-file` and `interned-file`,
+which can be used, respectively, to put arbitrary text or files from the
+filesystem in the store, returning the path to the created item.
+
+This doesn't work so well with g-expressions, though; you'd have to wrap each
+`ungexp`ed use of either of them with
+`(with-store store (run-with-store store …))`, which would be quite tedious.
+Thankfully, `(guix gexp)` provides the `plain-file` and `local-file` procedures,
+which return equivalent lowerable objects.  This code example builds a directory
+containing symlinks to files greeting the world:
+
+```scheme
+(use-modules (guix monads)
+             (ice-9 ftw)
+             (ice-9 textual-ports))
+
+(define (build-derivation monadic-drv)
+  (with-store store
+    (run-with-store store
+      (mlet* %store-monad ((drv monadic-drv))
+        (mbegin %store-monad
+          ;; BUILT-DERIVATIONS is the monadic version of BUILD-DERIVATIONS.
+          (built-derivations (list drv))
+          (return (derivation-output-path
+                   (assoc-ref (derivation-outputs drv) "out"))))))))
+                   
+(define world-greeting-output
+  (build-derivation
+   (gexp->derivation "world-greeting"
+     #~(begin
+         (mkdir #$output)
+         (symlink #$(plain-file "hi-world"
+                      "Hi, world!")
+                  (string-append #$output "/hi"))
+         (symlink #$(plain-file "hello-world"
+                      "Hello, world!")
+                  (string-append #$output "/hello"))
+         (symlink #$(plain-file "greetings-world"
+                      "Greetings, world!")
+                  (string-append #$output "/greetings"))))))
+
+;; We turn the list into multiple values using (APPLY VALUES …).
+(apply values
+       (map (lambda (file-path)
+              (let* ((path (string-append world-greeting-output "/" file-path))
+                     (contents (call-with-input-file path get-string-all)))
+                (list path contents)))
+            ;; SCANDIR from (ICE-9 FTW) returns the list of all files in a
+            ;; directory (including ``.'' and ``..'', so we remove them with the
+            ;; second argument, SELECT?, which specifies a predicate).
+            (scandir world-greeting-output
+                     (lambda (path)
+                       (not (or (string=? path ".")
+                                (string=? path "..")))))))
+⇒ ("/gnu/store/…-world-greeting/greetings" "Greetings, world!")
+⇒ ("/gnu/store/…-world-greeting/hello" "Hello, world!")
+⇒ ("/gnu/store/…-world-greeting/hi" "Hi, world!")
+```
+
+Note that we define a procedure for building the output; we will need to build
+more derivations in a very similar fashion later, so it helps to have this to
+reuse instead of copying the code in `world-greeting-output`.
+
+There are many other useful lowerable objects available as part of the gexp
+library.  These include `computed-file`, which accepts a gexp that builds
+the output file, `program-file`, which creates an executable Scheme script in
+the store using a g-expression, and `mixed-text-file`, which allows you to,
+well, mix text and lowerable objects; it creates a file from the concatenation
+of a sequence of strings and file-likes.  The
+[G-Expressions](https://guix.gnu.org/manual/en/html_node/G_002dExpressions.html)
+manual page has more details.
+
+So, you may be wondering, at this point: there's so many lowerable objects
+included with the g-expression library, surely there must be a way to define
+more?  Naturally, there is; this is Scheme, after all!  We simply need to
+acquaint ourselves with the `define-gexp-compiler` macro.
+
+The most basic usage of `define-gexp-compiler` essentially creates a procedure
+that takes as arguments a record to lower, the host system, and the target
+system, and returns a derivation or store item as a monadic value in
+`%store-monad`.
+
+Let's try implementing a lowerable object representing a file that greets the
+world.  First, we'll define the record type:
+
+```scheme
+(use-modules (srfi srfi-9))
+
+(define-record-type <greeting-file>
+  (greeting-file greeting)
+  greeting?
+  (greeting greeting-file-greeting))
+```
+
+Now we use `define-gexp-compiler` like so; note how we can use `lower-object`
+to compile down any sort of lowerable object into the equivalent store item or
+derivation; essentially, `lower-object` is just the procedure for applying the
+right gexp-compiler to an object:
+
+```scheme
+(use-modules (ice-9 i18n))
+
+(define-gexp-compiler (greeting-file-compiler
+                       (greeting-file <greeting-file>)
+                       system target)
+  (lower-object
+   (let ((greeting (greeting-file-greeting greeting-file)))
+     (plain-file (string-append greeting "-greeting")
+       (string-append (string-locale-titlecase greeting) ", world!")))))
+```
+
+Let's try it out now.  Here's how we could rewrite our greetings directory
+example from before using `<greeting-file>`:
+
+```scheme
+(define world-greeting-2-output
+  (build-derivation
+   (gexp->derivation "world-greeting-2"
+     #~(begin
+         (mkdir #$output)
+         (symlink #$(greeting-file "hi")
+                  (string-append #$output "/hi"))
+         (symlink #$(greeting-file "hello")
+                  (string-append #$output "/hello"))
+         (symlink #$(greeting-file "greetings")
+                  (string-append #$output "/greetings"))))))
+
+(apply values
+       (map (lambda (file-path)
+              (let* ((path (string-append world-greeting-2-output
+                                          "/" file-path))
+                     (contents (call-with-input-file path get-string-all)))
+                (list path contents)))
+            (scandir world-greeting-2-output
+                     (lambda (path)
+                       (not (or (string=? path ".")
+                                (string=? path "..")))))))
+⇒ ("/gnu/store/…-world-greeting-2/greetings" "Greetings, world!")
+⇒ ("/gnu/store/…-world-greeting-2/hello" "Hello, world!")
+⇒ ("/gnu/store/…-world-greeting-2/hi" "Hi, world!")
+```
+
+Now, this is probably not worth a whole new gexp-compiler.  How about something
+a bit more complex?  Sharp-eyed readers who are trying all this in the REPL may
+have noticed the following output when they used `define-gexp-compiler`
+(formatted for ease of reading):
+
+```scheme
+⇒ #<<gexp-compiler>
+    type: #<record-type <greeting-file>>
+    lower: #<procedure … (greeting-file system target)>
+    expand: #<procedure default-expander (thing obj output)>>
+```
+
+Now, the purpose of `type` and `lower` is self-explanatory, but what's this
+`expand` procedure here?  Well, if you recall `file-append`, you may realise
+that the text produced by a gexp-compiler for embedding into a g-expression
+doesn't necessarily have to be the exact output path of the produced derivation.
+
+There turns out to be another way to write a `define-gexp-compiler` form that
+allows you to specify _both_ the lowering procedure, which produces the
+derivation or store item, and the expanding procedure, which produces the text.
+
+Let's try making another new lowerable object; this one will let us build a
+Guile package and expand to the path to its module directory.  Here's our
+record:
+
+```scheme
+(define-record-type <module-directory>
+  (module-directory package)
+  module-directory?
+  (package module-directory-package))
+```
+
+Here's how we define both a compiler and expander for our new record:
+
+```scheme
+(use-modules (gnu packages guile)
+             (guix utils))
+
+(define lookup-expander (@@ (guix gexp) lookup-expander))
+
+(define-gexp-compiler module-directory-compiler <module-directory>
+  compiler => (lambda (obj system target)
+                (let ((package (module-directory-package obj)))
+                  (lower-object package system #:target target)))
+  expander => (lambda (obj drv output)
+                (let* ((package (module-directory-package obj))
+                       (expander (or (lookup-expander package)
+                                     (lookup-expander drv)))
+                       (out (expander package drv output))
+                       (guile (or (lookup-package-input package "guile")
+                                  guile-3.0))
+                       (version (version-major+minor
+                                 (package-version guile))))
+                  (string-append out "/share/guile/site/" version))))
+```
+
+Let's try this out now:
+
+```scheme
+(use-modules (gnu packages guile-xyz))
+
+(define module-directory-output/guile-webutils
+  (build-derivation
+   (gexp->derivation "module-directory-output"
+     #~(symlink #$(module-directory guile-webutils) #$output))))
+
+(readlink module-directory-output/guile-webutils)
+⇒ "/gnu/store/…-guile-webutils-0.1-1.d309d65/share/guile/site/3.0"
+
+(scandir module-directory-output/guile-webutils)
+⇒ ("." ".." "webutils")
+
+(define module-directory-output/guile2.2-webutils
+  (build-derivation
+   (gexp->derivation "module-directory-output"
+     #~(symlink #$(module-directory guile2.2-webutils) #$output))))
+
+(readlink module-directory-output/guile2.2-webutils)
+⇒ "/gnu/store/…-guile-webutils-0.1-1.d309d65/share/guile/site/2.2"
+
+(scandir module-directory-output/guile2.2-webutils)
+⇒ ("." ".." "webutils")
+```
+
+Who knows why you'd want to do this, but it certainly works!  We've looked at
+why we need g-expressions, how they work, and how to extend them, and we've now
+only got two more advanced features to cover: cross-build support, and modules.
+
+# Importing External Modules
+
+Let's try using one of the helpful procedures from the `(guix build utils)`
+module in a g-expression.
+
+```scheme
+(define simple-directory-output
+  (build-derivation
+   (gexp->derivation "simple-directory"
+     #~(begin
+         (use-modules (guix build utils))
+         (mkdir-p (string-append #$output "/a/rather/simple/directory"))))))
+```
+
+Looks fine, right?  We've even got a `use-modules` in th--
+
+```Scheme
+ERROR:
+  1. &store-protocol-error:
+      message: "build of `/gnu/store/…-simple-directory.drv' failed"
+      status: 100
+```
+
+OUTRAGEOUS.  Fortunately, there's an explanation to be found in the Guix build
+log directory, `/var/log/guix/drvs`; locate the file using the first two
+characters of the store hash as the subdirectory, and the rest as the file name,
+and remember to use `zcat` or `zless`, as the logs are gzipped:
+
+```scheme
+Backtrace:
+           9 (primitive-load "/gnu/store/…")
+In ice-9/eval.scm:
+   721:20  8 (primitive-eval (begin (use-modules (guix build #)) (?)))
+In ice-9/psyntax.scm:
+  1230:36  7 (expand-top-sequence ((begin (use-modules (guix ?)) #)) ?)
+  1090:25  6 (parse _ (("placeholder" placeholder)) ((top) #(# # ?)) ?)
+  1222:19  5 (parse _ (("placeholder" placeholder)) ((top) #(# # ?)) ?)
+   259:10  4 (parse _ (("placeholder" placeholder)) (()) _ c&e (eval) ?)
+In ice-9/boot-9.scm:
+  3927:20  3 (process-use-modules _)
+   222:17  2 (map1 (((guix build utils))))
+  3928:31  1 (_ ((guix build utils)))
+   3329:6  0 (resolve-interface (guix build utils) #:select _ #:hide ?)
+
+ice-9/boot-9.scm:3329:6: In procedure resolve-interface:
+no code for module (guix build utils)
+```
+
+It turns out `use-modules` can't actually find `(guix build utils)` at all.
+There's no typo; it's just that to ensure the build is isolated, Guix builds
+`module-import` and `module-importe-compiled` directories, and sets the
+_Guile module path_ within the build environment to contain said directories,
+along with those containing the Guile standard library modules.
+
+So, what to do?  Turns out one of the fields in `<gexp>` is `modules`, which,
+funnily enough, contains the names of the modules which will be used to build
+the aforementioned directories.  To add to this field, we use the
+`with-imported-modules` macro.  (`gexp->derivation` _does_ provide a `modules`
+parameter, but `with-imported-modules` lets you add the required modules
+directly to the g-expression value, rather than later on.)
+
+```scheme
+(define simple-directory-output
+  (build-derivation
+   (gexp->derivation "simple-directory"
+     (with-imported-modules '((guix build utils))
+       #~(begin
+           (use-modules (guix build utils))
+           (mkdir-p (string-append #$output "/a/rather/simple/directory")))))))
+           
+simple-directory-output
+⇒ "/gnu/store/…-simple-directory"
+```
+
+It works, yay.  It's worth noting that while passing just the list of modules to
+`with-imported-modules` works in this case, this is only because
+`(guix build utils)` has no dependencies on other Guix modules.  Were we to try
+adding, say, `(guix build emacs-build-system)`, we'd need to use the
+`source-module-closure` procedure to add its dependencies to the list:
+
+```scheme
+(use-modules (guix modules))
+
+(source-module-closure '((guix build emacs-build-system)))
+⇒ ((guix build emacs-build-system)
+   (guix build gnu-build-system)
+   (guix build utils)
+   (guix build gremlin)
+   (guix elf)
+   (guix build emacs-utils))
+```
+
+Here's another scenario: what if we want to use a module not from Guix or Guile
+but a third-party library?  In this example, we'll use [guile-json
+](https://github.com/aconchillo/guile-json), a library for converting between
+S-expressions and [JavaScript Object Notation](https://json.org).
+
+We can't just `with-imported-modules` its modules, since it's not part of Guix,
+so `<gexp>` provides another field for this purpose: `extensions`.  Each of
+these extensions is a lowerable object that produces a Guile package directory;
+so usually a package.  Let's try it out using the `guile-json-4` package to
+produce a JSON file from a Scheme value within a g-expression.
+
+```scheme
+(define helpful-guide-output
+  (build-derivation
+   (gexp->derivation "json-file"
+     (with-extensions (list guile-json-4)
+       #~(begin
+           (use-modules (json))
+           (mkdir #$output)
+           (call-with-output-file (string-append #$output "/helpful-guide.json")
+             (lambda (port)
+               (scm->json '((truth . "Guix is the best!")
+                            (lies . "Guix isn't the best!"))
+                          port))))))))
+
+(call-with-input-file
+    (string-append helpful-guide-output "/helpful-guide.json")
+  get-string-all)
+⇒ "{\"truth\":\"Guix is the best!\",\"lies\":\"Guix isn't the best!\"}"
+```
+
+Amen to that, `helpful-guide.json`.  Before we continue on to cross-compilation,
+there's one last feature of `with-imported-modules` you should note.  We can
+add modules to a g-expression by name, but we can also create entirely new ones
+using lowerable objects, such as in this pattern, which is used in several
+places in the Guix source code to make an appropriately-configured
+`(guix config)` module available:
+
+```scheme
+(with-imported-modules `(((guix config) => ,(make-config.scm))
+                         …)
+  …)
+```
+
+In case you're wondering, `make-config.scm` is found in `(guix self)` and
+returns a lowerable object that compiles to a version of the `(guix config)`
+module, which contains constants usually substituted into the source code at
+compile time.
+
+# Native `ungexp`
+
+There is another piece of syntax we can use with g-expressions, and it's called
+`ungexp-native`.  This helps us distinguish between native inputs and regular
+host-built inputs in cross-compilation situations.  We'll cover
+cross-compilation in detail at a later date, but the gist of it is that it
+allows you to compile a derivation for one architecture X, the target, using a
+machine of architecture Y, the host, and Guix has excellent support for it.
+
+If we cross-compile a g-expression G that _non-natively_ `ungexp`s L1, a
+lowerable object, from architecture Y to architecture X, both G and L1 will be
+compiled for architecture X.  However, if G _natively_ `ungexp`s L1, G will be
+compiled for X and L1 for Y.
+
+Essentially, we use `ungexp-native` in situations where there would be no
+difference between compiling on different architectures (for instance, if `L1`
+were a `plain-file`), or where using L1 built for X would actually _break_ G
+(for instance, if `L1` corresponds to a compiled executable that needs to be run
+during the build; the executable would fail to run on Y if it was built for X.)
+
+The `ungexp-native` macro naturally has a corresponding reader syntax, `#+`, and
+there's also `ungexp-native-splicing`, which is written as `#+@`.  These two
+pieces of syntax are used in the same way as their regular counterparts.
+
+# Conclusion
+
+What have we learned in this post?  To summarise:
+
++ G-expressions are essentially abstractions on top of s-expressions used in
+  Guix to stage code, often for execution within a build environment or a
+  Shepherd service script.
++ Much like you can `unquote` external values within a `quasiquote` form, you
+  can `ungexp` external values to access them within a `gexp` form.  The key
+  difference is that you may use not only s-expressions with `ungexp`, but other
+  g-expressions and lowerable objects too.
++ When a lowerable object is used with `ungexp`, the g-expression ultimately
+  receives the path to the object's store item (or whatever string the lowerable
+  object's expander produces), rather than the object itself.
++ A lowerable object is any record that has a "g-expression compiler" defined
+  for it using the `define-gexp-compiler` macro.  G-expression compilers always
+  contain a `compiler` procedure, which converts an appropriate record into a
+  derivation, and sometimes an `expander` procedure, which produces the string
+  that is to be expanded to within g-expressions when the object is `ungexp`ed.
++ G-expressions record the list of modules available in their environment, which
+  you may expand using `with-imported-modules` to add Guix modules, and
+  `with-extensions` to add modules from third-party Guile packages.
++ `ungexp-native` may be used within g-expressions to compile lowerable objects
+  for the host rather than the target system in cross-compilation scenarios.
+
+Mastering g-expressions is essential to understanding Guix's inner workings, so
+the aim of this blog post is to be as thorough as possible.  However, if you
+still find yourself with questions, please don't hesitate to stop by at the IRC
+channel `#guix:libera.chat` and mailing list `help-guix@gnu.org`; we'll be glad
+to assist you!
+
+Also note that due to the centrality of g-expressions to Guix, there exist a
+plethora of alternative resources on this topic; here are some which you may
+find useful:
+
++ Arun Isaac's
+  [post](https://www.systemreboot.net/post/deploy-scripts-using-g-expressions)
+  on using g-expressions with `guix deploy`.
++ Marius Bakke's
+  ["Guix Drops" post](https://gexp.no/blog/guix-drops-part-3-g-expressions.html)
+  which explains g-expressions in a more "top-down" way.
++ This 2020
+  [FOSDEM talk](https://archive.fosdem.org/2020/schedule/event/gexpressionsguile/)
+  by Christopher Marusich on the uses of g-expressions.
++ And, of course, the one and only original
+  [g-expression paper](https://hal.inria.fr/hal-01580582v1) by Ludovic Courtès,
+  the original author of Guix.
+
+#### About GNU Guix
+
+[GNU Guix](https://guix.gnu.org) is a transactional package manager and
+an advanced distribution of the GNU system that [respects user
+freedom](https://www.gnu.org/distros/free-system-distribution-guidelines.html).
+Guix can be used on top of any system running the Hurd or the Linux
+kernel, or it can be used as a standalone operating system distribution
+for i686, x86_64, ARMv7, AArch64 and POWER9 machines.
+
+In addition to standard package management features, Guix supports
+transactional upgrades and roll-backs, unprivileged package management,
+per-user profiles, and garbage collection.  When used as a standalone
+GNU/Linux distribution, Guix offers a declarative, stateless approach to
+operating system configuration management.  Guix is highly customizable
+and hackable through [Guile](https://www.gnu.org/software/guile)
+programming interfaces and extensions to the
+[Scheme](http://schemers.org) language.

base-commit: 9d8f36a722d33f75e2e081a8d8f04cf20c4d3511
-- 
2.39.2

[bug#62356] [PATCH guix-artwork] website: posts: Add Dissecting Guix, Part 3: G-Expressions.

[Ludovic Courtès]

Hello,

"(" <paren@disroot.org> skribis:

> * website/posts/dissecting-guix-3-gexps.md: New blog post.

This looks perfect to me, great job!

If there are no objections, I’ll push it tomorrow noon (GMT), which
should be better timing than now.

Thanks a lot, and thanks Simon for reviewing!

Ludo’.

[bug#62356] [PATCH guix-artwork] website: posts: Add Dissecting Guix, Part 3: G-Expressions.

[( via Guix-patches via]

Ludovic Courtès <ludo@gnu.org> writes:
> Hello,
>
> "(" <paren@disroot.org> skribis:
>
>> * website/posts/dissecting-guix-3-gexps.md: New blog post.
>
> This looks perfect to me, great job!

Thank you very much :D  It was a bit easier to write than the last one,
thank goodness :P

> If there are no objections, I’ll push it tomorrow noon (GMT), which
> should be better timing than now.

\o/

[bug#62356] [PATCH guix-artwork] website: posts: Add Dissecting Guix, Part 3: G-Expressions.

[( via Guix-patches via]

Ludovic Courtès <ludo@gnu.org> writes:
> This looks perfect to me, great job!
>
> If there are no objections, I’ll push it tomorrow noon (GMT), which
> should be better timing than now.

Just to be sure, you're talking about v2, right?  Because this showed up
in mu4e as a reply to v1 :)

[bug#62356] [PATCH guix-artwork] website: posts: Add Dissecting Guix, Part 3: G-Expressions.

[Simon Tournier]

Hi,

On Tue, 18 Apr 2023 at 21:08, "\( via Guix-patches" via <guix-patches@gnu.org> wrote:

> Just to be sure, you're talking about v2, right?  Because this showed up
> in mu4e as a reply to v1 :)

Well, if I am not missing a detail, Ludo (id:87edohhsmw.fsf_-_@gnu.org)
is replying to

    id:20230415222954.567-1-paren@disroot.org
    [bug#62356] [PAtCH guix-artwork] website: posts: Add Dissecting Guix, Part 3: G-Expressions.
    Sat, 15 Apr 2023 23:29:54 +0100

as shown by [1] or [2].  Maybe a bug of mue4? :-)

However, using the mboxes from [3] (bug in Mumi?), note that:

--8<---------------cut here---------------start------------->8---
$ for i in $(seq 0 6); do printf "$i "; cat 62356-$i.mbox | grep Message-I ;done

0 Message-Id: <20230321205749.4974-1-paren@disroot.org>
1 Message-ID: <87ile1glv6.fsf@gmail.com>
2 Message-ID: <87ile1glv6.fsf@gmail.com>
3 Message-ID: <87r0sp6ppe.fsf@disroot.org>
4 Message-ID: <87r0sp6ppe.fsf@disroot.org>
5 Message-Id: <20230415222954.567-1-paren@disroot.org>
6 Message-ID: <87edohhsmw.fsf_-_@gnu.org
--8<---------------cut here---------------end--------------->8---

which is different from notmuch or emacs-debbugs
(gnus-summary-show-raw-article)

    20230321205749.4974-1-paren@disroot.org
    87ile1glv6.fsf@gmail.com
    87r0sp6ppe.fsf@disroot.org
    20230415222954.567-1-paren@disroot.org
    87edohhsmw.fsf_-_@gnu.org
    877cu9ymwi.fsf@disroot.org
    87354xymu3.fsf@disroot.org

Last, using emacs-debbugs, the Ludo’s message contains:

--8<---------------cut here---------------start------------->8---
In-Reply-To: <20230415222954.567-1-paren@disroot.org> (paren@disroot.org's
 message of "Sat, 15 Apr 2023 23:29:54 +0100")
--8<---------------cut here---------------end--------------->8---

1: https://yhetil.org/guix/20230415222954.567-1-paren@disroot.org/#r
2: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62356;mbox=yes;msg=23
3: https://issues.guix.gnu.org/issue/62356

Cheers,
simon

[bug#62356] [PATCH guix-artwork] website: posts: Add Dissecting Guix, Part 3: G-Expressions.

[Ludovic Courtès]

"(" <paren@disroot.org> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>> This looks perfect to me, great job!
>>
>> If there are no objections, I’ll push it tomorrow noon (GMT), which
>> should be better timing than now.
>
> Just to be sure, you're talking about v2, right?  Because this showed up
> in mu4e as a reply to v1 :)

Yes, I’m talking about v2.  :-)

[bug#62356] [PAtCH guix-artwork] website: posts: Add Dissecting Guix, Part 3: G-Expressions.

[Théo Maxime Tyburn]

I like that V2 !
The references to other tutorials are quite nice :)

bug#62356: [PATCH guix-artwork] website: posts: Add Dissecting Guix, Part 3: G-Expressions.

[Ludovic Courtès]

Done!

  https://guix.gnu.org/en/blog/2023/dissecting-guix-part-3-g-expressions/

Thank you!

Ludo’.

[bug#62356] [PATCH guix-artwork] website: posts: Add Dissecting Guix, Part 3: G-Expressions.

[( via Guix-patches via]

Ludovic Courtès <ludo@gnu.org> writes:
> Done!
>
>   https://guix.gnu.org/en/blog/2023/dissecting-guix-part-3-g-expressions/
>
> Thank you!

\o/