* Dissecting Guix -- blog post series
@ 2022-12-08 18:24 (
2022-12-09 7:31 ` 宋文武
` (4 more replies)
0 siblings, 5 replies; 23+ messages in thread
From: ( @ 2022-12-08 18:24 UTC (permalink / raw)
To: guix-devel
[-- Attachment #1: Type: text/plain, Size: 21256 bytes --]
Heya!
Some of you may have seen on IRC that I've been writing a post for the Guix
blog that I hope will form the first part of a series. This series aims to
dissect the internals of Guix, from bottom to top. Perhaps they could go in
the cookbook once the series is done? My aim is to write the following posts
(complete with cheesy titles :P), hopefully one per week:
* Dissecting Guix, Part 1: Derivations
Discusses derivations, the bottom layer of the Guix compilation tower, and
dissects some example derivations.
A draft of this post may be found below. Please feel free to critique! :)
* Dissecting Guix, Part 2: The Humble G-Expression
Talks about g-expressions and why they're necessary, explains file-like objects,
and provides demonstrations of each kind of file-like.
* Dissecting Guix, Part 3: Packages
A walkthrough for package creation.
* Dissecting Guix, Part 4: Monads
``mlet'', ``>>='', ``define-monad'', ``return'', and all that.
* Dissecting Guix, Part 5: Profiles and Search Paths
Explores profiles and search paths.
* Dissecting Guix, Part 6: Goings-On in the Build Container
Explains build systems, examines a build script, and talks about what exactly
happens when a package is built. Also demonstrates some of the
``(guix build utils)'' APIs.
* Dissecting Guix, Part 7: Record Types
Demonstrates the ``(guix records)'' API in all its glory.
* Dissecting Guix, Part 8: Substitutes and Grafts
Discusses substitutes, and that persistent thorn in our sides, grafting.
* Dissecting Guix, Part 9: Cross-Compilation
Building packages on architecture X for architecture Y, and how that all
works.
* Dissecting Guix, Part 10: Services
Walks you through the process of creating a service, and thouroughly explains
system configuration.
* Dissecting Guix, Part 11: Home Services
Similar to Part 9, except it's about ``guix home'', of course.
* Dissecting Guix, Part 12: Writing a Subcommand
Guides you through the process of adding a new command to Guix with the
extensions feature, demonstrating several utility APIs in the process.
* Dissecting Guix, Part 13: Lending a Hand
How to edit the Guix source code and submit patches to be reviewed by the
lovely Guix community!
-- (
title: Dissecting Guix, Part 1: Derivations and Derivation
date: TBC
author: (
tags: Dissecting Guix, Functional package management, Programming interfaces, Scheme API
---
To a new user, Guix's functional architecture can seem quite alien, and
possibly offputting. With a combination of extensive `#guix`-querying,
determined manual-reading, and plenty of source-perusing, they may
eventually figure out how everything fits together by themselves, but this
can be frustrating and often takes a fairly long time.
However, once you peel back the layers, the "Nix way" is actually rather
elegant, if perhaps not as simple as the mutable, imperative style
implemented by the likes of [`dpkg`](https://wiki.debian.org/dpkg) and,
[`pacman`](https://wiki.archlinux.org/title/pacman). This series of blog
posts will cover basic Guix concepts, taking a "ground-up" approach by
dealing with lower-level concepts first, and hopefully make those months of
information-gathering unnecessary.
Before we dig in to Guix-specific concepts, we'll need to learn about one
inherited from [Nix](https://nixos.org), the original functional package
manager and the inspiration for Guix; the idea of a _derivation_ and its
corresponding _store items_.
These concepts were originally described by Eelco Dolstra, the author of Nix,
in their [PhD thesis](https://edolstra.github.io/pubs/phd-thesis.pdf); see
_§ 2.1 The Nix store_ and _§ 2.4 Store Derivations_.
# Store Items
As you almost certainly know, everything that Guix builds is stored in the
_store_, which is almost always the `/gnu/store` directory. It's the job of
the [`guix-daemon`](https://guix.gnu.org/manual/en/html_node/Invoking-guix_002ddaemon.html)
to manage the store and build things. If you run
[`guix build PKG`](https://guix.gnu.org/manual/en/html_node/Invoking-guix-build.html),
`PKG` will be built or downloaded from a substitute server if available, and
a path to an item in the store will be displayed.
```
$ guix build irssi
/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3
```
This item contains the final result of building [`irssi`](https://irssi.org).
Let's peek inside:
```
$ ls $(guix build irssi)
bin/ etc/ include/ lib/ share/
$ ls $(guix build irssi)/bin
irssi*
```
`irssi` is quite a simple package. What about a more complex one, like
[`glib`](https://docs.gtk.org/glib)?
```
$ guix build glib
/gnu/store/bx8qq76idlmjrlqf1faslsq6zjc6f426-glib-2.73.3-bin
/gnu/store/j65bhqwr7qq7l77nj0ahmk1f1ilnjr3a-glib-2.73.3-debug
/gnu/store/3pn4ll6qakgfvfpc4mw89qrrbsgj3jf3-glib-2.73.3-doc
/gnu/store/dvsk6x7d26nmwsqhnzws4iirb6dhhr1d-glib-2.73.3
/gnu/store/4c8ycz501n2d0xdi4blahvnbjhd5hpa8-glib-2.73.3-static
```
`glib` produces five `/gnu/store` items, because it's possible for a package
to produce multiple [outputs](https://guix.gnu.org/manual/en/html_node/Packages-with-Multiple-Outputs.html).
Each output can be referred to separately, by prefixing a package's name with
`:OUTPUT` where supported. For example, this
[`guix install`](https://guix.gnu.org/manual/en/html_node/Invoking-guix-package.html)
invocation will add `glib`'s `bin` output to your profile:
```
$ guix install glib:bin
```
The default output is `out`, so when you pass `glib` by itself to that
command, it will actually install `glib:out` to the profile.
`guix build` also provides the `--source` flag, which produces the store
item corresponding to the given package's downloaded source code.
```
$ guix build --source irssi
/gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz
$ guix build --source glib
/gnu/store/d22wzjq3xm3q8hwnhbgk2xd3ph7lb6ay-glib-2.73.3.tar.xz
```
But how does Guix know how to build these store outputs in the first place?
That's where derivations come in.
# `.drv` Files
You've probably seen these being printed by the Guix CLI now and again.
Derivations, represented in the daemon's eyes by `.drv` files, contain
instructions for building store items. We can retrieve the paths of
these `.drv` files with the `guix build --derivations` command:
```
$ guix build --derivations irssi
/gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv
```
`guix build` can actually also accept derivation paths as an argument, in
lieu of a package, like so:
```
$ guix build /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv
/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3
```
Let's look inside this derivation file.
```
Derive([("out","/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3","","")],[("/gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv",["out"]),("/gnu/store/baqpbl4wck7nkxrbyc9nlhma7kq5dyfl-guile-2.0.14.drv",["out"]),("/gnu/store/bfirgq65ndhf63nn4q6vlkbha9zd931q-openssl-1.1.1l.drv",["out"]),("/gnu/store/gjwpqzvfhz13shix6a6cs2hjc18pj7wy-module-import-compiled.drv",["out"]),("/gnu/store/ij8651x4yh53hhcn6qw2644nhh2s8kcn-glib-2.70.2.drv",["out"]),("/gnu/store/jg2vv6yc2yqzi3qzs82dxvqmi5k21lhy-irssi-1.4.3.drv",["out"]),("/gnu/store/qggpjl9g6ic3cq09qrwkm0dfsdjf7pyr-glibc-utf8-locales-2.33.drv",["out"]),("/gnu/store/zafabw13yyhz93jwrcz7axak1kn1f2cx-openssl-1.1.1s.drv",["out"])],["/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import","/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"],"x86_64-linux","/gnu/store/hnr4r2d0h0xarx52i6jq9gvsrlc3q81a-guile-2.0.14/bin/guile",["--no-auto-compile","-L","/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import","-C","/gnu/store/6rkkvvb7pl1l9ng8vvywvwf357vhm3va-module-import-compiled","/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"],[("allowSubstitutes","0"),("guix properties","((type . graft) (graft (count . 2)))"),("out","/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3"),("preferLocalBuild","1")])
```
It's... not exactly human-readable. We could try to format it and break
it down, but it'd still be pretty hard to understand, since `.drv` files
contain no labels for the "arguments" or any other human-readable indicators.
Instead, we're going to explore derivations in a Guile REPL.
# Exploring Guix Interactively
Before we continue, we'll want to start a REPL, so that we can try out
the Guix Guile API interactively. To run a REPL in the terminal, simply
[call `guix repl`](https://guix.gnu.org/manual/devel/en/html_node/Using-Guix-Interactively.html).
If you're using Emacs, you can instead install [Geiser](https://nongnu.org/geiser),
which provides a comfortable Emacs UI for various Lisp REPLs, invoke
`guix repl --listen=tcp:37146 &`, and type `M-x geiser-connect RET RET RET` to
connect to the running Guile instance.
There are a few Guix modules we'll need. Run this Scheme code to import
them:
```scheme
(use-modules (guix derivations)
(guix gexp)
(guix packages)
(guix store)
(gnu packages glib)
(gnu packages irc))
```
We now have access to the store, G-expression, package, and derivation
APIs, along with the `irssi` and `glib` `<package>` objects.
# Creating a `<derivation>`
The Guix API for derivations revolves around the `<derivation>` record,
which is the Scheme representation of that whole block of text surrounded by
`Derive(...)`. If we look in `guix/derivations.scm`, we can see that it's
defined like this:
```scheme
(define-immutable-record-type <derivation>
(make-derivation outputs inputs sources system builder args env-vars
file-name)
derivation?
(outputs derivation-outputs) ; list of name/<derivation-output> pairs
(inputs derivation-inputs) ; list of <derivation-input>
(sources derivation-sources) ; list of store paths
(system derivation-system) ; string
(builder derivation-builder) ; store path
(args derivation-builder-arguments) ; list of strings
(env-vars derivation-builder-environment-vars) ; list of name/value pairs
(file-name derivation-file-name)) ; the .drv file name
```
With the exception of `file-name`, each of those fields corresponds to
an "argument" in the `Derive(...)` form. Before we can examine them,
though, we need to figure out how to _lower_ that `irssi` `<package>`
object into a derivation.
The procedure we use to turn a high-level object like `<package>` into a
derivation is called `lower-object`; more on that in a future post.
However, this doesn't produce a derivation:
```scheme
(pk (lower-object irssi))
;;; (#<procedure 7fe17c7af540 at guix/store.scm:1994:2 (state)>)
```
`pk` is an abbreviation for the procedure `peek`, which takes the given
object, writes a representation of it to the output, and returns it.
It's especially handy when you want to view an intermediate value in a
complex expression.
The returned object is a procedure that needs to be evaluated in the
context of a store connection. We do this by first using `with-store`
to connect to the store and bind the connection to a name, then wrapping
the `lower-object` call with `run-with-store`:
```scheme
(define irssi-drv
(pk (with-store %store
(run-with-store %store
(lower-object irssi)))))
;;; (#<derivation /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7fe1902b6140>)
(define glib-drv
(pk (with-store %store
(run-with-store %store
(lower-object glib)))))
;;; (#<derivation /gnu/store/81qqs7xah2ln39znrji4r6xj85zi15bi-glib-2.70.2.drv => /gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin /gnu/store/22mkp8cr6rxg6w8br9q8dbymf51b44m8-glib-2.70.2-debug /gnu/store/a6qb5arvir4vm1zlkp4chnl7d8qzzd7x-glib-2.70.2 /gnu/store/y4ak268dcdwkc6lmqfk9g1dgk2jr9i34-glib-2.70.2-static 7fe17ca13b90>)
```
And we have liftoff! Now we've got two `<derivation>` records to play
with.
# Exploring `<derivation>`
The first "argument" in the `.drv` file is `outputs`, which tells the
Guix daemon about the outputs that this build can produce:
```scheme
(define irssi-outputs
(pk (derivation-outputs irssi-drv)))
;;; ((("out" . #<<derivation-output> path: "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3" hash-algo: #f hash: #f recursive?: #f>)))
(pk (assoc-ref irssi-outputs "out"))
(define glib-outputs
(pk (derivation-outputs glib-drv)))
;;; ((("bin" . #<<derivation-output> path: "/gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin" hash-algo: #f hash: #f recursive?: #f>) ("debug" . #<<derivation-output> path: "/gnu/store/22mkp8cr6rxg6w8br9q8dbymf51b44m8-glib-2.70.2-debug" hash-algo: #f hash: #f recursive?: #f>) ("out" . #<<derivation-output> path: "/gnu/store/a6qb5arvir4vm1zlkp4chnl7d8qzzd7x-glib-2.70.2" hash-algo: #f hash: #f recursive?: #f>) ("static" . #<<derivation-output> path: "/gnu/store/y4ak268dcdwkc6lmqfk9g1dgk2jr9i34-glib-2.70.2-static" hash-algo: #f hash: #f recursive?: #f>)))
(pk (assoc-ref glib-outputs "bin"))
;;; (#<<derivation-output> path: "/gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin" hash-algo: #f hash: #f recursive?: #f>)
```
It's a simple association list mapping output names to `<derivation-output>`
records, and it's equivalent to the first "argument" in the `.drv` file:
```
[ ("out", "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3", "", "")
]
```
The `hash-algo` and `hash` fields are for storing the content hash and the
algorithm used with that hash for what we term a _fixed-output derivation_,
which is essentially a derivation where we know what the hash of the content
will be in advance. For instance, `origin`s produce fixed-output derivations:
```scheme
(define irssi-src-drv
(pk (with-store %store
(run-with-store %store
(lower-object (package-source irssi))))))
;;; (#<derivation /gnu/store/mcz3vzq7lwwaqjb8dy7cd69lvmi6d241-irssi-1.4.3.tar.xz.drv => /gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz 7fe17b3c8d70>)
(define irssi-src-outputs
(pk (derivation-outputs irssi-src-drv)))
;;; ((("out" . #<<derivation-output> path: "/gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz" hash-algo: sha256 hash: #vu8(185 63 113 82 35 163 34 230 127 66 182 26 8 165 18 174 41 227 75 212 165 61 127 34 55 102 102 10 170 90 4 52) recursive?: #f>)))
(pk (assoc-ref irssi-src-outputs "out"))
;;; (#<<derivation-output> path: "/gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz" hash-algo: sha256 hash: #vu8(185 63 113 82 35 163 34 230 127 66 182 26 8 165 18 174 41 227 75 212 165 61 127 34 55 102 102 10 170 90 4 52) recursive?: #f>)
```
Note how the `hash` and `hash-algo` now have values.
Perceptive readers may note that the `<derivation-output>` has four fields,
whereas the tuple in the `.drv` file only has three (minus the label). If
we read the source for `write-derivation`, we can see that the `recursive?`
field is serialised by prefixing the `hash-algo` with `r:` if it's true:
```scheme
;;; guix/derivations.scm:630:2
(define (write-output output port)
(match output
((name . ($ <derivation-output> path hash-algo hash recursive?))
(write-tuple (list name path
(if hash-algo
(string-append (if recursive? "r:" "")
(symbol->string hash-algo))
"")
(or (and=> hash bytevector->base16-string)
""))
write-escaped-string
port))))
```
The purpose of `recursive?` is difficult to explain, and is out of scope for
this post.
The next field is `inputs`, which corresponds to, you guessed it, the
second pseudo-"argument" in the `.drv` file format:
```
[ ("/gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv", ["out"]),
("/gnu/store/baqpbl4wck7nkxrbyc9nlhma7kq5dyfl-guile-2.0.14.drv", ["out"]),
("/gnu/store/bfirgq65ndhf63nn4q6vlkbha9zd931q-openssl-1.1.1l.drv", ["out"]),
("/gnu/store/gjwpqzvfhz13shix6a6cs2hjc18pj7wy-module-import-compiled.drv", ["out"]),
("/gnu/store/ij8651x4yh53hhcn6qw2644nhh2s8kcn-glib-2.70.2.drv", ["out"]),
("/gnu/store/jg2vv6yc2yqzi3qzs82dxvqmi5k21lhy-irssi-1.4.3.drv", ["out"]),
("/gnu/store/qggpjl9g6ic3cq09qrwkm0dfsdjf7pyr-glibc-utf8-locales-2.33.drv", ["out"]),
("/gnu/store/zafabw13yyhz93jwrcz7axak1kn1f2cx-openssl-1.1.1s.drv", ["out"])
]
```
Here, each tuple specifies a derivation that needs to be built before this
derivation can be built, and the outputs of the derivation that the build
process of this derivation uses. Let's grab us the Scheme equivalent:
```scheme
(define irssi-inputs
(pk (derivation-inputs irssi-drv)))
;;; [a fairly large amount of output]
(pk (car irssi-inputs))
;;; (#<<derivation-input> drv: #<derivation /gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv => /gnu/store/2jj2mxn6wfrcw7i85nywk71mmqbnhzps-glib-2.70.2 7fe1902b6640> sub-derivations: ("out")>)
```
Unlike `derivation-outputs`, `derivation-inputs` maps 1:1 to the `.drv`
form; the `drv` field is a `<derivation>` to be built, and the
`sub-derivations` field is a list of outputs.
The other pseudo-"arguments" are pretty simple; none of them involve new
records. The third is `derivation-sources`, which contains a list of all
store items used in the build which aren't themselves built using
derivations, whereas `derivation-inputs` contains the dependencies which
are.
This list usually just contains the path to the Guile _build script_ that
realises the store items when run, which we'll examine in a later post, and
the path to a directory containing extra modules to add to the build script's
`%load-path`, called `/gnu/store/...-module-import`.
The next "argument" is self-explanatory: `derivation-system`, which specifies
the Nix system we're building for. Next is `derivation-builder`, pointing to
the `guile` executable that runs the script; and the second-to-last is
`derivation-args`, which is a list of arguments to pass to `derivation-builder`.
Note how we use `-L` and `-C` to extend the Guile `%load-path` and
`%load-compiled-path` to include the `module-import` and `module-import-compiled`
directories:
```scheme
(pk (derivation-system irssi-drv))
;;; ("x86_64-linux")
(pk (derivation-builder irrsi-drv))
;;; ("/gnu/store/hnr4r2d0h0xarx52i6jq9gvsrlc3q81a-guile-2.0.14/bin/guile")
(pk (derivation-builder-arguments irrsi-drv))
;;; (("--no-auto-compile" "-L" "/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import" "-C" "/gnu/store/6rkkvvb7pl1l9ng8vvywvwf357vhm3va-module-import-compiled" "/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"))
```
The final "argument" contains a list of environment variables to set before
we start the build process:
```scheme
(pk (derivation-builder-environment-vars irssi-drv))
;;; ((("allowSubstitutes" . "0") ("guix properties" . "((type . graft) (graft (count . 2)))") ("out" . "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3") ("preferLocalBuild" . "1")))
```
Obviously, the last record field, `derivation-file-name`, simply allows you to
retrieve the path to the `.drv` file in Scheme, and so isn't represented
in a serialised derivation. Speaking of serialisation, to convert between the
`.drv` text format and the Scheme `<derivation>` record, you can use
`write-derivation`, `read-derivation`, and `read-derivation-from-file`.
# Conclusion
Derivations are one of Guix's most important concepts, but are fairly easy to
understand once you get past the obtuse `.drv` file format. They provide the
Guix daemon with the initial instructions that it uses to build store items
like packages, origins, and other file-likes such as `computed-file` and
`local-file`, which will be discussed in a future post!
#### About GNU Guix
[GNU Guix](https://guix.gnu.org) is a transactional package manager and
an advanced distribution of the GNU system that [respects user
freedom](https://www.gnu.org/distros/free-system-distribution-guidelines.html).
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, AArch64 and POWER9 machines.
In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through [Guile](https://www.gnu.org/software/guile)
programming interfaces and extensions to the
[Scheme](http://schemers.org) language.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Dissecting Guix -- blog post series
2022-12-08 18:24 Dissecting Guix -- blog post series (
@ 2022-12-09 7:31 ` 宋文武
2022-12-09 7:33 ` (
2022-12-10 21:25 ` Mekeor Melire
` (3 subsequent siblings)
4 siblings, 1 reply; 23+ messages in thread
From: 宋文武 @ 2022-12-09 7:31 UTC (permalink / raw)
To: (; +Cc: guix-devel
"(" <paren@disroot.org> writes:
> Heya!
>
> Some of you may have seen on IRC that I've been writing a post for the Guix
> blog that I hope will form the first part of a series. This series aims to
> dissect the internals of Guix, from bottom to top. Perhaps they could go in
> the cookbook once the series is done? My aim is to write the following posts
> (complete with cheesy titles :P), hopefully one per week:
>
> * Dissecting Guix, Part 1: Derivations
>
> Discusses derivations, the bottom layer of the Guix compilation tower, and
> dissects some example derivations.
>
> A draft of this post may be found below. Please feel free to critique! :)
Great, thank you!
I think it's missing what "build-derivations" do, or "Part 0: Store".
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Dissecting Guix -- blog post series
2022-12-09 7:31 ` 宋文武
@ 2022-12-09 7:33 ` (
2022-12-09 9:32 ` bokr
0 siblings, 1 reply; 23+ messages in thread
From: ( @ 2022-12-09 7:33 UTC (permalink / raw)
To: 宋文武; +Cc: guix-devel
[-- Attachment #1: Type: text/plain, Size: 270 bytes --]
On Fri Dec 9, 2022 at 7:31 AM GMT, 宋文武 wrote:
> I think it's missing what "build-derivations" do, or "Part 0: Store".
Hmm, do you mean adding an example of building a derivation in Scheme with
``build-derivations''? I'll definitiely add that :)
-- (
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Dissecting Guix -- blog post series
2022-12-09 7:33 ` (
@ 2022-12-09 9:32 ` bokr
2022-12-09 17:25 ` (
0 siblings, 1 reply; 23+ messages in thread
From: bokr @ 2022-12-09 9:32 UTC (permalink / raw)
To: (; +Cc: 宋文武, guix-devel
Hi,
On +2022-12-09 07:33:16 +0000, ( wrote:
> On Fri Dec 9, 2022 at 7:31 AM GMT, 宋文武 wrote:
> > I think it's missing what "build-derivations" do, or "Part 0: Store".
>
> Hmm, do you mean adding an example of building a derivation in Scheme with
> ``build-derivations''? I'll definitiely add that :)
>
> -- (
Where is a "What can I trust?" section?
I.e., when some well-meaning enthusiast says,
--8<---------------cut here---------------start------------->8---
"Can we package the "warm-fuzzy-floss-name" package that they have at
http://www.lesser-known-but-nice-sounding-distro.org/fantabulous/"
--8<---------------cut here---------------end--------------->8---
How does a gullible noob like me know what the dangers might be, (e.g. http:)
and how to avoid (most of) them by finding a guix version that has been
gone through with a fine-tooth comb by trusted guix devs and has been
re-hosted at gitlab or gnu.org, etc ... for added security?
--
Regards,
Bengt Richter
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Dissecting Guix -- blog post series
2022-12-09 9:32 ` bokr
@ 2022-12-09 17:25 ` (
2022-12-12 13:46 ` Bengt Richter
0 siblings, 1 reply; 23+ messages in thread
From: ( @ 2022-12-09 17:25 UTC (permalink / raw)
To: bokr; +Cc: 宋文武, guix-devel
[-- Attachment #1: Type: text/plain, Size: 434 bytes --]
Heya,
On Fri Dec 9, 2022 at 9:32 AM GMT, wrote:
> How does a gullible noob like me know what the dangers might be, (e.g. http:)
> and how to avoid (most of) them by finding a guix version that has been
> gone through with a fine-tooth comb by trusted guix devs and has been
> re-hosted at gitlab or gnu.org, etc ... for added security?
Sorry, I don't really understand; how is this relevant to derivations? :)
-- (
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Dissecting Guix -- blog post series
2022-12-08 18:24 Dissecting Guix -- blog post series (
2022-12-09 7:31 ` 宋文武
@ 2022-12-10 21:25 ` Mekeor Melire
2022-12-11 10:08 ` (
2022-12-12 19:31 ` [PATCH guix-artwork v2] website: posts: Add Dissecting Guix, Part 1: Derivations (
` (2 subsequent siblings)
4 siblings, 1 reply; 23+ messages in thread
From: Mekeor Melire @ 2022-12-10 21:25 UTC (permalink / raw)
To: (; +Cc: guix-devel
[-- Attachment #1: Type: text/plain, Size: 23281 bytes --]
2022-12-08 18:24 paren@disroot.org:
> Some of you may have seen on IRC that I've been writing a post for the Guix blog that I hope will form the first part of a series. This series aims to dissect the internals of Guix, from bottom to top.
Great, I'm looking forward to read it! Especially, personally, I'm eager for the rather fundamental concepts of Guix (and Nix).
> Perhaps they could go in the cookbook once the series is done?
Yes, I personally think the content should be published at two places at the same time: OTOH, it should either be incorporated into either the manual or the cookbook; and OTOH, it should be published as a blog-post. The blog-post should also link to the respective section of the manual or cookbook in a preamble.
> * Dissecting Guix, Part 1: Derivations
> * Dissecting Guix, Part 2: The Humble G-Expression
> * Dissecting Guix, Part 3: Packages
> * Dissecting Guix, Part 4: Monads
> * Dissecting Guix, Part 5: Profiles and Search Paths
> * Dissecting Guix, Part 6: Goings-On in the Build Container
> * Dissecting Guix, Part 7: Record Types
> * Dissecting Guix, Part 8: Substitutes and Grafts
> * Dissecting Guix, Part 9: Cross-Compilation
I think it'd also be fine to come up with the titles during the process of writing as sometimes that process itself is insightful. E.g. I could imagine parts 2 and 4 to collapse. Maybe, maybe not, you'll see.
> * Dissecting Guix, Part 10: Services
>
> Walks you through the process of creating a service, and thouroughly explains system configuration.
How'd this part differ from section "12.18 Defining Services" of the manual?
> * Dissecting Guix, Part 11: Home Services
> * Dissecting Guix, Part 12: Writing a Subcommand
> * Dissecting Guix, Part 13: Lending a Hand
>
> How to edit the Guix source code and submit patches to be reviewed by the lovely Guix community!
How'd this part differ from sections 22 and "22.6 Submitting Patches" from the manual?
Now, as for the actual article. Firstly, I added some comments below. Secondly, I created a "patch" suggesting some changes.
By the way, the text does not seem to strictly fill columns at a certain line width. So I did not bother to fill columns in the "patch".
> title: Dissecting Guix, Part 1: Derivations and Derivation
> date: TBC
> author: (
> tags: Dissecting Guix, Functional package management, Programming interfaces, Scheme API
> ---
> To a new user, Guix's functional architecture can seem quite alien, and
> possibly offputting. With a combination of extensive `#guix`-querying,
> determined manual-reading, and plenty of source-perusing, they may
> eventually figure out how everything fits together by themselves, but this
> can be frustrating and often takes a fairly long time.
>
> However, once you peel back the layers, the "Nix way" is actually rather
> elegant, if perhaps not as simple as the mutable, imperative style
> implemented by the likes of [`dpkg`](https://wiki.debian.org/dpkg) and,
> [`pacman`](https://wiki.archlinux.org/title/pacman). This series of blog
> posts will cover basic Guix concepts, taking a "ground-up" approach by
> dealing with lower-level concepts first, and hopefully make those months of
> information-gathering unnecessary.
>
> Before we dig in to Guix-specific concepts, we'll need to learn about one
> inherited from [Nix](https://nixos.org), the original functional package
> manager and the inspiration for Guix; the idea of a _derivation_ and its
> corresponding _store items_.
>
> These concepts were originally described by Eelco Dolstra, the author of Nix,
> in their [PhD thesis](https://edolstra.github.io/pubs/phd-thesis.pdf); see
> _§ 2.1 The Nix store_ and _§ 2.4 Store Derivations_.
>
> # Store Items
>
> As you almost certainly know, everything that Guix builds is stored in the
> _store_, which is almost always the `/gnu/store` directory. It's the job of
> the [`guix-daemon`](https://guix.gnu.org/manual/en/html_node/Invoking-guix_002ddaemon.html)
> to manage the store and build things. If you run
> [`guix build PKG`](https://guix.gnu.org/manual/en/html_node/Invoking-guix-build.html),
> `PKG` will be built or downloaded from a substitute server if available, and
> a path to an item in the store will be displayed.
>
> ```
> $ guix build irssi
> /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3
> ```
>
> This item contains the final result of building [`irssi`](https://irssi.org).
> Let's peek inside:
>
> ```
> $ ls $(guix build irssi)
> bin/ etc/ include/ lib/ share/
> $ ls $(guix build irssi)/bin
> irssi*
> ```
>
> `irssi` is quite a simple package. What about a more complex one, like
> [`glib`](https://docs.gtk.org/glib)?
>
> ```
> $ guix build glib
> /gnu/store/bx8qq76idlmjrlqf1faslsq6zjc6f426-glib-2.73.3-bin
> /gnu/store/j65bhqwr7qq7l77nj0ahmk1f1ilnjr3a-glib-2.73.3-debug
> /gnu/store/3pn4ll6qakgfvfpc4mw89qrrbsgj3jf3-glib-2.73.3-doc
> /gnu/store/dvsk6x7d26nmwsqhnzws4iirb6dhhr1d-glib-2.73.3
> /gnu/store/4c8ycz501n2d0xdi4blahvnbjhd5hpa8-glib-2.73.3-static
> ```
>
> `glib` produces five `/gnu/store` items, because it's possible for a package
> to produce multiple [outputs](https://guix.gnu.org/manual/en/html_node/Packages-with-Multiple-Outputs.html).
> Each output can be referred to separately, by prefixing a package's name with
> `:OUTPUT` where supported. For example, this
> [`guix install`](https://guix.gnu.org/manual/en/html_node/Invoking-guix-package.html)
> invocation will add `glib`'s `bin` output to your profile:
>
> ```
> $ guix install glib:bin
> ```
>
> The default output is `out`, so when you pass `glib` by itself to that
> command, it will actually install `glib:out` to the profile.
>
> `guix build` also provides the `--source` flag, which produces the store
> item corresponding to the given package's downloaded source code.
>
> ```
> $ guix build --source irssi
> /gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz
> $ guix build --source glib
> /gnu/store/d22wzjq3xm3q8hwnhbgk2xd3ph7lb6ay-glib-2.73.3.tar.xz
> ```
>
> But how does Guix know how to build these store outputs in the first place?
> That's where derivations come in.
>
> # `.drv` Files
>
> You've probably seen these being printed by the Guix CLI now and again.
> Derivations, represented in the daemon's eyes by `.drv` files, contain
> instructions for building store items. We can retrieve the paths of
> these `.drv` files with the `guix build --derivations` command:
>
> ```
> $ guix build --derivations irssi
> /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv
> ```
>
> `guix build` can actually also accept derivation paths as an argument, in
> lieu of a package, like so:
>
> ```
> $ guix build /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv
> /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3
> ```
>
> Let's look inside this derivation file.
>
> ```
> Derive([("out","/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3","","")],[("/gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv",["out"]),("/gnu/store/baqpbl4wck7nkxrbyc9nlhma7kq5dyfl-guile-2.0.14.drv",["out"]),("/gnu/store/bfirgq65ndhf63nn4q6vlkbha9zd931q-openssl-1.1.1l.drv",["out"]),("/gnu/store/gjwpqzvfhz13shix6a6cs2hjc18pj7wy-module-import-compiled.drv",["out"]),("/gnu/store/ij8651x4yh53hhcn6qw2644nhh2s8kcn-glib-2.70.2.drv",["out"]),("/gnu/store/jg2vv6yc2yqzi3qzs82dxvqmi5k21lhy-irssi-1.4.3.drv",["out"]),("/gnu/store/qggpjl9g6ic3cq09qrwkm0dfsdjf7pyr-glibc-utf8-locales-2.33.drv",["out"]),("/gnu/store/zafabw13yyhz93jwrcz7axak1kn1f2cx-openssl-1.1.1s.drv",["out"])],["/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import","/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"],"x86_64-linux","/gnu/store/hnr4r2d0h0xarx52i6jq9gvsrlc3q81a-guile-2.0.14/bin/guile",["--no-auto-compile","-L","/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import","-C","/gnu/store/6rkkvvb7pl1l9ng8vvywvwf357vhm3va-module-import-compiled","/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"],[("allowSubstitutes","0"),("guix properties","((type . graft) (graft (count . 2)))"),("out","/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3"),("preferLocalBuild","1")])
> ```
>
> It's... not exactly human-readable. We could try to format it and break
> it down, but it'd still be pretty hard to understand, since `.drv` files
> contain no labels for the "arguments" or any other human-readable indicators.
> Instead, we're going to explore derivations in a Guile REPL.
>
> # Exploring Guix Interactively
>
> Before we continue, we'll want to start a REPL, so that we can try out
> the Guix Guile API interactively. To run a REPL in the terminal, simply
> [call `guix repl`](https://guix.gnu.org/manual/devel/en/html_node/Using-Guix-Interactively.html).
>
> If you're using Emacs, you can instead install [Geiser](https://nongnu.org/geiser),
> which provides a comfortable Emacs UI for various Lisp REPLs, invoke
> `guix repl --listen=tcp:37146 &`, and type `M-x geiser-connect RET RET RET` to
> connect to the running Guile instance.
This approach did not work nicely for me. In particular, the last command made Emacs freeze (even with and 'emacs -Q'). And then, when evaluating the first code block below, i.e. the module-import, I got the warning "Unknown meta command: geiser-no-values". That's why, in the "patch", I suggest to simply run 'M-x geiser RET' which works just fine.
(I guess this is due to the fact that "Geiser invokes geiser-guile-binary with certain flags": <https://gitlab.com/emacs-guix/emacs-guix/-/issues/16>.)
> There are a few Guix modules we'll need. Run this Scheme code to import
> them:
>
> ```scheme
> (use-modules (guix derivations)
> (guix gexp)
> (guix packages)
> (guix store)
> (gnu packages glib)
> (gnu packages irc))
> ```
>
> We now have access to the store, G-expression, package, and derivation
> APIs, along with the `irssi` and `glib` `<package>` objects.
>
> # Creating a `<derivation>`
>
> The Guix API for derivations revolves around the `<derivation>` record,
> which is the Scheme representation of that whole block of text surrounded by
> `Derive(...)`. If we look in `guix/derivations.scm`, we can see that it's
> defined like this:
>
> ```scheme
> (define-immutable-record-type <derivation>
> (make-derivation outputs inputs sources system builder args env-vars
> file-name)
> derivation?
> (outputs derivation-outputs) ; list of name/<derivation-output> pairs
> (inputs derivation-inputs) ; list of <derivation-input>
> (sources derivation-sources) ; list of store paths
> (system derivation-system) ; string
> (builder derivation-builder) ; store path
> (args derivation-builder-arguments) ; list of strings
> (env-vars derivation-builder-environment-vars) ; list of name/value pairs
> (file-name derivation-file-name)) ; the .drv file name
> ```
>
> With the exception of `file-name`, each of those fields corresponds to
> an "argument" in the `Derive(...)` form. Before we can examine them,
> though, we need to figure out how to _lower_ that `irssi` `<package>`
> object into a derivation.
>
> The procedure we use to turn a high-level object like `<package>` into a
> derivation is called `lower-object`; more on that in a future post.
> However, this doesn't produce a derivation:
>
> ```scheme
> (pk (lower-object irssi))
> ;;; (#<procedure 7fe17c7af540 at guix/store.scm:1994:2 (state)>)
> ```
>
> `pk` is an abbreviation for the procedure `peek`, which takes the given
> object, writes a representation of it to the output, and returns it.
> It's especially handy when you want to view an intermediate value in a
> complex expression.
>
> The returned object is a procedure that needs to be evaluated in the
> context of a store connection. We do this by first using `with-store`
> to connect to the store and bind the connection to a name, then wrapping
> the `lower-object` call with `run-with-store`:
>
> ```scheme
> (define irssi-drv
> (pk (with-store %store
> (run-with-store %store
> (lower-object irssi)))))
> ;;; (#<derivation /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7fe1902b6140>)
>
> (define glib-drv
> (pk (with-store %store
> (run-with-store %store
> (lower-object glib)))))
> ;;; (#<derivation /gnu/store/81qqs7xah2ln39znrji4r6xj85zi15bi-glib-2.70.2.drv => /gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin /gnu/store/22mkp8cr6rxg6w8br9q8dbymf51b44m8-glib-2.70.2-debug /gnu/store/a6qb5arvir4vm1zlkp4chnl7d8qzzd7x-glib-2.70.2 /gnu/store/y4ak268dcdwkc6lmqfk9g1dgk2jr9i34-glib-2.70.2-static 7fe17ca13b90>)
> ```
>
> And we have liftoff! Now we've got two `<derivation>` records to play
> with.
>
> # Exploring `<derivation>`
I suggest to use subheadings within this chapter because otherwise readers loose the overview.
> The first "argument" in the `.drv` file is `outputs`, which tells the
> Guix daemon about the outputs that this build can produce:
>
> ```scheme
> (define irssi-outputs
> (pk (derivation-outputs irssi-drv)))
> ;;; ((("out" . #<<derivation-output> path: "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3" hash-algo: #f hash: #f recursive?: #f>)))
>
> (pk (assoc-ref irssi-outputs "out"))
>
> (define glib-outputs
> (pk (derivation-outputs glib-drv)))
> ;;; ((("bin" . #<<derivation-output> path: "/gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin" hash-algo: #f hash: #f recursive?: #f>) ("debug" . #<<derivation-output> path: "/gnu/store/22mkp8cr6rxg6w8br9q8dbymf51b44m8-glib-2.70.2-debug" hash-algo: #f hash: #f recursive?: #f>) ("out" . #<<derivation-output> path: "/gnu/store/a6qb5arvir4vm1zlkp4chnl7d8qzzd7x-glib-2.70.2" hash-algo: #f hash: #f recursive?: #f>) ("static" . #<<derivation-output> path: "/gnu/store/y4ak268dcdwkc6lmqfk9g1dgk2jr9i34-glib-2.70.2-static" hash-algo: #f hash: #f recursive?: #f>)))
>
> (pk (assoc-ref glib-outputs "bin"))
> ;;; (#<<derivation-output> path: "/gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin" hash-algo: #f hash: #f recursive?: #f>)
> ```
>
> It's a simple association list mapping output names to `<derivation-output>`
> records, and it's equivalent to the first "argument" in the `.drv` file:
>
> ```
> [ ("out", "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3", "", "")
> ]
> ```
>
> The `hash-algo` and `hash` fields are for storing the content hash and the
> algorithm used with that hash for what we term a _fixed-output derivation_,
> which is essentially a derivation where we know what the hash of the content
> will be in advance. For instance, `origin`s produce fixed-output derivations:
>
> ```scheme
> (define irssi-src-drv
> (pk (with-store %store
> (run-with-store %store
> (lower-object (package-source irssi))))))
> ;;; (#<derivation /gnu/store/mcz3vzq7lwwaqjb8dy7cd69lvmi6d241-irssi-1.4.3.tar.xz.drv => /gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz 7fe17b3c8d70>)
>
> (define irssi-src-outputs
> (pk (derivation-outputs irssi-src-drv)))
> ;;; ((("out" . #<<derivation-output> path: "/gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz" hash-algo: sha256 hash: #vu8(185 63 113 82 35 163 34 230 127 66 182 26 8 165 18 174 41 227 75 212 165 61 127 34 55 102 102 10 170 90 4 52) recursive?: #f>)))
>
> (pk (assoc-ref irssi-src-outputs "out"))
> ;;; (#<<derivation-output> path: "/gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz" hash-algo: sha256 hash: #vu8(185 63 113 82 35 163 34 230 127 66 182 26 8 165 18 174 41 227 75 212 165 61 127 34 55 102 102 10 170 90 4 52) recursive?: #f>)
> ```
>
> Note how the `hash` and `hash-algo` now have values.
>
> Perceptive readers may note that the `<derivation-output>` has four fields,
> whereas the tuple in the `.drv` file only has three (minus the label). If
> we read the source for `write-derivation`, we can see that the `recursive?`
> field is serialised by prefixing the `hash-algo` with `r:` if it's true:
>
> ```scheme
> ;;; guix/derivations.scm:630:2
>
> (define (write-output output port)
> (match output
> ((name . ($ <derivation-output> path hash-algo hash recursive?))
> (write-tuple (list name path
> (if hash-algo
> (string-append (if recursive? "r:" "")
> (symbol->string hash-algo))
> "")
> (or (and=> hash bytevector->base16-string)
> ""))
> write-escaped-string
> port))))
> ```
>
> The purpose of `recursive?` is difficult to explain, and is out of scope for
> this post.
If the purpose is out of scope, then we should not dive in that deeply. Especially, I'd suggest skip the code snippet.
> The next field is `inputs`, which corresponds to, you guessed it, the
> second pseudo-"argument" in the `.drv` file format:
>
> ```
> [ ("/gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv", ["out"]),
> ("/gnu/store/baqpbl4wck7nkxrbyc9nlhma7kq5dyfl-guile-2.0.14.drv", ["out"]),
> ("/gnu/store/bfirgq65ndhf63nn4q6vlkbha9zd931q-openssl-1.1.1l.drv", ["out"]),
> ("/gnu/store/gjwpqzvfhz13shix6a6cs2hjc18pj7wy-module-import-compiled.drv", ["out"]),
> ("/gnu/store/ij8651x4yh53hhcn6qw2644nhh2s8kcn-glib-2.70.2.drv", ["out"]),
> ("/gnu/store/jg2vv6yc2yqzi3qzs82dxvqmi5k21lhy-irssi-1.4.3.drv", ["out"]),
> ("/gnu/store/qggpjl9g6ic3cq09qrwkm0dfsdjf7pyr-glibc-utf8-locales-2.33.drv", ["out"]),
> ("/gnu/store/zafabw13yyhz93jwrcz7axak1kn1f2cx-openssl-1.1.1s.drv", ["out"])
> ]
> ```
>
> Here, each tuple specifies a derivation that needs to be built before this
> derivation can be built, and the outputs of the derivation that the build
> process of this derivation uses. Let's grab us the Scheme equivalent:
>
> ```scheme
> (define irssi-inputs
> (pk (derivation-inputs irssi-drv)))
> ;;; [a fairly large amount of output]
>
> (pk (car irssi-inputs))
> ;;; (#<<derivation-input> drv: #<derivation /gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv => /gnu/store/2jj2mxn6wfrcw7i85nywk71mmqbnhzps-glib-2.70.2 7fe1902b6640> sub-derivations: ("out")>)
> ```
>
> Unlike `derivation-outputs`, `derivation-inputs` maps 1:1 to the `.drv`
> form; the `drv` field is a `<derivation>` to be built, and the
> `sub-derivations` field is a list of outputs.
>
> The other pseudo-"arguments" are pretty simple; none of them involve new
Here you write "pretty simple". Later you also write "self-explanatory" and "obviously". I suggest to let the reader decide what's simple.
> records. The third is `derivation-sources`, which contains a list of
> all store items used in the build which aren't themselves built using
> derivations, whereas `derivation-inputs` contains the dependencies
> which are.
>
> This list usually just contains the path to the Guile _build script_ that
> realises the store items when run, which we'll examine in a later post, and
> the path to a directory containing extra modules to add to the build script's
> `%load-path`, called `/gnu/store/...-module-import`.
>
> The next "argument" is self-explanatory: `derivation-system`, which specifies
> the Nix system we're building for. Next is `derivation-builder`, pointing to
> the `guile` executable that runs the script; and the second-to-last is
> `derivation-args`, which is a list of arguments to pass to `derivation-builder`.
> Note how we use `-L` and `-C` to extend the Guile `%load-path` and
> `%load-compiled-path` to include the `module-import` and `module-import-compiled`
> directories:
>
> ```scheme
> (pk (derivation-system irssi-drv))
> ;;; ("x86_64-linux")
>
> (pk (derivation-builder irrsi-drv))
> ;;; ("/gnu/store/hnr4r2d0h0xarx52i6jq9gvsrlc3q81a-guile-2.0.14/bin/guile")
>
> (pk (derivation-builder-arguments irrsi-drv))
> ;;; (("--no-auto-compile" "-L" "/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import" "-C" "/gnu/store/6rkkvvb7pl1l9ng8vvywvwf357vhm3va-module-import-compiled" "/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"))
> ```
>
> The final "argument" contains a list of environment variables to set before
> we start the build process:
>
> ```scheme
> (pk (derivation-builder-environment-vars irssi-drv))
> ;;; ((("allowSubstitutes" . "0") ("guix properties" . "((type . graft) (graft (count . 2)))") ("out" . "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3") ("preferLocalBuild" . "1")))
> ```
>
> Obviously, the last record field, `derivation-file-name`, simply allows you to
Before this point, the record fields have been called '"argument"'s all the time. I think it's not nice style to carry on a quoted term through many paragraphs like this. Let's simply write "record field" all the time, instead.
> retrieve the path to the `.drv` file in Scheme, and so isn't
> represented in a serialised derivation. Speaking of serialisation, to
> convert between the `.drv` text format and the Scheme `<derivation>`
> record, you can use `write-derivation`, `read-derivation`, and
> `read-derivation-from-file`.
I think an example for invoking read-derivation-from-file would round up this tutorial really nicely because it'd close the circle between .drv-files and <derivation>-objects.
> # Conclusion
>
> Derivations are one of Guix's most important concepts, but are fairly easy to
> understand once you get past the obtuse `.drv` file format. They provide the
> Guix daemon with the initial instructions that it uses to build store items
> like packages, origins, and other file-likes such as `computed-file` and
> `local-file`, which will be discussed in a future post!
Here, in the conclusion, IMHO, there could be another brief listing of all fields of a derivation.
> #### About GNU Guix
>
> [GNU Guix](https://guix.gnu.org) is a transactional package manager and
> an advanced distribution of the GNU system that [respects user
> freedom](https://www.gnu.org/distros/free-system-distribution-guidelines.html).
> Guix can be used on top of any system running the Hurd or the Linux
> kernel, or it can be used as a standalone operating system distribution
> for i686, x86_64, ARMv7, AArch64 and POWER9 machines.
>
> In addition to standard package management features, Guix supports
> transactional upgrades and roll-backs, unprivileged package management,
> per-user profiles, and garbage collection. When used as a standalone
> GNU/Linux distribution, Guix offers a declarative, stateless approach to
> operating system configuration management. Guix is highly customizable
> and hackable through [Guile](https://www.gnu.org/software/guile)
> programming interfaces and extensions to the
> [Scheme](http://schemers.org) language.
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-many-different-rather-small-changes.patch --]
[-- Type: text/x-patch, Size: 8259 bytes --]
From 2abcfaff2f3203d5e568044beffbb964684b99a0 Mon Sep 17 00:00:00 2001
From: user <>
Date: Sun, 11 Dec 2022 00:44:07 +0100
Subject: [PATCH] many different, rather small changes
---
one.md | 52 +++++++++++++++++++++++++++++++++-------------------
1 file changed, 33 insertions(+), 19 deletions(-)
diff --git a/one.md b/one.md
index cb2c9d9..ccd0397 100644
--- a/one.md
+++ b/one.md
@@ -4,7 +4,7 @@ author: (
tags: Dissecting Guix, Functional package management, Programming interfaces, Scheme API
---
To a new user, Guix's functional architecture can seem quite alien, and
-possibly offputting. With a combination of extensive `#guix`-querying,
+possibly be off-putting. With a combination of extensive `#guix`-querying,
determined manual-reading, and plenty of source-perusing, they may
eventually figure out how everything fits together by themselves, but this
can be frustrating and often takes a fairly long time.
@@ -118,7 +118,7 @@ Derive([("out","/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3","","")]
It's... not exactly human-readable. We could try to format it and break
it down, but it'd still be pretty hard to understand, since `.drv` files
-contain no labels for the "arguments" or any other human-readable indicators.
+contain no labels for their fields or any other human-readable indicators.
Instead, we're going to explore derivations in a Guile REPL.
# Exploring Guix Interactively
@@ -128,9 +128,8 @@ the Guix Guile API interactively. To run a REPL in the terminal, simply
[call `guix repl`](https://guix.gnu.org/manual/devel/en/html_node/Using-Guix-Interactively.html).
If you're using Emacs, you can instead install [Geiser](https://nongnu.org/geiser),
-which provides a comfortable Emacs UI for various Lisp REPLs, invoke
-`guix repl --listen=tcp:37146 &`, and type `M-x geiser-connect RET RET RET` to
-connect to the running Guile instance.
+which provides a comfortable Emacs UI for various Lisp REPLs, run
+`M-x geiser RET`.
There are a few Guix modules we'll need. Run this Scheme code to import
them:
@@ -149,7 +148,7 @@ APIs, along with the `irssi` and `glib` `<package>` objects.
# Creating a `<derivation>`
-The Guix API for derivations revolves around the `<derivation>` record,
+The Guix API for derivations revolves around the [`<derivation>` record](https://guix.gnu.org/en/manual/en/html_node/Derivations.html),
which is the Scheme representation of that whole block of text surrounded by
`Derive(...)`. If we look in `guix/derivations.scm`, we can see that it's
defined like this:
@@ -170,7 +169,7 @@ defined like this:
```
With the exception of `file-name`, each of those fields corresponds to
-an "argument" in the `Derive(...)` form. Before we can examine them,
+a field in the `Derive(...)` form. Before we can examine them,
though, we need to figure out how to _lower_ that `irssi` `<package>`
object into a derivation.
@@ -183,7 +182,7 @@ However, this doesn't produce a derivation:
;;; (#<procedure 7fe17c7af540 at guix/store.scm:1994:2 (state)>)
```
-`pk` is an abbreviation for the procedure `peek`, which takes the given
+`pk` is an abbreviation for the Guile procedure `peek`, which takes the given
object, writes a representation of it to the output, and returns it.
It's especially handy when you want to view an intermediate value in a
complex expression.
@@ -212,7 +211,9 @@ with.
# Exploring `<derivation>`
-The first "argument" in the `.drv` file is `outputs`, which tells the
+## Outputs
+
+The first derivation record field in the `.drv` file is `outputs`, which tells the
Guix daemon about the outputs that this build can produce:
```scheme
@@ -221,6 +222,7 @@ Guix daemon about the outputs that this build can produce:
;;; ((("out" . #<<derivation-output> path: "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3" hash-algo: #f hash: #f recursive?: #f>)))
(pk (assoc-ref irssi-outputs "out"))
+;;; (#<<derivation-output> path: "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.1" hash-algo: #f hash: #f recursive?: #f>)
(define glib-outputs
(pk (derivation-outputs glib-drv)))
@@ -231,7 +233,7 @@ Guix daemon about the outputs that this build can produce:
```
It's a simple association list mapping output names to `<derivation-output>`
-records, and it's equivalent to the first "argument" in the `.drv` file:
+records, and it's equivalent to the first field in the `.drv` file:
```
[ ("out", "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3", "", "")
@@ -260,7 +262,7 @@ will be in advance. For instance, `origin`s produce fixed-output derivations:
Note how the `hash` and `hash-algo` now have values.
-Perceptive readers may note that the `<derivation-output>` has four fields,
+You might have noted that the `<derivation-output>` has four fields,
whereas the tuple in the `.drv` file only has three (minus the label). If
we read the source for `write-derivation`, we can see that the `recursive?`
field is serialised by prefixing the `hash-algo` with `r:` if it's true:
@@ -285,8 +287,10 @@ field is serialised by prefixing the `hash-algo` with `r:` if it's true:
The purpose of `recursive?` is difficult to explain, and is out of scope for
this post.
+## Inputs
+
The next field is `inputs`, which corresponds to, you guessed it, the
-second pseudo-"argument" in the `.drv` file format:
+second field in the `.drv` file format:
```
[ ("/gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv", ["out"]),
@@ -317,8 +321,9 @@ Unlike `derivation-outputs`, `derivation-inputs` maps 1:1 to the `.drv`
form; the `drv` field is a `<derivation>` to be built, and the
`sub-derivations` field is a list of outputs.
-The other pseudo-"arguments" are pretty simple; none of them involve new
-records. The third is `derivation-sources`, which contains a list of all
+## Sources
+
+The third is `derivation-sources`, which contains a list of all
store items used in the build which aren't themselves built using
derivations, whereas `derivation-inputs` contains the dependencies which
are.
@@ -328,8 +333,10 @@ realises the store items when run, which we'll examine in a later post, and
the path to a directory containing extra modules to add to the build script's
`%load-path`, called `/gnu/store/...-module-import`.
-The next "argument" is self-explanatory: `derivation-system`, which specifies
-the Nix system we're building for. Next is `derivation-builder`, pointing to
+## System, Builder and Arguments
+
+The next record field is `derivation-system`, which specifies
+the Nix system we're building for. Then, there is `derivation-builder`, pointing to
the `guile` executable that runs the script; and the second-to-last is
`derivation-args`, which is a list of arguments to pass to `derivation-builder`.
Note how we use `-L` and `-C` to extend the Guile `%load-path` and
@@ -347,7 +354,7 @@ directories:
;;; (("--no-auto-compile" "-L" "/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import" "-C" "/gnu/store/6rkkvvb7pl1l9ng8vvywvwf357vhm3va-module-import-compiled" "/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"))
```
-The final "argument" contains a list of environment variables to set before
+The final record field contains a list of environment variables to set before
we start the build process:
```scheme
@@ -355,9 +362,16 @@ we start the build process:
;;; ((("allowSubstitutes" . "0") ("guix properties" . "((type . graft) (graft (count . 2)))") ("out" . "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3") ("preferLocalBuild" . "1")))
```
-Obviously, the last record field, `derivation-file-name`, simply allows you to
+## File Name
+
+The last record field, `derivation-file-name`, simply allows you to
retrieve the path to the `.drv` file in Scheme, and so isn't represented
-in a serialised derivation. Speaking of serialisation, to convert between the
+in a serialised derivation. But how to do the inverse? How to read a
+derivation from a path?
+
+# Reading a derivation from file
+
+Speaking of serialisation, to convert between the
`.drv` text format and the Scheme `<derivation>` record, you can use
`write-derivation`, `read-derivation`, and `read-derivation-from-file`.
--
2.38.1
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0000-initialize-markdown.patch --]
[-- Type: text/x-patch, Size: 0 bytes --]
^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: Dissecting Guix -- blog post series
2022-12-10 21:25 ` Mekeor Melire
@ 2022-12-11 10:08 ` (
2022-12-12 1:43 ` Mekeor Melire
0 siblings, 1 reply; 23+ messages in thread
From: ( @ 2022-12-11 10:08 UTC (permalink / raw)
To: Mekeor Melire; +Cc: guix-devel
[-- Attachment #1: Type: text/plain, Size: 2050 bytes --]
Heya,
On Sat Dec 10, 2022 at 9:25 PM GMT, Mekeor Melire wrote:
> I think it'd also be fine to come up with the titles during the process of writing as sometimes that process itself is insightful. E.g. I could imagine parts 2 and 4 to collapse. Maybe, maybe not, you'll see.
Perhaps. I think monads are complex enough to warrant their own post, though.
> How'd this part differ from section "12.18 Defining Services" of the manual?
Along with a low-level description of the workings of services, it'd contain a
complex, concrete example of a service. The manual mostly has an API reference
and some high-level-ish examples.
> How'd this part differ from sections 22 and "22.6 Submitting Patches" from the manual?
Again, it'd be more concrete than what the manual shows. It'd explain the
process by demonstrating the development of an *actual patch*, and showing how
it could be sent to guix-patches@gnu.org.
> By the way, the text does not seem to strictly fill columns at a certain line width. So I did not bother to fill columns in the "patch".
Right, I should set up FILL-COLUMNS-INDICATOR... :)
> If the purpose is out of scope, then we should not dive in that deeply. Especially, I'd suggest skip the code snippet.
I'm not sure about this.
> Here you write "pretty simple". Later you also write "self-explanatory" and "obviously". I suggest to let the reader decide what's simple.
Fair.
> Before this point, the record fields have been called '"argument"'s all the time. I think it's not nice style to carry on a quoted term through many paragraphs like this. Let's simply write "record field" all the time, instead.
This is fair too :)
> I think an example for invoking read-derivation-from-file would round up this tutorial really nicely because it'd close the circle between .drv-files and <derivation>-objects.
Okay! And one for write-derivation, too.
> Here, in the conclusion, IMHO, there could be another brief listing of all fields of a derivation.
Good idea.
-- (
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Dissecting Guix -- blog post series
2022-12-11 10:08 ` (
@ 2022-12-12 1:43 ` Mekeor Melire
2022-12-12 17:26 ` (
0 siblings, 1 reply; 23+ messages in thread
From: Mekeor Melire @ 2022-12-12 1:43 UTC (permalink / raw)
To: (; +Cc: guix-devel
2022-12-11 10:08 paren@disroot.org:
> On Sat Dec 10, 2022 at 9:25 PM GMT, Mekeor Melire wrote:
>
> > By the way, the text does not seem to strictly fill columns at
> > a certain line width. So I did not bother to fill columns in
> > the "patch".
>
> Right, I should set up FILL-COLUMNS-INDICATOR... :)
Or auto-fill-mode? :)
> > If the purpose is out of scope, then we should not dive in
> > that deeply. Especially, I'd suggest skip the code snippet.
>
> I'm not sure about this.
Personally, I felt kind of fooled at that point, because it felt
like I had made needless efforts to read an irrelevant code
snippet.
> > I think an example for invoking read-derivation-from-file
> > would round up this tutorial really nicely because it'd close
> > the circle between .drv-files and <derivation>-objects.
>
> Okay! And one for write-derivation, too.
Sure :)
By the way, sorry for the malformatted e-mail; I was tweaking
around with my mail-client's settings. (I'm now using f=f and
non-broken lines.) Hope my mails are well readable at your end.
Please let me know if they're not.
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Dissecting Guix -- blog post series
2022-12-09 17:25 ` (
@ 2022-12-12 13:46 ` Bengt Richter
0 siblings, 0 replies; 23+ messages in thread
From: Bengt Richter @ 2022-12-12 13:46 UTC (permalink / raw)
To: (, cancontain; +Cc: 宋文武, guix-devel
Hi,
On +2022-12-09 17:25:35 +0000, ( wrote:
> Heya,
>
> On Fri Dec 9, 2022 at 9:32 AM GMT, wrote:
> > How does a gullible noob like me know what the dangers might be, (e.g. http:)
> > and how to avoid (most of) them by finding a guix version that has been
> > gone through with a fine-tooth comb by trusted guix devs and has been
> > re-hosted at gitlab or gnu.org, etc ... for added security?
>
> Sorry, I don't really understand; how is this relevant to derivations? :)
>
> -- (
Maybe I mis-imagine your assumptions about your audience.
For myself, I would like an emacs M-x idiot-mode
so I could run a boot-bricker-test.sh script someone
has posted, without worrying that in plain cli context,
it will /actually/ brick my machine :)
I am assuming if your lowlevel examples are really good,
they will be used as bases for cut/paste variants that people
will then post and implicitly prompt each other to try..
I don't trust that everything thus posted
will be both benevolent and competently avoiding security vulns.
I can't even trust my own stuff. I make too many mistakes :)
So, narrowly focusing on derivations, maybe trust is not technically
relevant, but in the larger social context gullible noobs like me
need all the help we can get about recognizing potentially dangerous
code.
And I think derivations can potentially contain or generate or activate
code one should not trust.
So that's how I see asking for trust info being relevant to derivations :)
--
Regards,
Bengt Richter
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Dissecting Guix -- blog post series
2022-12-12 1:43 ` Mekeor Melire
@ 2022-12-12 17:26 ` (
0 siblings, 0 replies; 23+ messages in thread
From: ( @ 2022-12-12 17:26 UTC (permalink / raw)
To: Mekeor Melire; +Cc: guix-devel
[-- Attachment #1: Type: text/plain, Size: 872 bytes --]
Heya,
On Mon Dec 12, 2022 at 1:43 AM GMT, Mekeor Melire wrote:
> Or auto-fill-mode? :)
I'm pretty new to Emacs, so I dunno which way is best -.o.-
> Personally, I felt kind of fooled at that point, because it felt
> like I had made needless efforts to read an irrelevant code
> snippet.
Fair enough, but I also think if I remove it people will be left wondering how
``recursive?'' is serialised.
I think that code snippet might be irrelevant anyway, since I can just show the
result of serialising a derivation with ``recursive? #t''.
> By the way, sorry for the malformatted e-mail; I was tweaking
> around with my mail-client's settings. (I'm now using f=f and
> non-broken lines.) Hope my mails are well readable at your end.
> Please let me know if they're not.
They're perfectly readable, but the lines are quite short.
-- (
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH guix-artwork v2] website: posts: Add Dissecting Guix, Part 1: Derivations.
2022-12-08 18:24 Dissecting Guix -- blog post series (
2022-12-09 7:31 ` 宋文武
2022-12-10 21:25 ` Mekeor Melire
@ 2022-12-12 19:31 ` (
2022-12-12 20:59 ` Mekeor Melire
` (2 more replies)
2022-12-12 22:53 ` [PATCH guix-artwork v3] " (
2022-12-17 22:06 ` [PATCH guix-artwork v4] " (
4 siblings, 3 replies; 23+ messages in thread
From: ( @ 2022-12-12 19:31 UTC (permalink / raw)
To: guix-devel; +Cc: mekeor, (
* website/posts/dissecting-guix-1-derivations.md: New blog post.
---
Heya,
This patch adds an updated blog post, taking the critiques on this thread into
account.
* Changed post name to just "Derivations"
* Removed subjective "simple" language
* Added subheadings to "Exploring `<derivation>`"
* Added field recap to "Conclusion"
* Added "Utilising `<derivation>`", with examples for READ-DERIVATION,
READ-DERIVATION-FROM-FILE, WRITE-DERIVATION, and BUILD-DERIVATION
* Wrapped lines to 80 columns
.../posts/dissecting-guix-1-derivations.md | 426 ++++++++++++++++++
1 file changed, 426 insertions(+)
create mode 100644 website/posts/dissecting-guix-1-derivations.md
diff --git a/website/posts/dissecting-guix-1-derivations.md b/website/posts/dissecting-guix-1-derivations.md
new file mode 100644
index 0000000..18cdc3a
--- /dev/null
+++ b/website/posts/dissecting-guix-1-derivations.md
@@ -0,0 +1,426 @@
+title: Dissecting Guix, Part 1: Derivations
+date: TBC
+author: (
+tags: Dissecting Guix, Functional package management, Programming interfaces, Scheme API
+---
+To a new user, Guix's functional architecture can seem quite alien, and possibly
+offputting. With a combination of extensive `#guix`-querying, determined
+manual-reading, and plenty of source-perusing, they may eventually figure out
+how everything fits together by themselves, but this can be frustrating and
+often takes a fairly long time.
+
+However, once you peel back the layers, the "Nix way" is actually rather
+elegant, if perhaps not as simple as the mutable, imperative style implemented
+by the likes of [`dpkg`](https://wiki.debian.org/dpkg) and
+[`pacman`](https://wiki.archlinux.org/title/pacman).
+This series of blog posts will cover basic Guix concepts, taking a "ground-up"
+approach by dealing with lower-level concepts first, and hopefully make those
+months of information-gathering unnecessary.
+
+Before we dig in to Guix-specific concepts, we'll need to learn about one
+inherited from [Nix](https://nixos.org), the original functional package manager
+and the inspiration for Guix; the idea of a _derivation_ and its corresponding
+_store items_.
+
+These concepts were originally described by Eelco Dolstra, the author of Nix, in
+their [PhD thesis](https://edolstra.github.io/pubs/phd-thesis.pdf); see
+_§ 2.1 The Nix store_ and _§ 2.4 Store Derivations_.
+
+# Store Items
+
+As you almost certainly know, everything that Guix builds is stored in the
+_store_, which is almost always the `/gnu/store` directory. It's the job of the
+[`guix-daemon`](https://guix.gnu.org/manual/en/html_node/Invoking-guix_002ddaemon.html)
+to manage the store and build things. If you run
+[`guix build PKG`](https://guix.gnu.org/manual/en/html_node/Invoking-guix-build.html),
+`PKG` will be built or downloaded from a substitute server if available, and a
+path to an item in the store will be displayed.
+
+```
+$ guix build irssi
+/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3
+```
+
+This item contains the final result of building [`irssi`](https://irssi.org).
+Let's peek inside:
+
+```
+$ ls $(guix build irssi)
+bin/ etc/ include/ lib/ share/
+$ ls $(guix build irssi)/bin
+irssi*
+```
+
+`irssi` is quite a simple package. What about a more complex one, like
+[`glib`](https://docs.gtk.org/glib)?
+
+```
+$ guix build glib
+/gnu/store/bx8qq76idlmjrlqf1faslsq6zjc6f426-glib-2.73.3-bin
+/gnu/store/j65bhqwr7qq7l77nj0ahmk1f1ilnjr3a-glib-2.73.3-debug
+/gnu/store/3pn4ll6qakgfvfpc4mw89qrrbsgj3jf3-glib-2.73.3-doc
+/gnu/store/dvsk6x7d26nmwsqhnzws4iirb6dhhr1d-glib-2.73.3
+/gnu/store/4c8ycz501n2d0xdi4blahvnbjhd5hpa8-glib-2.73.3-static
+```
+
+`glib` produces five `/gnu/store` items, because it's possible for a package to
+produce multiple [outputs](https://guix.gnu.org/manual/en/html_node/Packages-with-Multiple-Outputs.html).
+Each output can be referred to separately, by prefixing a package's name with
+`:OUTPUT` where supported. For example, this
+[`guix install`](https://guix.gnu.org/manual/en/html_node/Invoking-guix-package.html)
+invocation will add `glib`'s `bin` output to your profile:
+
+```
+$ guix install glib:bin
+```
+
+The default output is `out`, so when you pass `glib` by itself to that command,
+it will actually install `glib:out` to the profile.
+
+`guix build` also provides the `--source` flag, which produces the store item
+corresponding to the given package's downloaded source code.
+
+```
+$ guix build --source irssi
+/gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz
+$ guix build --source glib
+/gnu/store/d22wzjq3xm3q8hwnhbgk2xd3ph7lb6ay-glib-2.73.3.tar.xz
+```
+
+But how does Guix know how to build these store outputs in the first place?
+That's where derivations come in.
+
+# `.drv` Files
+
+You've probably seen these being printed by the Guix CLI now and again.
+Derivations, represented in the daemon's eyes by `.drv` files, contain
+instructions for building store items. We can retrieve the paths of these
+`.drv` files with the `guix build --derivations` command:
+
+```
+$ guix build --derivations irssi
+/gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv
+```
+
+`guix build` can actually also accept derivation paths as an argument, in lieu
+of a package, like so:
+
+```
+$ guix build /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv
+/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3
+```
+
+Let's look inside this derivation file.
+
+```
+Derive([("out","/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3","","")],[("/gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv",["out"]),("/gnu/store/baqpbl4wck7nkxrbyc9nlhma7kq5dyfl-guile-2.0.14.drv",["out"]),("/gnu/store/bfirgq65ndhf63nn4q6vlkbha9zd931q-openssl-1.1.1l.drv",["out"]),("/gnu/store/gjwpqzvfhz13shix6a6cs2hjc18pj7wy-module-import-compiled.drv",["out"]),("/gnu/store/ij8651x4yh53hhcn6qw2644nhh2s8kcn-glib-2.70.2.drv",["out"]),("/gnu/store/jg2vv6yc2yqzi3qzs82dxvqmi5k21lhy-irssi-1.4.3.drv",["out"]),("/gnu/store/qggpjl9g6ic3cq09qrwkm0dfsdjf7pyr-glibc-utf8-locales-2.33.drv",["out"]),("/gnu/store/zafabw13yyhz93jwrcz7axak1kn1f2cx-openssl-1.1.1s.drv",["out"])],["/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import","/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"],"x86_64-linux","/gnu/store/hnr4r2d0h0xarx52i6jq9gvsrlc3q81a-guile-2.0.14/bin/guile",["--no-auto-compile","-L","/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import","-C","/gnu/store/6rkkvvb7pl1l9ng8vvywvwf357vhm3va-module-import-compiled","/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"],[("allowSubstitutes","0"),("guix properties","((type . graft) (graft (count . 2)))"),("out","/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3"),("preferLocalBuild","1")])
+```
+
+It's... not exactly human-readable. We could try to format it and break it
+down, but it'd still be pretty hard to understand, since `.drv` files contain no
+labels for the fields or any other human-readable indicators. Instead, we're
+going to explore derivations in a Guile REPL.
+
+# Exploring Guix Interactively
+
+Before we continue, we'll want to start a REPL, so that we can try out the Guix
+Guile API interactively. To run a REPL in the terminal, simply
+[call `guix repl`](https://guix.gnu.org/manual/devel/en/html_node/Using-Guix-Interactively.html).
+
+If you're using Emacs, you can instead install
+[Geiser](https://nongnu.org/geiser), which provides a comfortable Emacs UI for
+various Lisp REPLs, invoke `guix repl --listen=tcp:37146 &`, and type
+`M-x geiser-connect RET RET RET` to connect to the running Guile instance.
+
+There are a few Guix modules we'll need. Run this Scheme code to import them:
+
+```scheme
+(use-modules (guix derivations)
+ (guix gexp)
+ (guix packages)
+ (guix store)
+ (gnu packages glib)
+ (gnu packages irc))
+```
+
+We now have access to the store, G-expression, package, and derivation APIs,
+along with the `irssi` and `glib` `<package>` objects.
+
+# Creating a `<derivation>`
+
+The Guix API for derivations revolves around the `<derivation>` record, which is
+the Scheme representation of that whole block of text surrounded by
+`Derive(...)`. If we look in `guix/derivations.scm`, we can see that it's
+defined like this:
+
+```scheme
+(define-immutable-record-type <derivation>
+ (make-derivation outputs inputs sources system builder args env-vars
+ file-name)
+ derivation?
+ (outputs derivation-outputs) ; list of name/<derivation-output> pairs
+ (inputs derivation-inputs) ; list of <derivation-input>
+ (sources derivation-sources) ; list of store paths
+ (system derivation-system) ; string
+ (builder derivation-builder) ; store path
+ (args derivation-builder-arguments) ; list of strings
+ (env-vars derivation-builder-environment-vars) ; list of name/value pairs
+ (file-name derivation-file-name)) ; the .drv file name
+```
+
+With the exception of `file-name`, each of those fields corresponds to a field
+in the `Derive(...)` form. Before we can examine them, though, we need to
+figure out how to _lower_ that `irssi` `<package>` object into a derivation.
+
+The procedure we use to turn a high-level object like `<package>` into a
+derivation is called `lower-object`; more on that in a future post. However,
+this doesn't produce a derivation:
+
+```scheme
+(pk (lower-object irssi))
+;;; (#<procedure 7fe17c7af540 at guix/store.scm:1994:2 (state)>)
+```
+
+`pk` is an abbreviation for the procedure `peek`, which takes the given object,
+writes a representation of it to the output, and returns it. It's especially
+handy when you want to view an intermediate value in a complex expression.
+
+The returned object is a procedure that needs to be evaluated in the context of
+a store connection. We do this by first using `with-store` to connect to the
+store and bind the connection to a name, then wrapping the `lower-object` call
+with `run-with-store`:
+
+```scheme
+(define irssi-drv
+ (pk (with-store %store
+ (run-with-store %store
+ (lower-object irssi)))))
+;;; (#<derivation /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7fe1902b6140>)
+
+(define glib-drv
+ (pk (with-store %store
+ (run-with-store %store
+ (lower-object glib)))))
+;;; (#<derivation /gnu/store/81qqs7xah2ln39znrji4r6xj85zi15bi-glib-2.70.2.drv => /gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin /gnu/store/22mkp8cr6rxg6w8br9q8dbymf51b44m8-glib-2.70.2-debug /gnu/store/a6qb5arvir4vm1zlkp4chnl7d8qzzd7x-glib-2.70.2 /gnu/store/y4ak268dcdwkc6lmqfk9g1dgk2jr9i34-glib-2.70.2-static 7fe17ca13b90>)
+```
+
+And we have liftoff! Now we've got two `<derivation>` records to play with.
+
+# Exploring `<derivation>`
+
+## `<derivation-output>`
+
+The first "argument" in the `.drv` file is `outputs`, which tells the Guix
+daemon about the outputs that this build can produce:
+
+```scheme
+(define irssi-outputs
+ (pk (derivation-outputs irssi-drv)))
+;;; ((("out" . #<<derivation-output> path: "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3" hash-algo: #f hash: #f recursive?: #f>)))
+
+(pk (assoc-ref irssi-outputs "out"))
+
+(define glib-outputs
+ (pk (derivation-outputs glib-drv)))
+;;; ((("bin" . #<<derivation-output> path: "/gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin" hash-algo: #f hash: #f recursive?: #f>) ("debug" . #<<derivation-output> path: "/gnu/store/22mkp8cr6rxg6w8br9q8dbymf51b44m8-glib-2.70.2-debug" hash-algo: #f hash: #f recursive?: #f>) ("out" . #<<derivation-output> path: "/gnu/store/a6qb5arvir4vm1zlkp4chnl7d8qzzd7x-glib-2.70.2" hash-algo: #f hash: #f recursive?: #f>) ("static" . #<<derivation-output> path: "/gnu/store/y4ak268dcdwkc6lmqfk9g1dgk2jr9i34-glib-2.70.2-static" hash-algo: #f hash: #f recursive?: #f>)))
+
+(pk (assoc-ref glib-outputs "bin"))
+;;; (#<<derivation-output> path: "/gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin" hash-algo: #f hash: #f recursive?: #f>)
+```
+
+It's a simple association list mapping output names to `<derivation-output>`
+records, and it's equivalent to the first "argument" in the `.drv` file:
+
+```
+[ ("out", "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3", "", "")
+]
+```
+
+The `hash-algo` and `hash` fields are for storing the content hash and the
+algorithm used with that hash for what we term a _fixed-output derivation_,
+which is essentially a derivation where we know what the hash of the content
+will be in advance. For instance, `origin`s produce fixed-output derivations:
+
+```scheme
+(define irssi-src-drv
+ (pk (with-store %store
+ (run-with-store %store
+ (lower-object (package-source irssi))))))
+;;; (#<derivation /gnu/store/mcz3vzq7lwwaqjb8dy7cd69lvmi6d241-irssi-1.4.3.tar.xz.drv => /gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz 7fe17b3c8d70>)
+
+(define irssi-src-outputs
+ (pk (derivation-outputs irssi-src-drv)))
+;;; ((("out" . #<<derivation-output> path: "/gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz" hash-algo: sha256 hash: #vu8(185 63 113 82 35 163 34 230 127 66 182 26 8 165 18 174 41 227 75 212 165 61 127 34 55 102 102 10 170 90 4 52) recursive?: #f>)))
+
+(pk (assoc-ref irssi-src-outputs "out"))
+;;; (#<<derivation-output> path: "/gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz" hash-algo: sha256 hash: #vu8(185 63 113 82 35 163 34 230 127 66 182 26 8 165 18 174 41 227 75 212 165 61 127 34 55 102 102 10 170 90 4 52) recursive?: #f>)
+```
+
+Note how the `hash` and `hash-algo` now have values.
+
+Perceptive readers may note that the `<derivation-output>` has four fields,
+whereas the tuple in the `.drv` file only has three (minus the label). The
+serialisation of `recursive?` is done by adding the prefix `r:` to the
+`hash-algo` field, though its actual purpose is difficult to explain, and is out
+of scope for this post.
+
+## `<derivation-input>`
+
+The next field is `inputs`, which corresponds to the second field in the `.drv`
+file format:
+
+```
+[ ("/gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv", ["out"]),
+ ("/gnu/store/baqpbl4wck7nkxrbyc9nlhma7kq5dyfl-guile-2.0.14.drv", ["out"]),
+ ("/gnu/store/bfirgq65ndhf63nn4q6vlkbha9zd931q-openssl-1.1.1l.drv", ["out"]),
+ ("/gnu/store/gjwpqzvfhz13shix6a6cs2hjc18pj7wy-module-import-compiled.drv", ["out"]),
+ ("/gnu/store/ij8651x4yh53hhcn6qw2644nhh2s8kcn-glib-2.70.2.drv", ["out"]),
+ ("/gnu/store/jg2vv6yc2yqzi3qzs82dxvqmi5k21lhy-irssi-1.4.3.drv", ["out"]),
+ ("/gnu/store/qggpjl9g6ic3cq09qrwkm0dfsdjf7pyr-glibc-utf8-locales-2.33.drv", ["out"]),
+ ("/gnu/store/zafabw13yyhz93jwrcz7axak1kn1f2cx-openssl-1.1.1s.drv", ["out"])
+]
+```
+
+Here, each tuple specifies a derivation that needs to be built before this
+derivation can be built, and the outputs of the derivation that the build
+process of this derivation uses. Let's grab us the Scheme equivalent:
+
+```scheme
+(define irssi-inputs
+ (pk (derivation-inputs irssi-drv)))
+;;; [a fairly large amount of output]
+
+(pk (car irssi-inputs))
+;;; (#<<derivation-input> drv: #<derivation /gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv => /gnu/store/2jj2mxn6wfrcw7i85nywk71mmqbnhzps-glib-2.70.2 7fe1902b6640> sub-derivations: ("out")>)
+```
+
+Unlike `derivation-outputs`, `derivation-inputs` maps 1:1 to the `.drv`
+form; the `drv` field is a `<derivation>` to be built, and the
+`sub-derivations` field is a list of outputs.
+
+## Builder Configuration
+
+The other fields are simpler; none of them involve new records. The third is
+`derivation-sources`, which contains a list of all store items used in the build
+which aren't themselves built using derivations, whereas `derivation-inputs`
+contains the dependencies which are.
+
+This list usually just contains the path to the Guile _build script_ that
+realises the store items when run, which we'll examine in a later post, and
+the path to a directory containing extra modules to add to the build script's
+`%load-path`, called `/gnu/store/...-module-import`.
+
+The next field is `derivation-system`, which specifies the Nix system we're
+building for. Then we have `derivation-builder`, pointing to the `guile`
+executable that runs the build script; and the second-to-last is
+`derivation-builder-arguments`, which is a list of arguments to pass to
+`derivation-builder`. Note how we use `-L` and `-C` to extend the Guile
+`%load-path` and `%load-compiled-path` to include the `module-import` and
+`module-import-compiled` directories:
+
+```scheme
+(pk (derivation-system irssi-drv))
+;;; ("x86_64-linux")
+
+(pk (derivation-builder irrsi-drv))
+;;; ("/gnu/store/hnr4r2d0h0xarx52i6jq9gvsrlc3q81a-guile-2.0.14/bin/guile")
+
+(pk (derivation-builder-arguments irrsi-drv))
+;;; (("--no-auto-compile" "-L" "/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import" "-C" "/gnu/store/6rkkvvb7pl1l9ng8vvywvwf357vhm3va-module-import-compiled" "/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"))
+```
+
+The final field contains a list of environment variables to set before we start
+the build process:
+
+```scheme
+(pk (derivation-builder-environment-vars irssi-drv))
+;;; ((("allowSubstitutes" . "0") ("guix properties" . "((type . graft) (graft (count . 2)))") ("out" . "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3") ("preferLocalBuild" . "1")))
+```
+
+The last record field, `derivation-file-name` contains the path to the `.drv`
+file, and so isn't represented in a serialised derivation.
+
+# Utilising `<derivation>`
+
+Speaking of serialisation, to convert between the `.drv` text format and the
+Scheme `<derivation>` record, you can use `write-derivation`, `read-derivation`,
+and `read-derivation-from-file`:
+
+```scheme
+(define manual-drv
+ (with-store %store
+ (derivation %store "manual"
+ "/bin/sh" '())))
+
+(write-derivation drv (current-output-port))
+;;; -| Derive([("out","/gnu/store/kh7fais2zab22fd8ar0ywa4767y6xyak-example","","")],[],[],"x86_64-linux","/bin/sh",[],[("out","/gnu/store/kh7fais2zab22fd8ar0ywa4767y6xyak-example")])
+
+(pk (read-derivation-from-file (derivation-file-name irssi-drv)))
+;;; (#<derivation /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7fb3798788c0>)
+
+(call-with-input-file (derivation-file-name irssi-drv)
+ read-derivation)
+;;; (#<derivation /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7fb37ad19e10>)
+```
+
+You can realise `<derivation>`s as store items using the `build-derivations`
+procedure:
+
+```scheme
+(use-modules (ice-9 ftw))
+
+(define irssi-drv-out
+ (pk (derivation-output-path
+ (assoc-ref (derivation-outputs irssi-drv) "out"))))
+;;; ("/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3")
+
+(pk (scandir irssi-drv-out))
+;;; (#f)
+
+(pk (with-store %store (build-derivations %store (list irssi-drv))))
+;;; (#t)
+
+(pk (scandir irssi-drv-out))
+;;; (("." ".." "bin" "etc" "include" "lib" "share"))
+```
+
+# Conclusion
+
+Derivations are one of Guix's most important concepts, but are fairly easy to
+understand once you get past the obtuse `.drv` file format. They provide the
+Guix daemon with the initial instructions that it uses to build store items
+like packages, origins, and other file-likes such as `computed-file` and
+`local-file`, which will be discussed in a future post!
+
+To recap, a derivation contains the following fields:
+
+1. `derivation-outputs`, describing the various output paths that the derivation
+ builds
+2. `derivation-inputs`, describing the other derivations that need to be built
+ before this one is
+3. `derivation-sources`, listing the non-derivation store items that the
+ derivation depends on
+4. `derivation-system`, specifying the Nix system a derivation will be compiled
+ for
+5. `derivation-builder`, the executable to run the build script with
+6. `derivation-builder-arguments`, arguments to pass to the builder
+7. `derivation-builder-environment-vars`, variables to set in the builder's
+ environment
+
+#### About GNU Guix
+
+[GNU Guix](https://guix.gnu.org) is a transactional package manager and
+an advanced distribution of the GNU system that [respects user
+freedom](https://www.gnu.org/distros/free-system-distribution-guidelines.html).
+Guix can be used on top of any system running the Hurd or the Linux
+kernel, or it can be used as a standalone operating system distribution
+for i686, x86_64, ARMv7, AArch64 and POWER9 machines.
+
+In addition to standard package management features, Guix supports
+transactional upgrades and roll-backs, unprivileged package management,
+per-user profiles, and garbage collection. When used as a standalone
+GNU/Linux distribution, Guix offers a declarative, stateless approach to
+operating system configuration management. Guix is highly customizable
+and hackable through [Guile](https://www.gnu.org/software/guile)
+programming interfaces and extensions to the
+[Scheme](http://schemers.org) language.
--
2.38.1
^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: [PATCH guix-artwork v2] website: posts: Add Dissecting Guix, Part 1: Derivations.
2022-12-12 19:31 ` [PATCH guix-artwork v2] website: posts: Add Dissecting Guix, Part 1: Derivations (
@ 2022-12-12 20:59 ` Mekeor Melire
2022-12-12 21:06 ` Mekeor Melire
2022-12-15 14:26 ` Ludovic Courtès
2 siblings, 0 replies; 23+ messages in thread
From: Mekeor Melire @ 2022-12-12 20:59 UTC (permalink / raw)
To: (; +Cc: guix-devel
2022-12-12 19:31 paren@disroot.org:
> This patch adds an updated blog post, taking the critiques on
> this thread into
> account.
>
> * Changed post name to just "Derivations"
> * Removed subjective "simple" language
> * Added subheadings to "Exploring `<derivation>`"
> * Added field recap to "Conclusion"
> * Added "Utilising `<derivation>`", with examples for
> READ-DERIVATION,
> READ-DERIVATION-FROM-FILE, WRITE-DERIVATION, and
> BUILD-DERIVATION
> * Wrapped lines to 80 columns
looks good to me :)
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH guix-artwork v2] website: posts: Add Dissecting Guix, Part 1: Derivations.
2022-12-12 19:31 ` [PATCH guix-artwork v2] website: posts: Add Dissecting Guix, Part 1: Derivations (
2022-12-12 20:59 ` Mekeor Melire
@ 2022-12-12 21:06 ` Mekeor Melire
2022-12-12 21:24 ` (
2022-12-15 14:26 ` Ludovic Courtès
2 siblings, 1 reply; 23+ messages in thread
From: Mekeor Melire @ 2022-12-12 21:06 UTC (permalink / raw)
To: (; +Cc: guix-devel
I have one remark.
2022-12-12 19:31 paren@disroot.org:
> +If you're using Emacs, you can instead install
> +[Geiser](https://nongnu.org/geiser), which provides a
> comfortable Emacs UI for
> +various Lisp REPLs, invoke `guix repl --listen=tcp:37146 &`,
> and type
> +`M-x geiser-connect RET RET RET` to connect to the running
> Guile instance.
As noted in <https://issues.guix.gnu.org/35727>, Guix System
users, who use the =~/.guile= file which is created by default,
will need to pass the =INSIDE_EMACS= environment variable to =guix
repl=, like this:
#+begin_src sh
INSIDE_EMACS=true guix repl --listen=tcp:37146 &
#+end_src
Otherwise, they'll experience a freeze (which they'll need to
break with C-g).
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH guix-artwork v2] website: posts: Add Dissecting Guix, Part 1: Derivations.
2022-12-12 21:06 ` Mekeor Melire
@ 2022-12-12 21:24 ` (
0 siblings, 0 replies; 23+ messages in thread
From: ( @ 2022-12-12 21:24 UTC (permalink / raw)
To: Mekeor Melire; +Cc: guix-devel
[-- Attachment #1: Type: text/plain, Size: 431 bytes --]
Heya,
On Mon Dec 12, 2022 at 9:06 PM GMT, Mekeor Melire wrote:
> As noted in <https://issues.guix.gnu.org/35727>, Guix System
> users, who use the =~/.guile= file which is created by default,
> will need to pass the =INSIDE_EMACS= environment variable to =guix
> repl=, like this:
>
> #+begin_src sh
> INSIDE_EMACS=true guix repl --listen=tcp:37146 &
> #+end_src
Ah! That was the problem :) Thanks!
-- (
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH guix-artwork v3] website: posts: Add Dissecting Guix, Part 1: Derivations.
2022-12-08 18:24 Dissecting Guix -- blog post series (
` (2 preceding siblings ...)
2022-12-12 19:31 ` [PATCH guix-artwork v2] website: posts: Add Dissecting Guix, Part 1: Derivations (
@ 2022-12-12 22:53 ` (
2022-12-17 22:06 ` [PATCH guix-artwork v4] " (
4 siblings, 0 replies; 23+ messages in thread
From: ( @ 2022-12-12 22:53 UTC (permalink / raw)
To: guix-devel; +Cc: mekeor, (
* website/posts/dissecting-guix-1-derivations.md: New blog post.
---
Heya,
This patch adds instructions for suppressing colours and readline support in
``guix repl'' on Guix System.
.../posts/dissecting-guix-1-derivations.md | 435 ++++++++++++++++++
1 file changed, 435 insertions(+)
create mode 100644 website/posts/dissecting-guix-1-derivations.md
diff --git a/website/posts/dissecting-guix-1-derivations.md b/website/posts/dissecting-guix-1-derivations.md
new file mode 100644
index 0000000..e3b6b71
--- /dev/null
+++ b/website/posts/dissecting-guix-1-derivations.md
@@ -0,0 +1,435 @@
+title: Dissecting Guix, Part 1: Derivations
+date: TBC
+author: (
+tags: Dissecting Guix, Functional package management, Programming interfaces, Scheme API
+---
+To a new user, Guix's functional architecture can seem quite alien, and possibly
+offputting. With a combination of extensive `#guix`-querying, determined
+manual-reading, and plenty of source-perusing, they may eventually figure out
+how everything fits together by themselves, but this can be frustrating and
+often takes a fairly long time.
+
+However, once you peel back the layers, the "Nix way" is actually rather
+elegant, if perhaps not as simple as the mutable, imperative style implemented
+by the likes of [`dpkg`](https://wiki.debian.org/dpkg) and
+[`pacman`](https://wiki.archlinux.org/title/pacman).
+This series of blog posts will cover basic Guix concepts, taking a "ground-up"
+approach by dealing with lower-level concepts first, and hopefully make those
+months of information-gathering unnecessary.
+
+Before we dig in to Guix-specific concepts, we'll need to learn about one
+inherited from [Nix](https://nixos.org), the original functional package manager
+and the inspiration for Guix; the idea of a _derivation_ and its corresponding
+_store items_.
+
+These concepts were originally described by Eelco Dolstra, the author of Nix, in
+their [PhD thesis](https://edolstra.github.io/pubs/phd-thesis.pdf); see
+_§ 2.1 The Nix store_ and _§ 2.4 Store Derivations_.
+
+# Store Items
+
+As you almost certainly know, everything that Guix builds is stored in the
+_store_, which is almost always the `/gnu/store` directory. It's the job of the
+[`guix-daemon`](https://guix.gnu.org/manual/en/html_node/Invoking-guix_002ddaemon.html)
+to manage the store and build things. If you run
+[`guix build PKG`](https://guix.gnu.org/manual/en/html_node/Invoking-guix-build.html),
+`PKG` will be built or downloaded from a substitute server if available, and a
+path to an item in the store will be displayed.
+
+```
+$ guix build irssi
+/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3
+```
+
+This item contains the final result of building [`irssi`](https://irssi.org).
+Let's peek inside:
+
+```
+$ ls $(guix build irssi)
+bin/ etc/ include/ lib/ share/
+$ ls $(guix build irssi)/bin
+irssi*
+```
+
+`irssi` is quite a simple package. What about a more complex one, like
+[`glib`](https://docs.gtk.org/glib)?
+
+```
+$ guix build glib
+/gnu/store/bx8qq76idlmjrlqf1faslsq6zjc6f426-glib-2.73.3-bin
+/gnu/store/j65bhqwr7qq7l77nj0ahmk1f1ilnjr3a-glib-2.73.3-debug
+/gnu/store/3pn4ll6qakgfvfpc4mw89qrrbsgj3jf3-glib-2.73.3-doc
+/gnu/store/dvsk6x7d26nmwsqhnzws4iirb6dhhr1d-glib-2.73.3
+/gnu/store/4c8ycz501n2d0xdi4blahvnbjhd5hpa8-glib-2.73.3-static
+```
+
+`glib` produces five `/gnu/store` items, because it's possible for a package to
+produce multiple [outputs](https://guix.gnu.org/manual/en/html_node/Packages-with-Multiple-Outputs.html).
+Each output can be referred to separately, by prefixing a package's name with
+`:OUTPUT` where supported. For example, this
+[`guix install`](https://guix.gnu.org/manual/en/html_node/Invoking-guix-package.html)
+invocation will add `glib`'s `bin` output to your profile:
+
+```
+$ guix install glib:bin
+```
+
+The default output is `out`, so when you pass `glib` by itself to that command,
+it will actually install `glib:out` to the profile.
+
+`guix build` also provides the `--source` flag, which produces the store item
+corresponding to the given package's downloaded source code.
+
+```
+$ guix build --source irssi
+/gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz
+$ guix build --source glib
+/gnu/store/d22wzjq3xm3q8hwnhbgk2xd3ph7lb6ay-glib-2.73.3.tar.xz
+```
+
+But how does Guix know how to build these store outputs in the first place?
+That's where derivations come in.
+
+# `.drv` Files
+
+You've probably seen these being printed by the Guix CLI now and again.
+Derivations, represented in the daemon's eyes by `.drv` files, contain
+instructions for building store items. We can retrieve the paths of these
+`.drv` files with the `guix build --derivations` command:
+
+```
+$ guix build --derivations irssi
+/gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv
+```
+
+`guix build` can actually also accept derivation paths as an argument, in lieu
+of a package, like so:
+
+```
+$ guix build /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv
+/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3
+```
+
+Let's look inside this derivation file.
+
+```
+Derive([("out","/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3","","")],[("/gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv",["out"]),("/gnu/store/baqpbl4wck7nkxrbyc9nlhma7kq5dyfl-guile-2.0.14.drv",["out"]),("/gnu/store/bfirgq65ndhf63nn4q6vlkbha9zd931q-openssl-1.1.1l.drv",["out"]),("/gnu/store/gjwpqzvfhz13shix6a6cs2hjc18pj7wy-module-import-compiled.drv",["out"]),("/gnu/store/ij8651x4yh53hhcn6qw2644nhh2s8kcn-glib-2.70.2.drv",["out"]),("/gnu/store/jg2vv6yc2yqzi3qzs82dxvqmi5k21lhy-irssi-1.4.3.drv",["out"]),("/gnu/store/qggpjl9g6ic3cq09qrwkm0dfsdjf7pyr-glibc-utf8-locales-2.33.drv",["out"]),("/gnu/store/zafabw13yyhz93jwrcz7axak1kn1f2cx-openssl-1.1.1s.drv",["out"])],["/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import","/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"],"x86_64-linux","/gnu/store/hnr4r2d0h0xarx52i6jq9gvsrlc3q81a-guile-2.0.14/bin/guile",["--no-auto-compile","-L","/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import","-C","/gnu/store/6rkkvvb7pl1l9ng8vvywvwf357vhm3va-module-import-compiled","/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"],[("allowSubstitutes","0"),("guix properties","((type . graft) (graft (count . 2)))"),("out","/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3"),("preferLocalBuild","1")])
+```
+
+It's... not exactly human-readable. We could try to format it and break it
+down, but it'd still be pretty hard to understand, since `.drv` files contain no
+labels for the fields or any other human-readable indicators. Instead, we're
+going to explore derivations in a Guile REPL.
+
+# Exploring Guix Interactively
+
+Before we continue, we'll want to start a REPL, so that we can try out the Guix
+Guile API interactively. To run a REPL in the terminal, simply
+[call `guix repl`](https://guix.gnu.org/manual/devel/en/html_node/Using-Guix-Interactively.html).
+
+If you're using Emacs, you can instead install
+[Geiser](https://nongnu.org/geiser), which provides a comfortable Emacs UI for
+various Lisp REPLs, invoke `guix repl --listen=tcp:37146 &`, and type
+`M-x geiser-connect RET RET RET` to connect to the running Guile instance.
+
+Your `.guile` file may contain code for enabling colours and readline bindings
+that Geiser will choke on. The default Guix System `.guile` contains code to
+suppress these features when `INSIDE_EMACS` is set, so you'll need to run
+`guix repl` like this:
+
+```sh
+INSIDE_EMACS=1 guix repl --listen=tcp:37146 &
+```
+
+There are a few Guix modules we'll need. Run this Scheme code to import them:
+
+```scheme
+(use-modules (guix derivations)
+ (guix gexp)
+ (guix packages)
+ (guix store)
+ (gnu packages glib)
+ (gnu packages irc))
+```
+
+We now have access to the store, G-expression, package, and derivation APIs,
+along with the `irssi` and `glib` `<package>` objects.
+
+# Creating a `<derivation>`
+
+The Guix API for derivations revolves around the `<derivation>` record, which is
+the Scheme representation of that whole block of text surrounded by
+`Derive(...)`. If we look in `guix/derivations.scm`, we can see that it's
+defined like this:
+
+```scheme
+(define-immutable-record-type <derivation>
+ (make-derivation outputs inputs sources system builder args env-vars
+ file-name)
+ derivation?
+ (outputs derivation-outputs) ; list of name/<derivation-output> pairs
+ (inputs derivation-inputs) ; list of <derivation-input>
+ (sources derivation-sources) ; list of store paths
+ (system derivation-system) ; string
+ (builder derivation-builder) ; store path
+ (args derivation-builder-arguments) ; list of strings
+ (env-vars derivation-builder-environment-vars) ; list of name/value pairs
+ (file-name derivation-file-name)) ; the .drv file name
+```
+
+With the exception of `file-name`, each of those fields corresponds to a field
+in the `Derive(...)` form. Before we can examine them, though, we need to
+figure out how to _lower_ that `irssi` `<package>` object into a derivation.
+
+The procedure we use to turn a high-level object like `<package>` into a
+derivation is called `lower-object`; more on that in a future post. However,
+this doesn't produce a derivation:
+
+```scheme
+(pk (lower-object irssi))
+;;; (#<procedure 7fe17c7af540 at guix/store.scm:1994:2 (state)>)
+```
+
+`pk` is an abbreviation for the procedure `peek`, which takes the given object,
+writes a representation of it to the output, and returns it. It's especially
+handy when you want to view an intermediate value in a complex expression.
+
+The returned object is a procedure that needs to be evaluated in the context of
+a store connection. We do this by first using `with-store` to connect to the
+store and bind the connection to a name, then wrapping the `lower-object` call
+with `run-with-store`:
+
+```scheme
+(define irssi-drv
+ (pk (with-store %store
+ (run-with-store %store
+ (lower-object irssi)))))
+;;; (#<derivation /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7fe1902b6140>)
+
+(define glib-drv
+ (pk (with-store %store
+ (run-with-store %store
+ (lower-object glib)))))
+;;; (#<derivation /gnu/store/81qqs7xah2ln39znrji4r6xj85zi15bi-glib-2.70.2.drv => /gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin /gnu/store/22mkp8cr6rxg6w8br9q8dbymf51b44m8-glib-2.70.2-debug /gnu/store/a6qb5arvir4vm1zlkp4chnl7d8qzzd7x-glib-2.70.2 /gnu/store/y4ak268dcdwkc6lmqfk9g1dgk2jr9i34-glib-2.70.2-static 7fe17ca13b90>)
+```
+
+And we have liftoff! Now we've got two `<derivation>` records to play with.
+
+# Exploring `<derivation>`
+
+## `<derivation-output>`
+
+The first "argument" in the `.drv` file is `outputs`, which tells the Guix
+daemon about the outputs that this build can produce:
+
+```scheme
+(define irssi-outputs
+ (pk (derivation-outputs irssi-drv)))
+;;; ((("out" . #<<derivation-output> path: "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3" hash-algo: #f hash: #f recursive?: #f>)))
+
+(pk (assoc-ref irssi-outputs "out"))
+
+(define glib-outputs
+ (pk (derivation-outputs glib-drv)))
+;;; ((("bin" . #<<derivation-output> path: "/gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin" hash-algo: #f hash: #f recursive?: #f>) ("debug" . #<<derivation-output> path: "/gnu/store/22mkp8cr6rxg6w8br9q8dbymf51b44m8-glib-2.70.2-debug" hash-algo: #f hash: #f recursive?: #f>) ("out" . #<<derivation-output> path: "/gnu/store/a6qb5arvir4vm1zlkp4chnl7d8qzzd7x-glib-2.70.2" hash-algo: #f hash: #f recursive?: #f>) ("static" . #<<derivation-output> path: "/gnu/store/y4ak268dcdwkc6lmqfk9g1dgk2jr9i34-glib-2.70.2-static" hash-algo: #f hash: #f recursive?: #f>)))
+
+(pk (assoc-ref glib-outputs "bin"))
+;;; (#<<derivation-output> path: "/gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin" hash-algo: #f hash: #f recursive?: #f>)
+```
+
+It's a simple association list mapping output names to `<derivation-output>`
+records, and it's equivalent to the first "argument" in the `.drv` file:
+
+```
+[ ("out", "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3", "", "")
+]
+```
+
+The `hash-algo` and `hash` fields are for storing the content hash and the
+algorithm used with that hash for what we term a _fixed-output derivation_,
+which is essentially a derivation where we know what the hash of the content
+will be in advance. For instance, `origin`s produce fixed-output derivations:
+
+```scheme
+(define irssi-src-drv
+ (pk (with-store %store
+ (run-with-store %store
+ (lower-object (package-source irssi))))))
+;;; (#<derivation /gnu/store/mcz3vzq7lwwaqjb8dy7cd69lvmi6d241-irssi-1.4.3.tar.xz.drv => /gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz 7fe17b3c8d70>)
+
+(define irssi-src-outputs
+ (pk (derivation-outputs irssi-src-drv)))
+;;; ((("out" . #<<derivation-output> path: "/gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz" hash-algo: sha256 hash: #vu8(185 63 113 82 35 163 34 230 127 66 182 26 8 165 18 174 41 227 75 212 165 61 127 34 55 102 102 10 170 90 4 52) recursive?: #f>)))
+
+(pk (assoc-ref irssi-src-outputs "out"))
+;;; (#<<derivation-output> path: "/gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz" hash-algo: sha256 hash: #vu8(185 63 113 82 35 163 34 230 127 66 182 26 8 165 18 174 41 227 75 212 165 61 127 34 55 102 102 10 170 90 4 52) recursive?: #f>)
+```
+
+Note how the `hash` and `hash-algo` now have values.
+
+Perceptive readers may note that the `<derivation-output>` has four fields,
+whereas the tuple in the `.drv` file only has three (minus the label). The
+serialisation of `recursive?` is done by adding the prefix `r:` to the
+`hash-algo` field, though its actual purpose is difficult to explain, and is out
+of scope for this post.
+
+## `<derivation-input>`
+
+The next field is `inputs`, which corresponds to the second field in the `.drv`
+file format:
+
+```
+[ ("/gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv", ["out"]),
+ ("/gnu/store/baqpbl4wck7nkxrbyc9nlhma7kq5dyfl-guile-2.0.14.drv", ["out"]),
+ ("/gnu/store/bfirgq65ndhf63nn4q6vlkbha9zd931q-openssl-1.1.1l.drv", ["out"]),
+ ("/gnu/store/gjwpqzvfhz13shix6a6cs2hjc18pj7wy-module-import-compiled.drv", ["out"]),
+ ("/gnu/store/ij8651x4yh53hhcn6qw2644nhh2s8kcn-glib-2.70.2.drv", ["out"]),
+ ("/gnu/store/jg2vv6yc2yqzi3qzs82dxvqmi5k21lhy-irssi-1.4.3.drv", ["out"]),
+ ("/gnu/store/qggpjl9g6ic3cq09qrwkm0dfsdjf7pyr-glibc-utf8-locales-2.33.drv", ["out"]),
+ ("/gnu/store/zafabw13yyhz93jwrcz7axak1kn1f2cx-openssl-1.1.1s.drv", ["out"])
+]
+```
+
+Here, each tuple specifies a derivation that needs to be built before this
+derivation can be built, and the outputs of the derivation that the build
+process of this derivation uses. Let's grab us the Scheme equivalent:
+
+```scheme
+(define irssi-inputs
+ (pk (derivation-inputs irssi-drv)))
+;;; [a fairly large amount of output]
+
+(pk (car irssi-inputs))
+;;; (#<<derivation-input> drv: #<derivation /gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv => /gnu/store/2jj2mxn6wfrcw7i85nywk71mmqbnhzps-glib-2.70.2 7fe1902b6640> sub-derivations: ("out")>)
+```
+
+Unlike `derivation-outputs`, `derivation-inputs` maps 1:1 to the `.drv`
+form; the `drv` field is a `<derivation>` to be built, and the
+`sub-derivations` field is a list of outputs.
+
+## Builder Configuration
+
+The other fields are simpler; none of them involve new records. The third is
+`derivation-sources`, which contains a list of all store items used in the build
+which aren't themselves built using derivations, whereas `derivation-inputs`
+contains the dependencies which are.
+
+This list usually just contains the path to the Guile _build script_ that
+realises the store items when run, which we'll examine in a later post, and
+the path to a directory containing extra modules to add to the build script's
+`%load-path`, called `/gnu/store/...-module-import`.
+
+The next field is `derivation-system`, which specifies the Nix system we're
+building for. Then we have `derivation-builder`, pointing to the `guile`
+executable that runs the build script; and the second-to-last is
+`derivation-builder-arguments`, which is a list of arguments to pass to
+`derivation-builder`. Note how we use `-L` and `-C` to extend the Guile
+`%load-path` and `%load-compiled-path` to include the `module-import` and
+`module-import-compiled` directories:
+
+```scheme
+(pk (derivation-system irssi-drv))
+;;; ("x86_64-linux")
+
+(pk (derivation-builder irrsi-drv))
+;;; ("/gnu/store/hnr4r2d0h0xarx52i6jq9gvsrlc3q81a-guile-2.0.14/bin/guile")
+
+(pk (derivation-builder-arguments irrsi-drv))
+;;; (("--no-auto-compile" "-L" "/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import" "-C" "/gnu/store/6rkkvvb7pl1l9ng8vvywvwf357vhm3va-module-import-compiled" "/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"))
+```
+
+The final field contains a list of environment variables to set before we start
+the build process:
+
+```scheme
+(pk (derivation-builder-environment-vars irssi-drv))
+;;; ((("allowSubstitutes" . "0") ("guix properties" . "((type . graft) (graft (count . 2)))") ("out" . "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3") ("preferLocalBuild" . "1")))
+```
+
+The last record field, `derivation-file-name` contains the path to the `.drv`
+file, and so isn't represented in a serialised derivation.
+
+# Utilising `<derivation>`
+
+Speaking of serialisation, to convert between the `.drv` text format and the
+Scheme `<derivation>` record, you can use `write-derivation`, `read-derivation`,
+and `read-derivation-from-file`:
+
+```scheme
+(define manual-drv
+ (with-store %store
+ (derivation %store "manual"
+ "/bin/sh" '())))
+
+(write-derivation drv (current-output-port))
+;;; -| Derive([("out","/gnu/store/kh7fais2zab22fd8ar0ywa4767y6xyak-example","","")],[],[],"x86_64-linux","/bin/sh",[],[("out","/gnu/store/kh7fais2zab22fd8ar0ywa4767y6xyak-example")])
+
+(pk (read-derivation-from-file (derivation-file-name irssi-drv)))
+;;; (#<derivation /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7fb3798788c0>)
+
+(call-with-input-file (derivation-file-name irssi-drv)
+ read-derivation)
+;;; (#<derivation /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7fb37ad19e10>)
+```
+
+You can realise `<derivation>`s as store items using the `build-derivations`
+procedure:
+
+```scheme
+(use-modules (ice-9 ftw))
+
+(define irssi-drv-out
+ (pk (derivation-output-path
+ (assoc-ref (derivation-outputs irssi-drv) "out"))))
+;;; ("/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3")
+
+(pk (scandir irssi-drv-out))
+;;; (#f)
+
+(pk (with-store %store (build-derivations %store (list irssi-drv))))
+;;; (#t)
+
+(pk (scandir irssi-drv-out))
+;;; (("." ".." "bin" "etc" "include" "lib" "share"))
+```
+
+# Conclusion
+
+Derivations are one of Guix's most important concepts, but are fairly easy to
+understand once you get past the obtuse `.drv` file format. They provide the
+Guix daemon with the initial instructions that it uses to build store items
+like packages, origins, and other file-likes such as `computed-file` and
+`local-file`, which will be discussed in a future post!
+
+To recap, a derivation contains the following fields:
+
+1. `derivation-outputs`, describing the various output paths that the derivation
+ builds
+2. `derivation-inputs`, describing the other derivations that need to be built
+ before this one is
+3. `derivation-sources`, listing the non-derivation store items that the
+ derivation depends on
+4. `derivation-system`, specifying the Nix system a derivation will be compiled
+ for
+5. `derivation-builder`, the executable to run the build script with
+6. `derivation-builder-arguments`, arguments to pass to the builder
+7. `derivation-builder-environment-vars`, variables to set in the builder's
+ environment
+
+#### About GNU Guix
+
+[GNU Guix](https://guix.gnu.org) is a transactional package manager and
+an advanced distribution of the GNU system that [respects user
+freedom](https://www.gnu.org/distros/free-system-distribution-guidelines.html).
+Guix can be used on top of any system running the Hurd or the Linux
+kernel, or it can be used as a standalone operating system distribution
+for i686, x86_64, ARMv7, AArch64 and POWER9 machines.
+
+In addition to standard package management features, Guix supports
+transactional upgrades and roll-backs, unprivileged package management,
+per-user profiles, and garbage collection. When used as a standalone
+GNU/Linux distribution, Guix offers a declarative, stateless approach to
+operating system configuration management. Guix is highly customizable
+and hackable through [Guile](https://www.gnu.org/software/guile)
+programming interfaces and extensions to the
+[Scheme](http://schemers.org) language.
--
2.38.1
^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: [PATCH guix-artwork v2] website: posts: Add Dissecting Guix, Part 1: Derivations.
2022-12-12 19:31 ` [PATCH guix-artwork v2] website: posts: Add Dissecting Guix, Part 1: Derivations (
2022-12-12 20:59 ` Mekeor Melire
2022-12-12 21:06 ` Mekeor Melire
@ 2022-12-15 14:26 ` Ludovic Courtès
2 siblings, 0 replies; 23+ messages in thread
From: Ludovic Courtès @ 2022-12-15 14:26 UTC (permalink / raw)
To: (; +Cc: guix-devel, mekeor
Hello,
"(" <paren@disroot.org> skribis:
> * website/posts/dissecting-guix-1-derivations.md: New blog post.
Neat! I think your plan for blog posts is really great; I’m sure it
will be of great use to many.
Perhaps we could schedule the first one a few days after the release
(which should be on Monday, 19th)?
And then schedule subsequent posts once per month or something?
Anyway, I like this first post! Here are cosmetic suggestions:
> +These concepts were originally described by Eelco Dolstra, the author of Nix, in
s/author/original author/
> +# `.drv` Files
> +
> +You've probably seen these being printed by the Guix CLI now and again.
Please avoid abbreviations; if you want to use them, you can write “Guix
command-line interface (CLI)” the first time and just “CLI” on
subsequent iterations. Some abbreviations may look obvious to us, but I
think it’s important to reduce assumptions about what the reader already
knows, especially since the goal is to help them learn.
> +Derivations, represented in the daemon's eyes by `.drv` files, contain
> +instructions for building store items. We can retrieve the paths of these
> +`.drv` files with the `guix build --derivations` command:
Perhaps you can add a paragraph showing package definitions, explaining
that they get “lowered” to derivations, but that packagers and users
only ever manipulate high-level objects such as packages or operating
systems, not derivations?
> +The procedure we use to turn a high-level object like `<package>` into a
> +derivation is called `lower-object`; more on that in a future post. However,
> +this doesn't produce a derivation:
> +
> +```scheme
> +(pk (lower-object irssi))
> +;;; (#<procedure 7fe17c7af540 at guix/store.scm:1994:2 (state)>)
> +```
Maybe beforehand you can mention “,lower irssi”, to show that it exists
and that there’s a simple way to peek at derivations, and only then
dissect “,lower”?
> +The returned object is a procedure that needs to be evaluated in the context of
s/is a procedure/is a monadic value [in the store monad](…)/
> +a store connection. We do this by first using `with-store` to connect to the
> +store and bind the connection to a name, then wrapping the `lower-object` call
> +with `run-with-store`:
> +
> +```scheme
> +(define irssi-drv
> + (pk (with-store %store
> + (run-with-store %store
> + (lower-object irssi)))))
> +;;; (#<derivation /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7fe1902b6140>)
> +
> +(define glib-drv
> + (pk (with-store %store
> + (run-with-store %store
> + (lower-object glib)))))
> +;;; (#<derivation /gnu/store/81qqs7xah2ln39znrji4r6xj85zi15bi-glib-2.70.2.drv => /gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin /gnu/store/22mkp8cr6rxg6w8br9q8dbymf51b44m8-glib-2.70.2-debug /gnu/store/a6qb5arvir4vm1zlkp4chnl7d8qzzd7x-glib-2.70.2 /gnu/store/y4ak268dcdwkc6lmqfk9g1dgk2jr9i34-glib-2.70.2-static 7fe17ca13b90>)
> +```
Personally I’d leave ‘with-store’, ‘run-with-store’, and ‘lower-object’
for a subsequent post, to avoid muddying the waters. In this one I’d
just use “,lower” and “,build”, which is enough to peek at derivations
(we discussed it earlier on IRC and ISTR you were not quite convinced
though :-)).
> +Derivations are one of Guix's most important concepts, but are fairly easy to
Maybe: [Derivations](https://guix.gnu.org/manual/devel/en/html_node/Derivations.html)
> +4. `derivation-system`, specifying the Nix system a derivation will be compiled
> + for
s/Nix system(.*)/system type\1—e.g., `"x86_64-linux"`/
This is great, thanks a lot for your work!
Ludo’.
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH guix-artwork v4] website: posts: Add Dissecting Guix, Part 1: Derivations.
2022-12-08 18:24 Dissecting Guix -- blog post series (
` (3 preceding siblings ...)
2022-12-12 22:53 ` [PATCH guix-artwork v3] " (
@ 2022-12-17 22:06 ` (
2023-01-04 12:00 ` Ludovic Courtès
4 siblings, 1 reply; 23+ messages in thread
From: ( @ 2022-12-17 22:06 UTC (permalink / raw)
To: guix-devel; +Cc: mekeor, ludo, (
* website/posts/dissecting-guix-1-derivations.md: New blog post.
---
Heya,
Here, I've addressed Ludo's criticisms.
I do still think we should use LOWER-OBJECT rather than ,LOWER most of the time,
but there is now a paragraph noting that it exists.
.../posts/dissecting-guix-1-derivations.md | 450 ++++++++++++++++++
1 file changed, 450 insertions(+)
create mode 100644 website/posts/dissecting-guix-1-derivations.md
diff --git a/website/posts/dissecting-guix-1-derivations.md b/website/posts/dissecting-guix-1-derivations.md
new file mode 100644
index 0000000..7226029
--- /dev/null
+++ b/website/posts/dissecting-guix-1-derivations.md
@@ -0,0 +1,450 @@
+title: Dissecting Guix, Part 1: Derivations
+date: TBC
+author: (
+tags: Dissecting Guix, Functional package management, Programming interfaces, Scheme API
+---
+To a new user, Guix's functional architecture can seem quite alien, and possibly
+offputting. With a combination of extensive `#guix`-querying, determined
+manual-reading, and plenty of source-perusing, they may eventually figure out
+how everything fits together by themselves, but this can be frustrating and
+often takes a fairly long time.
+
+However, once you peel back the layers, the "Nix way" is actually rather
+elegant, if perhaps not as simple as the mutable, imperative style implemented
+by the likes of [`dpkg`](https://wiki.debian.org/dpkg) and
+[`pacman`](https://wiki.archlinux.org/title/pacman).
+This series of blog posts will cover basic Guix concepts, taking a "ground-up"
+approach by dealing with lower-level concepts first, and hopefully make those
+months of information-gathering unnecessary.
+
+Before we dig in to Guix-specific concepts, we'll need to learn about one
+inherited from [Nix](https://nixos.org), the original functional package manager
+and the inspiration for Guix; the idea of a
+[_derivation_](https://guix.gnu.org/manual/devel/en/html_node/Derivations.html)
+and its corresponding _store items_.
+
+These concepts were originally described by Eelco Dolstra, the original author
+of Nix, in their [PhD thesis](https://edolstra.github.io/pubs/phd-thesis.pdf);
+see _§ 2.1 The Nix store_ and _§ 2.4 Store Derivations_.
+
+# Store Items
+
+As you probably know, everything that Guix builds is stored in the _store_,
+which is almost always the `/gnu/store` directory. It's the job of the
+[`guix-daemon`](https://guix.gnu.org/manual/en/html_node/Invoking-guix_002ddaemon.html)
+to manage the store and build things. If you run
+[`guix build PKG`](https://guix.gnu.org/manual/en/html_node/Invoking-guix-build.html),
+`PKG` will be built or downloaded from a substitute server if available, and a
+path to an item in the store will be displayed.
+
+```
+$ guix build irssi
+/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3
+```
+
+This item contains the final result of building [`irssi`](https://irssi.org).
+Let's peek inside:
+
+```
+$ ls $(guix build irssi)
+bin/ etc/ include/ lib/ share/
+$ ls $(guix build irssi)/bin
+irssi*
+```
+
+`irssi` is quite a simple package. What about a more complex one, like
+[`glib`](https://docs.gtk.org/glib)?
+
+```
+$ guix build glib
+/gnu/store/bx8qq76idlmjrlqf1faslsq6zjc6f426-glib-2.73.3-bin
+/gnu/store/j65bhqwr7qq7l77nj0ahmk1f1ilnjr3a-glib-2.73.3-debug
+/gnu/store/3pn4ll6qakgfvfpc4mw89qrrbsgj3jf3-glib-2.73.3-doc
+/gnu/store/dvsk6x7d26nmwsqhnzws4iirb6dhhr1d-glib-2.73.3
+/gnu/store/4c8ycz501n2d0xdi4blahvnbjhd5hpa8-glib-2.73.3-static
+```
+
+`glib` produces five `/gnu/store` items, because it's possible for a package to
+produce multiple [outputs](https://guix.gnu.org/manual/en/html_node/Packages-with-Multiple-Outputs.html).
+Each output can be referred to separately, by prefixing a package's name with
+`:OUTPUT` where supported. For example, this
+[`guix install`](https://guix.gnu.org/manual/en/html_node/Invoking-guix-package.html)
+invocation will add `glib`'s `bin` output to your profile:
+
+```
+$ guix install glib:bin
+```
+
+The default output is `out`, so when you pass `glib` by itself to that command,
+it will actually install `glib:out` to the profile.
+
+`guix build` also provides the `--source` flag, which produces the store item
+corresponding to the given package's downloaded source code.
+
+```
+$ guix build --source irssi
+/gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz
+$ guix build --source glib
+/gnu/store/d22wzjq3xm3q8hwnhbgk2xd3ph7lb6ay-glib-2.73.3.tar.xz
+```
+
+But how does Guix know how to build these store outputs in the first place?
+That's where derivations come in.
+
+# `.drv` Files
+
+You've probably seen these being printed by the Guix program now and again.
+Derivations, represented in the daemon's eyes by `.drv` files, contain
+instructions for building store items. We can retrieve the paths of these
+`.drv` files with the `guix build --derivations` command:
+
+```
+$ guix build --derivations irssi
+/gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv
+```
+
+`guix build` can actually also accept derivation paths as an argument, in lieu
+of a package, like so:
+
+```
+$ guix build /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv
+/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3
+```
+
+Let's look inside this derivation file.
+
+```
+Derive([("out","/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3","","")],[("/gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv",["out"]),("/gnu/store/baqpbl4wck7nkxrbyc9nlhma7kq5dyfl-guile-2.0.14.drv",["out"]),("/gnu/store/bfirgq65ndhf63nn4q6vlkbha9zd931q-openssl-1.1.1l.drv",["out"]),("/gnu/store/gjwpqzvfhz13shix6a6cs2hjc18pj7wy-module-import-compiled.drv",["out"]),("/gnu/store/ij8651x4yh53hhcn6qw2644nhh2s8kcn-glib-2.70.2.drv",["out"]),("/gnu/store/jg2vv6yc2yqzi3qzs82dxvqmi5k21lhy-irssi-1.4.3.drv",["out"]),("/gnu/store/qggpjl9g6ic3cq09qrwkm0dfsdjf7pyr-glibc-utf8-locales-2.33.drv",["out"]),("/gnu/store/zafabw13yyhz93jwrcz7axak1kn1f2cx-openssl-1.1.1s.drv",["out"])],["/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import","/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"],"x86_64-linux","/gnu/store/hnr4r2d0h0xarx52i6jq9gvsrlc3q81a-guile-2.0.14/bin/guile",["--no-auto-compile","-L","/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import","-C","/gnu/store/6rkkvvb7pl1l9ng8vvywvwf357vhm3va-module-import-compiled","/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"],[("allowSubstitutes","0"),("guix properties","((type . graft) (graft (count . 2)))"),("out","/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3"),("preferLocalBuild","1")])
+```
+
+It's... not exactly human-readable. We could try to format it and break it
+down, but it'd still be pretty hard to understand, since `.drv` files contain no
+labels for the fields or any other human-readable indicators. Instead, we're
+going to explore derivations in a Guile REPL.
+
+# Exploring Guix Interactively
+
+Before we continue, we'll want to start a REPL, so that we can try out the Guix
+Guile API interactively. To run a REPL in the terminal, simply
+[call `guix repl`](https://guix.gnu.org/manual/devel/en/html_node/Using-Guix-Interactively.html).
+
+If you're using Emacs, you can instead install
+[Geiser](https://nongnu.org/geiser), which provides a comfortable Emacs UI for
+various Lisp REPLs, invoke `guix repl --listen=tcp:37146 &`, and type
+`M-x geiser-connect RET RET RET` to connect to the running Guile instance.
+
+Your `.guile` file may contain code for enabling colours and readline bindings
+that Geiser will choke on. The default Guix System `.guile` contains code to
+suppress these features when `INSIDE_EMACS` is set, so you'll need to run
+`guix repl` like this:
+
+```sh
+INSIDE_EMACS=1 guix repl --listen=tcp:37146 &
+```
+
+There are a few Guix modules we'll need. Run this Scheme code to import them:
+
+```scheme
+(use-modules (guix)
+ (guix derivations)
+ (guix gexp)
+ (guix packages)
+ (guix store)
+ (gnu packages glib)
+ (gnu packages irc))
+```
+
+We now have access to the store, G-expression, package, and derivation APIs,
+along with the `irssi` and `glib` `<package>` objects.
+
+# Creating a `<derivation>`
+
+The Guix API for derivations revolves around the `<derivation>` record, which is
+the Scheme representation of that whole block of text surrounded by
+`Derive(...)`. If we look in `guix/derivations.scm`, we can see that it's
+defined like this:
+
+```scheme
+(define-immutable-record-type <derivation>
+ (make-derivation outputs inputs sources system builder args env-vars
+ file-name)
+ derivation?
+ (outputs derivation-outputs) ; list of name/<derivation-output> pairs
+ (inputs derivation-inputs) ; list of <derivation-input>
+ (sources derivation-sources) ; list of store paths
+ (system derivation-system) ; string
+ (builder derivation-builder) ; store path
+ (args derivation-builder-arguments) ; list of strings
+ (env-vars derivation-builder-environment-vars) ; list of name/value pairs
+ (file-name derivation-file-name)) ; the .drv file name
+```
+
+With the exception of `file-name`, each of those fields corresponds to a field
+in the `Derive(...)` form. Before we can examine them, though, we need to
+figure out how to _lower_ that `irssi` `<package>` object into a derivation.
+
+`guix repl` provides the `,lower` command to create derivations quickly:
+
+```scheme
+,lower irssi
+(pk $1) ;use the $N variable automatically bound by the REPL
+;;; (#<derivation /gnu/store/drjfddvlblpr635jazrg9kn5azd9hsbj-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7f3bff4a6370>)
+```
+
+Since `,lower` is a REPL command, however, we can't use it in proper Scheme
+code. It's quite useful for exploring specific derivations interactively, but
+since the purpose of this blog post is to explain how things work inside, we're
+going to use the pure-Scheme approach here.
+
+The procedure we need to use to turn a high-level object like `<package>` into a
+derivation is called `lower-object`; more on that in a future post. However,
+this doesn't initially produce a derivation:
+
+```scheme
+(pk (lower-object irssi))
+;;; (#<procedure 7fe17c7af540 at guix/store.scm:1994:2 (state)>)
+```
+
+`pk` is an abbreviation for the procedure `peek`, which takes the given object,
+writes a representation of it to the output, and returns it. It's especially
+handy when you want to view an intermediate value in a complex expression.
+
+The returned object is a monadic value (more on those in the next post on
+monads) that needs to be evaluated in the context of a store connection. We do
+this by first using `with-store` to connect to the store and bind the connection
+to a name, then wrapping the `lower-object` call with `run-with-store`:
+
+```scheme
+(define irssi-drv
+ (pk (with-store %store
+ (run-with-store %store
+ (lower-object irssi)))))
+;;; (#<derivation /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7fe1902b6140>)
+
+(define glib-drv
+ (pk (with-store %store
+ (run-with-store %store
+ (lower-object glib)))))
+;;; (#<derivation /gnu/store/81qqs7xah2ln39znrji4r6xj85zi15bi-glib-2.70.2.drv => /gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin /gnu/store/22mkp8cr6rxg6w8br9q8dbymf51b44m8-glib-2.70.2-debug /gnu/store/a6qb5arvir4vm1zlkp4chnl7d8qzzd7x-glib-2.70.2 /gnu/store/y4ak268dcdwkc6lmqfk9g1dgk2jr9i34-glib-2.70.2-static 7fe17ca13b90>)
+```
+
+And we have liftoff! Now we've got two `<derivation>` records to play with.
+
+# Exploring `<derivation>`
+
+## `<derivation-output>`
+
+The first "argument" in the `.drv` file is `outputs`, which tells the Guix
+daemon about the outputs that this build can produce:
+
+```scheme
+(define irssi-outputs
+ (pk (derivation-outputs irssi-drv)))
+;;; ((("out" . #<<derivation-output> path: "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3" hash-algo: #f hash: #f recursive?: #f>)))
+
+(pk (assoc-ref irssi-outputs "out"))
+
+(define glib-outputs
+ (pk (derivation-outputs glib-drv)))
+;;; ((("bin" . #<<derivation-output> path: "/gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin" hash-algo: #f hash: #f recursive?: #f>) ("debug" . #<<derivation-output> path: "/gnu/store/22mkp8cr6rxg6w8br9q8dbymf51b44m8-glib-2.70.2-debug" hash-algo: #f hash: #f recursive?: #f>) ("out" . #<<derivation-output> path: "/gnu/store/a6qb5arvir4vm1zlkp4chnl7d8qzzd7x-glib-2.70.2" hash-algo: #f hash: #f recursive?: #f>) ("static" . #<<derivation-output> path: "/gnu/store/y4ak268dcdwkc6lmqfk9g1dgk2jr9i34-glib-2.70.2-static" hash-algo: #f hash: #f recursive?: #f>)))
+
+(pk (assoc-ref glib-outputs "bin"))
+;;; (#<<derivation-output> path: "/gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin" hash-algo: #f hash: #f recursive?: #f>)
+```
+
+It's a simple association list mapping output names to `<derivation-output>`
+records, and it's equivalent to the first "argument" in the `.drv` file:
+
+```
+[ ("out", "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3", "", "")
+]
+```
+
+The `hash-algo` and `hash` fields are for storing the content hash and the
+algorithm used with that hash for what we term a _fixed-output derivation_,
+which is essentially a derivation where we know what the hash of the content
+will be in advance. For instance, `origin`s produce fixed-output derivations:
+
+```scheme
+(define irssi-src-drv
+ (pk (with-store %store
+ (run-with-store %store
+ (lower-object (package-source irssi))))))
+;;; (#<derivation /gnu/store/mcz3vzq7lwwaqjb8dy7cd69lvmi6d241-irssi-1.4.3.tar.xz.drv => /gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz 7fe17b3c8d70>)
+
+(define irssi-src-outputs
+ (pk (derivation-outputs irssi-src-drv)))
+;;; ((("out" . #<<derivation-output> path: "/gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz" hash-algo: sha256 hash: #vu8(185 63 113 82 35 163 34 230 127 66 182 26 8 165 18 174 41 227 75 212 165 61 127 34 55 102 102 10 170 90 4 52) recursive?: #f>)))
+
+(pk (assoc-ref irssi-src-outputs "out"))
+;;; (#<<derivation-output> path: "/gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz" hash-algo: sha256 hash: #vu8(185 63 113 82 35 163 34 230 127 66 182 26 8 165 18 174 41 227 75 212 165 61 127 34 55 102 102 10 170 90 4 52) recursive?: #f>)
+```
+
+Note how the `hash` and `hash-algo` now have values.
+
+Perceptive readers may note that the `<derivation-output>` has four fields,
+whereas the tuple in the `.drv` file only has three (minus the label). The
+serialisation of `recursive?` is done by adding the prefix `r:` to the
+`hash-algo` field, though its actual purpose is difficult to explain, and is out
+of scope for this post.
+
+## `<derivation-input>`
+
+The next field is `inputs`, which corresponds to the second field in the `.drv`
+file format:
+
+```
+[ ("/gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv", ["out"]),
+ ("/gnu/store/baqpbl4wck7nkxrbyc9nlhma7kq5dyfl-guile-2.0.14.drv", ["out"]),
+ ("/gnu/store/bfirgq65ndhf63nn4q6vlkbha9zd931q-openssl-1.1.1l.drv", ["out"]),
+ ("/gnu/store/gjwpqzvfhz13shix6a6cs2hjc18pj7wy-module-import-compiled.drv", ["out"]),
+ ("/gnu/store/ij8651x4yh53hhcn6qw2644nhh2s8kcn-glib-2.70.2.drv", ["out"]),
+ ("/gnu/store/jg2vv6yc2yqzi3qzs82dxvqmi5k21lhy-irssi-1.4.3.drv", ["out"]),
+ ("/gnu/store/qggpjl9g6ic3cq09qrwkm0dfsdjf7pyr-glibc-utf8-locales-2.33.drv", ["out"]),
+ ("/gnu/store/zafabw13yyhz93jwrcz7axak1kn1f2cx-openssl-1.1.1s.drv", ["out"])
+]
+```
+
+Here, each tuple specifies a derivation that needs to be built before this
+derivation can be built, and the outputs of the derivation that the build
+process of this derivation uses. Let's grab us the Scheme equivalent:
+
+```scheme
+(define irssi-inputs
+ (pk (derivation-inputs irssi-drv)))
+;;; [a fairly large amount of output]
+
+(pk (car irssi-inputs))
+;;; (#<<derivation-input> drv: #<derivation /gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv => /gnu/store/2jj2mxn6wfrcw7i85nywk71mmqbnhzps-glib-2.70.2 7fe1902b6640> sub-derivations: ("out")>)
+```
+
+Unlike `derivation-outputs`, `derivation-inputs` maps 1:1 to the `.drv`
+form; the `drv` field is a `<derivation>` to be built, and the
+`sub-derivations` field is a list of outputs.
+
+## Builder Configuration
+
+The other fields are simpler; none of them involve new records. The third is
+`derivation-sources`, which contains a list of all store items used in the build
+which aren't themselves built using derivations, whereas `derivation-inputs`
+contains the dependencies which are.
+
+This list usually just contains the path to the Guile _build script_ that
+realises the store items when run, which we'll examine in a later post, and
+the path to a directory containing extra modules to add to the build script's
+`%load-path`, called `/gnu/store/...-module-import`.
+
+The next field is `derivation-system`, which specifies the system type (such as
+`x86_64-linux`) we're building for. Then we have `derivation-builder`, pointing
+to the `guile` executable that runs the build script; and the second-to-last is
+`derivation-builder-arguments`, which is a list of arguments to pass to
+`derivation-builder`. Note how we use `-L` and `-C` to extend the Guile
+`%load-path` and `%load-compiled-path` to include the `module-import` and
+`module-import-compiled` directories:
+
+```scheme
+(pk (derivation-system irssi-drv))
+;;; ("x86_64-linux")
+
+(pk (derivation-builder irrsi-drv))
+;;; ("/gnu/store/hnr4r2d0h0xarx52i6jq9gvsrlc3q81a-guile-2.0.14/bin/guile")
+
+(pk (derivation-builder-arguments irrsi-drv))
+;;; (("--no-auto-compile" "-L" "/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import" "-C" "/gnu/store/6rkkvvb7pl1l9ng8vvywvwf357vhm3va-module-import-compiled" "/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"))
+```
+
+The final field contains a list of environment variables to set before we start
+the build process:
+
+```scheme
+(pk (derivation-builder-environment-vars irssi-drv))
+;;; ((("allowSubstitutes" . "0") ("guix properties" . "((type . graft) (graft (count . 2)))") ("out" . "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3") ("preferLocalBuild" . "1")))
+```
+
+The last record field, `derivation-file-name` contains the path to the `.drv`
+file, and so isn't represented in a serialised derivation.
+
+# Utilising `<derivation>`
+
+Speaking of serialisation, to convert between the `.drv` text format and the
+Scheme `<derivation>` record, you can use `write-derivation`, `read-derivation`,
+and `read-derivation-from-file`:
+
+```scheme
+(define manual-drv
+ (with-store %store
+ (derivation %store "manual"
+ "/bin/sh" '())))
+
+(write-derivation drv (current-output-port))
+;;; -| Derive([("out","/gnu/store/kh7fais2zab22fd8ar0ywa4767y6xyak-example","","")],[],[],"x86_64-linux","/bin/sh",[],[("out","/gnu/store/kh7fais2zab22fd8ar0ywa4767y6xyak-example")])
+
+(pk (read-derivation-from-file (derivation-file-name irssi-drv)))
+;;; (#<derivation /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7fb3798788c0>)
+
+(call-with-input-file (derivation-file-name irssi-drv)
+ read-derivation)
+;;; (#<derivation /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7fb37ad19e10>)
+```
+
+You can realise `<derivation>`s as store items using the `build-derivations`
+procedure:
+
+```scheme
+(use-modules (ice-9 ftw))
+
+(define irssi-drv-out
+ (pk (derivation-output-path
+ (assoc-ref (derivation-outputs irssi-drv) "out"))))
+;;; ("/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3")
+
+(pk (scandir irssi-drv-out))
+;;; (#f)
+
+(pk (with-store %store (build-derivations %store (list irssi-drv))))
+;;; (#t)
+
+(pk (scandir irssi-drv-out))
+;;; (("." ".." "bin" "etc" "include" "lib" "share"))
+```
+
+# Conclusion
+
+Derivations are one of Guix's most important concepts, but are fairly easy to
+understand once you get past the obtuse `.drv` file format. They provide the
+Guix daemon with the initial instructions that it uses to build store items
+like packages, origins, and other file-likes such as `computed-file` and
+`local-file`, which will be discussed in a future post!
+
+To recap, a derivation contains the following fields:
+
+1. `derivation-outputs`, describing the various output paths that the derivation
+ builds
+2. `derivation-inputs`, describing the other derivations that need to be built
+ before this one is
+3. `derivation-sources`, listing the non-derivation store items that the
+ derivation depends on
+4. `derivation-system`, specifying the system type a derivation will be compiled
+ for
+5. `derivation-builder`, the executable to run the build script with
+6. `derivation-builder-arguments`, arguments to pass to the builder
+7. `derivation-builder-environment-vars`, variables to set in the builder's
+ environment
+
+#### About GNU Guix
+
+[GNU Guix](https://guix.gnu.org) is a transactional package manager and
+an advanced distribution of the GNU system that [respects user
+freedom](https://www.gnu.org/distros/free-system-distribution-guidelines.html).
+Guix can be used on top of any system running the Hurd or the Linux
+kernel, or it can be used as a standalone operating system distribution
+for i686, x86_64, ARMv7, AArch64 and POWER9 machines.
+
+In addition to standard package management features, Guix supports
+transactional upgrades and roll-backs, unprivileged package management,
+per-user profiles, and garbage collection. When used as a standalone
+GNU/Linux distribution, Guix offers a declarative, stateless approach to
+operating system configuration management. Guix is highly customizable
+and hackable through [Guile](https://www.gnu.org/software/guile)
+programming interfaces and extensions to the
+[Scheme](http://schemers.org) language.
--
2.38.1
^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: [PATCH guix-artwork v4] website: posts: Add Dissecting Guix, Part 1: Derivations.
2022-12-17 22:06 ` [PATCH guix-artwork v4] " (
@ 2023-01-04 12:00 ` Ludovic Courtès
2023-01-05 12:00 ` (
0 siblings, 1 reply; 23+ messages in thread
From: Ludovic Courtès @ 2023-01-04 12:00 UTC (permalink / raw)
To: (; +Cc: guix-devel, mekeor
Heya (,
"(" <paren@disroot.org> skribis:
> * website/posts/dissecting-guix-1-derivations.md: New blog post.
Finally pushed! It should show up online soon. Looking forward to
part 2. :-)
> +`guix repl` provides the `,lower` command to create derivations quickly:
> +
> +```scheme
> +,lower irssi
> +(pk $1) ;use the $N variable automatically bound by the REPL
> +;;; (#<derivation /gnu/store/drjfddvlblpr635jazrg9kn5azd9hsbj-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7f3bff4a6370>)
> +```
I took the liberty to turn this into a sample session to give a better
idea of what people would type and see at their REPL.
Thanks,
Ludo’.
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH guix-artwork v4] website: posts: Add Dissecting Guix, Part 1: Derivations.
2023-01-04 12:00 ` Ludovic Courtès
@ 2023-01-05 12:00 ` (
2023-01-06 1:12 ` 宋文武
2023-01-09 11:13 ` Ludovic Courtès
0 siblings, 2 replies; 23+ messages in thread
From: ( @ 2023-01-05 12:00 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: guix-devel, mekeor
[-- Attachment #1: Type: text/plain, Size: 222 bytes --]
On Wed Jan 4, 2023 at 12:00 PM GMT, Ludovic Courtès wrote:
> Finally pushed! It should show up online soon. Looking forward to
> part 2. :-)
\o/
Next part will be about monads and G-expressions :)
-- (
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH guix-artwork v4] website: posts: Add Dissecting Guix, Part 1: Derivations.
2023-01-05 12:00 ` (
@ 2023-01-06 1:12 ` 宋文武
2023-01-09 11:13 ` Ludovic Courtès
1 sibling, 0 replies; 23+ messages in thread
From: 宋文武 @ 2023-01-06 1:12 UTC (permalink / raw)
To: (; +Cc: Ludovic Courtès, guix-devel, mekeor
"(" <paren@disroot.org> writes:
> On Wed Jan 4, 2023 at 12:00 PM GMT, Ludovic Courtès wrote:
>> Finally pushed! It should show up online soon. Looking forward to
>> part 2. :-)
>
> \o/
>
> Next part will be about monads and G-expressions :)
>
Great, I enjoy read Part 1, and waiting for next... Thank you!
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH guix-artwork v4] website: posts: Add Dissecting Guix, Part 1: Derivations.
2023-01-05 12:00 ` (
2023-01-06 1:12 ` 宋文武
@ 2023-01-09 11:13 ` Ludovic Courtès
2023-01-10 6:59 ` (
1 sibling, 1 reply; 23+ messages in thread
From: Ludovic Courtès @ 2023-01-09 11:13 UTC (permalink / raw)
To: (; +Cc: guix-devel, mekeor
"(" <paren@disroot.org> skribis:
> On Wed Jan 4, 2023 at 12:00 PM GMT, Ludovic Courtès wrote:
>> Finally pushed! It should show up online soon. Looking forward to
>> part 2. :-)
>
> \o/
>
> Next part will be about monads and G-expressions :)
Yay! That can even be parts 2 and 3. :-)
Ludo’.
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH guix-artwork v4] website: posts: Add Dissecting Guix, Part 1: Derivations.
2023-01-09 11:13 ` Ludovic Courtès
@ 2023-01-10 6:59 ` (
2023-01-11 21:59 ` zimoun
0 siblings, 1 reply; 23+ messages in thread
From: ( @ 2023-01-10 6:59 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: guix-devel, mekeor
[-- Attachment #1: Type: text/plain, Size: 290 bytes --]
On Mon Jan 9, 2023 at 11:13 AM GMT, Ludovic Courtès wrote:
> Yay! That can even be parts 2 and 3. :-)
Probably, yeah. It might take me longer to write the monads post, since i didn't
understand Guix's monads when I started :) (I do understand them a bit now, though.)
-- (
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH guix-artwork v4] website: posts: Add Dissecting Guix, Part 1: Derivations.
2023-01-10 6:59 ` (
@ 2023-01-11 21:59 ` zimoun
0 siblings, 0 replies; 23+ messages in thread
From: zimoun @ 2023-01-11 21:59 UTC (permalink / raw)
To: (, Ludovic Courtès; +Cc: guix-devel, mekeor
Hi,
On Tue, 10 Jan 2023 at 06:59, "(" <paren@disroot.org> wrote:
> Probably, yeah. It might take me longer to write the monads post, since i didn't
> understand Guix's monads when I started :) (I do understand them a bit now, though.)
Feel free to post to guix-blog@gnu.org even an early draft if you want
some early feedback. Well, if this private alias is still working. :-)
(Hum, I do not remember who is behind in addition to me. ;-)))
Cheers,
simon
^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2023-01-11 22:50 UTC | newest]
Thread overview: 23+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-12-08 18:24 Dissecting Guix -- blog post series (
2022-12-09 7:31 ` 宋文武
2022-12-09 7:33 ` (
2022-12-09 9:32 ` bokr
2022-12-09 17:25 ` (
2022-12-12 13:46 ` Bengt Richter
2022-12-10 21:25 ` Mekeor Melire
2022-12-11 10:08 ` (
2022-12-12 1:43 ` Mekeor Melire
2022-12-12 17:26 ` (
2022-12-12 19:31 ` [PATCH guix-artwork v2] website: posts: Add Dissecting Guix, Part 1: Derivations (
2022-12-12 20:59 ` Mekeor Melire
2022-12-12 21:06 ` Mekeor Melire
2022-12-12 21:24 ` (
2022-12-15 14:26 ` Ludovic Courtès
2022-12-12 22:53 ` [PATCH guix-artwork v3] " (
2022-12-17 22:06 ` [PATCH guix-artwork v4] " (
2023-01-04 12:00 ` Ludovic Courtès
2023-01-05 12:00 ` (
2023-01-06 1:12 ` 宋文武
2023-01-09 11:13 ` Ludovic Courtès
2023-01-10 6:59 ` (
2023-01-11 21:59 ` zimoun
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).