Re: Blog: Guix packaging tutorial

[Pierre Neidhardt <mail@ambrevar.xyz>]

[-- Attachment #1.1: Type: text/plain, Size: 769 bytes --]

Alright, here is a first draft!

It's currently in Org, I'll export it to Markdown later.

It's much longer than I expected.  About 5000 words!  I guess some parts could
go, but I'm not sure which.  Feedback would be very much appreciated here.

Let me know if it's approachable enough and if you find any missing/tangled
parts in the progression.

I would also love feedback about the section titles.  Some could be made more
explicit I suppose.

I'm not satisfied with the advanced example: I took ~mg~ because it has a lot of
typical customization (snippets, inputs, arguments) but it does not have
- in-input sources
- %build-inputs
- propagated-inputs

I've made up a "my-mg" package but it relies on made-up inputs, so it won't
build.
Better suggestions anyone?


[-- Attachment #1.2: index.org --]
[-- Type: text/x-org, Size: 39984 bytes --]

#+TITLE: Guix packaging tutorial
#+AUTHOR: Pierre Neidhardt
#+date: <2018-09-24 Mon>

* Introduction

GNU Guix stands out as the /hackable/ package manager, mostly because it uses
[[https://www.gnu.org/software/guile/][GNU Guile]], a powerful high-level programming language, one of the [[https://en.wikipedia.org/wiki/Scheme_(programming_language)][Scheme]]
dialects from the [[https://en.wikipedia.org/wiki/Lisp_(programming_language)][Lisp family]].

Package definitions are also written in Scheme, which empowers Guix in some very
unique ways, unlike most other package managers that use shell scripts or
simplistic languages.

- Use functions, structures, macros and all of Scheme expressiveness into your
  package definitions.

- Inheritance makes it easy to customize a package by inheriting from it and
  modifying only what is needed.

- Batch processing: the whole package collection can be parsed, filtered and
  processed.  Building a headless server with all graphical interfaces stripped
  out?  It's possible.  Want to rebuild everything from source for a specific
  architecture?  Pass the ~#:system "YOUR-ARCH"~ argument to the list of
  packages.  It wouldn't be a stretch to think [[https://wiki.gentoo.org/wiki/USE_flag][Gentoo USE flags]] here, but this
  goes even further: the changes don't have to be thought out beforehand by the
  packager, they can be /programmed/ by the user!

The following tutorial covers all the basics around package creation with Guix.
It does not assume much knowledge of the Guix system nor of the Lisp language.
The reader is only expected to be familiar with the commandline and to have some
basic programming knowledge.

* A "Hello World" package

=GNU hello= is a dummy project that serves as an idiomatic example for
packaging.  If uses the GNU build chain (~./configure && make && make install~).
Guix already provides a package definition which is a perfect example to start
with.  You can look up its declaration with ~guix edit hello~ from the
commandline.  Let's see how it looks:

#+BEGIN_SRC scheme
(define-public hello
  (package
    (name "hello")
    (version "2.10")
    (source (origin
              (method url-fetch)
              (uri (string-append "mirror://gnu/hello/hello-" version
                                  ".tar.gz"))
              (sha256
               (base32
                "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
    (build-system gnu-build-system)
    (synopsis "Hello, GNU world: An example GNU package")
    (description
     "GNU Hello prints the message \"Hello, world!\" and then exits.  It
serves as an example of standard GNU coding practices.  As such, it supports
command-line arguments, multiple languages, and so on.")
    (home-page "https://www.gnu.org/software/hello/")
    (license gpl3+)))
#+END_SRC

As you can see, most of it is rather straightforward.  But let's review the
fields together:

- name :: The project name.  Using Scheme conventions, we prefer to keep it
          lower case, without underscore and using dash-separated words.
- source :: The ~origin~ record gather all the information required for a
            source.  Among which:
  1. The method, here ~url-fetch~ to download via HTTP/FTP, but other methods
     exist, such as ~git-fetch~ for Git repositories.
  2. The URI, which is typically some =https://= location for ~url-fetch~.  Here
     the special =mirror://gnu= refers to a set of well known locations, all of
     which can be used by Guix to fetch the source, should some of them fail.
  3. The ~sha256~ checksum of the requested file.  This is essential to ensure
     the source is not corrupted.  Note that Guix works with base32 strings,
     hence the call to the ~base32~ function.
- build-system :: This is where the power of abstraction provided by the Scheme
                  language really shine: in this case, the ~gnu-build-system~
                  abstracts away the famous ~./configure && make && make
                  install~ shell invocations: If the package can be built just
                  those commands, then Guix can often handle the complete
                  packaging process automatically.  Other build systems include
                  the ~trivial-build-system~ which does not do anything and
                  leaves all actions to the packager, the ~python-build-system~,
                  the ~emacs-build-system~, and many more.
- synopsis :: It should be a concise and explicit summary of what the package
              does.  Many projects homepage already provide a synopsis, it's
              fine to re-use it.
- description :: Same as for the synopsis, it's fine to re-use the project
                 description from the homepage.  Note that Guix uses Texinfo
                 syntax.
- home-page :: Use HTTPS if available.
- license :: See =$GUIX_CHECKOUT/guix/licenses.scm= for a full list.

Time to build our first package!  Nothing fancy here for now: we will stick to a
dummy "my-hello", a copycat of the above declaration.

As with the ritualistic "Hello World" taught with most programming language,
this will possibly be the most "manual" approach.  We will work out an ideal
setup later; for now we will go the simplest route.

Save the following to a file =my-hello.scm=.

#+BEGIN_SRC scheme
(use-modules (guix packages)
             (guix download)
             (guix build-system gnu)
             (guix licenses))

(define-public my-hello
  (package
    (name "my-hello")
    (version "2.10")
    (source (origin
              (method url-fetch)
              (uri (string-append "mirror://gnu/hello/hello-" version
                                  ".tar.gz"))
              (sha256
               (base32
                "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
    (build-system gnu-build-system)
    (synopsis "Hello, Guix world: An example custom Guix package")
    (description
     "GNU Hello prints the message \"Hello, world!\" and then exits.  It
serves as an example of standard GNU coding practices.  As such, it supports
command-line arguments, multiple languages, and so on.")
    (home-page "https://www.gnu.org/software/hello/")
    (license gpl3+)))

my-hello
#+END_SRC

We will explain the extra code in a moment.

Feel free to play with the different values of the various fields.  If you
change the source, you'll need to update the checksum.
Indeed, Guix refuses to build anything if the source fails to verify the checksum.
To compute the correct checksum of the package declaration, we need to download the
source, compute the sha256 checksum and convert it to base32.

Thankfully Guix can automate this task for us, all we need is to provide the URI:

#+BEGIN_SRC sh
$ guix download mirror://gnu/hello/hello-2.10.tar.gz

Starting download of /tmp/guix-file.JLYgL7
From https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz...
following redirection to `https://mirror.ibcp.fr/pub/gnu/hello/hello-2.10.tar.gz'...
 …10.tar.gz  709KiB                                 2.5MiB/s 00:00 [##################] 100.0%
/gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i
#+END_SRC

Note in this specific case that the output tells us which mirror was chosen.

If the result of the above command is not the same as in the above snippet,
update you =my-hello= declaration accordingly.

Now you can happily run

#+BEGIN_SRC sh
$ guix package --install-from-file=my-hello.scm
#+END_SRC

You should now have =my-hello= in your profile!

#+BEGIN_SRC sh
$ guix package --list-installed=my-hello
my-hello	2.10	out	/gnu/store/f1db2mfm8syb8qvc357c53slbvf1g9m9-my-hello-2.10
#+END_SRC

We've gone as far as we could without any knowledge of Scheme.  Now is the right
time to introduce the minimum we need of the language before we can proceed.

* A Scheme crash-course

As we've seen above, basic packages don't require much Scheme knowledge, if none
at all.  But as you'll progress and your desire to write more and more complex
packages will grow, it will become both necessary and empowering to hone your
Lisper skills.

Since this is very much out of the scope of this tutorial, we will only cover
some basics here.

Guix uses the Guile implementation of Scheme.  To start playing with the
language, install it with ~guix package --install guile~ and start a [[https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop][REPL]] by
running ~guile~ from the commandline.

Alternatively you can also run ~guix environment --ad-hoc guile -- guile~ if
you'd rather not have Guile installed in your user profile.

- Scheme syntax boils down to a tree of expressions (or /s-expression/ in Lisp
  lingo) It can be a literal such numbers and strings, or a compound which is a
  parenthesized list of compounds and literals.
  ~#t~ and ~#f~ stand for the booleans "true" and "false", respectively.

  Examples of valid expressions:
  #+BEGIN_SRC scheme
scheme@(guile-user)> "Hello World!"
$1 = "Hello World!"
scheme@(guile-user)> 17
$2 = 17
scheme@(guile-user)> (format #f "Hello ~a!" "Guix")
$3 = "Hello Guix!"
  #+END_SRC

- This last example is a function call.  When a parenthesized expression is
  evaluated, the first term is the function and the rest are the arguments
  passed to the function.  Every function returns the last
  evaluated expression as value.

- Anonymous functions are declared with the ~lambda~ term:
  #+BEGIN_SRC scheme
scheme@(guile-user)> (lambda (x) (* x x))
$4 = #<procedure 120e348 at <unknown port>:24:0 (x)>
  #+END_SRC
  The above lambda returns the square of its argument.  Since everything is an
  expression, the ~lambda~ expression returns an anonymous function, which can
  in turn be called over an argument:
  #+BEGIN_SRC scheme
scheme@(guile-user)> ((lambda (x) (* x x)) 3)
$5 = 9
  #+END_SRC

- Anything can be assigned a global name with ~define~:
  #+BEGIN_SRC scheme
scheme@(guile-user)> (define a 4)
scheme@(guile-user)> (define square (lambda (x) (* x x)))
scheme@(guile-user)> (square a)
$6 = 9
  #+END_SRC

- Procedures can be defined more concisely with the following syntax:
  #+BEGIN_SRC scheme
  (define (square x) (* x x))
  #+END_SRC

- A list structure can be created with the ~list~ procedure:
  #+BEGIN_SRC scheme
scheme@(guile-user)> (list 2 a 5 7)
$7 = (2 3 5 7)
  #+END_SRC

- The /quote/ disables evaluation of a parenthesized expression: the first term
  is not called over the other terms.  Thus it effectively returns a list of
  terms.
  #+BEGIN_SRC scheme
scheme@(guile-user)> '(format #f "Hello ~a!" "Guix")
$8 = (format #f "Hello ~a!" "Guix")
scheme@(guile-user)> '(2 a 5 7)
$9 = (2 a 5 7)
  #+END_SRC

- The /quasiquote/ disables evaluation of a parenthesized expression until a
  colon re-enables it.  Thus it enables us with fine-grained control over what
  is evaluated and what is not.
  #+BEGIN_SRC scheme
scheme@(guile-user)> `(2 a 5 7 (2 ,a 5 ,(+ a 4)))
$10 = (2 a 5 7 (2 3 5 7))
  #+END_SRC
  Note that the above result is a list of mixed elements: numbers, symbols (here
  ~a~) and the last element is a list itself.

- Multiple variables can be named locally with ~let~:
  #+BEGIN_SRC scheme
scheme@(guile-user)> (define x 10)
scheme@(guile-user)> (let ((x 2)
                           (y 3))
                       (list x y))
$11 = (2 3)
scheme@(guile-user)> x
$12 = 10
scheme@(guile-user)> y
ERROR: In procedure module-lookup: Unbound variable: y
  #+END_SRC
  Use ~let*~ to re-allows the initializers of later variables to refer to the
  earlier variables.
  #+BEGIN_SRC scheme
scheme@(guile-user)> (let* ((x 2)
                            (y (* x 3)))
                       (list x y))
$13 = (2 6)
  #+END_SRC

- The keyword syntax is ~#:~, it is used to create unique identifiers.  See also
  http://practical-scheme.net/wiliki/schemexref.cgi?keyword%3F.

- The percentage ~%~ is conventionally used as a prefix for variables generated
  in scope.  Note that it is merely a convention, like ~_~ in C.  Scheme Lisp
  treats ~%~ exactly the same as any other letter.

- Modules are created with ~define-module~.  For instance
  #+BEGIN_SRC scheme
  (define-module (guix build-system ruby)
    #:use-module (guix store)
    #:export (ruby-build
              ruby-build-system))
  #+END_SRC
  defines the module ~ruby~ which must be located in
  ~guix/build-system/ruby.scm~ somewhere in =GUILE_LOAD_PATH=.  It depends on
  the ~(guix store)~ module and it exports two symbols, ~ruby-build~ and
  ~ruby-build-system~.

For a more detailed introduction, check out /[[http://www.troubleshooters.com/codecorn/scheme_guile/hello.htm][Scheme at a Glance]]/, by Steve Litt.

One of the reference Scheme books is the seminal /Structure and Interpretation
of Computer Programs/, by Harold Abelson and Gerald Jay Sussman, with Julie
Sussman.  You'll find a free copy [[https://mitpress.mit.edu/sites/default/files/sicp/index.html][online]], together with [[https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-001-structure-and-interpretation-of-computer-programs-spring-2005/video-lectures/][videos of the lectures
by the authors]].  The book is available in Texinfo format as the =sicp= Guix
package.  Go ahead, run ~guix package --install sicp~ and start reading with
~info sicp~ (or with the Emacs Info reader).  An unofficial ebook [[https://sarabander.github.io/sicp/][is also
available]].

You'll find more books, tutorials and other resources at https://schemers.org/.

* Setup

Now that we know some Scheme basics we can detail the different possible setups
for working on Guix packages.

There are several ways to set up a Guix packaging environment.

We recommend you work directly on the Guix source checkout since it makes it
easier for everyone to contribute to the project.

But first, let's look at other possibilities.

*** Local file

This is what we previously did with =my-hello=.  Now that we know more Scheme,
let's explain the leading and trailing chunks.  As stated in ~guix package
--help~:

#+BEGIN_SRC sh
  -f, --install-from-file=FILE
                         install the package that the code within FILE
                         evaluates to
#+END_SRC

This explains why we had to add ~my-hello~ as a last value: we need to return a
package.
Indeed, if you remove that line, ~guix package --install-from-file=my-hello.scm~
will fail because the last expression, ~define-public~, does not return a package.

The ~use-modules~ expression tells which of the modules we need in the file.
Modules are a collection of values and procedures.  They are commonly called
"libraries" or "packages" in other programming languages.

*** GUIX_PACKAGE_PATH

It can be tedious to specify the file from the commandline instead of simply
calling ~guix package --install my-hello~ as you would do with the official
packages.

Guix makes it possible to streamline the process by adding as many "package
declaration paths" as you want.

Create a folder, say =~./guix-packages= and add it to the GUIX_PACKAGE_PATH
environment variable:

#+BEGIN_SRC sh
$ mkdir ~/guix-packages
$ export GUIX_PACKAGE_PATH=~/guix-packages
#+END_SRC

To add several folders, separate them with a colon (~:~).

Our previous =my-hello= needs some adjustments though:

#+BEGIN_SRC scheme
(define-module (my-hello)
  #:use-module (guix licenses)
  #:use-module (guix packages)
  #:use-module (guix build-system gnu)
  #:use-module (guix download))

(define-public my-hello
  (package
    (name "my-hello")
    (version "2.10")
    (source (origin
              (method url-fetch)
              (uri (string-append "mirror://gnu/hello/hello-" version
                                  ".tar.gz"))
              (sha256
               (base32
                "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
    (build-system gnu-build-system)
    (synopsis "Hello, Guix world: An example custom Guix package")
    (description
     "GNU Hello prints the message \"Hello, world!\" and then exits.  It
serves as an example of standard GNU coding practices.  As such, it supports
command-line arguments, multiple languages, and so on.")
    (home-page "https://www.gnu.org/software/hello/")
    (license gpl3+)))
#+END_SRC

Now =my-hello= should be part of the package collection like all other official
packages.  You can verify this with:

#+BEGIN_SRC sh
$ guix package --show=my-hello
#+END_SRC

*** Direct checkout hacking

Working directly on the Guix project is recommended: it reduces the friction
when the time comes to submit your changes upstream to let the community benefit
from your hard work!

Check out the official [[https://git-scm.com/][Git]] repository:

#+BEGIN_SRC sh
$ git clone https://git.savannah.gnu.org/git/guix.git
#+END_SRC

Follow the instruction from the [[https://www.gnu.org/software/guix/manual/en/html_node/Contributing.html]["Contributing" chapter]] in the manual to set up the
repository environment.

Once ready, you should be able to use the package definitions from the
repository environment.

Feel free to edit package definitions found in =$GUIX_CHECKOUT/gnu/packages=.

The =$GUIX_CHECKOUT/pre-inst-env= script lets you use =guix= over the repository
package collection.

- Search packages:

  #+BEGIN_SRC sh
  $ ./pre-inst-env guix package --list-available=ruby
      ruby    1.8.7-p374      out     gnu/packages/ruby.scm:119:2
      ruby    2.1.6   out     gnu/packages/ruby.scm:91:2
      ruby    2.2.2   out     gnu/packages/ruby.scm:39:2
  #+END_SRC

- Build a package:

  #+BEGIN_SRC sh
  $ ./pre-inst-env guix build --keep-failed ruby@2.1
    /gnu/store/c13v73jxmj2nir2xjqaz5259zywsa9zi-ruby-2.1.6
  #+END_SRC

- Install it to your user profile:

  #+BEGIN_SRC sh
  $ ./pre-inst-env guix package --install ruby@2.1
  #+END_SRC

- Check for common mistakes:

  #+BEGIN_SRC sh
  $ ./pre-inst-env guix lint ruby@2.1
  #+END_SRC

Once you are happy with the result, you are welcome to send your contribution so
that make it part of Guix.  This process is also detailed in the [[https://www.gnu.org/software/guix/manual/en/html_node/Contributing.html][manual]].

It's a community effort so the more join in, the better Guix becomes!

* Extended example

The first example was as simple as it goes.  Packages can be more complex than
that and Guix can handle more advanced scenarios.
Let's looks at another, more sophisticated package (slightly modified from the
source):

#+BEGIN_SRC scheme
  (define-public my-mg
    (package
      (name "my-mg")
      (version "20180408")
      (source (origin
                (method git-fetch)
                (uri (git-reference
                      (url "https://github.com/hboetes/mg")
                      (commit version)))
                (file-name (git-file-name name version))
                (sha256
                 (base32
                  "06w86xk7sjl2x2h3z6msn8kpmwj05qdimcym77wzhz5s94dzh1bl"))
                (modules '((guix build utils)))
                (snippet '(begin
                            (substitute* "GNUmakefile"
                              (("/usr/bin/") ""))
                            #t))))
      (build-system gnu-build-system)
      (native-inputs
       `(("pkg-config" ,pkg-config)))
      (inputs
       `(("libbsd" ,libbsd)
         ("ncurses" ,ncurses)))
      (propagated-inputs
       `(("mg-extensions"
          ,(origin
             (method url-fetch)
             (uri "https://example.org/~doe/mg-extensions.tar.gz"
             (sha256
              (base32
               "0kfhpj5rnh24hz2714qhfmxk281vwc2w50sm73ggw5d15af7zfsw"))))))
      (outputs '("out" "doc"))
      (arguments
       '(#:tests? #f ; No test suite available.
         #:make-flags (list (string-append "prefix=" %output)
                            "CC=gcc")
         #:configure-flags `(,(string-append "--with-ext="
                                             (assoc-ref %build-inputs "mg-extensions")))
         #:phases (modify-phases %standard-phases
                    (delete 'configure)   ; no configure script
                    (add-before 'build 'correct-location-of-difftool
                      (lambda _
                        (substitute* "buffer.c"
                          (("/usr/bin/diff")
                           (which "diff")))
                        #t))
                    (add-before 'install 'patch-tutorial-location
                      (lambda* (#:key outputs #:allow-other-keys)
                        (substitute* "mg.1"
                          (("/usr") (assoc-ref outputs "out")))
                        #t))
                    (add-after 'install 'install-tutorial
                      (lambda* (#:key outputs #:allow-other-keys)
                        (let* ((doc-root (assoc-ref outputs "doc"))
                               (doc (string-append out-root "/share/doc/mg")))
                          (install-file "tutorial" doc)
                          #t))))))
      (home-page "https://homepage.boetes.org/software/mg/")
      (synopsis "Microscopic GNU Emacs clone with custom extensions")
      (description
       "Mg (mg) is a GNU Emacs style editor, with which it is \"broadly\"
  compatible.  This is a portable version of the mg maintained by the OpenBSD
  team.
  This package adds some custom extensions to the original.")
      (license license:public-domain)))
#+END_SRC

(In those cases were you only want to tweak a few fields from a package
definition, you should rely on inheritance instead of copy-pasting everything.
See below.)

Let's discuss those fields in depth.

** ~git-fetch~ method

Unlike the ~url-fetch~ method, ~git-fetch~ expects a ~git-reference~ which takes
a Git repository and a commit.  The commit can be any Git reference such as
tags, so if the ~version~ is tagged, then it can be used directly.  Sometimes
the tag is prefixed with a ~v~, in which case you'd use ~(commit (string-append
"v" version))~.

When downloaded to the store, the source should be different from version to
version.  Since Git repositories don't contain a specific version in their name,
we've got to force the download folder using ~(file-name (git-file-name name
version))~.

Note that there is also a ~git-version~ procedure that can be used to derive the
version when packaging programs for a specific commit.

** Snippets

Snippets are quoted (i.e. non-evaluated) Scheme code that are a mean of patching
the source.  They are a Guix-y alternative to the traditional =.patch= files.
Because of the quote, the code in only evaluated when passed to the Guix daemon
for building.

There can be as many snippet as needed.

Snippets might need additional Guile modules which can be imported from the
~modules~ field.

** Inputs

First, a syntactic comment: See the quasi-quote / comma syntax?

#+BEGIN_SRC scheme
    (native-inputs
     `(("pkg-config" ,pkg-config)))
#+END_SRC

is equivalent to

#+BEGIN_SRC scheme
    (native-inputs
     (list "pkg-config" pkg-config))
#+END_SRC

You'll mostly see the former because it's shorter.

There are 3 different input types.  In short:

- native-inputs :: Required for building but not runtime -- installing a package
                   through a substitute won't install these inputs.
- inputs :: Installed in the store but not in the profile, as well as being
            present at build time.
- propagated-inputs :: Installed in the store and in the profile, as well as
     being present at build time.

The distinction between the various inputs is important: if a dependency can be
handled as an /input/ instead of a /propagated input/, it should be done so, or
else it "pollutes" the user profile for no reason.

For instance, a user installing a graphical program that depends on a
commandline tool might only be interested in the graphical part, so there is no
need to force the commandline tool into the user profile.  The dependency is a
concern to the package, no to the user.  /Inputs/ make it possible to handle
dependencies without bugging the using by adding undesired executable files (or
libraries) to their profile.

Same goes for /native-inputs/: once the program is installed, build-time
dependencies can be safely garbage collected.
It also matters when a substitute is available, in which case only the /inputs/
and /propagated inputs/ will be fetched: the /native inputs/ are not required to
install a package from a substitute.

** Build system arguments

The ~arguments~ is a keyword-value list used to configure the build process.

The simplest argument ~#:tests?~ can be used to disable the test suite when
building the package.  This is mostly useful when the package does not feature
any test suite.  It's strongly recommended to keep the test suite on if there is
one.

Another  common argument is ~:make-flags~, which specifies a list of flags to
append when running make, as you would from the commandline:

#+BEGIN_SRC sh
$ make FLAGS...

#+END_SRC

In the above package, the argument

#+BEGIN_SRC scheme
         #:make-flags (list (string-append "prefix=" %output)
                            "CC=gcc")
#+END_SRC

sets the C compiler to ~gcc~ and the ~prefix~ variable (the installation folder
in Make parlance) to ~%output~, which is a variable generated in scope and
pointing to the destination folder in the store (something like
=/GNU/store/...-my-mg-20180408=).

Similarly, it's possible to set the "configure" flags.  In the above example,
we set the extensions to the store download folder of our custom mg-extensions.

#+BEGIN_SRC scheme
         #:configure-flags `(,(string-append "--with-ext="
                                             (assoc-ref %build-inputs "mg-extensions")))
#+END_SRC

The ~%build-inputs~ variable is also generated in scope.  It's an associated
table that maps the input names to their store folders.

The ~phases~ keyword lists the sequential steps of the build system.  Typically
phases include ~unpack~, ~configure~, ~build~, ~install~ and ~check~.  To know
more about those phases, you need to work out the appropriate build system
definition in =$GUIX_CHECKOUT/guix/build/gnu-build-system.scm=:

#+BEGIN_SRC scheme
(define %standard-phases
  ;; Standard build phases, as a list of symbol/procedure pairs.
  (let-syntax ((phases (syntax-rules ()
                         ((_ p ...) `((p . ,p) ...)))))
    (phases set-SOURCE-DATE-EPOCH set-paths install-locale unpack
            bootstrap
            patch-usr-bin-file
            patch-source-shebangs configure patch-generated-file-shebangs
            build check install
            patch-shebangs strip
            validate-runpath
            validate-documentation-location
            delete-info-dir-file
            patch-dot-desktop-files
            install-license-files
            reset-gzip-timestamps
            compress-documentation)))
#+END_SRC

Or from the REPL (assuming the Guix source is in your Guile load path):

#+BEGIN_SRC scheme
scheme@(guile-user)> ,module (guix build gnu-build-system)
scheme@(guix build gnu-build-system)> (map first %standard-phases)
$1 = (set-SOURCE-DATE-EPOCH set-paths install-locale unpack bootstrap patch-usr-bin-file patch-source-shebangs configure patch-generated-file-shebangs build check install patch-shebangs strip validate-runpath validate-documentation-location delete-info-dir-file patch-dot-desktop-files install-license-files reset-gzip-timestamps compress-documentation)
#+END_SRC

If you want to know more about what happens during those phases, consult the
associated functions.

For instance, as of this writing the definition of ~unpack~ for the GNU build
system is

#+BEGIN_SRC scheme
(define* (unpack #:key source #:allow-other-keys)
  "Unpack SOURCE in the working directory, and change directory within the
source.  When SOURCE is a directory, copy it in a sub-directory of the current
working directory."
  (if (file-is-directory? source)
      (begin
        (mkdir "source")
        (chdir "source")

        ;; Preserve timestamps (set to the Epoch) on the copied tree so that
        ;; things work deterministically.
        (copy-recursively source "."
                          #:keep-mtime? #t))
      (begin
        (if (string-suffix? ".zip" source)
            (invoke "unzip" source)
            (invoke "tar" "xvf" source))
        (chdir (first-subdirectory "."))))
  #t)
#+END_SRC

Note the ~chdir~ call: it changes the working directory to where the source was
unpacked.
Thus every phase following the ~unpack~ will use the source as a working
directory, which is why we can directly work on the source files.
That is to say, unless a later phase changes the working directory to something
else.

To manipulate the phases,

- ~add-before PHASE NEW-PHASE PROCEDURE~: Run ~PROCEDURE~ named ~NEW-PHASE~ before ~PHASE~.
- ~add-after PHASE NEW-PHASE PROCEDURE~: Same, but afterwards.
- ~delete PHASE~.

The ~PROCEDURE~ supports the keyword arguments ~inputs~ and ~outputs~.  Each
input (whether /native/, /propagated/ or not) and output directory is referenced
by their name in those variables.  Thus ~(assoc-ref outputs "out")~ is the store
directory of the main output of the package.  A phase procedure may look like
this:

#+BEGIN_SRC scheme
(lambda* (#:key inputs outputs #:allow-other-keys)
  (let (((bash-folder (assoc-ref inputs "bash"))
         (output-folder (assoc-ref outputs "out"))
         (doc-folder (assoc-ref outputs "doc"))
  ; ...
  #t)
#+END_SRC

The procedure must return ~#t~ on success.  It's brittle to rely on the return
value of the last expression used to tweak the phase because there is no
guarantee it would be a ~#t~.  Hence the trailing ~#t~ to ensure the right value
is returned on success.

** Code staging

The astute reader may have noticed the quasi-quote and comma syntax in the
argument field.  Indeed, the build code in the package declaration should not be
evaluated on the client side, but only when passed to the Guix daemon.  This
mechanism of passing code around two running processes is called [[https://arxiv.org/abs/1709.00833][code staging]].
See [[https://www.gnu.org/software/guix/manual/en/html_node/G_002dExpressions.html][the "G-Expressions" chapter]] from the manual.

** "Utils" functions

When customizing ~phases~, we often need to write code that mimics the
equivalent system invocations (~make~, ~mkdir~, ~cp~, etc.) commonly used during
regular "Unix-style" installations.

Some like ~chmod~ are native to Guile.  See the [[https://www.gnu.org/software/guile/manual/guile.html][Guile reference manual]] for a
complete list.

Guix provides additional helper functions which prove especially handy in the
context of package management.

Some of those functions can be found in
=$GUIX_CHECKOUT/guix/guix/build/utils.scm=.  Most of them mirror the behaviour
of the traditional Unix system commands:

- which :: Like the =which= system command.
- find-files :: Akin to the =find= system command.
- mkdir-p :: Like =mkdir -p=, which creates all parents as needed.
- install-file :: Similar to =install= when installing a file to a (possibly
                  non-existing) directory.  Guile has ~copy-file~ which works
                  like =cp=.
- copy-recursively :: Like =cp -r=.
- delete-file-recursively :: Like =rm -rf=.
- invoke :: This should be used instead of ~system*~.
- with-directory-excursion :: Run the body in a different working directory,
     then restore the previous working directory.
- substitute* :: A "sed-like" function.

** Module prefix

The license now needs a prefix: this is because of how the ~license~ module was
important in the package, as ~#:use-module ((guix licenses) #:prefix
license:)~.  This gives the user full control over namespacing.

* Other build systems

What we've seen so far covers the majority of packages using a build system
other than the ~trivial-build-system~.  The latter does not automate anything
and leaves you to build everything manually.  This can be more demanding and we
won't cover it here for now, but thankfully it is rarely necessary to fall back
on this system.

For the other build systems, such as ASDF, Emacs, Perl, Ruby and many more, the
process is very similar to the GNU build system but for a few specialized
arguments.

Find more about build systems in
- [[https://www.gnu.org/software/guix/manual/en/html_node/Build-Systems.html#Build-Systems][the manual, section 4.2 Build systems]],
- the source code in the =$GUIX_CHECKOUT/guix/build= and
  =$GUIX_CHECKOUT/guix/build-system= folders.

* Common pitfalls and best practices

The following check list is important to maintain a high packaging standard, in
particular when contributing to the Guix project.

- Use the /linter/: ~guix lint $PACKAGES~.
- Use mirrors when possible in the source URL.
- Use reliable URLs, not generated ones (e.g. GitHub archives).
- Sort inputs alphabetically.
- Synopsis should be as concise as possible, don't start with "The" or "A", don't repeat
  the package name.
- Description uses Texinfo syntax with two spaces at the end of sentences.
- Reproducibility: use ~guix build --check~ and ~guix build --rounds=N~.
- Follow the [[https://www.gnu.org/software/guix/manual/en/html_node/Coding-Style.html][coding style]] from the manual.
- Review the [[https://www.gnu.org/software/guix/manual/en/html_node/Submitting-Patches.html][check list]] from the manual.

* Programmable and automated package definition

We can't repeat it enough: having a full-fledged programming language at hand
empowers us in ways that reach far beyond traditional package management.

Let's illustrate this with some awesome features of Guix!

** Recursive importers

You might find some build systems good enough that there is little to do at all
to write a package, to the point that it becomes repetitive and tedious after a
while.

Humans should not operate tedious repetitive taks manuall, computers are meant
to automate that job!  Let's tell Guix to create the package definition of an
ELPA package:

#+BEGIN_SRC sh
$ guix import elpa exwm

(package
  (name "emacs-exwm")
  (version "0.19")
  (source
    (origin
      (method url-fetch)
      (uri (string-append
             "https://elpa.gnu.org/packages/exwm-"
             version
             ".tar"))
      (sha256
        (base32
          "11xd2w4h3zdwkdxypvmcz8s7q72cn76lfr9js77jbizyj6b04lr0"))))
  (build-system emacs-build-system)
  (propagated-inputs `(("emacs-xelb" ,emacs-xelb)))
  (home-page "https://github.com/ch11ng/exwm")
  (synopsis "Emacs X Window Manager")
  (description
    "EXWM (Emacs X Window Manager) is a full-featured tiling X window manager
for Emacs built on top of [XELB](https://github.com/ch11ng/xelb).
It features:
+ Fully keyboard-driven operations
+ Hybrid layout modes (tiling & stacking)
+ Dynamic workspace support
+ ICCCM/EWMH compliance
+ (Optional) RandR (multi-monitor) support
+ (Optional) Built-in system tray")
  (license license:gpl3+))
#+END_SRC

Not all applications can be packaged this way, on those relying on a select
number of systems.  Read about the full list of importers in the [[https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-import.html][guix import
section]] of the manual.

** Automatic update

Guix can be smart enough to check for updates on systems it knows.  It can
report outdated package definition with

#+BEGIN_SRC sh
$ guix refresh hello
#+END_SRC

In most cases, updating a package to a newer version requires little more than
changing the version number and the checksum.  Guix can do that automatically as
well:

#+BEGIN_SRC
$ guix refresh hello --update
#+END_SRC

** Inheritance

If you've started browsing the existing package definitions, you might have
noticed that a significant number of them have a ~inherit~ field:

#+BEGIN_SRC scheme
(define-public adwaita-icon-theme
  (package (inherit gnome-icon-theme)
    (name "adwaita-icon-theme")
    (version "3.26.1")
    (source (origin
              (method url-fetch)
              (uri (string-append "mirror://gnome/sources/" name "/"
                                  (version-major+minor version) "/"
                                  name "-" version ".tar.xz"))
              (sha256
               (base32
                "17fpahgh5dyckgz7rwqvzgnhx53cx9kr2xw0szprc6bnqy977fi8"))))
    (native-inputs
     `(("gtk-encode-symbolic-svg" ,gtk+ "bin")))))
#+END_SRC

All unspecified fields inherit from the parent package.  This is very convenient
to create alternative packages, for instance with different source, version or
compilation options.

Version-specific with different source and other arguments.

* Getting help

Sadly some applications can be tough to package.  Sometimes they need a patch to
work with the non-standard filesystem hierarchy enforced by the store.
Sometimes the tests won't run properly.  (They can be skipped but this is not
recommended.)  Other times the resulting package won't be reproducible.

Should you be stuck, unable to figure out how to fix any sort of packaging
issue, don't hesitate to ask the community for help.

See https://guix.info/contact/ for the mailing lists, IRC, etc.

* Conclusion

This tutorial is an introductory showcase of how Guix can bring automation to
the field of packaging.  The GNU build system is the flagship of the high-level
abstraction layers Guix makes possible to build.

Now where do we go from here?  Next we ought to dissect the innards of the build
system by removing all abstractions, using the ~trivial-build-system~: this
should give us a thorough understanding of the process before investigating some
more advanced packaging techniques and edge cases.

Other features worth exploring are the interactive editing and debugging
capabilities of Guix provided by the Guile REPL.

* References

- [[https://gitlab.com/pjotrp/guix-notes/blob/master/HACKING.org][Pjotr’s hacking guide to GNU Guix]]

- "Guix Guix: Package without a scheme!", by Andreas Enge (in
  =guix-maintenance.git/talks/ghm-2013/andreas/slides.pdf=)

* About GNU Guix

[[https://www.gnu.org/software/guix][GNU Guix]] is a transactional package manager for the GNU system.  The Guix System
Distribution or GuixSD is an advanced distribution of the GNU system that relies
on GNU Guix and [[https://www.gnu.org/distros/free-system-distribution-guidelines.html][respects the user's freedom]].

In addition to standard package management features, Guix supports transactional
upgrades and roll-backs, unprivileged package management, per-user profiles, and
garbage collection.  Guix uses low-level mechanisms from the Nix package
manager, except that packages are defined as native [[https://www.gnu.org/software/guile][Guile]] modules, using
extensions to the [[http://schemers.org][Scheme]] language.  GuixSD offers a declarative approach to
operating system configuration management, and is highly customizable and
hackable.

GuixSD can be used on an i686, x86_64 and armv7 machines.  It is also possible
to use Guix on top of an already installed GNU/Linux system, including on
mips64el and aarch64.

[-- Attachment #1.3: Type: text/plain, Size: 57 bytes --]

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]