Blog: Guix packaging tutorial

Blog: Guix packaging tutorial

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1628 bytes --]

Hi!

I'd like to post a blog article which would serve as a packaging
tutorial.

A rough outline:

--8<---------------cut here---------------start------------->8---
0. Introduction
The perks of Guix, what it means to have a full-fledged programming
language at hand for package management (e.g. inheritance, en-masse
modifications).

1. General
Go through the various fields of the `package` record.
GUIX_PACKAGE_PATH, `guix build` and `guix package`.

Where to read about the build systems: Guix source code.

Explain some common pitfalls, e.g. the difference between
`define-public` and the `package` record (this has come up many times on
the mailing list), the Texinfo syntax in descriptions, the linter, the
licenses.

2. Trivial build system
Provide a complete example.

3. GNU build system
- Provide a complete, argument-free example.
- Provide a complete example with arguments such as extra configure/make
flags, some substitution, a snippet, etc.

4. Conclusion
Further reading, more build systems, tips & tricks.
Maybe mention a few complex package declarations
--8<---------------cut here---------------end--------------->8---

What do you think?

I'm mostly wondering whether the GNU build system should be introduced
before the trivial build system: it's simpler and would let the user
have a first package ready within minutes with fewer lines of code to
understand; conversely, it also means that we would first take some
black-box systems for granted ("What the hell happens under the hood?",
might some readers wonder) which would only be explained later.

Opinions?

--
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Blog: Guix packaging tutorial

[Pjotr Prins]

On Thu, Sep 13, 2018 at 12:50:58PM +0200, Pierre Neidhardt wrote:
> Hi!
> 
> I'd like to post a blog article which would serve as a packaging
> tutorial.

Great idea. I have some material on Guix-notes already you may want to
reuse:

https://gitlab.com/pjotrp/guix-notes/blob/master/HACKING.org

There is more in other files in that repo which I mostly add because I
don't trust my memory ;)

Be good to have a document that is part of the main documentation.

Pj.

Re: Blog: Guix packaging tutorial

[Ricardo Wurmus]

Hi Pierre,

> I'd like to post a blog article which would serve as a packaging
> tutorial.

This is an excellent idea!

> --8<---------------cut here---------------start------------->8---
> 0. Introduction
> The perks of Guix, what it means to have a full-fledged programming
> language at hand for package management (e.g. inheritance, en-masse
> modifications).
>
> 1. General
> Go through the various fields of the `package` record.
> GUIX_PACKAGE_PATH, `guix build` and `guix package`.

I have mixed feelings about recommending GUIX_PACKAGE_PATH to new
contributors.  I think that generally it is better to guide them to
working on a git checkout directly, because moving packages from a
separate GUIX_PACKAGE_PATH module to Guix proper isn’t frictionless.

> Where to read about the build systems: Guix source code.

The manual is probably a better place to read about the build systems
(section 4.2 Build systems).

> 2. Trivial build system
> Provide a complete example.

I would skip the trivial build system because it is rather complicated
and may scare readers off.

> 3. GNU build system
> - Provide a complete, argument-free example.
> - Provide a complete example with arguments such as extra configure/make
> flags, some substitution, a snippet, etc.

Sounds good!

> 4. Conclusion
> Further reading, more build systems, tips & tricks.
> Maybe mention a few complex package declarations

Maybe also mention the recursive importers to automate a large part of
the work for packages from Pypi, CRAN, Bioconductor, etc.

> I'm mostly wondering whether the GNU build system should be introduced
> before the trivial build system: it's simpler and would let the user
> have a first package ready within minutes with fewer lines of code to
> understand; conversely, it also means that we would first take some
> black-box systems for granted ("What the hell happens under the hood?",
> might some readers wonder) which would only be explained later.

In my opinion, the power of abstraction over conventional build systems
is very impressive.  Instead of having to write a shell snippet that
does “./configure --prefix=/foo && make && make install” you can lean
back and use “gnu-build-system” instead.

From there it’s easy to talk about packages that *don’t* use
conventional build systems or that otherwise deviate from conventions,
which is one reason for using modify-phases.

Thanks for your initiative!

--
Ricardo

Re: Blog: Guix packaging tutorial

[Pjotr Prins]

On Thu, Sep 13, 2018 at 01:53:16PM +0200, Ricardo Wurmus wrote:
> > 1. General
> > Go through the various fields of the `package` record.
> > GUIX_PACKAGE_PATH, `guix build` and `guix package`.
> 
> I have mixed feelings about recommending GUIX_PACKAGE_PATH to new
> contributors.  I think that generally it is better to guide them to
> working on a git checkout directly, because moving packages from a
> separate GUIX_PACKAGE_PATH module to Guix proper isn’t frictionless.

Yes, the GUIX_PACKAGE_PATH serves a different purpose. Main thing you
want to capture is package definitions and perhaps the workflow of
submitting a patch.

> Maybe also mention the recursive importers to automate a large part of
> the work for packages from Pypi, CRAN, Bioconductor, etc.
> 
> > I'm mostly wondering whether the GNU build system should be introduced
> > before the trivial build system: it's simpler and would let the user
> > have a first package ready within minutes with fewer lines of code to
> > understand; conversely, it also means that we would first take some
> > black-box systems for granted ("What the hell happens under the hood?",
> > might some readers wonder) which would only be explained later.
> 
> In my opinion, the power of abstraction over conventional build systems
> is very impressive.  Instead of having to write a shell snippet that
> does “./configure --prefix=/foo && make && make install” you can lean
> back and use “gnu-build-system” instead.

And it is less specialized. Hard coding configure options is the road
to madness - at least where it comes to targetting multiple outputs.
That is where packaging systems like conda fall short.

> From there it’s easy to talk about packages that *don’t* use
> conventional build systems or that otherwise deviate from conventions,
> which is one reason for using modify-phases.

A second advanced hacking tutorial could discuss more advanced
features.

Pj.

Re: Blog: Guix packaging tutorial

[Andreas Enge]

Hello Pierre,

nice idea! I agree with Ricardo's suggestions.

You could have a look at the slides of a talk I gave at GHM 2013 in the
"maintenance" git repo, inside the talks/ghm-2013/andreas directory.
Some things are outdated (we did not have the macros to manipulate build
phases yet).

There I start actually from an even more basic view point, since the
target audience was people not knowing Scheme at all: first, a few basic
notions of Scheme (quoting and unqoting is needed all the time); then
how to modify an existing package without inputs using gnu-build-system;
then how to handle inputs; and so on.

Andreas

Re: Blog: Guix packaging tutorial

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 2487 bytes --]

> https://gitlab.com/pjotrp/guix-notes/blob/master/HACKING.org

Wow, this is a fantastic document!  Is it mentioned anywhere?  I wish I had
known about this before.  I guess you've done most of the work already.
I'll make sure to mention your work and credit you in the blog article.

I could spot some mistakes:

- it's "%outputs" with and "s", not "%output".

- "–no-substites".

Ricardo has a very good point.  Initially I did not think of orienting the blog
towards contributing to upstream, and I think you are right, it's something very
important we should do, especially considering that Guix is one of the very
distributions that welcomes user contributions regardless of their "status".

On the other hand, I can also see a different population of users, those who
simply don't intend to contribute for whatever reasons (think the "fear of the
first commitment" when newborn hackers send their first patch).  Those users
might get a feeling that Guix is not "hackable for their own purposes" if they
can't define a package for their own private use.

My suggestion is to quickly mention GUIX_PACKAGE_PATH, insist on contributing
upstream and underline the friction that results from moving from
GUIX_PACKAGE_PATH to upstream, as Ricardo pointed out.

Complex packages and trivial build system: I think a tutorial is not so useful
if it only covers the basics that many users can figure out by themselves.  An
advanced tutorial covering the parts that are hard to learn alone would be more
than welcome in my opinion.
In my personal experience, I got started with "inxi" which uses the trivial
build system, and it was _very hard_, even while I could get a grasp of the
gnu-build-system.

Pjotr Prins <pjotr.public12@thebird.nl> writes:
> A second advanced hacking tutorial could discuss more advanced
> features.

I also agree with your point that starting off with complex packaging could
scare people away.  So my suggestion is as follows:

- Part 1: basics + gnu-build-system + importers
- Part 2: trivial-build-system + debug and REPL interaction.

What do you think?

Andreas Enge <andreas@enge.fr> writes:
> You could have a look at the slides of a talk I gave at GHM 2013 in the
> "maintenance" git repo, inside the talks/ghm-2013/andreas directory.
> Some things are outdated (we did not have the macros to manipulate build
> phases yet).

Will have a look, thanks!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Blog: Guix packaging tutorial

[Pjotr Prins]

On Fri, Sep 14, 2018 at 01:07:44PM +0200, Pierre Neidhardt wrote:
> > https://gitlab.com/pjotrp/guix-notes/blob/master/HACKING.org
> 
> Wow, this is a fantastic document!  Is it mentioned anywhere?  I wish I had
> known about this before.  I guess you've done most of the work already.
> I'll make sure to mention your work and credit you in the blog article.

:)

Also check out the DLANG and Ruby docs. They contain some gems. In
particular where it comes to using a guix environment container to fix
build issues (I just pushed that).

> I could spot some mistakes:
> 
> - it's "%outputs" with and "s", not "%output".
> 
> - "–no-substites".

Feel free to send me a patch/PR. There will be more.

> I also agree with your point that starting off with complex packaging could
> scare people away.  So my suggestion is as follows:
> 
> - Part 1: basics + gnu-build-system + importers
> - Part 2: trivial-build-system + debug and REPL interaction.
> 
> What do you think?

Great.

Pj.

Re: Blog: Guix packaging tutorial

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 157 bytes --]

Err... Actually I see that a few packages like "mg" uses %output and not
%outputs.  What's the difference?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Blog: Guix packaging tutorial

[Pierre Neidhardt]

[-- Attachment #1.1: Type: text/plain, Size: 769 bytes --]

Alright, here is a first draft!

It's currently in Org, I'll export it to Markdown later.

It's much longer than I expected.  About 5000 words!  I guess some parts could
go, but I'm not sure which.  Feedback would be very much appreciated here.

Let me know if it's approachable enough and if you find any missing/tangled
parts in the progression.

I would also love feedback about the section titles.  Some could be made more
explicit I suppose.

I'm not satisfied with the advanced example: I took ~mg~ because it has a lot of
typical customization (snippets, inputs, arguments) but it does not have
- in-input sources
- %build-inputs
- propagated-inputs

I've made up a "my-mg" package but it relies on made-up inputs, so it won't
build.
Better suggestions anyone?


[-- Attachment #1.2: index.org --]
[-- Type: text/x-org, Size: 39984 bytes --]

#+TITLE: Guix packaging tutorial
#+AUTHOR: Pierre Neidhardt
#+date: <2018-09-24 Mon>

* Introduction

GNU Guix stands out as the /hackable/ package manager, mostly because it uses
[[https://www.gnu.org/software/guile/][GNU Guile]], a powerful high-level programming language, one of the [[https://en.wikipedia.org/wiki/Scheme_(programming_language)][Scheme]]
dialects from the [[https://en.wikipedia.org/wiki/Lisp_(programming_language)][Lisp family]].

Package definitions are also written in Scheme, which empowers Guix in some very
unique ways, unlike most other package managers that use shell scripts or
simplistic languages.

- Use functions, structures, macros and all of Scheme expressiveness into your
  package definitions.

- Inheritance makes it easy to customize a package by inheriting from it and
  modifying only what is needed.

- Batch processing: the whole package collection can be parsed, filtered and
  processed.  Building a headless server with all graphical interfaces stripped
  out?  It's possible.  Want to rebuild everything from source for a specific
  architecture?  Pass the ~#:system "YOUR-ARCH"~ argument to the list of
  packages.  It wouldn't be a stretch to think [[https://wiki.gentoo.org/wiki/USE_flag][Gentoo USE flags]] here, but this
  goes even further: the changes don't have to be thought out beforehand by the
  packager, they can be /programmed/ by the user!

The following tutorial covers all the basics around package creation with Guix.
It does not assume much knowledge of the Guix system nor of the Lisp language.
The reader is only expected to be familiar with the commandline and to have some
basic programming knowledge.

* A "Hello World" package

=GNU hello= is a dummy project that serves as an idiomatic example for
packaging.  If uses the GNU build chain (~./configure && make && make install~).
Guix already provides a package definition which is a perfect example to start
with.  You can look up its declaration with ~guix edit hello~ from the
commandline.  Let's see how it looks:

#+BEGIN_SRC scheme
(define-public hello
  (package
    (name "hello")
    (version "2.10")
    (source (origin
              (method url-fetch)
              (uri (string-append "mirror://gnu/hello/hello-" version
                                  ".tar.gz"))
              (sha256
               (base32
                "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
    (build-system gnu-build-system)
    (synopsis "Hello, GNU world: An example GNU package")
    (description
     "GNU Hello prints the message \"Hello, world!\" and then exits.  It
serves as an example of standard GNU coding practices.  As such, it supports
command-line arguments, multiple languages, and so on.")
    (home-page "https://www.gnu.org/software/hello/")
    (license gpl3+)))
#+END_SRC

As you can see, most of it is rather straightforward.  But let's review the
fields together:

- name :: The project name.  Using Scheme conventions, we prefer to keep it
          lower case, without underscore and using dash-separated words.
- source :: The ~origin~ record gather all the information required for a
            source.  Among which:
  1. The method, here ~url-fetch~ to download via HTTP/FTP, but other methods
     exist, such as ~git-fetch~ for Git repositories.
  2. The URI, which is typically some =https://= location for ~url-fetch~.  Here
     the special =mirror://gnu= refers to a set of well known locations, all of
     which can be used by Guix to fetch the source, should some of them fail.
  3. The ~sha256~ checksum of the requested file.  This is essential to ensure
     the source is not corrupted.  Note that Guix works with base32 strings,
     hence the call to the ~base32~ function.
- build-system :: This is where the power of abstraction provided by the Scheme
                  language really shine: in this case, the ~gnu-build-system~
                  abstracts away the famous ~./configure && make && make
                  install~ shell invocations: If the package can be built just
                  those commands, then Guix can often handle the complete
                  packaging process automatically.  Other build systems include
                  the ~trivial-build-system~ which does not do anything and
                  leaves all actions to the packager, the ~python-build-system~,
                  the ~emacs-build-system~, and many more.
- synopsis :: It should be a concise and explicit summary of what the package
              does.  Many projects homepage already provide a synopsis, it's
              fine to re-use it.
- description :: Same as for the synopsis, it's fine to re-use the project
                 description from the homepage.  Note that Guix uses Texinfo
                 syntax.
- home-page :: Use HTTPS if available.
- license :: See =$GUIX_CHECKOUT/guix/licenses.scm= for a full list.

Time to build our first package!  Nothing fancy here for now: we will stick to a
dummy "my-hello", a copycat of the above declaration.

As with the ritualistic "Hello World" taught with most programming language,
this will possibly be the most "manual" approach.  We will work out an ideal
setup later; for now we will go the simplest route.

Save the following to a file =my-hello.scm=.

#+BEGIN_SRC scheme
(use-modules (guix packages)
             (guix download)
             (guix build-system gnu)
             (guix licenses))

(define-public my-hello
  (package
    (name "my-hello")
    (version "2.10")
    (source (origin
              (method url-fetch)
              (uri (string-append "mirror://gnu/hello/hello-" version
                                  ".tar.gz"))
              (sha256
               (base32
                "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
    (build-system gnu-build-system)
    (synopsis "Hello, Guix world: An example custom Guix package")
    (description
     "GNU Hello prints the message \"Hello, world!\" and then exits.  It
serves as an example of standard GNU coding practices.  As such, it supports
command-line arguments, multiple languages, and so on.")
    (home-page "https://www.gnu.org/software/hello/")
    (license gpl3+)))

my-hello
#+END_SRC

We will explain the extra code in a moment.

Feel free to play with the different values of the various fields.  If you
change the source, you'll need to update the checksum.
Indeed, Guix refuses to build anything if the source fails to verify the checksum.
To compute the correct checksum of the package declaration, we need to download the
source, compute the sha256 checksum and convert it to base32.

Thankfully Guix can automate this task for us, all we need is to provide the URI:

#+BEGIN_SRC sh
$ guix download mirror://gnu/hello/hello-2.10.tar.gz

Starting download of /tmp/guix-file.JLYgL7
From https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz...
following redirection to `https://mirror.ibcp.fr/pub/gnu/hello/hello-2.10.tar.gz'...
 …10.tar.gz  709KiB                                 2.5MiB/s 00:00 [##################] 100.0%
/gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i
#+END_SRC

Note in this specific case that the output tells us which mirror was chosen.

If the result of the above command is not the same as in the above snippet,
update you =my-hello= declaration accordingly.

Now you can happily run

#+BEGIN_SRC sh
$ guix package --install-from-file=my-hello.scm
#+END_SRC

You should now have =my-hello= in your profile!

#+BEGIN_SRC sh
$ guix package --list-installed=my-hello
my-hello	2.10	out	/gnu/store/f1db2mfm8syb8qvc357c53slbvf1g9m9-my-hello-2.10
#+END_SRC

We've gone as far as we could without any knowledge of Scheme.  Now is the right
time to introduce the minimum we need of the language before we can proceed.

* A Scheme crash-course

As we've seen above, basic packages don't require much Scheme knowledge, if none
at all.  But as you'll progress and your desire to write more and more complex
packages will grow, it will become both necessary and empowering to hone your
Lisper skills.

Since this is very much out of the scope of this tutorial, we will only cover
some basics here.

Guix uses the Guile implementation of Scheme.  To start playing with the
language, install it with ~guix package --install guile~ and start a [[https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop][REPL]] by
running ~guile~ from the commandline.

Alternatively you can also run ~guix environment --ad-hoc guile -- guile~ if
you'd rather not have Guile installed in your user profile.

- Scheme syntax boils down to a tree of expressions (or /s-expression/ in Lisp
  lingo) It can be a literal such numbers and strings, or a compound which is a
  parenthesized list of compounds and literals.
  ~#t~ and ~#f~ stand for the booleans "true" and "false", respectively.

  Examples of valid expressions:
  #+BEGIN_SRC scheme
scheme@(guile-user)> "Hello World!"
$1 = "Hello World!"
scheme@(guile-user)> 17
$2 = 17
scheme@(guile-user)> (format #f "Hello ~a!" "Guix")
$3 = "Hello Guix!"
  #+END_SRC

- This last example is a function call.  When a parenthesized expression is
  evaluated, the first term is the function and the rest are the arguments
  passed to the function.  Every function returns the last
  evaluated expression as value.

- Anonymous functions are declared with the ~lambda~ term:
  #+BEGIN_SRC scheme
scheme@(guile-user)> (lambda (x) (* x x))
$4 = #<procedure 120e348 at <unknown port>:24:0 (x)>
  #+END_SRC
  The above lambda returns the square of its argument.  Since everything is an
  expression, the ~lambda~ expression returns an anonymous function, which can
  in turn be called over an argument:
  #+BEGIN_SRC scheme
scheme@(guile-user)> ((lambda (x) (* x x)) 3)
$5 = 9
  #+END_SRC

- Anything can be assigned a global name with ~define~:
  #+BEGIN_SRC scheme
scheme@(guile-user)> (define a 4)
scheme@(guile-user)> (define square (lambda (x) (* x x)))
scheme@(guile-user)> (square a)
$6 = 9
  #+END_SRC

- Procedures can be defined more concisely with the following syntax:
  #+BEGIN_SRC scheme
  (define (square x) (* x x))
  #+END_SRC

- A list structure can be created with the ~list~ procedure:
  #+BEGIN_SRC scheme
scheme@(guile-user)> (list 2 a 5 7)
$7 = (2 3 5 7)
  #+END_SRC

- The /quote/ disables evaluation of a parenthesized expression: the first term
  is not called over the other terms.  Thus it effectively returns a list of
  terms.
  #+BEGIN_SRC scheme
scheme@(guile-user)> '(format #f "Hello ~a!" "Guix")
$8 = (format #f "Hello ~a!" "Guix")
scheme@(guile-user)> '(2 a 5 7)
$9 = (2 a 5 7)
  #+END_SRC

- The /quasiquote/ disables evaluation of a parenthesized expression until a
  colon re-enables it.  Thus it enables us with fine-grained control over what
  is evaluated and what is not.
  #+BEGIN_SRC scheme
scheme@(guile-user)> `(2 a 5 7 (2 ,a 5 ,(+ a 4)))
$10 = (2 a 5 7 (2 3 5 7))
  #+END_SRC
  Note that the above result is a list of mixed elements: numbers, symbols (here
  ~a~) and the last element is a list itself.

- Multiple variables can be named locally with ~let~:
  #+BEGIN_SRC scheme
scheme@(guile-user)> (define x 10)
scheme@(guile-user)> (let ((x 2)
                           (y 3))
                       (list x y))
$11 = (2 3)
scheme@(guile-user)> x
$12 = 10
scheme@(guile-user)> y
ERROR: In procedure module-lookup: Unbound variable: y
  #+END_SRC
  Use ~let*~ to re-allows the initializers of later variables to refer to the
  earlier variables.
  #+BEGIN_SRC scheme
scheme@(guile-user)> (let* ((x 2)
                            (y (* x 3)))
                       (list x y))
$13 = (2 6)
  #+END_SRC

- The keyword syntax is ~#:~, it is used to create unique identifiers.  See also
  http://practical-scheme.net/wiliki/schemexref.cgi?keyword%3F.

- The percentage ~%~ is conventionally used as a prefix for variables generated
  in scope.  Note that it is merely a convention, like ~_~ in C.  Scheme Lisp
  treats ~%~ exactly the same as any other letter.

- Modules are created with ~define-module~.  For instance
  #+BEGIN_SRC scheme
  (define-module (guix build-system ruby)
    #:use-module (guix store)
    #:export (ruby-build
              ruby-build-system))
  #+END_SRC
  defines the module ~ruby~ which must be located in
  ~guix/build-system/ruby.scm~ somewhere in =GUILE_LOAD_PATH=.  It depends on
  the ~(guix store)~ module and it exports two symbols, ~ruby-build~ and
  ~ruby-build-system~.

For a more detailed introduction, check out /[[http://www.troubleshooters.com/codecorn/scheme_guile/hello.htm][Scheme at a Glance]]/, by Steve Litt.

One of the reference Scheme books is the seminal /Structure and Interpretation
of Computer Programs/, by Harold Abelson and Gerald Jay Sussman, with Julie
Sussman.  You'll find a free copy [[https://mitpress.mit.edu/sites/default/files/sicp/index.html][online]], together with [[https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-001-structure-and-interpretation-of-computer-programs-spring-2005/video-lectures/][videos of the lectures
by the authors]].  The book is available in Texinfo format as the =sicp= Guix
package.  Go ahead, run ~guix package --install sicp~ and start reading with
~info sicp~ (or with the Emacs Info reader).  An unofficial ebook [[https://sarabander.github.io/sicp/][is also
available]].

You'll find more books, tutorials and other resources at https://schemers.org/.

* Setup

Now that we know some Scheme basics we can detail the different possible setups
for working on Guix packages.

There are several ways to set up a Guix packaging environment.

We recommend you work directly on the Guix source checkout since it makes it
easier for everyone to contribute to the project.

But first, let's look at other possibilities.

*** Local file

This is what we previously did with =my-hello=.  Now that we know more Scheme,
let's explain the leading and trailing chunks.  As stated in ~guix package
--help~:

#+BEGIN_SRC sh
  -f, --install-from-file=FILE
                         install the package that the code within FILE
                         evaluates to
#+END_SRC

This explains why we had to add ~my-hello~ as a last value: we need to return a
package.
Indeed, if you remove that line, ~guix package --install-from-file=my-hello.scm~
will fail because the last expression, ~define-public~, does not return a package.

The ~use-modules~ expression tells which of the modules we need in the file.
Modules are a collection of values and procedures.  They are commonly called
"libraries" or "packages" in other programming languages.

*** GUIX_PACKAGE_PATH

It can be tedious to specify the file from the commandline instead of simply
calling ~guix package --install my-hello~ as you would do with the official
packages.

Guix makes it possible to streamline the process by adding as many "package
declaration paths" as you want.

Create a folder, say =~./guix-packages= and add it to the GUIX_PACKAGE_PATH
environment variable:

#+BEGIN_SRC sh
$ mkdir ~/guix-packages
$ export GUIX_PACKAGE_PATH=~/guix-packages
#+END_SRC

To add several folders, separate them with a colon (~:~).

Our previous =my-hello= needs some adjustments though:

#+BEGIN_SRC scheme
(define-module (my-hello)
  #:use-module (guix licenses)
  #:use-module (guix packages)
  #:use-module (guix build-system gnu)
  #:use-module (guix download))

(define-public my-hello
  (package
    (name "my-hello")
    (version "2.10")
    (source (origin
              (method url-fetch)
              (uri (string-append "mirror://gnu/hello/hello-" version
                                  ".tar.gz"))
              (sha256
               (base32
                "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
    (build-system gnu-build-system)
    (synopsis "Hello, Guix world: An example custom Guix package")
    (description
     "GNU Hello prints the message \"Hello, world!\" and then exits.  It
serves as an example of standard GNU coding practices.  As such, it supports
command-line arguments, multiple languages, and so on.")
    (home-page "https://www.gnu.org/software/hello/")
    (license gpl3+)))
#+END_SRC

Now =my-hello= should be part of the package collection like all other official
packages.  You can verify this with:

#+BEGIN_SRC sh
$ guix package --show=my-hello
#+END_SRC

*** Direct checkout hacking

Working directly on the Guix project is recommended: it reduces the friction
when the time comes to submit your changes upstream to let the community benefit
from your hard work!

Check out the official [[https://git-scm.com/][Git]] repository:

#+BEGIN_SRC sh
$ git clone https://git.savannah.gnu.org/git/guix.git
#+END_SRC

Follow the instruction from the [[https://www.gnu.org/software/guix/manual/en/html_node/Contributing.html]["Contributing" chapter]] in the manual to set up the
repository environment.

Once ready, you should be able to use the package definitions from the
repository environment.

Feel free to edit package definitions found in =$GUIX_CHECKOUT/gnu/packages=.

The =$GUIX_CHECKOUT/pre-inst-env= script lets you use =guix= over the repository
package collection.

- Search packages:

  #+BEGIN_SRC sh
  $ ./pre-inst-env guix package --list-available=ruby
      ruby    1.8.7-p374      out     gnu/packages/ruby.scm:119:2
      ruby    2.1.6   out     gnu/packages/ruby.scm:91:2
      ruby    2.2.2   out     gnu/packages/ruby.scm:39:2
  #+END_SRC

- Build a package:

  #+BEGIN_SRC sh
  $ ./pre-inst-env guix build --keep-failed ruby@2.1
    /gnu/store/c13v73jxmj2nir2xjqaz5259zywsa9zi-ruby-2.1.6
  #+END_SRC

- Install it to your user profile:

  #+BEGIN_SRC sh
  $ ./pre-inst-env guix package --install ruby@2.1
  #+END_SRC

- Check for common mistakes:

  #+BEGIN_SRC sh
  $ ./pre-inst-env guix lint ruby@2.1
  #+END_SRC

Once you are happy with the result, you are welcome to send your contribution so
that make it part of Guix.  This process is also detailed in the [[https://www.gnu.org/software/guix/manual/en/html_node/Contributing.html][manual]].

It's a community effort so the more join in, the better Guix becomes!

* Extended example

The first example was as simple as it goes.  Packages can be more complex than
that and Guix can handle more advanced scenarios.
Let's looks at another, more sophisticated package (slightly modified from the
source):

#+BEGIN_SRC scheme
  (define-public my-mg
    (package
      (name "my-mg")
      (version "20180408")
      (source (origin
                (method git-fetch)
                (uri (git-reference
                      (url "https://github.com/hboetes/mg")
                      (commit version)))
                (file-name (git-file-name name version))
                (sha256
                 (base32
                  "06w86xk7sjl2x2h3z6msn8kpmwj05qdimcym77wzhz5s94dzh1bl"))
                (modules '((guix build utils)))
                (snippet '(begin
                            (substitute* "GNUmakefile"
                              (("/usr/bin/") ""))
                            #t))))
      (build-system gnu-build-system)
      (native-inputs
       `(("pkg-config" ,pkg-config)))
      (inputs
       `(("libbsd" ,libbsd)
         ("ncurses" ,ncurses)))
      (propagated-inputs
       `(("mg-extensions"
          ,(origin
             (method url-fetch)
             (uri "https://example.org/~doe/mg-extensions.tar.gz"
             (sha256
              (base32
               "0kfhpj5rnh24hz2714qhfmxk281vwc2w50sm73ggw5d15af7zfsw"))))))
      (outputs '("out" "doc"))
      (arguments
       '(#:tests? #f ; No test suite available.
         #:make-flags (list (string-append "prefix=" %output)
                            "CC=gcc")
         #:configure-flags `(,(string-append "--with-ext="
                                             (assoc-ref %build-inputs "mg-extensions")))
         #:phases (modify-phases %standard-phases
                    (delete 'configure)   ; no configure script
                    (add-before 'build 'correct-location-of-difftool
                      (lambda _
                        (substitute* "buffer.c"
                          (("/usr/bin/diff")
                           (which "diff")))
                        #t))
                    (add-before 'install 'patch-tutorial-location
                      (lambda* (#:key outputs #:allow-other-keys)
                        (substitute* "mg.1"
                          (("/usr") (assoc-ref outputs "out")))
                        #t))
                    (add-after 'install 'install-tutorial
                      (lambda* (#:key outputs #:allow-other-keys)
                        (let* ((doc-root (assoc-ref outputs "doc"))
                               (doc (string-append out-root "/share/doc/mg")))
                          (install-file "tutorial" doc)
                          #t))))))
      (home-page "https://homepage.boetes.org/software/mg/")
      (synopsis "Microscopic GNU Emacs clone with custom extensions")
      (description
       "Mg (mg) is a GNU Emacs style editor, with which it is \"broadly\"
  compatible.  This is a portable version of the mg maintained by the OpenBSD
  team.
  This package adds some custom extensions to the original.")
      (license license:public-domain)))
#+END_SRC

(In those cases were you only want to tweak a few fields from a package
definition, you should rely on inheritance instead of copy-pasting everything.
See below.)

Let's discuss those fields in depth.

** ~git-fetch~ method

Unlike the ~url-fetch~ method, ~git-fetch~ expects a ~git-reference~ which takes
a Git repository and a commit.  The commit can be any Git reference such as
tags, so if the ~version~ is tagged, then it can be used directly.  Sometimes
the tag is prefixed with a ~v~, in which case you'd use ~(commit (string-append
"v" version))~.

When downloaded to the store, the source should be different from version to
version.  Since Git repositories don't contain a specific version in their name,
we've got to force the download folder using ~(file-name (git-file-name name
version))~.

Note that there is also a ~git-version~ procedure that can be used to derive the
version when packaging programs for a specific commit.

** Snippets

Snippets are quoted (i.e. non-evaluated) Scheme code that are a mean of patching
the source.  They are a Guix-y alternative to the traditional =.patch= files.
Because of the quote, the code in only evaluated when passed to the Guix daemon
for building.

There can be as many snippet as needed.

Snippets might need additional Guile modules which can be imported from the
~modules~ field.

** Inputs

First, a syntactic comment: See the quasi-quote / comma syntax?

#+BEGIN_SRC scheme
    (native-inputs
     `(("pkg-config" ,pkg-config)))
#+END_SRC

is equivalent to

#+BEGIN_SRC scheme
    (native-inputs
     (list "pkg-config" pkg-config))
#+END_SRC

You'll mostly see the former because it's shorter.

There are 3 different input types.  In short:

- native-inputs :: Required for building but not runtime -- installing a package
                   through a substitute won't install these inputs.
- inputs :: Installed in the store but not in the profile, as well as being
            present at build time.
- propagated-inputs :: Installed in the store and in the profile, as well as
     being present at build time.

The distinction between the various inputs is important: if a dependency can be
handled as an /input/ instead of a /propagated input/, it should be done so, or
else it "pollutes" the user profile for no reason.

For instance, a user installing a graphical program that depends on a
commandline tool might only be interested in the graphical part, so there is no
need to force the commandline tool into the user profile.  The dependency is a
concern to the package, no to the user.  /Inputs/ make it possible to handle
dependencies without bugging the using by adding undesired executable files (or
libraries) to their profile.

Same goes for /native-inputs/: once the program is installed, build-time
dependencies can be safely garbage collected.
It also matters when a substitute is available, in which case only the /inputs/
and /propagated inputs/ will be fetched: the /native inputs/ are not required to
install a package from a substitute.

** Build system arguments

The ~arguments~ is a keyword-value list used to configure the build process.

The simplest argument ~#:tests?~ can be used to disable the test suite when
building the package.  This is mostly useful when the package does not feature
any test suite.  It's strongly recommended to keep the test suite on if there is
one.

Another  common argument is ~:make-flags~, which specifies a list of flags to
append when running make, as you would from the commandline:

#+BEGIN_SRC sh
$ make FLAGS...

#+END_SRC

In the above package, the argument

#+BEGIN_SRC scheme
         #:make-flags (list (string-append "prefix=" %output)
                            "CC=gcc")
#+END_SRC

sets the C compiler to ~gcc~ and the ~prefix~ variable (the installation folder
in Make parlance) to ~%output~, which is a variable generated in scope and
pointing to the destination folder in the store (something like
=/GNU/store/...-my-mg-20180408=).

Similarly, it's possible to set the "configure" flags.  In the above example,
we set the extensions to the store download folder of our custom mg-extensions.

#+BEGIN_SRC scheme
         #:configure-flags `(,(string-append "--with-ext="
                                             (assoc-ref %build-inputs "mg-extensions")))
#+END_SRC

The ~%build-inputs~ variable is also generated in scope.  It's an associated
table that maps the input names to their store folders.

The ~phases~ keyword lists the sequential steps of the build system.  Typically
phases include ~unpack~, ~configure~, ~build~, ~install~ and ~check~.  To know
more about those phases, you need to work out the appropriate build system
definition in =$GUIX_CHECKOUT/guix/build/gnu-build-system.scm=:

#+BEGIN_SRC scheme
(define %standard-phases
  ;; Standard build phases, as a list of symbol/procedure pairs.
  (let-syntax ((phases (syntax-rules ()
                         ((_ p ...) `((p . ,p) ...)))))
    (phases set-SOURCE-DATE-EPOCH set-paths install-locale unpack
            bootstrap
            patch-usr-bin-file
            patch-source-shebangs configure patch-generated-file-shebangs
            build check install
            patch-shebangs strip
            validate-runpath
            validate-documentation-location
            delete-info-dir-file
            patch-dot-desktop-files
            install-license-files
            reset-gzip-timestamps
            compress-documentation)))
#+END_SRC

Or from the REPL (assuming the Guix source is in your Guile load path):

#+BEGIN_SRC scheme
scheme@(guile-user)> ,module (guix build gnu-build-system)
scheme@(guix build gnu-build-system)> (map first %standard-phases)
$1 = (set-SOURCE-DATE-EPOCH set-paths install-locale unpack bootstrap patch-usr-bin-file patch-source-shebangs configure patch-generated-file-shebangs build check install patch-shebangs strip validate-runpath validate-documentation-location delete-info-dir-file patch-dot-desktop-files install-license-files reset-gzip-timestamps compress-documentation)
#+END_SRC

If you want to know more about what happens during those phases, consult the
associated functions.

For instance, as of this writing the definition of ~unpack~ for the GNU build
system is

#+BEGIN_SRC scheme
(define* (unpack #:key source #:allow-other-keys)
  "Unpack SOURCE in the working directory, and change directory within the
source.  When SOURCE is a directory, copy it in a sub-directory of the current
working directory."
  (if (file-is-directory? source)
      (begin
        (mkdir "source")
        (chdir "source")

        ;; Preserve timestamps (set to the Epoch) on the copied tree so that
        ;; things work deterministically.
        (copy-recursively source "."
                          #:keep-mtime? #t))
      (begin
        (if (string-suffix? ".zip" source)
            (invoke "unzip" source)
            (invoke "tar" "xvf" source))
        (chdir (first-subdirectory "."))))
  #t)
#+END_SRC

Note the ~chdir~ call: it changes the working directory to where the source was
unpacked.
Thus every phase following the ~unpack~ will use the source as a working
directory, which is why we can directly work on the source files.
That is to say, unless a later phase changes the working directory to something
else.

To manipulate the phases,

- ~add-before PHASE NEW-PHASE PROCEDURE~: Run ~PROCEDURE~ named ~NEW-PHASE~ before ~PHASE~.
- ~add-after PHASE NEW-PHASE PROCEDURE~: Same, but afterwards.
- ~delete PHASE~.

The ~PROCEDURE~ supports the keyword arguments ~inputs~ and ~outputs~.  Each
input (whether /native/, /propagated/ or not) and output directory is referenced
by their name in those variables.  Thus ~(assoc-ref outputs "out")~ is the store
directory of the main output of the package.  A phase procedure may look like
this:

#+BEGIN_SRC scheme
(lambda* (#:key inputs outputs #:allow-other-keys)
  (let (((bash-folder (assoc-ref inputs "bash"))
         (output-folder (assoc-ref outputs "out"))
         (doc-folder (assoc-ref outputs "doc"))
  ; ...
  #t)
#+END_SRC

The procedure must return ~#t~ on success.  It's brittle to rely on the return
value of the last expression used to tweak the phase because there is no
guarantee it would be a ~#t~.  Hence the trailing ~#t~ to ensure the right value
is returned on success.

** Code staging

The astute reader may have noticed the quasi-quote and comma syntax in the
argument field.  Indeed, the build code in the package declaration should not be
evaluated on the client side, but only when passed to the Guix daemon.  This
mechanism of passing code around two running processes is called [[https://arxiv.org/abs/1709.00833][code staging]].
See [[https://www.gnu.org/software/guix/manual/en/html_node/G_002dExpressions.html][the "G-Expressions" chapter]] from the manual.

** "Utils" functions

When customizing ~phases~, we often need to write code that mimics the
equivalent system invocations (~make~, ~mkdir~, ~cp~, etc.) commonly used during
regular "Unix-style" installations.

Some like ~chmod~ are native to Guile.  See the [[https://www.gnu.org/software/guile/manual/guile.html][Guile reference manual]] for a
complete list.

Guix provides additional helper functions which prove especially handy in the
context of package management.

Some of those functions can be found in
=$GUIX_CHECKOUT/guix/guix/build/utils.scm=.  Most of them mirror the behaviour
of the traditional Unix system commands:

- which :: Like the =which= system command.
- find-files :: Akin to the =find= system command.
- mkdir-p :: Like =mkdir -p=, which creates all parents as needed.
- install-file :: Similar to =install= when installing a file to a (possibly
                  non-existing) directory.  Guile has ~copy-file~ which works
                  like =cp=.
- copy-recursively :: Like =cp -r=.
- delete-file-recursively :: Like =rm -rf=.
- invoke :: This should be used instead of ~system*~.
- with-directory-excursion :: Run the body in a different working directory,
     then restore the previous working directory.
- substitute* :: A "sed-like" function.

** Module prefix

The license now needs a prefix: this is because of how the ~license~ module was
important in the package, as ~#:use-module ((guix licenses) #:prefix
license:)~.  This gives the user full control over namespacing.

* Other build systems

What we've seen so far covers the majority of packages using a build system
other than the ~trivial-build-system~.  The latter does not automate anything
and leaves you to build everything manually.  This can be more demanding and we
won't cover it here for now, but thankfully it is rarely necessary to fall back
on this system.

For the other build systems, such as ASDF, Emacs, Perl, Ruby and many more, the
process is very similar to the GNU build system but for a few specialized
arguments.

Find more about build systems in
- [[https://www.gnu.org/software/guix/manual/en/html_node/Build-Systems.html#Build-Systems][the manual, section 4.2 Build systems]],
- the source code in the =$GUIX_CHECKOUT/guix/build= and
  =$GUIX_CHECKOUT/guix/build-system= folders.

* Common pitfalls and best practices

The following check list is important to maintain a high packaging standard, in
particular when contributing to the Guix project.

- Use the /linter/: ~guix lint $PACKAGES~.
- Use mirrors when possible in the source URL.
- Use reliable URLs, not generated ones (e.g. GitHub archives).
- Sort inputs alphabetically.
- Synopsis should be as concise as possible, don't start with "The" or "A", don't repeat
  the package name.
- Description uses Texinfo syntax with two spaces at the end of sentences.
- Reproducibility: use ~guix build --check~ and ~guix build --rounds=N~.
- Follow the [[https://www.gnu.org/software/guix/manual/en/html_node/Coding-Style.html][coding style]] from the manual.
- Review the [[https://www.gnu.org/software/guix/manual/en/html_node/Submitting-Patches.html][check list]] from the manual.

* Programmable and automated package definition

We can't repeat it enough: having a full-fledged programming language at hand
empowers us in ways that reach far beyond traditional package management.

Let's illustrate this with some awesome features of Guix!

** Recursive importers

You might find some build systems good enough that there is little to do at all
to write a package, to the point that it becomes repetitive and tedious after a
while.

Humans should not operate tedious repetitive taks manuall, computers are meant
to automate that job!  Let's tell Guix to create the package definition of an
ELPA package:

#+BEGIN_SRC sh
$ guix import elpa exwm

(package
  (name "emacs-exwm")
  (version "0.19")
  (source
    (origin
      (method url-fetch)
      (uri (string-append
             "https://elpa.gnu.org/packages/exwm-"
             version
             ".tar"))
      (sha256
        (base32
          "11xd2w4h3zdwkdxypvmcz8s7q72cn76lfr9js77jbizyj6b04lr0"))))
  (build-system emacs-build-system)
  (propagated-inputs `(("emacs-xelb" ,emacs-xelb)))
  (home-page "https://github.com/ch11ng/exwm")
  (synopsis "Emacs X Window Manager")
  (description
    "EXWM (Emacs X Window Manager) is a full-featured tiling X window manager
for Emacs built on top of [XELB](https://github.com/ch11ng/xelb).
It features:
+ Fully keyboard-driven operations
+ Hybrid layout modes (tiling & stacking)
+ Dynamic workspace support
+ ICCCM/EWMH compliance
+ (Optional) RandR (multi-monitor) support
+ (Optional) Built-in system tray")
  (license license:gpl3+))
#+END_SRC

Not all applications can be packaged this way, on those relying on a select
number of systems.  Read about the full list of importers in the [[https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-import.html][guix import
section]] of the manual.

** Automatic update

Guix can be smart enough to check for updates on systems it knows.  It can
report outdated package definition with

#+BEGIN_SRC sh
$ guix refresh hello
#+END_SRC

In most cases, updating a package to a newer version requires little more than
changing the version number and the checksum.  Guix can do that automatically as
well:

#+BEGIN_SRC
$ guix refresh hello --update
#+END_SRC

** Inheritance

If you've started browsing the existing package definitions, you might have
noticed that a significant number of them have a ~inherit~ field:

#+BEGIN_SRC scheme
(define-public adwaita-icon-theme
  (package (inherit gnome-icon-theme)
    (name "adwaita-icon-theme")
    (version "3.26.1")
    (source (origin
              (method url-fetch)
              (uri (string-append "mirror://gnome/sources/" name "/"
                                  (version-major+minor version) "/"
                                  name "-" version ".tar.xz"))
              (sha256
               (base32
                "17fpahgh5dyckgz7rwqvzgnhx53cx9kr2xw0szprc6bnqy977fi8"))))
    (native-inputs
     `(("gtk-encode-symbolic-svg" ,gtk+ "bin")))))
#+END_SRC

All unspecified fields inherit from the parent package.  This is very convenient
to create alternative packages, for instance with different source, version or
compilation options.

Version-specific with different source and other arguments.

* Getting help

Sadly some applications can be tough to package.  Sometimes they need a patch to
work with the non-standard filesystem hierarchy enforced by the store.
Sometimes the tests won't run properly.  (They can be skipped but this is not
recommended.)  Other times the resulting package won't be reproducible.

Should you be stuck, unable to figure out how to fix any sort of packaging
issue, don't hesitate to ask the community for help.

See https://guix.info/contact/ for the mailing lists, IRC, etc.

* Conclusion

This tutorial is an introductory showcase of how Guix can bring automation to
the field of packaging.  The GNU build system is the flagship of the high-level
abstraction layers Guix makes possible to build.

Now where do we go from here?  Next we ought to dissect the innards of the build
system by removing all abstractions, using the ~trivial-build-system~: this
should give us a thorough understanding of the process before investigating some
more advanced packaging techniques and edge cases.

Other features worth exploring are the interactive editing and debugging
capabilities of Guix provided by the Guile REPL.

* References

- [[https://gitlab.com/pjotrp/guix-notes/blob/master/HACKING.org][Pjotr’s hacking guide to GNU Guix]]

- "Guix Guix: Package without a scheme!", by Andreas Enge (in
  =guix-maintenance.git/talks/ghm-2013/andreas/slides.pdf=)

* About GNU Guix

[[https://www.gnu.org/software/guix][GNU Guix]] is a transactional package manager for the GNU system.  The Guix System
Distribution or GuixSD is an advanced distribution of the GNU system that relies
on GNU Guix and [[https://www.gnu.org/distros/free-system-distribution-guidelines.html][respects the user's freedom]].

In addition to standard package management features, Guix supports transactional
upgrades and roll-backs, unprivileged package management, per-user profiles, and
garbage collection.  Guix uses low-level mechanisms from the Nix package
manager, except that packages are defined as native [[https://www.gnu.org/software/guile][Guile]] modules, using
extensions to the [[http://schemers.org][Scheme]] language.  GuixSD offers a declarative approach to
operating system configuration management, and is highly customizable and
hackable.

GuixSD can be used on an i686, x86_64 and armv7 machines.  It is also possible
to use Guix on top of an already installed GNU/Linux system, including on
mips64el and aarch64.

[-- Attachment #1.3: Type: text/plain, Size: 57 bytes --]

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Blog: Guix packaging tutorial

[Ludovic Courtès]

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> Err... Actually I see that a few packages like "mg" uses %output and not
> %outputs.  What's the difference?

‘%outputs’ is the recommended way, ‘%output’ is not.  :-)

Ludo’.

Re: Blog: Guix packaging tutorial

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 136 bytes --]

My understanding it that

  %output = (assoc-ref "out" %outputs)

Is this correct?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Blog: Guix packaging tutorial

[Ludovic Courtès]

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> My understanding it that
>
>   %output = (assoc-ref "out" %outputs)
>
> Is this correct?

Yes!

Ludo’.

Re: Blog: Guix packaging tutorial

[Ludovic Courtès]

Hi Pierre,

Thanks a lot for writing this tutorial!  I think it’ll be very useful to
many.

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> Let me know if it's approachable enough and if you find any missing/tangled
> parts in the progression.

I’m not in a good position to say if it’s approachable I’m afraid.  :-)
It would be nice to get feedback from newcomers or users of other
packaging tools.

It’s a bit long indeed, but I think it covers lots of useful things,
going from simple packages to more complex things.  So overall I think
it’s very good and nicely complements our doc!

Some inline comments below, but nothing crucial.

> #+TITLE: Guix packaging tutorial
> #+AUTHOR: Pierre Neidhardt
> #+date: <2018-09-24 Mon>

[...]

> Package definitions are also written in Scheme, which empowers Guix in some very

Perhaps link to
<https://www.gnu.org/software/guix/manual/en/html_node/Defining-Packages.html>.
This part overlaps with the “Defining Packages” section, but it probably
doesn’t hurt.

> unique ways, unlike most other package managers that use shell scripts or
> simplistic languages.

s/simplistic/simple/ (let’s not be judgmental)

> - build-system :: This is where the power of abstraction provided by the Scheme
>                   language really shine: in this case, the ~gnu-build-system~

s/shine/shines/

>                   the ~emacs-build-system~, and many more.

[[https://www.gnu.org/software/guix/manual/en/html_node/Build-Systems.html][and many more]]

> Guix uses the Guile implementation of Scheme.  To start playing with the
> language, install it with ~guix package --install guile~ and start a [[https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop][REPL]] by
> running ~guile~ from the commandline.

s/commandline/command line/

Maybe link to
<https://www.gnu.org/software/guile/manual/html_node/Using-Guile-Interactively.html>
for the REPL.

> - The keyword syntax is ~#:~, it is used to create unique identifiers.  See also
>   http://practical-scheme.net/wiliki/schemexref.cgi?keyword%3F.

Rather link to
<https://www.gnu.org/software/guile/manual/html_node/Keywords.html> as
each Scheme has its own flavor of keywords…

> - The percentage ~%~ is conventionally used as a prefix for variables generated
>   in scope.

s/conventionally.*/typically used for read-only global variables/

(it’s a disputed convention :-))

> *** GUIX_PACKAGE_PATH

You’ll have to mention that the upcoming Guix version features a new
mechanism, “channels”, which provides better integration and provenance
tracking than ‘GUIX_PACKAGE_PATH’.  :-)

> *** Direct checkout hacking
>
> Working directly on the Guix project is recommended: it reduces the friction
> when the time comes to submit your changes upstream to let the community benefit
> from your hard work!
>
> Check out the official [[https://git-scm.com/][Git]] repository:
>
> #+BEGIN_SRC sh
> $ git clone https://git.savannah.gnu.org/git/guix.git
> #+END_SRC

Maybe we should mention that the repo contains both
Guix-the-tools/libraries and the package definitions?  This often comes
as a surprise to people used to toolsets where both are clearly
separate, such as APT vs. Debian packages.


[...]

>       (build-system gnu-build-system)
>       (native-inputs
>        `(("pkg-config" ,pkg-config)))
>       (inputs
>        `(("libbsd" ,libbsd)
>          ("ncurses" ,ncurses)))
>       (propagated-inputs
>        `(("mg-extensions"
>           ,(origin
>              (method url-fetch)
>              (uri "https://example.org/~doe/mg-extensions.tar.gz"
>              (sha256
>               (base32
>                "0kfhpj5rnh24hz2714qhfmxk281vwc2w50sm73ggw5d15af7zfsw"))))))

It doesn’t make much sense to propagate a tarball, does it?

>       (arguments
>        '(#:tests? #f ; No test suite available.
>          #:make-flags (list (string-append "prefix=" %output)
>                             "CC=gcc")
>          #:configure-flags `(,(string-append "--with-ext="
>                                              (assoc-ref %build-inputs "mg-extensions")))

I recommend using ‘list’ rather than quasiquote/unquote here.

> There are 3 different input types.  In short:

Perhaps link to
<https://www.gnu.org/software/guix/manual/en/html_node/package-Reference.html>?

> ** Code staging
>
> The astute reader may have noticed the quasi-quote and comma syntax in the
> argument field.  Indeed, the build code in the package declaration should not be
> evaluated on the client side, but only when passed to the Guix daemon.  This
> mechanism of passing code around two running processes is called [[https://arxiv.org/abs/1709.00833][code staging]].
> See [[https://www.gnu.org/software/guix/manual/en/html_node/G_002dExpressions.html][the "G-Expressions" chapter]] from the manual.

Though precisely package definitions don’t use gexps yet…  Not sure if
we should mention it; maybe it’s outside the scope of this tutorial.

> ** Module prefix
>
> The license now needs a prefix: this is because of how the ~license~ module was
> important in the package, as ~#:use-module ((guix licenses) #:prefix
> license:)~.  This gives the user full control over namespacing.

Perhaps mention that the prefix is needed to avoid classes between,
say, the ‘zlib’ variable from licenses.scm and the ‘zlib’ variable from
compression.scm?

Perhaps you can link to
<https://www.gnu.org/software/guile/manual/html_node/Using-Guile-Modules.html>.

> See https://guix.info/contact/ for the mailing lists, IRC, etc.

For now please use gnu.org/software/guix URLs.

> * References
>
> - [[https://gitlab.com/pjotrp/guix-notes/blob/master/HACKING.org][Pjotr’s hacking guide to GNU Guix]]
>
> - "Guix Guix: Package without a scheme!", by Andreas Enge (in
>   =guix-maintenance.git/talks/ghm-2013/andreas/slides.pdf=)

And the “Defining Packages” section of the manual!  ;-)

Thank you!

Ludo’.

Re: Blog: Guix packaging tutorial

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1762 bytes --]

Thanks for this thorough review, Ludovic!

I've taken most of your comments into account.  A few more questions:

> s/conventionally.*/typically used for read-only global variables/

But aren't they generated in scope?  I think we should mention this, or else
new users try to access them out of scope.

> It doesn’t make much sense to propagate a tarball, does it?

Not really, but I just wanted to use "propagated input".  We need a better
example.  Any idea?

> > The astute reader may have noticed the quasi-quote and comma syntax in the
> > argument field.  Indeed, the build code in the package declaration should not be
> > evaluated on the client side, but only when passed to the Guix daemon.  This
> > mechanism of passing code around two running processes is called [[https://arxiv.org/abs/1709.00833][code staging]].
> > See [[https://www.gnu.org/software/guix/manual/en/html_node/G_002dExpressions.html][the "G-Expressions" chapter]] from the manual.
>
> Though precisely package definitions don’t use gexps yet…  Not sure if
> we should mention it; maybe it’s outside the scope of this tutorial.

Hmmm... I think it's important to mention why code is not evaluated.  Maybe this
rather obscure paragraph could be simplified?
I'll remove the mention to G-exp, it does not belong here indeed.

> > See https://guix.info/contact/ for the mailing lists, IRC, etc.
>
> For now please use gnu.org/software/guix URLs.

Ok for this one, but I'd also like to link to the channels section in the
manual, but it's not on gnu.org.  Or is it?

Last but not least, what should I do next?  Should we wait for more reviews?
Should I go ahead and push to master?

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Blog: Guix packaging tutorial

[Ludovic Courtès]

Hello!

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

>> s/conventionally.*/typically used for read-only global variables/
>
> But aren't they generated in scope?  I think we should mention this, or else
> new users try to access them out of scope.

I’m not sure what you mean by “generated in scope”.

The ‘%’ convention is just a convention (and Andy and I realized a while
back we interpreted the convention differently :-)) so we shouldn’t draw
too much from that.

Regarding ‘%build-inputs’ and ‘%outputs’, I think it’s enough to say
that these are global variables.  Whether they are in scope depends on
whether a local variable shadows them.

Does that make sense?

>> It doesn’t make much sense to propagate a tarball, does it?
>
> Not really, but I just wanted to use "propagated input".  We need a better
> example.  Any idea?

The typical example are C, Python, or Guile libraries that propagate
libraries they depend on.

Another option is to skip propagated inputs altogether.

Thoughts?

>> > The astute reader may have noticed the quasi-quote and comma syntax in the
>> > argument field.  Indeed, the build code in the package declaration should not be
>> > evaluated on the client side, but only when passed to the Guix daemon.  This
>> > mechanism of passing code around two running processes is called [[https://arxiv.org/abs/1709.00833][code staging]].
>> > See [[https://www.gnu.org/software/guix/manual/en/html_node/G_002dExpressions.html][the "G-Expressions" chapter]] from the manual.
>>
>> Though precisely package definitions don’t use gexps yet…  Not sure if
>> we should mention it; maybe it’s outside the scope of this tutorial.
>
> Hmmm... I think it's important to mention why code is not evaluated.  Maybe this
> rather obscure paragraph could be simplified?
> I'll remove the mention to G-exp, it does not belong here indeed.

I think it’s good to mention code staging and the fact that there’s
“build-side code”, but the G-Expressions chapter says more than this,
which could be confusing.

>> > See https://guix.info/contact/ for the mailing lists, IRC, etc.
>>
>> For now please use gnu.org/software/guix URLs.
>
> Ok for this one, but I'd also like to link to the channels section in the
> manual, but it's not on gnu.org.  Or is it?

Not yet.  So yeah, you can use guix.info for this one if you need it.

> Last but not least, what should I do next?  Should we wait for more reviews?
> Should I go ahead and push to master?

I suppose you can push to guix-artwork.git master; Ricardo?  Please try
to use tags already used by the other articles.

When would you like it to be on line?

Thanks,
Ludo’.

Re: Blog: Guix packaging tutorial

[Ricardo Wurmus]

Hi Pierre,

thank you for writing this draft!  This is a good start.  Below are some
comments about the beginning and some general thoughts about the idea of
a tutorial.

> I'm not satisfied with the advanced example: I took ~mg~ because it has a lot of
> typical customization (snippets, inputs, arguments) but it does not have
> - in-input sources
> - %build-inputs
> - propagated-inputs
>
> I've made up a "my-mg" package but it relies on made-up inputs, so it won't
> build.
> Better suggestions anyone?

I wonder if a complicated example like that really has a place in a
tutorial.  To me it seems a bit scary.  I image people who are used to
Conda to browse the tutorial and recoil in horror at seeing a
complicated example with all sorts of customizations.

Here some more comments:

> Want to rebuild everything from source for a specific
>   architecture?  Pass the ~#:system "YOUR-ARCH"~ argument to the list of
>   packages.

People would rather use the “--system” option on the command line
instead of overwriting the arguments field, I think, so maybe that’s not
a good example.

> =GNU hello= is a dummy project that serves as an idiomatic example for
> packaging.  If uses the GNU build chain (~./configure && make && make install~).

s/If uses/It uses/

I would also replace “build chain” with “build system”.

> - source :: The ~origin~ record gather all the information required for a
>             source.  Among which:

How about: “This field contains a description of the source code origin.
The ~origin~ record contains these fields:” ?

> abstracts away the famous ~./configure && make && make
> install~ shell invocations: If the package can be built just
> those commands, then Guix can often handle the complete
> packaging process automatically. […]

“be built [+with] just those commands”.

What do you mean by “packaging process”?  What does Guix “handle” here?
I think it is less confusing to remove the sentence after the colon.

> - synopsis :: It should be a concise and explicit summary of what the package
>               does.  Many projects homepage already provide a synopsis, it's
>               fine to re-use it.

I’d remove “and explicit”.  I’d also remove the “Many projects
homepage..” sentence as we often tweak synopses to remove advertising
statements or to turn funny taglines into more informative, descriptive
statements.  You could also say “For many packages a tagline from the
project’s home page can be used as the synopsis.”

> dummy "my-hello", a copycat of the above declaration.

s/copycat/copy/

“copycat” is an imitator, not an imitation.

> Indeed, Guix refuses to build anything if the source fails to verify
> the checksum.

It isn’t clear here who verifies what.  The checksum and the source are
both passive in this situation, but the sentence makes it seem like the
source is active.  How about “if the given checksum does not match the
computed checksum of the source code”?

> To compute the correct checksum of the package declaration, we need to download the
> source, compute the sha256 checksum and convert it to base32.

To avoid the duplication of “compute” let’s replace the first instance
with “obtain”.

> Thankfully Guix can automate this task for us, all we need is to
> provide the URI:

There should be a comma after the first word; I’d replace the comma
before “all” with a semicolon.

> update you =my-hello= declaration accordingly.

s/you/your/

> your desire to write more and more complex packages will grow

s/will grow/grows/

> returns an anonymous function, which can in turn be called over an argument:

Maybe better: “can in turn be applied to an argument:”

> Thus it enables us with fine-grained control over what is evaluated and what is not.

s/enables/provides/

FWIW, I think that maybe we should avoid quasi-quotation in the
tutorial, or at least at this point.  It could be a little overwhelming
to be introduced to so many new concepts at once before seeing how they
might be useful.

Maybe this could be interleaved and motivated by packaging features.
For example, at the very beginning we define a package and then evaluate
it at the end so that we can use it with “guix package -f”.  Maybe it
would be better to avoid defining things and using the plain “package”
form.

Definitions could then be introduced later.

What do you think?

(I need to go now, but if you want I can comment on the next sections
tomorrow.)

--
Ricardo

Re: Blog: Guix packaging tutorial

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 3694 bytes --]


ludo@gnu.org (Ludovic Courtès) writes:
> I’m not sure what you mean by “generated in scope”.
>
> The ‘%’ convention is just a convention (and Andy and I realized a while
> back we interpreted the convention differently :-)) so we shouldn’t draw
> too much from that.
>
> Regarding ‘%build-inputs’ and ‘%outputs’, I think it’s enough to say
> that these are global variables.  Whether they are in scope depends on
> whether a local variable shadows them.
>
> Does that make sense?

I borrowed the expression "generated in scope" from Pjotr's
https://gitlab.com/pjotrp/guix-notes/blob/master/HACKING.org.

Are they really "global" variables?  My (quick) understanding from
derivations.scm is that the %-prefixed variables are defined at
derivation-time.  I think it's important to make this explicit.  What do you
think?

ludo@gnu.org (Ludovic Courtès) writes:
> Another option is to skip propagated inputs altogether.

I think it's important to highlight the distinction between the three kinds of
inputs.
When I got started, I was tempted to use propagated-inputs all the way.  The
distinction is very Guix-specific (Nix?) and is one of the lesser known strength
of Guix in my opinion.

ludo@gnu.org (Ludovic Courtès) writes:
> When would you like it to be on line?

Well, as soon as possible :)  Monday 1st, maybe?

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> writes:
> I wonder if a complicated example like that really has a place in a
> tutorial.  To me it seems a bit scary.  I image people who are used to
> Conda to browse the tutorial and recoil in horror at seeing a
> complicated example with all sorts of customizations.

In my opinion the "my-mg" example is not so long, although I could probably
remove some phase code because they don't add much to the tutorial.
But for the rest of it, it's rather widespread, idiomatic code.  I don't think
we can afford to remove much of it lest we make this tutorial useless for any
serious work.

Is there any specific element you'd like to cut off?

Also consider that the first part of this tutorial focuses on a hello-world
example that puts emphasis and getting an example up and running as soon as
possible.

What about splitting this tutorial in two separate blog entries and leaving the
advanced example for the second one?

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> writes:
> FWIW, I think that maybe we should avoid quasi-quotation in the
> tutorial, or at least at this point.  It could be a little overwhelming
> to be introduced to so many new concepts at once before seeing how they
> might be useful.

I am not sure about this.  On the one hand, it adds some complexity especially
to users unfamiliar with Lisp.  On the other hand, quasi-quotes are all over the
place.

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> writes:
> Maybe this could be interleaved and motivated by packaging features.
> For example, at the very beginning we define a package and then evaluate
> it at the end so that we can use it with “guix package -f”.  Maybe it
> would be better to avoid defining things and using the plain “package”
> form.

It would be more structured (and thus clearer), but it could also make the
tutorial a few lines longer...  I'll see if I can manage without adding too
much.

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> writes:
> (I need to go now, but if you want I can comment on the next sections
> tomorrow.)

Please do! :)

Ludovic, Ricardo, I've taken all your other considerations into account.  Thanks
a lot for your time reviewing this!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Blog: Guix packaging tutorial

[Ludovic Courtès]

Hello,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>> I’m not sure what you mean by “generated in scope”.
>>
>> The ‘%’ convention is just a convention (and Andy and I realized a while
>> back we interpreted the convention differently :-)) so we shouldn’t draw
>> too much from that.
>>
>> Regarding ‘%build-inputs’ and ‘%outputs’, I think it’s enough to say
>> that these are global variables.  Whether they are in scope depends on
>> whether a local variable shadows them.
>>
>> Does that make sense?
>
> I borrowed the expression "generated in scope" from Pjotr's
> https://gitlab.com/pjotrp/guix-notes/blob/master/HACKING.org.
>
> Are they really "global" variables?  My (quick) understanding from
> derivations.scm is that the %-prefixed variables are defined at
> derivation-time.  I think it's important to make this explicit.  What do you
> think?

‘%build-inputs’ etc. are global variables of package derivation build
scripts; see ‘build-expression->derivation’ in (guix derivations).

To view the code of one of these scripts, open the file returned by:

  $ guix gc --references $(guix build -d coreutils) | grep builder
  /gnu/store/v02xky6f5rvjywd7ficzi5pyibbmk6cq-coreutils-8.29-guile-builder

> ludo@gnu.org (Ludovic Courtès) writes:
>> When would you like it to be on line?
>
> Well, as soon as possible :)  Monday 1st, maybe?

Alright, let’s see how far we go; I’ll be AFK most of the day tomorrow
though, so it might have to be on Tuesday.

Thanks,
Ludo’.

Re: Blog: Guix packaging tutorial

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 670 bytes --]


> ‘%build-inputs’ etc. are global variables of package derivation build
> scripts; see ‘build-expression->derivation’ in (guix derivations).
>
> To view the code of one of these scripts, open the file returned by:
>
>   $ guix gc --references $(guix build -d coreutils) | grep builder
>   /gnu/store/v02xky6f5rvjywd7ficzi5pyibbmk6cq-coreutils-8.29-guile-builder

Nice, I had never looked into the builders before.  Good to know :)

Maybe we keep the concept of derivations out of this tutorial for now.  So I
think I'll explain this as "variables global to the package definition".  Sounds
alright?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Blog: Guix packaging tutorial

[Ludovic Courtès]

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

>> ‘%build-inputs’ etc. are global variables of package derivation build
>> scripts; see ‘build-expression->derivation’ in (guix derivations).
>>
>> To view the code of one of these scripts, open the file returned by:
>>
>>   $ guix gc --references $(guix build -d coreutils) | grep builder
>>   /gnu/store/v02xky6f5rvjywd7ficzi5pyibbmk6cq-coreutils-8.29-guile-builder
>
> Nice, I had never looked into the builders before.  Good to know :)
>
> Maybe we keep the concept of derivations out of this tutorial for now.  So I
> think I'll explain this as "variables global to the package definition".  Sounds
> alright?

Or global variables in the build stage, to be more accurate.  :-)

It’s really a fine point though and shouldn’t block the whole tutorial!

Thanks,
Ludo’.

Re: Blog: Guix packaging tutorial

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1036 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Or global variables in the build stage, to be more accurate.  :-)
>
> It’s really a fine point though and shouldn’t block the whole tutorial!

OK, I'll go ahead then.

Another Guix user told me he got confused about the Guile example.
`format` seemed confusing.  I suggested replacing it with

--8<---------------cut here---------------start------------->8---
(define foo "World")
(display (string-append "Hello " foo "\n"))
--8<---------------cut here---------------end--------------->8---

Apparently the formatting of the REPL was also confusing ("What is this
dollar thing, is it some result?" kind of confusion).
What about changing the formatting to

--8<---------------cut here---------------start------------->8---
(display (string-append "Hello " foo "\n"))
=> Hello World
--8<---------------cut here---------------end--------------->8---

Or should I use a Unicode arrow (e.g. ⇒ in the Guile manual)?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Blog: Guix packaging tutorial

[Ricardo Wurmus]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> Apparently the formatting of the REPL was also confusing ("What is this
> dollar thing, is it some result?" kind of confusion).
> What about changing the formatting to
>
> --8<---------------cut here---------------start------------->8---
> (display (string-append "Hello " foo "\n"))
> => Hello World
> --8<---------------cut here---------------end--------------->8---
>
> Or should I use a Unicode arrow (e.g. ⇒ in the Guile manual)?

The dollar thing is for return values.  The Guile manual uses the arrow,
but that would be incorrect in the above example as “Hello world” is not
returned by “display” — it is merely a side effect.  In the Guile manual
this would be indicated by “⊣”.

-- 
Ricardo

Re: Blog: Guix packaging tutorial

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 472 bytes --]


> The Guile manual uses the arrow,
> but that would be incorrect in the above example as “Hello world” is not
> returned by “display” — it is merely a side effect.  In the Guile manual
> this would be indicated by “⊣”.

Absolutely, and the distinction is not so useful in the context of this
tutorial.
So I think I'll simply skip it and remove the "$n = " prefix.
I'll only leave the prompt then.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Blog: Guix packaging tutorial

[Pierre Neidhardt]

[-- Attachment #1.1: Type: text/plain, Size: 1073 bytes --]

Find the last revision attached.  I've taken all comments into
consideration and I'm quite happy with the result.

I've found a much better advanced example: libgit2.
The only thing is... it does not pass the tests.

In guix, libgit2 uses an archive upstream.  I've changed the package to
check out the git repository.  Only then the test fail:

--8<---------------cut here---------------start------------->8---
starting phase `check'
Loaded 341 suites: 
Started

apply::fromdiff...................
apply::fromfile..........................................
attr::file....
attr::flags...
attr::ignoreF

  1) Failure:
attr::ignore::honor_temporary_rules [/tmp/guix-build-my-libgit2-0.26.6-1.e98d0a3.drv-0/my-libgit2-0.26.6-1.e98d0a3-checkout/tests/clar_libgit2.c:47]
  Expression is not true: (fd = p_open(path, flags, mode)) >= 0
--8<---------------cut here---------------end--------------->8---

Any clue?
I'll keep investigating tomorrow.  Once this is fixed, it should be
ready for publication.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #1.2: index.org --]
[-- Type: text/x-org, Size: 43683 bytes --]

#+TITLE: A packaging tutorial for Guix
#+AUTHOR: Pierre Neidhardt
#+date: <2018-10-03 Wed>
#+tags: Software development, Programming interfaces, Scheme API

* Introduction

GNU Guix stands out as the /hackable/ package manager, mostly because it uses
[[https://www.gnu.org/software/guile/][GNU Guile]], a powerful high-level programming language, one of the [[https://en.wikipedia.org/wiki/Scheme_(programming_language)][Scheme]]
dialects from the [[https://en.wikipedia.org/wiki/Lisp_(programming_language)][Lisp family]].

Package definitions are also written in Scheme, which empowers Guix in some very
unique ways, unlike most other package managers that use shell scripts or
simple languages.

- Use functions, structures, macros and all of Scheme expressiveness for your
  package definitions.

- Inheritance makes it easy to customize a package by inheriting from it and
  modifying only what is needed.

- Batch processing: the whole package collection can be parsed, filtered and
  processed.  Building a headless server with all graphical interfaces stripped
  out?  It's possible.  Want to rebuild everything from source using specific
  compiler optimization flags?  Pass the ~#:make-flags "..."~ argument to the
  list of packages.  It wouldn't be a stretch to think [[https://wiki.gentoo.org/wiki/USE_flag][Gentoo USE flags]] here,
  but this goes even further: the changes don't have to be thought out
  beforehand by the packager, they can be /programmed/ by the user!

The following tutorial covers all the basics around package creation with Guix.
It does not assume much knowledge of the Guix system nor of the Lisp language.
The reader is only expected to be familiar with the commandline and to have some
basic programming knowledge.

* A "Hello World" package

The [[https://www.gnu.org/software/guix/manual/en/html_node/Defining-Packages.html][Defining Packages section of the manual]] introduces the basics of Guix
packaging.  In the following section, we will partly go over those basics again.

=GNU hello= is a dummy project that serves as an idiomatic example for
packaging.  It uses the GNU build system (~./configure && make && make install~).
Guix already provides a package definition which is a perfect example to start
with.  You can look up its declaration with ~guix edit hello~ from the
commandline.  Let's see how it looks:

#+BEGIN_SRC scheme
(define-public hello
  (package
    (name "hello")
    (version "2.10")
    (source (origin
              (method url-fetch)
              (uri (string-append "mirror://gnu/hello/hello-" version
                                  ".tar.gz"))
              (sha256
               (base32
                "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
    (build-system gnu-build-system)
    (synopsis "Hello, GNU world: An example GNU package")
    (description
     "GNU Hello prints the message \"Hello, world!\" and then exits.  It
serves as an example of standard GNU coding practices.  As such, it supports
command-line arguments, multiple languages, and so on.")
    (home-page "https://www.gnu.org/software/hello/")
    (license gpl3+)))
#+END_SRC

As you can see, most of it is rather straightforward.  But let's review the
fields together:

- name :: The project name.  Using Scheme conventions, we prefer to keep it
          lower case, without underscore and using dash-separated words.
- source :: This field contains a description of the source code origin.  The
            ~origin~ record contains these fields:
  1. The method, here ~url-fetch~ to download via HTTP/FTP, but other methods
     exist, such as ~git-fetch~ for Git repositories.
  2. The URI, which is typically some =https://= location for ~url-fetch~.  Here
     the special =mirror://gnu= refers to a set of well known locations, all of
     which can be used by Guix to fetch the source, should some of them fail.
  3. The ~sha256~ checksum of the requested file.  This is essential to ensure
     the source is not corrupted.  Note that Guix works with base32 strings,
     hence the call to the ~base32~ function.
- build-system :: This is where the power of abstraction provided by the Scheme
                  language really shines: in this case, the ~gnu-build-system~
                  abstracts away the famous ~./configure && make && make
                  install~ shell invocations.  Other build systems include the
                  ~trivial-build-system~ which does not do anything and requires
                  from the packager to program all the build steps, the
                  ~python-build-system~, the ~emacs-build-system~, [[https://www.gnu.org/software/guix/manual/en/html_node/Build-Systems.html][and many
                  more]].
- synopsis :: It should be a concise summary of what the package
              does.  For many packages a tagline from the
              project's home page can be used as the synopsis.
- description :: Same as for the synopsis, it's fine to re-use the project
                 description from the homepage.  Note that Guix uses Texinfo
                 syntax.
- home-page :: Use HTTPS if available.
- license :: See =$GUIX_CHECKOUT/guix/licenses.scm= for a full list.

Time to build our first package!  Nothing fancy here for now: we will stick to a
dummy "my-hello", a copy of the above declaration.

As with the ritualistic "Hello World" taught with most programming language,
this will possibly be the most "manual" approach.  We will work out an ideal
setup later; for now we will go the simplest route.

Save the following to a file =my-hello.scm=.

#+BEGIN_SRC scheme
(use-modules (guix packages)
             (guix download)
             (guix build-system gnu)
             (guix licenses))

(package
  (name "my-hello")
  (version "2.10")
  (source (origin
            (method url-fetch)
            (uri (string-append "mirror://gnu/hello/hello-" version
                                ".tar.gz"))
            (sha256
             (base32
              "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
  (build-system gnu-build-system)
  (synopsis "Hello, Guix world: An example custom Guix package")
  (description
   "GNU Hello prints the message \"Hello, world!\" and then exits.  It
serves as an example of standard GNU coding practices.  As such, it supports
command-line arguments, multiple languages, and so on.")
  (home-page "https://www.gnu.org/software/hello/")
  (license gpl3+))
#+END_SRC

We will explain the extra code in a moment.

Feel free to play with the different values of the various fields.  If you
change the source, you'll need to update the checksum.  Indeed, Guix refuses to
build anything if the given checksum does not match the computed checksum of the
source code.  To obtain the correct checksum of the package declaration, we
need to download the source, compute the sha256 checksum and convert it to
base32.

Thankfully, Guix can automate this task for us; all we need is to provide the
URI:

#+BEGIN_SRC sh
$ guix download mirror://gnu/hello/hello-2.10.tar.gz

Starting download of /tmp/guix-file.JLYgL7
From https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz...
following redirection to `https://mirror.ibcp.fr/pub/gnu/hello/hello-2.10.tar.gz'...
 …10.tar.gz  709KiB                                 2.5MiB/s 00:00 [##################] 100.0%
/gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i
#+END_SRC

Note in this specific case that the output tells us which mirror was chosen.

If the result of the above command is not the same as in the above snippet,
update your =my-hello= declaration accordingly.

Now you can happily run

#+BEGIN_SRC sh
$ guix package --install-from-file=my-hello.scm
#+END_SRC

You should now have =my-hello= in your profile!

#+BEGIN_SRC sh
$ guix package --list-installed=my-hello
my-hello	2.10	out	/gnu/store/f1db2mfm8syb8qvc357c53slbvf1g9m9-my-hello-2.10
#+END_SRC

We've gone as far as we could without any knowledge of Scheme.  Now is the right
time to introduce the minimum we need from the language before we can proceed.

* A Scheme crash-course

As we've seen above, basic packages don't require much Scheme knowledge, if none
at all.  But as you progress and your desire to write more and more complex
packages grows, it will become both necessary and empowering to hone your Lisper
skills.

Since an extensive Lisp course is very much out of the scope of this tutorial,
we will only cover some basics here.

Guix uses the Guile implementation of Scheme.  To start playing with the
language, install it with ~guix package --install guile~ and start a [[https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop][REPL]] by
running ~guile~ from the commandline.

Alternatively you can also run ~guix environment --ad-hoc guile -- guile~ if
you'd rather not have Guile installed in your user profile.

In the following examples we use the =>= symbol to denote the REPL prompt, that
is, the line reserved for user input.  See [[https://www.gnu.org/software/guile/manual/html_node/Using-Guile-Interactively.html][the Guile manual]] for more details on
the REPL.

- Scheme syntax boils down to a tree of expressions (or /s-expression/ in Lisp
  lingo) It can be a literal such numbers and strings, or a compound which is a
  parenthesized list of compounds and literals.
  ~#t~ and ~#f~ stand for the booleans "true" and "false", respectively.

  Examples of valid expressions:
  #+BEGIN_SRC scheme
> "Hello World!"
"Hello World!"
> 17
17
> (display (string-append "Hello " "Guix" "\n"))
"Hello Guix!"
  #+END_SRC

- This last example is a function call embedded in another function call.  When
  a parenthesized expression is evaluated, the first term is the function and
  the rest are the arguments passed to the function.  Every function returns the
  last evaluated expression as value.

- Anonymous functions are declared with the ~lambda~ term:
  #+BEGIN_SRC scheme
> (lambda (x) (* x x))
#<procedure 120e348 at <unknown port>:24:0 (x)>
  #+END_SRC
  The above lambda returns the square of its argument.  Since everything is an
  expression, the ~lambda~ expression returns an anonymous function, which can
  in turn be applied to an argument:
  #+BEGIN_SRC scheme
> ((lambda (x) (* x x)) 3)
9
  #+END_SRC

- Anything can be assigned a global name with ~define~:
  #+BEGIN_SRC scheme
> (define a 3)
> (define square (lambda (x) (* x x)))
> (square a)
9
  #+END_SRC

- Procedures can be defined more concisely with the following syntax:
  #+BEGIN_SRC scheme
  (define (square x) (* x x))
  #+END_SRC

- A list structure can be created with the ~list~ procedure:
  #+BEGIN_SRC scheme
> (list 2 a 5 7)
(2 3 5 7)
  #+END_SRC

- The /quote/ disables evaluation of a parenthesized expression: the first term
  is not called over the other terms.  Thus it effectively returns a list of
  terms.
  #+BEGIN_SRC scheme
> '(display (string-append "Hello " "Guix" "\n"))
(display (string-append "Hello " "Guix" "\n"))
> '(2 a 5 7)
(2 a 5 7)
  #+END_SRC

- The /quasiquote/ disables evaluation of a parenthesized expression until a
  colon re-enables it.  Thus it provides us with fine-grained control over what
  is evaluated and what is not.
  #+BEGIN_SRC scheme
> `(2 a 5 7 (2 ,a 5 ,(+ a 4)))
(2 a 5 7 (2 3 5 7))
  #+END_SRC
  Note that the above result is a list of mixed elements: numbers, symbols (here
  ~a~) and the last element is a list itself.

- Multiple variables can be named locally with ~let~:
  #+BEGIN_SRC scheme
> (define x 10)
> (let ((x 2)
        (y 3))
    (list x y))
(2 3)
> x
10
> y
ERROR: In procedure module-lookup: Unbound variable: y
  #+END_SRC
  Use ~let*~ to re-allows the initializers of later variables to refer to the
  earlier variables.
  #+BEGIN_SRC scheme
> (let* ((x 2)
         (y (* x 3)))
    (list x y))
(2 6)
  #+END_SRC

- The keyword syntax is ~#:~, it is used to create unique identifiers.  See also
  the [[https://www.gnu.org/software/guile/manual/html_node/Keywords.html][Keywords section in the Guile manual]].

- The percentage ~%~ is typically used for read-only global variables in the
  build stage.  Note that it is merely a convention, like ~_~ in C.  Scheme Lisp
  treats ~%~ exactly the same as any other letter.

- Modules are created with ~define-module~.  For instance
  #+BEGIN_SRC scheme
  (define-module (guix build-system ruby)
    #:use-module (guix store)
    #:export (ruby-build
              ruby-build-system))
  #+END_SRC
  defines the module ~ruby~ which must be located in
  ~guix/build-system/ruby.scm~ somewhere in =GUILE_LOAD_PATH=.  It depends on
  the ~(guix store)~ module and it exports two symbols, ~ruby-build~ and
  ~ruby-build-system~.

For a more detailed introduction, check out [[http://www.troubleshooters.com/codecorn/scheme_guile/hello.htm][Scheme at a Glance]], by Steve Litt.

One of the reference Scheme books is the seminal /Structure and Interpretation
of Computer Programs/, by Harold Abelson and Gerald Jay Sussman, with Julie
Sussman.  You'll find a free copy [[https://mitpress.mit.edu/sites/default/files/sicp/index.html][online]], together with [[https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-001-structure-and-interpretation-of-computer-programs-spring-2005/video-lectures/][videos of the lectures
by the authors]].  The book is available in Texinfo format as the =sicp= Guix
package.  Go ahead, run ~guix package --install sicp~ and start reading with
~info sicp~ (or with the Emacs Info reader).  An unofficial ebook [[https://sarabander.github.io/sicp/][is also
available]].

You'll find more books, tutorials and other resources at https://schemers.org/.

* Setup

Now that we know some Scheme basics we can detail the different possible setups
for working on Guix packages.

There are several ways to set up a Guix packaging environment.

We recommend you work directly on the Guix source checkout since it makes it
easier for everyone to contribute to the project.

But first, let's look at other possibilities.

*** Local file

This is what we previously did with =my-hello=.  Now that we know more Scheme,
let's explain the leading chunks.  As stated in ~guix package --help~:

#+BEGIN_SRC sh
  -f, --install-from-file=FILE
                         install the package that the code within FILE
                         evaluates to
#+END_SRC

Thus the last expression /must/ return a package, which is the case in our
earlier example.

The ~use-modules~ expression tells which of the modules we need in the file.
Modules are a collection of values and procedures.  They are commonly called
"libraries" or "packages" in other programming languages.

*** GUIX_PACKAGE_PATH

/Note: Starting from Guix 0.16, the more flexible Guix "channels" are the
preferred way and supersede =GUIX_PACKAGE_PATH=.  See below./

It can be tedious to specify the file from the commandline instead of simply
calling ~guix package --install my-hello~ as you would do with the official
packages.

Guix makes it possible to streamline the process by adding as many "package
declaration paths" as you want.

Create a folder, say =~./guix-packages= and add it to the =GUIX_PACKAGE_PATH=
environment variable:

#+BEGIN_SRC sh
$ mkdir ~/guix-packages
$ export GUIX_PACKAGE_PATH=~/guix-packages
#+END_SRC

To add several folders, separate them with a colon (~:~).

Our previous =my-hello= needs some adjustments though:

#+BEGIN_SRC scheme
(define-module (my-hello)
  #:use-module (guix licenses)
  #:use-module (guix packages)
  #:use-module (guix build-system gnu)
  #:use-module (guix download))

(define-public my-hello
  (package
    (name "my-hello")
    (version "2.10")
    (source (origin
              (method url-fetch)
              (uri (string-append "mirror://gnu/hello/hello-" version
                                  ".tar.gz"))
              (sha256
               (base32
                "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
    (build-system gnu-build-system)
    (synopsis "Hello, Guix world: An example custom Guix package")
    (description
     "GNU Hello prints the message \"Hello, world!\" and then exits.  It
serves as an example of standard GNU coding practices.  As such, it supports
command-line arguments, multiple languages, and so on.")
    (home-page "https://www.gnu.org/software/hello/")
    (license gpl3+)))
#+END_SRC

Notice that we have wrapped the package into a ~define-public~ record.  This is
effectively assigning the package to the ~my-hello~ variable so that it can be
referenced, among other as dependency of other packages.

If you use ~guix package --install-from-file=my-hello.scm~ on the above file, it
will fail because the last expression, ~define-public~, does not return a
package.  If you want to use ~define-public~ in this use-case nonetheless, make
sure the file ends with an evaluation of ~my-hello~:

#+BEGIN_SRC scheme
; ...
(define-public my-hello
  ; ...
  )

my-hello
#+END_SRC

This last example is not very typical.

Now =my-hello= should be part of the package collection like all other official
packages.  You can verify this with:

#+BEGIN_SRC sh
$ guix package --show=my-hello
#+END_SRC

*** Guix channels

Guix 0.16 features channels, which is very similar to =GUIX_PACKAGE_PATH= but
provides better integration and provenance tracking.  Channels are not
necessarily local, they can be maintained as a public Git repository for
instance.  Of course, several channels can be used at the same time.

See the [[http://guix.info/manual/en/Channels.html][Channel secion in the manual]] for setup details.

*** Direct checkout hacking

Working directly on the Guix project is recommended: it reduces the friction
when the time comes to submit your changes upstream to let the community benefit
from your hard work!

Unlike most software distributions, the Guix repository holds in one place both
the tooling (including the package manager) and the package definitions.  This
choice was made so that it would give developers the flexibility to modify the
API without breakage by updating all packages at the same time.  This reduces
development inertia.

Check out the official [[https://git-scm.com/][Git]] repository:

#+BEGIN_SRC sh
$ git clone https://git.savannah.gnu.org/git/guix.git
#+END_SRC

Follow the instruction from the [[https://www.gnu.org/software/guix/manual/en/html_node/Contributing.html]["Contributing" chapter]] in the manual to set up the
repository environment.

Once ready, you should be able to use the package definitions from the
repository environment.

Feel free to edit package definitions found in =$GUIX_CHECKOUT/gnu/packages=.

The =$GUIX_CHECKOUT/pre-inst-env= script lets you use =guix= over the package
collection of the repository.

- Search packages:

  #+BEGIN_SRC sh
  $ cd $GUIX_CHECKOUT
  $ ./pre-inst-env guix package --list-available=ruby
      ruby    1.8.7-p374      out     gnu/packages/ruby.scm:119:2
      ruby    2.1.6   out     gnu/packages/ruby.scm:91:2
      ruby    2.2.2   out     gnu/packages/ruby.scm:39:2
  #+END_SRC

- Build a package:

  #+BEGIN_SRC sh
  $ ./pre-inst-env guix build --keep-failed ruby@2.1
  /gnu/store/c13v73jxmj2nir2xjqaz5259zywsa9zi-ruby-2.1.6
  #+END_SRC

- Install it to your user profile:

  #+BEGIN_SRC sh
  $ ./pre-inst-env guix package --install ruby@2.1
  #+END_SRC

- Check for common mistakes:

  #+BEGIN_SRC sh
  $ ./pre-inst-env guix lint ruby@2.1
  #+END_SRC

Once you are happy with the result, you are welcome to send your contribution to
make it part of Guix.  This process is also detailed in the [[https://www.gnu.org/software/guix/manual/en/html_node/Contributing.html][manual]].

It's a community effort so the more join in, the better Guix becomes!

* Extended example

The above "Hello World" example is as simple as it goes.  Packages can be more
complex than that and Guix can handle more advanced scenarios.  Let's looks at
another, more sophisticated package (slightly modified from the source):

# REVIEW: Find a better example?
# qpdf: lacks input.
# qtbase: too long.
# bigloo: no configure, no make flag.
# libpng-apng misses snippets and git.
# minetest (65): needs outputs.
# pulseaudio (80): needs outputs.
# libgit2 (54): need git + makeflags

# TODO: The following does not pass the tests:
#
# starting phase `check'
# Loaded 341 suites:
# Started
#
# apply::fromdiff...................
# apply::fromfile..........................................
# attr::file....
# attr::flags...
# attr::ignoreF
#
#   1) Failure:
# attr::ignore::honor_temporary_rules [/tmp/guix-build-my-libgit2-0.26.6-1.e98d0a3.drv-0/my-libgit2-0.26.6-1.e98d0a3-checkout/tests/clar_libgit2.c:47]
#   Expression is not true: (fd = p_open(path, flags, mode)) >= 0


#+BEGIN_SRC scheme
(define-module (gnu packages version-control)
  #:use-module ((guix licenses) #:prefix license:)
  #:use-module (guix utils)
  #:use-module (guix packages)
  #:use-module (guix git-download)
  #:use-module (guix build-system cmake)
  #:use-module (gnu packages ssh)
  #:use-module (gnu packages web)
  #:use-module (gnu packages pkg-config)
  #:use-module (gnu packages python)
  #:use-module (gnu packages compression)
  #:use-module (gnu packages tls))

(define-public my-libgit2
  (let ((commit "e98d0a37c93574d2c6107bf7f31140b548c6a7bf")
        (revision "1"))
    (package
      (name "my-libgit2")
      (version (git-version "0.26.6" revision commit))
      (source (origin
                (method git-fetch)
                (uri (git-reference
                      (url "https://github.com/libgit2/libgit2/")
                      (commit commit)))
                (file-name (git-file-name name version))
                (sha256
                 (base32
                  "17pjvprmdrx4h6bb1hhc98w9qi6ki7yl57f090n9kbhswxqfs7s3"))
                (patches (search-patches "libgit2-mtime-0.patch"))
                (modules '((guix build utils)))
                (snippet '(begin
                            ;; Remove bundled software.
                            (delete-file-recursively "deps")
                            #t))))
      (build-system cmake-build-system)
      (outputs '("out" "debug"))
      (arguments
       `(#:tests? #t                            ; Run the test suite (this is the default)
         #:configure-flags '("-DUSE_SHA1DC=ON") ; SHA-1 collision detection
         #:phases
         (modify-phases %standard-phases
           (add-after 'unpack 'fix-hardcoded-paths
             (lambda _
               (substitute* "tests/repo/init.c"
                 (("#!/bin/sh") (string-append "#!" (which "sh"))))
               (substitute* "tests/clar/fs.h"
                 (("/bin/cp") (which "cp"))
                 (("/bin/rm") (which "rm")))
               #t))
           ;; Run checks more verbosely.
           (replace 'check
             (lambda _ (invoke "./libgit2_clar" "-v" "-Q"))))))
      (inputs
       `(("libssh2" ,libssh2)
         ("http-parser" ,http-parser)
         ("python" ,python-wrapper)))
      (native-inputs
       `(("pkg-config" ,pkg-config)))
      (propagated-inputs
       ;; These two libraries are in 'Requires.private' in libgit2.pc.
       `(("openssl" ,openssl)
         ("zlib" ,zlib)))
      (home-page "https://libgit2.github.com/")
      (synopsis "Library providing Git core methods")
      (description
       "Libgit2 is a portable, pure C implementation of the Git core methods
provided as a re-entrant linkable library with a solid API, allowing you to
write native speed custom Git applications in any language with bindings.")
      ;; GPLv2 with linking exception
      (license license:gpl2))))
#+END_SRC

(In those cases were you only want to tweak a few fields from a package
definition, you should rely on inheritance instead of copy-pasting everything.
See below.)

Let's discuss those fields in depth.

** ~git-fetch~ method

Unlike the ~url-fetch~ method, ~git-fetch~ expects a ~git-reference~ which takes
a Git repository and a commit.  The commit can be any Git reference such as
tags, so if the ~version~ is tagged, then it can be used directly.  Sometimes
the tag is prefixed with a ~v~, in which case you'd use ~(commit (string-append
"v" version))~.

When downloaded to the store, the source should be different from version to
version.  Since Git repositories don't contain a specific version in their name,
we've got to force the download folder using ~(file-name (git-file-name name
version))~.

Note that there is also a ~git-version~ procedure that can be used to derive the
version when packaging programs for a specific commit.

** Snippets

Snippets are quoted (i.e. non-evaluated) Scheme code that are a mean of patching
the source.  They are a Guix-y alternative to the traditional =.patch= files.
Because of the quote, the code in only evaluated when passed to the Guix daemon
for building.

There can be as many snippet as needed.

Snippets might need additional Guile modules which can be imported from the
~modules~ field.

** Inputs

First, a syntactic comment: See the quasi-quote / comma syntax?

#+BEGIN_SRC scheme
    (native-inputs
     `(("pkg-config" ,pkg-config)))
#+END_SRC

is equivalent to

#+BEGIN_SRC scheme
    (native-inputs
     (list (list "pkg-config" pkg-config)))
#+END_SRC

You'll mostly see the former because it's shorter.

There are 3 different input types.  In short:

- native-inputs :: Required for building but not runtime -- installing a package
                   through a substitute won't install these inputs.
- inputs :: Installed in the store but not in the profile, as well as being
            present at build time.
- propagated-inputs :: Installed in the store and in the profile, as well as
     being present at build time.

See [[https://www.gnu.org/software/guix/manual/en/html_node/package-Reference.html][the package reference in the manual]] for more details.

The distinction between the various inputs is important: if a dependency can be
handled as an /input/ instead of a /propagated input/, it should be done so, or
else it "pollutes" the user profile for no good reason.

For instance, a user installing a graphical program that depends on a
commandline tool might only be interested in the graphical part, so there is no
need to force the commandline tool into the user profile.  The dependency is a
concern to the package, no to the user.  /Inputs/ make it possible to handle
dependencies without bugging the user by adding undesired executable files (or
libraries) to their profile.

Same goes for /native-inputs/: once the program is installed, build-time
dependencies can be safely garbage-collected.
It also matters when a substitute is available, in which case only the /inputs/
and /propagated inputs/ will be fetched: the /native inputs/ are not required to
install a package from a substitute.

** Outputs

Just like how a package can have multiple inputs, it can also produce multiple
outputs.

Each output corresponds to a separate folder in the store.

The end user can choose which output to install; this is useful to save space or
to avoid polluting the user profile with unwanted executables or libraries.

Output separation is optional.  When the ~outputs~ field is left out, the
default and only output (the complete package) is referred to as ~"out"~.

Typical separate output names include ~debug~ and ~doc~.

It's advised to separate outputs only when you've shown it worth: if the output
size is significant (compare with ~guix size~) or in case the package is
modular.

** Build system arguments

The ~arguments~ is a keyword-value list used to configure the build process.

The simplest argument ~#:tests?~ can be used to disable the test suite when
building the package.  This is mostly useful when the package does not feature
any test suite.  It's strongly recommended to keep the test suite on if there is
one.

Another  common argument is ~:make-flags~, which specifies a list of flags to
append when running make, as you would from the commandline.  For instance, the
following flags

#+BEGIN_SRC scheme
         #:make-flags (list (string-append "prefix=" (assoc-ref %outputs "out"))
                            "CC=gcc")
#+END_SRC

translate into

#+BEGIN_SRC sh
$ make CC=gcc prefix=/gnu/store/...-<out>
#+END_SRC

This sets the C compiler to ~gcc~ and the ~prefix~ variable (the installation
folder in Make parlance) to ~(assoc-ref %outputs "out")~, which is a build-stage
global variable pointing to the destination folder in the store (something like
=/gnu/store/...-my-mg-20180408=).

Similarly, it's possible to set the "configure" flags.  In the above example, we
set the extensions to the store download folder of our fictitious mg-extensions.

#+BEGIN_SRC scheme
         #:configure-flags `(,(string-append "--with-ext="
                                             (assoc-ref %build-inputs "mg-extensions")))
#+END_SRC

The ~%build-inputs~ variable is also generated in scope.  It's an association
table that maps the input names to their store folders.

The ~phases~ keyword lists the sequential steps of the build system.  Typically
phases include ~unpack~, ~configure~, ~build~, ~install~ and ~check~.  To know
more about those phases, you need to work out the appropriate build system
definition in =$GUIX_CHECKOUT/guix/build/gnu-build-system.scm=:

#+BEGIN_SRC scheme
(define %standard-phases
  ;; Standard build phases, as a list of symbol/procedure pairs.
  (let-syntax ((phases (syntax-rules ()
                         ((_ p ...) `((p . ,p) ...)))))
    (phases set-SOURCE-DATE-EPOCH set-paths install-locale unpack
            bootstrap
            patch-usr-bin-file
            patch-source-shebangs configure patch-generated-file-shebangs
            build check install
            patch-shebangs strip
            validate-runpath
            validate-documentation-location
            delete-info-dir-file
            patch-dot-desktop-files
            install-license-files
            reset-gzip-timestamps
            compress-documentation)))
#+END_SRC

Or from the REPL (assuming the Guix source is in your Guile load path):

#+BEGIN_SRC scheme
> ,module (guix build gnu-build-system)
> (map first %standard-phases)
(set-SOURCE-DATE-EPOCH set-paths install-locale unpack bootstrap patch-usr-bin-file patch-source-shebangs configure patch-generated-file-shebangs build check install patch-shebangs strip validate-runpath validate-documentation-location delete-info-dir-file patch-dot-desktop-files install-license-files reset-gzip-timestamps compress-documentation)
#+END_SRC

If you want to know more about what happens during those phases, consult the
associated functions.

For instance, as of this writing the definition of ~unpack~ for the GNU build
system is

#+BEGIN_SRC scheme
(define* (unpack #:key source #:allow-other-keys)
  "Unpack SOURCE in the working directory, and change directory within the
source.  When SOURCE is a directory, copy it in a sub-directory of the current
working directory."
  (if (file-is-directory? source)
      (begin
        (mkdir "source")
        (chdir "source")

        ;; Preserve timestamps (set to the Epoch) on the copied tree so that
        ;; things work deterministically.
        (copy-recursively source "."
                          #:keep-mtime? #t))
      (begin
        (if (string-suffix? ".zip" source)
            (invoke "unzip" source)
            (invoke "tar" "xvf" source))
        (chdir (first-subdirectory "."))))
  #t)
#+END_SRC

Note the ~chdir~ call: it changes the working directory to where the source was
unpacked.
Thus every phase following the ~unpack~ will use the source as a working
directory, which is why we can directly work on the source files.
That is to say, unless a later phase changes the working directory to something
else.

To manipulate the phases,

- ~add-before PHASE NEW-PHASE PROCEDURE~: Run ~PROCEDURE~ named ~NEW-PHASE~ before ~PHASE~.
- ~add-after PHASE NEW-PHASE PROCEDURE~: Same, but afterwards.
- ~replace PHASE PROCEDURE~.
- ~delete PHASE~.

The ~PROCEDURE~ supports the keyword arguments ~inputs~ and ~outputs~.  Each
input (whether /native/, /propagated/ or not) and output directory is referenced
by their name in those variables.  Thus ~(assoc-ref outputs "out")~ is the store
directory of the main output of the package.  A phase procedure may look like
this:

#+BEGIN_SRC scheme
(lambda* (#:key inputs outputs #:allow-other-keys)
  (let (((bash-folder (assoc-ref inputs "bash"))
         (output-folder (assoc-ref outputs "out"))
         (doc-folder (assoc-ref outputs "doc"))
  ; ...
  #t)
#+END_SRC

The procedure must return ~#t~ on success.  It's brittle to rely on the return
value of the last expression used to tweak the phase because there is no
guarantee it would be a ~#t~.  Hence the trailing ~#t~ to ensure the right value
is returned on success.

** Code staging

The astute reader may have noticed the quasi-quote and comma syntax in the
argument field.  Indeed, the build code in the package declaration should not be
evaluated on the client side, but only when passed to the Guix daemon.  This
mechanism of passing code around two running processes is called [[https://arxiv.org/abs/1709.00833][code staging]].

** "Utils" functions

When customizing ~phases~, we often need to write code that mimics the
equivalent system invocations (~make~, ~mkdir~, ~cp~, etc.) commonly used during
regular "Unix-style" installations.

Some like ~chmod~ are native to Guile.  See the [[https://www.gnu.org/software/guile/manual/guile.html][Guile reference manual]] for a
complete list.

Guix provides additional helper functions which prove especially handy in the
context of package management.

Some of those functions can be found in
=$GUIX_CHECKOUT/guix/guix/build/utils.scm=.  Most of them mirror the behaviour
of the traditional Unix system commands:

- which :: Like the =which= system command.
- find-files :: Akin to the =find= system command.
- mkdir-p :: Like =mkdir -p=, which creates all parents as needed.
- install-file :: Similar to =install= when installing a file to a (possibly
                  non-existing) directory.  Guile has ~copy-file~ which works
                  like =cp=.
- copy-recursively :: Like =cp -r=.
- delete-file-recursively :: Like =rm -rf=.
- invoke :: Run an executable.  This should be used instead of ~system*~.
- with-directory-excursion :: Run the body in a different working directory,
     then restore the previous working directory.
- substitute* :: A "sed-like" function.

** Module prefix

The license in our last example needs a prefix: this is because of how the
~license~ module was imported in the package, as ~#:use-module ((guix licenses)
#:prefix license:)~.  The [[https://www.gnu.org/software/guile/manual/html_node/Using-Guile-Modules.html][Guile module import mechanism]] gives the user full
control over namespacing: this is needed to avoid clashes between, say, the
=zlib= variable from =licenses.scm= and the =zlib= variable from
=compression.scm=.

* Other build systems

What we've seen so far covers the majority of packages using a build system
other than the ~trivial-build-system~.  The latter does not automate anything
and leaves you to build everything manually.  This can be more demanding and we
won't cover it here for now, but thankfully it is rarely necessary to fall back
on this system.

For the other build systems, such as ASDF, Emacs, Perl, Ruby and many more, the
process is very similar to the GNU build system but for a few specialized
arguments.

Find more about build systems in
- [[https://www.gnu.org/software/guix/manual/en/html_node/Build-Systems.html#Build-Systems][the manual, section 4.2 Build systems]],
- the source code in the =$GUIX_CHECKOUT/guix/build= and
  =$GUIX_CHECKOUT/guix/build-system= folders.

* Common pitfalls and best practices

The following check list is important to maintain a high packaging standard, in
particular when contributing to the Guix project.

- Use the /linter/: ~guix lint $PACKAGES~.
- Use mirrors when possible in the source URL.
- Use reliable URLs, not generated ones (e.g. GitHub archives).
  Don't use the ~name~ field in the URL: it is not very useful and if the name
  changes, the URL will probably be wrong.
- Sort inputs alphabetically.
- Synopsis should be as concise as possible, don't start with "The" or "A", don't repeat
  the package name.
- Description uses Texinfo syntax with two spaces at the end of sentences.
- Reproducibility: use ~guix build --check~ and ~guix build --rounds=N~.
- Follow the [[https://www.gnu.org/software/guix/manual/en/html_node/Coding-Style.html][coding style]] from the manual.
- Review the [[https://www.gnu.org/software/guix/manual/en/html_node/Submitting-Patches.html][check list]] from the manual.

* Programmable and automated package definition

We can't repeat it enough: having a full-fledged programming language at hand
empowers us in ways that reach far beyond traditional package management.

Let's illustrate this with some awesome features of Guix!

** Recursive importers

You might find some build systems good enough that there is little to do at all
to write a package, to the point that it becomes repetitive and tedious after a
while.  A /raison d'être/ of computers is to replace human beings at those
boring tasks.  So let's tell Guix to do this for us and create the package
definition of an ELPA package:

#+BEGIN_SRC sh
$ guix import elpa exwm

(package
  (name "emacs-exwm")
  (version "0.19")
  (source
    (origin
      (method url-fetch)
      (uri (string-append
             "https://elpa.gnu.org/packages/exwm-"
             version
             ".tar"))
      (sha256
        (base32
          "11xd2w4h3zdwkdxypvmcz8s7q72cn76lfr9js77jbizyj6b04lr0"))))
  (build-system emacs-build-system)
  (propagated-inputs `(("emacs-xelb" ,emacs-xelb)))
  (home-page "https://github.com/ch11ng/exwm")
  (synopsis "Emacs X Window Manager")
  (description
    "EXWM (Emacs X Window Manager) is a full-featured tiling X window manager
for Emacs built on top of [XELB](https://github.com/ch11ng/xelb).
It features:
+ Fully keyboard-driven operations
+ Hybrid layout modes (tiling & stacking)
+ Dynamic workspace support
+ ICCCM/EWMH compliance
+ (Optional) RandR (multi-monitor) support
+ (Optional) Built-in system tray")
  (license license:gpl3+))
#+END_SRC

Not all applications can be packaged this way, only those relying on a select
number of supported systems.  Read about the full list of importers in the [[https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-import.html][guix
import section]] of the manual.

** Automatic update

Guix can be smart enough to check for updates on systems it knows.  It can
report outdated package definitions with

#+BEGIN_SRC sh
$ guix refresh hello
#+END_SRC

In most cases, updating a package to a newer version requires little more than
changing the version number and the checksum.  Guix can do that automatically as
well:

#+BEGIN_SRC
$ guix refresh hello --update
#+END_SRC

** Inheritance

If you've started browsing the existing package definitions, you might have
noticed that a significant number of them have a ~inherit~ field:

#+BEGIN_SRC scheme
(define-public adwaita-icon-theme
  (package (inherit gnome-icon-theme)
    (name "adwaita-icon-theme")
    (version "3.26.1")
    (source (origin
              (method url-fetch)
              (uri (string-append "mirror://gnome/sources/" name "/"
                                  (version-major+minor version) "/"
                                  name "-" version ".tar.xz"))
              (sha256
               (base32
                "17fpahgh5dyckgz7rwqvzgnhx53cx9kr2xw0szprc6bnqy977fi8"))))
    (native-inputs
     `(("gtk-encode-symbolic-svg" ,gtk+ "bin")))))
#+END_SRC

All unspecified fields inherit from the parent package.  This is very convenient
to create alternative packages, for instance with different source, version or
compilation options.

* Getting help

Sadly some applications can be tough to package.  Sometimes they need a patch to
work with the non-standard filesystem hierarchy enforced by the store.
Sometimes the tests won't run properly.  (They can be skipped but this is not
recommended.)  Other times the resulting package won't be reproducible.

Should you be stuck, unable to figure out how to fix any sort of packaging
issue, don't hesitate to ask the community for help.

See the [[https://www.gnu.org/software/guix/contact/][Guix homepage]] for information on the mailing lists, IRC, etc.

* Conclusion

This tutorial is an introductory showcase of how Guix can bring automation to
the field of packaging.  The GNU build system is the flagship of the high-level
abstraction layers Guix makes possible to build.

Now where do we go from here?  Next we ought to dissect the innards of the build
system by removing all abstractions, using the ~trivial-build-system~: this
should give us a thorough understanding of the process before investigating some
more advanced packaging techniques and edge cases.

Other features worth exploring are the interactive editing and debugging
capabilities of Guix provided by the Guile REPL.

* References

- The [[https://www.gnu.org/software/guix/manual/en/html_node/Defining-Packages.html][package reference in the manual]]

- [[https://gitlab.com/pjotrp/guix-notes/blob/master/HACKING.org][Pjotr’s hacking guide to GNU Guix]]

- "Guix Guix: Package without a scheme!", by Andreas Enge (in
  =guix-maintenance.git/talks/ghm-2013/andreas/slides.pdf=)

* About GNU Guix

[[https://www.gnu.org/software/guix][GNU Guix]] is a transactional package manager for the GNU system.  The Guix System
Distribution or GuixSD is an advanced distribution of the GNU system that relies
on GNU Guix and [[https://www.gnu.org/distros/free-system-distribution-guidelines.html][respects the user's freedom]].

In addition to standard package management features, Guix supports transactional
upgrades and roll-backs, unprivileged package management, per-user profiles, and
garbage collection.  Guix uses low-level mechanisms from the Nix package
manager, except that packages are defined as native [[https://www.gnu.org/software/guile][Guile]] modules, using
extensions to the [[http://schemers.org][Scheme]] language.  GuixSD offers a declarative approach to
operating system configuration management, and is highly customizable and
hackable.

GuixSD can be used on an i686, x86_64 and armv7 machines.  It is also possible
to use Guix on top of an already installed GNU/Linux system, including on
mips64el and aarch64.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Blog: Guix packaging tutorial

[Ludovic Courtès]

Hello Pierre,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> Find the last revision attached.  I've taken all comments into
> consideration and I'm quite happy with the result.

Neat!  I like it too, I’d say we can go frmo here.

> I've found a much better advanced example: libgit2.
> The only thing is... it does not pass the tests.

Heh.  :-)

> In guix, libgit2 uses an archive upstream.  I've changed the package to
> check out the git repository.  Only then the test fail:
>
> starting phase `check'
> Loaded 341 suites: 
> Started
>
> apply::fromdiff...................
> apply::fromfile..........................................
> attr::file....
> attr::flags...
> attr::ignoreF
>
>   1) Failure:
> attr::ignore::honor_temporary_rules [/tmp/guix-build-my-libgit2-0.26.6-1.e98d0a3.drv-0/my-libgit2-0.26.6-1.e98d0a3-checkout/tests/clar_libgit2.c:47]
>   Expression is not true: (fd = p_open(path, flags, mode)) >= 0

Could it be because the file it tries to open here is read-only?

Usually files resulting from a checkout are read-only, so perhaps you
need to make a bunch of files writable before the ‘check’ phase.  For a
start you could do:

  (for-each make-file-writable (find-files "tests"))

Once you’ve sorted that out I’m happy to publish if there are no
objections!

Thanks,
Ludo’.

Re: Blog: Guix packaging tutorial

[Ricardo Wurmus]

Hi Pierre,

> Find the last revision attached.  I've taken all comments into
> consideration and I'm quite happy with the result.

I like it!  Thanks for your patience.

Here are some more comments:

* In the section “A "Hello World" package” you use the word
  “commandline”, but it should be “command line”.  (Same in the Scheme
  tutorial and other places throughout.)

* The sentence: “As with the ritualistic "Hello World" taught with most
  programming language, […]” should say “programming languages”.

* I wonder if maybe we should avoid using “guix package” for the first
  example and use “guix build” instead.  The reason is that “guix
  package” might cause additional builds to be performed dependent on
  the state of the user’s default profile (because of profile hooks).
  Using “guix build” bypasses this issue completely.  What do you think?

* Missing period after “(or /s-expression/ in Lisp lingo)”.

* “It can be a literal[…]” –> “An expression can be a literal[…]”,
  because in the previous sentence we used the plural “expressions”.

* “Every function returns the last evaluated expression as value.” –>
  “Every function returns the value of the last evaluated expression.”

* “Use ~let*~ to re-allows the initializers of later variables to refer
  to the earlier variables.” –> “Use ~let*~ to allow later variable
  declarations to refer to earlier definitions.”

* “Create a folder” –> “Create a directory”; “folder” is a GUI metaphor,
  but that’s nitpicking to be fair.  I prefer “directory”, but use
  “folder” if you find it better :)

* “Notice that we have wrapped the package into a ~define-public~
  record” – I would not use the term “record” here, because it could be
  confused with Scheme records.  How about “Note that we have assigned
  the package value to an exported variable name with ~define-public~.”
  You’re saying as much in the following sentence, but I think it would
  be better to avoid the words “warp” and “record” here.

* “$GUIX_CHECKOUT” is not defined anywhere.  You reference it in the
  introduction before telling people to clone the git repository:

   “license :: See =$GUIX_CHECKOUT/guix/licenses.scm= for a full list.”

  I think it would be best to remove any mention of this from the
  introduction.  After telling the readers to “git clone” I think the
  meaning of $GUIX_CHECKOUT would be obvious, but earlier it is not.

* Why use “ruby” as an example after checking out?  Should this be
  mentioned e.g. “Search packages, such as Ruby:” to show that this is
  arbitrary?

* “Let's looks at another” –> “Let's look at another”

* This paragraph is a little difficult to understand for me:

     When downloaded to the store, the source should be different from
     version to version.  Since Git repositories don't contain a
     specific version in their name, we've got to force the download
     folder using ~(file-name (git-file-name name version))~.

  How about this:

     To ensure that the source code from the git repository is stored in
     a unique directory with a readable name we use ~(file-name
     (git-file-name name version))~.

* “Snippets are quoted (i.e. non-evaluated) Scheme code that are a mean
  of patching the source.”  –> “…a means of patching…”

* “native-inputs :: Required for building but not runtime -- installing
                   a package through a substitute won't install these
                   inputs.”

  This is usually true, but the meaning of “native-inputs” is primarily
  important for cross-building packages.  A native input will be used on
  the build machine and should thus be available for the build machine’s
  architecture, not for the target machine.  Since Guix scans for
  references, packages that are only needed at build time will not be
  installed even when they are among the plain “inputs”.  Likewise,
  packages that are in the “native-inputs” may end up leaving a
  reference behind, which will cause them to be downloaded.  (This is
  often a bug, such as in the icedtea packages, which keep unwanted
  references.)

  Not sure if we should change this in the tutorial, because it’s a
  finer point.  The distinction between inputs and propagated-inputs is
  more important and I think it’s done well in your draft.

* “concern to the package, no to the user” –> “…not to the user”

* “The end user can choose” –> I’d prefer just “user” instead of “end
  user”.

* “It's advised to separate outputs only when you've shown it worth:” –>
  “…it’s worth it:”.

* In the configure-flags example I would not use quasiquotation, so
  this:

    #:configure-flags `(,(string-append …

  would become this:

    #:configure-flags (list (string-append …

* “Or from the REPL (assuming the Guix source is in your Guile load
  path):” –> you could remove the assumption by adding an
  add-to-load-path expression to the example.

* “consult the associated functions” –> nitpick: these are “procedures”
  not “functions”.

* The segment starting with “To manipulate the phases,” seems incomplete
  as it does not introduce the “modify-phases” macro itself.

* “this is needed to avoid clashes between, say, the =zlib= variable
  from =licenses.scm= and the =zlib= variable from =compression.scm=.” —
  I suggest adding parenthetical remarks after the modules to clarify
  that these are different types: “(a license value)” and “(a package
  value)”.

* “…process is very similar to the GNU build system but for a few
  specialized arguments.” –> Maybe replace “but” with “except”.e

* “Find more about build systems in…” –> Maybe replace “Find” with
  “Learn” or “Find out”.

* I don’t know if there should be a section “Common pitfalls and best
  practices” other than to point to the checklist in the manual.  If the
  list in the manual misses one of the points I’d rather add them there
  and only link to the list.

* The section on “Recursive importers” does not mention any recursive
  importers :)  How about using the CRAN importer here, which supports
  recursive importing with the “-r” option?

* “All unspecified fields inherit from the parent package” – I think
  “are inherited from” would be better.

* “Sadly some applications can be tough to package” — missing comma
  after “Sadly”.

* I find this sentence a bit confusing: “The GNU build system is the
  flagship of the high-level abstraction layers Guix makes possible to
  build.”  — what does it mean to “build … an abstraction layer”?

Finally, I think the conclusion is a bit anti-climatic :)  I feel like
it misses a sentence after presenting the outlook to the future; maybe
an announcement of a follow-up blog post?  Or encouragement to get
started packaging today?

I’m very happy to see this tutorial so close to being published!  Thanks
again for your efforts!

--
Ricardo

Re: Blog: Guix packaging tutorial

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 4084 bytes --]


ludo@gnu.org (Ludovic Courtès) writes:
> Usually files resulting from a checkout are read-only, so perhaps you
> need to make a bunch of files writable before the ‘check’ phase.  For a
> start you could do:
>
>   (for-each make-file-writable (find-files "tests"))

Damn, bitten by this again!  It's not the first time and it seems that I fail to
learn from my mistakes :p

I've modified the 'unpack phase instead to avoid changing directory:

--8<---------------cut here---------------start------------->8---
           (add-after 'unpack 'make-files-writable-for-tests
               (lambda _ (for-each make-file-writable (find-files "."
           ".*")))))))
--8<---------------cut here---------------end--------------->8---

It works now, thanks!

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> writes:
> * I wonder if maybe we should avoid using “guix package” for the first
>   example and use “guix build” instead.  The reason is that “guix
>   package” might cause additional builds to be performed dependent on
>   the state of the user’s default profile (because of profile hooks).
>   Using “guix build” bypasses this issue completely.  What do you think?

Hmm... Installing a package is something more familiar than just "building".
Besides, with the new "cleaner" output, it's only a few lines which the user is
going to see all the time, so I don't think there is much complexity we can
trim here.

If we first build, then package, it makes the "easy" example an extra step more
complicated.

> * “Create a folder” –> “Create a directory”; “folder” is a GUI metaphor,
>   but that’s nitpicking to be fair.  I prefer “directory”, but use
>   “folder” if you find it better :)

Funny, I did not know that :p What is the GUI metaphor precisely?  I personally
prefer "folder" for the simple reason that it's shorter, but I don't mind
changing ;)

>   This is usually true, but the meaning of “native-inputs” is primarily
>   important for cross-building packages.  A native input will be used on
>   the build machine and should thus be available for the build machine’s
>   architecture, not for the target machine.  Since Guix scans for
>   references, packages that are only needed at build time will not be
>   installed even when they are among the plain “inputs”.  Likewise,
>   packages that are in the “native-inputs” may end up leaving a
>   reference behind, which will cause them to be downloaded.  (This is
>   often a bug, such as in the icedtea packages, which keep unwanted
>   references.)
>
>   Not sure if we should change this in the tutorial, because it’s a
>   finer point.  The distinction between inputs and propagated-inputs is
>   more important and I think it’s done well in your draft.

Indeed, I think it's too complicated at this stage.  Case in point, your
paragraph is quite long ;)  I'd rather skip this.

> * I don’t know if there should be a section “Common pitfalls and best
>   practices” other than to point to the checklist in the manual.  If the
>   list in the manual misses one of the points I’d rather add them there
>   and only link to the list.

I agree.  I'll merge this list with the manual then.

> * The section on “Recursive importers” does not mention any recursive
>   importers :)  How about using the CRAN importer here, which supports
>   recursive importing with the “-r” option?

Hmm... I can't get this to work.

--8<---------------cut here---------------start------------->8---
guix import cran --recursive lubridate
--8<---------------cut here---------------end--------------->8---

returns only one package.  In fact, the "-r,--recursive" flag is missing from
the `guix import cran -h` documentation (while it's there for pipy).  I cannot
get it to work for pipy either.

Do you have a working example?

I've applied all you other suggestions.  Thank you Ricardo for this second
thorough review!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Blog: Guix packaging tutorial

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 156 bytes --]

Oh, and is it customary to add a "Special thanks" section?
I'd like to thank Ludovic and Ricardo there :)

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Blog: Guix packaging tutorial

[Ricardo Wurmus]

Hi Pierre,

> Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> writes:
>> * I wonder if maybe we should avoid using “guix package” for the first
>>   example and use “guix build” instead.  The reason is that “guix
>>   package” might cause additional builds to be performed dependent on
>>   the state of the user’s default profile (because of profile hooks).
>>   Using “guix build” bypasses this issue completely.  What do you think?
>
> Hmm... Installing a package is something more familiar than just "building".
> Besides, with the new "cleaner" output, it's only a few lines which the user is
> going to see all the time, so I don't think there is much complexity we can
> trim here.
>
> If we first build, then package, it makes the "easy" example an extra step more
> complicated.

I assumed we would just build the package, not build and install.  When
packaging something, the common thing to do is to build, change, build
again, … and in the end maybe install (or submit to the mailing list).

>> * “Create a folder” –> “Create a directory”; “folder” is a GUI metaphor,
>>   but that’s nitpicking to be fair.  I prefer “directory”, but use
>>   “folder” if you find it better :)
>
> Funny, I did not know that :p What is the GUI metaphor precisely?  I personally
> prefer "folder" for the simple reason that it's shorter, but I don't mind
> changing ;)

A directory is a file system concept: a special kind of file that is a
directory of files.  IIRC the term “folder” was introduced with Windows
and the metaphor is that of a physical folder to store paper documents;
it came together with the icon of a cardboard folder, which is still
used in modern GUIs.

>> * The section on “Recursive importers” does not mention any recursive
>>   importers :)  How about using the CRAN importer here, which supports
>>   recursive importing with the “-r” option?
>
> Hmm... I can't get this to work.
>
> --8<---------------cut here---------------start------------->8---
> guix import cran --recursive lubridate
> --8<---------------cut here---------------end--------------->8---
>
> returns only one package.

That’s because the recursive importer won’t import packages for which
Guix already has package definitions (except for the very first).
“r-lubridate” already exists, and so do all its dependencies.

Currently, “walrus” and some of its dependencies have not been packaged
yet, so this command will give you representative output:

    guix import cran -r walrus

> In fact, the "-r,--recursive" flag is missing from
> the `guix import cran -h` documentation (while it's there for pipy).  I cannot
> get it to work for pipy either.

Oh, that’s a bug then.  We should fix that.

--
Ricardo

Re: Blog: Guix packaging tutorial

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 831 bytes --]

> I assumed we would just build the package, not build and install.  When
> packaging something, the common thing to do is to build, change, build
> again, … and in the end maybe install (or submit to the mailing list).

Maybe you are right.  I'll see if I can adjust this.

> Finally, I think the conclusion is a bit anti-climatic :)  I feel like
> it misses a sentence after presenting the outlook to the future; maybe
> an announcement of a follow-up blog post?  Or encouragement to get
> started packaging today?

I don't want to announce a follow-up blog post because I'm very busy at the
moment and I don't want to promise something I'm not sure I can offer.
That said, you are right, I forget to edit the conclusion and it feels
unfinished.  I'll work on it.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Blog: Guix packaging tutorial

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 659 bytes --]


> > I assumed we would just build the package, not build and install.  When
> > packaging something, the common thing to do is to build, change, build
> > again, … and in the end maybe install (or submit to the mailing list).
> 
> Maybe you are right.  I'll see if I can adjust this.

Hmm... The problem is that I give example on how to see the effect of installing
a package.  Actually I think a good tutorial should first give an example of
something to _play_ with, so it needs to have some immediate effect, that is, to
be be installed to the user profile.

I'll stick to that if that's OK.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Blog: Guix packaging tutorial

[Benjamin Slade]

This will be extremely helpful, Pierre! Thanks for putting it together.

I looked at an early version of this; I don't know if it has changed
since, but one thing that would be useful to a brief checklist of steps
outlined somewhere. 

 —Ben

-- 
Benjamin Slade - https://babbagefiles.xyz
  `(pgp_fp: ,(21BA 2AE1 28F6 DF36 110A 0E9C A320 BBE8 2B52 EE19))
    '(sent by mu4e on Emacs running under GNU/Linux . https://gnu.org )
       `(Choose Linux ,(Choose Freedom) . https://linux.com )

Re: Blog: Guix packaging tutorial

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1655 bytes --]

Hi Ben!

I've sent two versions so far and I'm going to send the final version just now.

As for the checklists: hmm... Which steps do you have in mind?  The trick is
that the is little to repeat once the Git checkout has been set up.
Pjotr has a checklist however, let me know what you think:

From https://gitlab.com/pjotrp/guix-notes/blob/master/HACKING.org:

--8<---------------cut here---------------start------------->8---
Workflow for packaging

The general workflow for adding and maintaining packages is a bit
  complex. Everything goes via de guix-dev mailing list and includes a
  review process which can be discouraging and is more geared towards
  reviewers than towards newbies. This should not discourage you because
  GNU Guix is great. Note that the reviewers do this work voluntarily
  and most ‘rules’ have been agreed by the community. In the end your
  labours will get rewarded. So, how to start?

 1 - Work on a recent git checkout of guix
 2 - Use ‘guix import’ if you can (e.g. for python and R modules)
 3 - Build the package yourself
 4 - If tests are there, make sure they pass
 5 - Test your work with ‘guix lint’
 6 - Create a patch as described above
 7 - Send it to debbugs as described above
 8 - Submit one patch at a time and submit the next one when it goes in
 9 - Be patient, review can take a while - monitor debbugs
--8<---------------cut here---------------end--------------->8---

I'm not sure those are exactly steps, while some items actually belong the
manual's check list "(guix) Submitting Patches".

Opinions?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Blog: Guix packaging tutorial

[Pierre Neidhardt]

[-- Attachment #1.1: Type: text/plain, Size: 75 bytes --]

And here is the last version, in Markdown (hopefully the syntax is right).

[-- Attachment #1.2: index.md --]
[-- Type: text/plain, Size: 43901 bytes --]

title: A packaging tutorial for Guix
date: 2018-10-09
author: Pierre Neidhardt
tags: Software development, Programming interfaces, Scheme API
---


# Introduction

GNU Guix stands out as the *hackable* package manager, mostly because it uses
[GNU Guile](https://www.gnu.org/software/guile/), a powerful high-level programming language, one of the [Scheme](https://en.wikipedia.org/wiki/Scheme_(programming_language))
dialects from the [Lisp family](https://en.wikipedia.org/wiki/Lisp_(programming_language)).

Package definitions are also written in Scheme, which empowers Guix in some very
unique ways, unlike most other package managers that use shell scripts or
simple languages.

-   Use functions, structures, macros and all of Scheme expressiveness for your
    package definitions.

-   Inheritance makes it easy to customize a package by inheriting from it and
    modifying only what is needed.

-   Batch processing: the whole package collection can be parsed, filtered and
    processed.  Building a headless server with all graphical interfaces stripped
    out?  It's possible.  Want to rebuild everything from source using specific
    compiler optimization flags?  Pass the `#:make-flags "..."` argument to the
    list of packages.  It wouldn't be a stretch to think [Gentoo USE flags](https://wiki.gentoo.org/wiki/USE_flag) here,
    but this goes even further: the changes don't have to be thought out
    beforehand by the packager, they can be *programmed* by the user!

The following tutorial covers all the basics around package creation with Guix.
It does not assume much knowledge of the Guix system nor of the Lisp language.
The reader is only expected to be familiar with the command line and to have some
basic programming knowledge.


# A "Hello World" package

The [Defining Packages section of the manual](https://www.gnu.org/software/guix/manual/en/html_node/Defining-Packages.html) introduces the basics of Guix
packaging.  In the following section, we will partly go over those basics again.

`GNU hello` is a dummy project that serves as an idiomatic example for
packaging.  It uses the GNU build system (`./configure && make && make install`).
Guix already provides a package definition which is a perfect example to start
with.  You can look up its declaration with `guix edit hello` from the
command line.  Let's see how it looks:

```scheme
    (define-public hello
      (package
        (name "hello")
        (version "2.10")
        (source (origin
                  (method url-fetch)
                  (uri (string-append "mirror://gnu/hello/hello-" version
                                      ".tar.gz"))
                  (sha256
                   (base32
                    "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
        (build-system gnu-build-system)
        (synopsis "Hello, GNU world: An example GNU package")
        (description
         "GNU Hello prints the message \"Hello, world!\" and then exits.  It
    serves as an example of standard GNU coding practices.  As such, it supports
    command-line arguments, multiple languages, and so on.")
        (home-page "https://www.gnu.org/software/hello/")
        (license gpl3+)))
```

As you can see, most of it is rather straightforward.  But let's review the
fields together:

-   **name:** The project name.  Using Scheme conventions, we prefer to keep it
    lower case, without underscore and using dash-separated words.
-   **source:** This field contains a description of the source code origin.  The
    `origin` record contains these fields:
    1.  The method, here `url-fetch` to download via HTTP/FTP, but other methods
        exist, such as `git-fetch` for Git repositories.
    2.  The URI, which is typically some `https://` location for `url-fetch`.  Here
        the special `mirror://gnu` refers to a set of well known locations, all of
        which can be used by Guix to fetch the source, should some of them fail.
    3.  The `sha256` checksum of the requested file.  This is essential to ensure
        the source is not corrupted.  Note that Guix works with base32 strings,
        hence the call to the `base32` function.
-   **build-system:** This is where the power of abstraction provided by the Scheme
    language really shines: in this case, the `gnu-build-system`
    abstracts away the famous `./configure && make && make
                      install` shell invocations.  Other build systems include the
    `trivial-build-system` which does not do anything and requires
    from the packager to program all the build steps, the
    `python-build-system`, the `emacs-build-system`, [and many
    more](https://www.gnu.org/software/guix/manual/en/html_node/Build-Systems.html).
-   **synopsis:** It should be a concise summary of what the package
    does.  For many packages a tagline from the
    project's home page can be used as the synopsis.
-   **description:** Same as for the synopsis, it's fine to re-use the project
    description from the homepage.  Note that Guix uses Texinfo
    syntax.
-   **home-page:** Use HTTPS if available.
-   **license:** See `guix/licenses.scm` in the project source for a full list.

Time to build our first package!  Nothing fancy here for now: we will stick to a
dummy "my-hello", a copy of the above declaration.

As with the ritualistic "Hello World" taught with most programming languages,
this will possibly be the most "manual" approach.  We will work out an ideal
setup later; for now we will go the simplest route.

Save the following to a file `my-hello.scm`.

```scheme
    (use-modules (guix packages)
                 (guix download)
                 (guix build-system gnu)
                 (guix licenses))

    (package
      (name "my-hello")
      (version "2.10")
      (source (origin
                (method url-fetch)
                (uri (string-append "mirror://gnu/hello/hello-" version
                                    ".tar.gz"))
                (sha256
                 (base32
                  "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
      (build-system gnu-build-system)
      (synopsis "Hello, Guix world: An example custom Guix package")
      (description
       "GNU Hello prints the message \"Hello, world!\" and then exits.  It
    serves as an example of standard GNU coding practices.  As such, it supports
    command-line arguments, multiple languages, and so on.")
      (home-page "https://www.gnu.org/software/hello/")
      (license gpl3+))
```

We will explain the extra code in a moment.

Feel free to play with the different values of the various fields.  If you
change the source, you'll need to update the checksum.  Indeed, Guix refuses to
build anything if the given checksum does not match the computed checksum of the
source code.  To obtain the correct checksum of the package declaration, we
need to download the source, compute the sha256 checksum and convert it to
base32.

Thankfully, Guix can automate this task for us; all we need is to provide the
URI:

```sh
    $ guix download mirror://gnu/hello/hello-2.10.tar.gz

    Starting download of /tmp/guix-file.JLYgL7
    From https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz...
    following redirection to `https://mirror.ibcp.fr/pub/gnu/hello/hello-2.10.tar.gz'...
     …10.tar.gz  709KiB                                 2.5MiB/s 00:00 [##################] 100.0%
    /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
    0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i
```

Note in this specific case that the output tells us which mirror was chosen.

If the result of the above command is not the same as in the above snippet,
update your `my-hello` declaration accordingly.

Now you can happily run

```sh
    $ guix package --install-from-file=my-hello.scm
```

You should now have `my-hello` in your profile!

```sh
    $ guix package --list-installed=my-hello
    my-hello	2.10	out
    /gnu/store/f1db2mfm8syb8qvc357c53slbvf1g9m9-my-hello-2.10
```

We've gone as far as we could without any knowledge of Scheme.  Now is the right
time to introduce the minimum we need from the language before we can proceed.


# A Scheme crash-course

As we've seen above, basic packages don't require much Scheme knowledge, if none
at all.  But as you progress and your desire to write more and more complex
packages grows, it will become both necessary and empowering to hone your Lisper
skills.

Since an extensive Lisp course is very much out of the scope of this tutorial,
we will only cover some basics here.

Guix uses the Guile implementation of Scheme.  To start playing with the
language, install it with `guix package --install guile` and start a [REPL](https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop) by
running `guile` from the command line.

Alternatively you can also run `guix environment --ad-hoc guile -- guile` if
you'd rather not have Guile installed in your user profile.

In the following examples we use the `>` symbol to denote the REPL prompt, that
is, the line reserved for user input.  See [the Guile manual](https://www.gnu.org/software/guile/manual/html_node/Using-Guile-Interactively.html) for more details on
the REPL.

-   Scheme syntax boils down to a tree of expressions (or *s-expression* in Lisp
    lingo).  An expression can be a literal such numbers and strings, or a
    compound which is a parenthesized list of compounds and literals.  `#t` and
    `#f` stand for the booleans "true" and "false", respectively.

    Examples of valid expressions:

        > "Hello World!"
        "Hello World!"
        > 17
        17
        > (display (string-append "Hello " "Guix" "\n"))
        "Hello Guix!"

-   This last example is a function call embedded in another function call.  When
    a parenthesized expression is evaluated, the first term is the function and
    the rest are the arguments passed to the function.  Every function returns the
    last evaluated expression as value.

-   Anonymous functions are declared with the `lambda` term:

        > (lambda (x) (* x x))
        #<procedure 120e348 at <unknown port>:24:0 (x)>

    The above lambda returns the square of its argument.  Since everything is an
    expression, the `lambda` expression returns an anonymous function, which can
    in turn be applied to an argument:

        > ((lambda (x) (* x x)) 3)
        9

-   Anything can be assigned a global name with `define`:

        > (define a 3)
        > (define square (lambda (x) (* x x)))
        > (square a)
        9

-   Procedures can be defined more concisely with the following syntax:

        (define (square x) (* x x))

-   A list structure can be created with the `list` procedure:

        > (list 2 a 5 7)
        (2 3 5 7)

-   The *quote* disables evaluation of a parenthesized expression: the first term
    is not called over the other terms.  Thus it effectively returns a list of
    terms.

        > '(display (string-append "Hello " "Guix" "\n"))
        (display (string-append "Hello " "Guix" "\n"))
        > '(2 a 5 7)
        (2 a 5 7)

-   The *quasiquote* disables evaluation of a parenthesized expression until a
    colon re-enables it.  Thus it provides us with fine-grained control over what
    is evaluated and what is not.

        > `(2 a 5 7 (2 ,a 5 ,(+ a 4)))
        (2 a 5 7 (2 3 5 7))

    Note that the above result is a list of mixed elements: numbers, symbols (here
    `a`) and the last element is a list itself.

-   Multiple variables can be named locally with `let`:

        > (define x 10)
        > (let ((x 2)
                (y 3))
            (list x y))
        (2 3)
        > x
        10
        > y
        ERROR: In procedure module-lookup: Unbound variable: y

    Use `let*` to allow later variable declarations to refer to earlier
    definitions.

        > (let* ((x 2)
                 (y (* x 3)))
            (list x y))
        (2 6)

-   The keyword syntax is `#:`, it is used to create unique identifiers.  See also
    the [Keywords section in the Guile manual](https://www.gnu.org/software/guile/manual/html_node/Keywords.html).

-   The percentage `%` is typically used for read-only global variables in the
    build stage.  Note that it is merely a convention, like `_` in C.  Scheme Lisp
    treats `%` exactly the same as any other letter.

-   Modules are created with `define-module`.  For instance

        (define-module (guix build-system ruby)
          #:use-module (guix store)
          #:export (ruby-build
                    ruby-build-system))

    defines the module `ruby` which must be located in
    `guix/build-system/ruby.scm` somewhere in `GUILE_LOAD_PATH`.  It depends on
    the `(guix store)` module and it exports two symbols, `ruby-build` and
    `ruby-build-system`.

For a more detailed introduction, check out [Scheme at a Glance](http://www.troubleshooters.com/codecorn/scheme_guile/hello.htm), by Steve Litt.

One of the reference Scheme books is the seminal *Structure and Interpretation
of Computer Programs*, by Harold Abelson and Gerald Jay Sussman, with Julie
Sussman.  You'll find a free copy [online](https://mitpress.mit.edu/sites/default/files/sicp/index.html), together with [videos of the lectures
by the authors](https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-001-structure-and-interpretation-of-computer-programs-spring-2005/video-lectures/).  The book is available in Texinfo format as the `sicp` Guix
package.  Go ahead, run `guix package --install sicp` and start reading with
`info sicp` (or with the Emacs Info reader).  An unofficial ebook [is also
available](https://sarabander.github.io/sicp/).

You'll find more books, tutorials and other resources at <https://schemers.org/>.


# Setup

Now that we know some Scheme basics we can detail the different possible setups
for working on Guix packages.

There are several ways to set up a Guix packaging environment.

We recommend you work directly on the Guix source checkout since it makes it
easier for everyone to contribute to the project.

But first, let's look at other possibilities.


### Local file

This is what we previously did with `my-hello`.  Now that we know more Scheme,
let's explain the leading chunks.  As stated in `guix package --help`:

```scheme
    -f, --install-from-file=FILE
                           install the package that the code within FILE
                           evaluates to
```

Thus the last expression *must* return a package, which is the case in our
earlier example.

The `use-modules` expression tells which of the modules we need in the file.
Modules are a collection of values and procedures.  They are commonly called
"libraries" or "packages" in other programming languages.


### GUIX_PACKAGE_PATH

*Note: Starting from Guix 0.16, the more flexible Guix "channels" are the
preferred way and supersede `GUIX_PACKAGE_PATH`.  See below.*

It can be tedious to specify the file from the command line instead of simply
calling `guix package --install my-hello` as you would do with the official
packages.

Guix makes it possible to streamline the process by adding as many "package
declaration paths" as you want.

Create a directory, say `~./guix-packages` and add it to the `GUIX_PACKAGE_PATH`
environment variable:

```sh
    $ mkdir ~/guix-packages
    $ export GUIX_PACKAGE_PATH=~/guix-packages
```

To add several directories, separate them with a colon (`:`).

Our previous `my-hello` needs some adjustments though:

```scheme
    (define-module (my-hello)
      #:use-module (guix licenses)
      #:use-module (guix packages)
      #:use-module (guix build-system gnu)
      #:use-module (guix download))

    (define-public my-hello
      (package
        (name "my-hello")
        (version "2.10")
        (source (origin
                  (method url-fetch)
                  (uri (string-append "mirror://gnu/hello/hello-" version
                                      ".tar.gz"))
                  (sha256
                   (base32
                    "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
        (build-system gnu-build-system)
        (synopsis "Hello, Guix world: An example custom Guix package")
        (description
         "GNU Hello prints the message \"Hello, world!\" and then exits.  It
    serves as an example of standard GNU coding practices.  As such, it supports
    command-line arguments, multiple languages, and so on.")
        (home-page "https://www.gnu.org/software/hello/")
        (license gpl3+)))
```

Note that we have assigned the package value to an exported variable name with
`define-public`.  This is effectively assigning the package to the `my-hello`
variable so that it can be referenced, among other as dependency of other
packages.

If you use `guix package --install-from-file=my-hello.scm` on the above file, it
will fail because the last expression, `define-public`, does not return a
package.  If you want to use `define-public` in this use-case nonetheless, make
sure the file ends with an evaluation of `my-hello`:

```scheme
    ; ...
    (define-public my-hello
      ; ...
      )

    my-hello
```

This last example is not very typical.

Now `my-hello` should be part of the package collection like all other official
packages.  You can verify this with:

```sh
    $ guix package --show=my-hello
```


### Guix channels

Guix 0.16 features channels, which is very similar to `GUIX_PACKAGE_PATH` but
provides better integration and provenance tracking.  Channels are not
necessarily local, they can be maintained as a public Git repository for
instance.  Of course, several channels can be used at the same time.

See the [Channel secion in the manual](http://guix.info/manual/en/Channels.html) for setup details.


### Direct checkout hacking

Working directly on the Guix project is recommended: it reduces the friction
when the time comes to submit your changes upstream to let the community benefit
from your hard work!

Unlike most software distributions, the Guix repository holds in one place both
the tooling (including the package manager) and the package definitions.  This
choice was made so that it would give developers the flexibility to modify the
API without breakage by updating all packages at the same time.  This reduces
development inertia.

Check out the official [Git](https://git-scm.com/) repository:

```sh
    $ git clone https://git.savannah.gnu.org/git/guix.git
```

In the rest of this article, we use `$GUIX_CHECKOUT` to refer to the location of
the checkout.

Follow the instruction from the ["Contributing" chapter](https://www.gnu.org/software/guix/manual/en/html_node/Contributing.html) in the manual to set up the
repository environment.

Once ready, you should be able to use the package definitions from the
repository environment.

Feel free to edit package definitions found in `$GUIX_CHECKOUT/gnu/packages`.

The `$GUIX_CHECKOUT/pre-inst-env` script lets you use `guix` over the package
collection of the repository.

-   Search packages, such as Ruby:

        $ cd $GUIX_CHECKOUT
        $ ./pre-inst-env guix package --list-available=ruby
            ruby    1.8.7-p374      out     gnu/packages/ruby.scm:119:2
            ruby    2.1.6   out     gnu/packages/ruby.scm:91:2
            ruby    2.2.2   out     gnu/packages/ruby.scm:39:2

-   Build a package, here Ruby version 2.1:

        $ ./pre-inst-env guix build --keep-failed ruby@2.1
        /gnu/store/c13v73jxmj2nir2xjqaz5259zywsa9zi-ruby-2.1.6

-   Install it to your user profile:

        $ ./pre-inst-env guix package --install ruby@2.1

-   Check for common mistakes:

        $ ./pre-inst-env guix lint ruby@2.1

Guix strives at maintaining a high packaging standard; when contributing to the
Guix project, remember to

-   follow the [coding style](https://www.gnu.org/software/guix/manual/en/html_node/Coding-Style.html),
-   and review the [check list](https://www.gnu.org/software/guix/manual/en/html_node/Submitting-Patches.html) from the manual.

Once you are happy with the result, you are welcome to send your contribution to
make it part of Guix.  This process is also detailed in the [manual](https://www.gnu.org/software/guix/manual/en/html_node/Contributing.html).

It's a community effort so the more join in, the better Guix becomes!


# Extended example

The above "Hello World" example is as simple as it goes.  Packages can be more
complex than that and Guix can handle more advanced scenarios.  Let's look at
another, more sophisticated package (slightly modified from the source):

```scheme
    (define-module (gnu packages version-control)
      #:use-module ((guix licenses) #:prefix license:)
      #:use-module (guix utils)
      #:use-module (guix packages)
      #:use-module (guix git-download)
      #:use-module (guix build-system cmake)
      #:use-module (gnu packages ssh)
      #:use-module (gnu packages web)
      #:use-module (gnu packages pkg-config)
      #:use-module (gnu packages python)
      #:use-module (gnu packages compression)
      #:use-module (gnu packages tls))

    (define-public my-libgit2
      (let ((commit "e98d0a37c93574d2c6107bf7f31140b548c6a7bf")
            (revision "1"))
        (package
          (name "my-libgit2")
          (version (git-version "0.26.6" revision commit))
          (source (origin
                    (method git-fetch)
                    (uri (git-reference
                          (url "https://github.com/libgit2/libgit2/")
                          (commit commit)))
                    (file-name (git-file-name name version))
                    (sha256
                     (base32
                      "17pjvprmdrx4h6bb1hhc98w9qi6ki7yl57f090n9kbhswxqfs7s3"))
                    (patches (search-patches "libgit2-mtime-0.patch"))
                    (modules '((guix build utils)))
                    (snippet '(begin
                                ;; Remove bundled software.
                                (delete-file-recursively "deps")
                                #t))))
          (build-system cmake-build-system)
          (outputs '("out" "debug"))
          (arguments
           `(#:tests? #t                            ; Run the test suite (this is the default)
             #:configure-flags '("-DUSE_SHA1DC=ON") ; SHA-1 collision detection
             #:phases
             (modify-phases %standard-phases
               (add-after 'unpack 'fix-hardcoded-paths
                 (lambda _
                   (substitute* "tests/repo/init.c"
                     (("#!/bin/sh") (string-append "#!" (which "sh"))))
                   (substitute* "tests/clar/fs.h"
                     (("/bin/cp") (which "cp"))
                     (("/bin/rm") (which "rm")))
                   #t))
               ;; Run checks more verbosely.
               (replace 'check
                 (lambda _ (invoke "./libgit2_clar" "-v" "-Q")))
               (add-after 'unpack 'make-files-writable-for-tests
                   (lambda _ (for-each make-file-writable (find-files "." ".*")))))))
          (inputs
           `(("libssh2" ,libssh2)
             ("http-parser" ,http-parser)
             ("python" ,python-wrapper)))
          (native-inputs
           `(("pkg-config" ,pkg-config)))
          (propagated-inputs
           ;; These two libraries are in 'Requires.private' in libgit2.pc.
           `(("openssl" ,openssl)
             ("zlib" ,zlib)))
          (home-page "https://libgit2.github.com/")
          (synopsis "Library providing Git core methods")
          (description
           "Libgit2 is a portable, pure C implementation of the Git core methods
    provided as a re-entrant linkable library with a solid API, allowing you to
    write native speed custom Git applications in any language with bindings.")
          ;; GPLv2 with linking exception
          (license license:gpl2))))
```

(In those cases were you only want to tweak a few fields from a package
definition, you should rely on inheritance instead of copy-pasting everything.
See below.)

Let's discuss those fields in depth.


## `git-fetch` method

Unlike the `url-fetch` method, `git-fetch` expects a `git-reference` which takes
a Git repository and a commit.  The commit can be any Git reference such as
tags, so if the `version` is tagged, then it can be used directly.  Sometimes
the tag is prefixed with a `v`, in which case you'd use `(commit (string-append
"v" version))`.

To ensure that the source code from the Git repository is stored in a unique
directory with a readable name we use `(file-name (git-file-name name
version))`.

Note that there is also a `git-version` procedure that can be used to derive the
version when packaging programs for a specific commit.


## Snippets

Snippets are quoted (i.e. non-evaluated) Scheme code that are a means of patching
the source.  They are a Guix-y alternative to the traditional `.patch` files.
Because of the quote, the code in only evaluated when passed to the Guix daemon
for building.

There can be as many snippet as needed.

Snippets might need additional Guile modules which can be imported from the
`modules` field.


## Inputs

First, a syntactic comment: See the quasi-quote / comma syntax?

```scheme
    (native-inputs
     `(("pkg-config" ,pkg-config)))
```

is equivalent to

```scheme
    (native-inputs
     (list (list "pkg-config" pkg-config)))
```

You'll mostly see the former because it's shorter.

There are 3 different input types.  In short:

-   **native-inputs:** Required for building but not runtime &#x2013; installing a package
    through a substitute won't install these inputs.
-   **inputs:** Installed in the store but not in the profile, as well as being
    present at build time.
-   **propagated-inputs:** Installed in the store and in the profile, as well as
    being present at build time.

See [the package reference in the manual](https://www.gnu.org/software/guix/manual/en/html_node/package-Reference.html) for more details.

The distinction between the various inputs is important: if a dependency can be
handled as an *input* instead of a *propagated input*, it should be done so, or
else it "pollutes" the user profile for no good reason.

For instance, a user installing a graphical program that depends on a
command line tool might only be interested in the graphical part, so there is no
need to force the command line tool into the user profile.  The dependency is a
concern to the package, not to the user.  *Inputs* make it possible to handle
dependencies without bugging the user by adding undesired executable files (or
libraries) to their profile.

Same goes for *native-inputs*: once the program is installed, build-time
dependencies can be safely garbage-collected.
It also matters when a substitute is available, in which case only the *inputs*
and *propagated inputs* will be fetched: the *native inputs* are not required to
install a package from a substitute.


## Outputs

Just like how a package can have multiple inputs, it can also produce multiple
outputs.

Each output corresponds to a separate directory in the store.

The user can choose which output to install; this is useful to save space or
to avoid polluting the user profile with unwanted executables or libraries.

Output separation is optional.  When the `outputs` field is left out, the
default and only output (the complete package) is referred to as `"out"`.

Typical separate output names include `debug` and `doc`.

It's advised to separate outputs only when you've shown it's worth it: if the
output size is significant (compare with `guix size`) or in case the package is
modular.


## Build system arguments

The `arguments` is a keyword-value list used to configure the build process.

The simplest argument `#:tests?` can be used to disable the test suite when
building the package.  This is mostly useful when the package does not feature
any test suite.  It's strongly recommended to keep the test suite on if there is
one.

Another  common argument is `:make-flags`, which specifies a list of flags to
append when running make, as you would from the command line.  For instance, the
following flags

```scheme
    #:make-flags (list (string-append "prefix=" (assoc-ref %outputs "out"))
                       "CC=gcc")
```

translate into

```sh
    $ make CC=gcc prefix=/gnu/store/...-<out>
```

This sets the C compiler to `gcc` and the `prefix` variable (the installation
directory in Make parlance) to `(assoc-ref %outputs "out")`, which is a build-stage
global variable pointing to the destination directory in the store (something like
`/gnu/store/...-my-libgit2-20180408`).

Similarly, it's possible to set the "configure" flags.

```scheme
    #:configure-flags '("-DUSE_SHA1DC=ON")
```

The `%build-inputs` variable is also generated in scope.  It's an association
table that maps the input names to their store directories.

The `phases` keyword lists the sequential steps of the build system.  Typically
phases include `unpack`, `configure`, `build`, `install` and `check`.  To know
more about those phases, you need to work out the appropriate build system
definition in `$GUIX_CHECKOUT/guix/build/gnu-build-system.scm`:

```scheme
    (define %standard-phases
      ;; Standard build phases, as a list of symbol/procedure pairs.
      (let-syntax ((phases (syntax-rules ()
                             ((_ p ...) `((p . ,p) ...)))))
        (phases set-SOURCE-DATE-EPOCH set-paths install-locale unpack
                bootstrap
                patch-usr-bin-file
                patch-source-shebangs configure patch-generated-file-shebangs
                build check install
                patch-shebangs strip
                validate-runpath
                validate-documentation-location
                delete-info-dir-file
                patch-dot-desktop-files
                install-license-files
                reset-gzip-timestamps
                compress-documentation)))
```

Or from the REPL:

```scheme
    > (add-to-load-path "/path/to/guix/checkout")
    > ,module (guix build gnu-build-system)
    > (map first %standard-phases)
    (set-SOURCE-DATE-EPOCH set-paths install-locale unpack bootstrap
	patch-usr-bin-file patch-source-shebangs configure
	patch-generated-file-shebangs build check install patch-shebangs strip
	validate-runpath validate-documentation-location delete-info-dir-file
	patch-dot-desktop-files install-license-files reset-gzip-timestamps
	compress-documentation)
```

If you want to know more about what happens during those phases, consult the
associated procedures.

For instance, as of this writing the definition of `unpack` for the GNU build
system is

```scheme
    (define* (unpack #:key source #:allow-other-keys)
      "Unpack SOURCE in the working directory, and change directory within the
    source.  When SOURCE is a directory, copy it in a sub-directory of the current
    working directory."
      (if (file-is-directory? source)
          (begin
            (mkdir "source")
            (chdir "source")

            ;; Preserve timestamps (set to the Epoch) on the copied tree so that
            ;; things work deterministically.
            (copy-recursively source "."
                              #:keep-mtime? #t))
          (begin
            (if (string-suffix? ".zip" source)
                (invoke "unzip" source)
                (invoke "tar" "xvf" source))
            (chdir (first-subdirectory "."))))
      #t)
```

Note the `chdir` call: it changes the working directory to where the source was
unpacked.
Thus every phase following the `unpack` will use the source as a working
directory, which is why we can directly work on the source files.
That is to say, unless a later phase changes the working directory to something
else.

We modify the list of `%standard-phases` of the build system with the
`modify-phases` macro as per the list of specified modifications, which may have
the following forms:

-   `(add-before PHASE NEW-PHASE PROCEDURE)`: Run `PROCEDURE` named `NEW-PHASE` before `PHASE`.
-   `(add-after PHASE NEW-PHASE PROCEDURE)`: Same, but afterwards.
-   `(replace PHASE PROCEDURE)`.
-   `(delete PHASE)`.

The `PROCEDURE` supports the keyword arguments `inputs` and `outputs`.  Each
input (whether *native*, *propagated* or not) and output directory is referenced
by their name in those variables.  Thus `(assoc-ref outputs "out")` is the store
directory of the main output of the package.  A phase procedure may look like
this:

```scheme
    (lambda* (#:key inputs outputs #:allow-other-keys)
      (let (((bash-directory (assoc-ref inputs "bash"))
             (output-directory (assoc-ref outputs "out"))
             (doc-directory (assoc-ref outputs "doc"))
      ; ...
      #t)
```

The procedure must return `#t` on success.  It's brittle to rely on the return
value of the last expression used to tweak the phase because there is no
guarantee it would be a `#t`.  Hence the trailing `#t` to ensure the right value
is returned on success.


## Code staging

The astute reader may have noticed the quasi-quote and comma syntax in the
argument field.  Indeed, the build code in the package declaration should not be
evaluated on the client side, but only when passed to the Guix daemon.  This
mechanism of passing code around two running processes is called [code staging](https://arxiv.org/abs/1709.00833).


## "Utils" functions

When customizing `phases`, we often need to write code that mimics the
equivalent system invocations (`make`, `mkdir`, `cp`, etc.) commonly used during
regular "Unix-style" installations.

Some like `chmod` are native to Guile.  See the [Guile reference manual](https://www.gnu.org/software/guile/manual/guile.html) for a
complete list.

Guix provides additional helper functions which prove especially handy in the
context of package management.

Some of those functions can be found in
`$GUIX_CHECKOUT/guix/guix/build/utils.scm`.  Most of them mirror the behaviour
of the traditional Unix system commands:

-   **which:** Like the `which` system command.
-   **find-files:** Akin to the `find` system command.
-   **mkdir-p:** Like `mkdir -p`, which creates all parents as needed.
-   **install-file:** Similar to `install` when installing a file to a (possibly
    non-existing) directory.  Guile has `copy-file` which works
    like `cp`.
-   **copy-recursively:** Like `cp -r`.
-   **delete-file-recursively:** Like `rm -rf`.
-   **invoke:** Run an executable.  This should be used instead of `system*`.
-   **with-directory-excursion:** Run the body in a different working directory,
    then restore the previous working directory.
-   **substitute\*:** A "sed-like" function.


## Module prefix

The license in our last example needs a prefix: this is because of how the
`license` module was imported in the package, as `#:use-module ((guix licenses)
#:prefix license:)`.  The [Guile module import mechanism](https://www.gnu.org/software/guile/manual/html_node/Using-Guile-Modules.html) gives the user full
control over namespacing: this is needed to avoid clashes between, say, the
`zlib` variable from `licenses.scm` (a *license* value) and the `zlib` variable
from `compression.scm` (a *package* value).


# Other build systems

What we've seen so far covers the majority of packages using a build system
other than the `trivial-build-system`.  The latter does not automate anything
and leaves you to build everything manually.  This can be more demanding and we
won't cover it here for now, but thankfully it is rarely necessary to fall back
on this system.

For the other build systems, such as ASDF, Emacs, Perl, Ruby and many more, the
process is very similar to the GNU build system except for a few specialized
arguments.

Learn more about build systems in

-   [the manual, section 4.2 Build systems](https://www.gnu.org/software/guix/manual/en/html_node/Build-Systems.html#Build-Systems),
-   the source code in the `$GUIX_CHECKOUT/guix/build` and
    `$GUIX_CHECKOUT/guix/build-system` directories.


# Programmable and automated package definition

We can't repeat it enough: having a full-fledged programming language at hand
empowers us in ways that reach far beyond traditional package management.

Let's illustrate this with some awesome features of Guix!


## Recursive importers

You might find some build systems good enough that there is little to do at all
to write a package, to the point that it becomes repetitive and tedious after a
while.  A *raison d'être* of computers is to replace human beings at those
boring tasks.  So let's tell Guix to do this for us and create the package
definition of an R package from CRAN (the output is trimmed for conciseness):

```sh
    $ guix import cran --recursive walrus

    (define-public r-mc2d
        ; ...
        (license gpl2+)))

    (define-public r-jmvcore
        ; ...
        (license gpl2+)))

    (define-public r-wrs2
        ; ...
        (license gpl3)))

    (define-public r-walrus
      (package
        (name "r-walrus")
        (version "1.0.3")
        (source
          (origin
            (method url-fetch)
            (uri (cran-uri "walrus" version))
            (sha256
              (base32
                "1nk2glcvy4hyksl5ipq2mz8jy4fss90hx6cq98m3w96kzjni6jjj"))))
        (build-system r-build-system)
        (propagated-inputs
          `(("r-ggplot2" ,r-ggplot2)
            ("r-jmvcore" ,r-jmvcore)
            ("r-r6" ,r-r6)
            ("r-wrs2" ,r-wrs2)))
        (home-page "https://github.com/jamovi/walrus")
        (synopsis "Robust Statistical Methods")
        (description
          "This package provides a toolbox of common robust statistical tests, including robust descriptives, robust t-tests, and robust ANOVA.  It is also available as a module for 'jamovi' (see <https://www.jamovi.org> for more information).  Walrus is based on the WRS2 package by Patrick Mair, which is in turn based on the scripts and work of Rand Wilcox.  These analyses are described in depth in the book 'Introduction to Robust Estimation & Hypothesis Testing'.")
        (license gpl3)))
```

The recursive importer won't import packages for which Guix already has package
definitions, except for the very first.

Not all applications can be packaged this way, only those relying on a select
number of supported systems.  Read about the full list of importers in the [guix
import section](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-import.html) of the manual.


## Automatic update

Guix can be smart enough to check for updates on systems it knows.  It can
report outdated package definitions with

```sh
    $ guix refresh hello
```

In most cases, updating a package to a newer version requires little more than
changing the version number and the checksum.  Guix can do that automatically as
well:

```sh
    $ guix refresh hello --update
```


## Inheritance

If you've started browsing the existing package definitions, you might have
noticed that a significant number of them have a `inherit` field:

```scheme
    (define-public adwaita-icon-theme
      (package (inherit gnome-icon-theme)
        (name "adwaita-icon-theme")
        (version "3.26.1")
        (source (origin
                  (method url-fetch)
                  (uri (string-append "mirror://gnome/sources/" name "/"
                                      (version-major+minor version) "/"
                                      name "-" version ".tar.xz"))
                  (sha256
                   (base32
                    "17fpahgh5dyckgz7rwqvzgnhx53cx9kr2xw0szprc6bnqy977fi8"))))
        (native-inputs
         `(("gtk-encode-symbolic-svg" ,gtk+ "bin")))))
```

All unspecified fields are inherited from the parent package.  This is very
convenient to create alternative packages, for instance with different source,
version or compilation options.


# Getting help

Sadly, some applications can be tough to package.  Sometimes they need a patch to
work with the non-standard filesystem hierarchy enforced by the store.
Sometimes the tests won't run properly.  (They can be skipped but this is not
recommended.)  Other times the resulting package won't be reproducible.

Should you be stuck, unable to figure out how to fix any sort of packaging
issue, don't hesitate to ask the community for help.

See the [Guix homepage](https://www.gnu.org/software/guix/contact/) for information on the mailing lists, IRC, etc.


# Conclusion

This tutorial was an showcase of the sophisticated package management that Guix
boasts.  At this point we have mostly restricted this introduction to the
`gnu-build-system` which is a core abstraction layer on which more advanced
abstractions are based.

Now where do we go from here?  Next we ought to dissect the innards of the build
system by removing all abstractions, using the `trivial-build-system`: this
should give us a thorough understanding of the process before investigating some
more advanced packaging techniques and edge cases.

Other features worth exploring are the interactive editing and debugging
capabilities of Guix provided by the Guile REPL.

Those fancy features are no strict requirements and this is a good point to stop
for now.  With what we've introduced here you should be well armed to package
lots of programs.  You can get started right away and hopefully we will see your
contributions soon!


# References

-   The [package reference in the manual](https://www.gnu.org/software/guix/manual/en/html_node/Defining-Packages.html)

-   [Pjotr’s hacking guide to GNU Guix](https://gitlab.com/pjotrp/guix-notes/blob/master/HACKING.org)

-   "Guix Guix: Package without a scheme!", by Andreas Enge (in
    `guix-maintenance.git/talks/ghm-2013/andreas/slides.pdf`)


# About GNU Guix

[GNU Guix](https://www.gnu.org/software/guix) is a transactional package manager for the GNU system.  The Guix System
Distribution or GuixSD is an advanced distribution of the GNU system that relies
on GNU Guix and [respects the user's freedom](https://www.gnu.org/distros/free-system-distribution-guidelines.html).

In addition to standard package management features, Guix supports transactional
upgrades and roll-backs, unprivileged package management, per-user profiles, and
garbage collection.  Guix uses low-level mechanisms from the Nix package
manager, except that packages are defined as native [Guile](https://www.gnu.org/software/guile) modules, using
extensions to the [Scheme](http://schemers.org) language.  GuixSD offers a declarative approach to
operating system configuration management, and is highly customizable and
hackable.

GuixSD can be used on an i686, x86_64 and armv7 machines.  It is also possible
to use Guix on top of an already installed GNU/Linux system, including on
mips64el and aarch64.

[-- Attachment #1.3: Type: text/plain, Size: 48 bytes --]


-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Blog: Guix packaging tutorial

[Benjamin Slade]

Hi Pierre,

Maybe it really falls outside of the scope of this packaging tutorial
since some of it is not about creating packages per se but about how to
submit them....

...But I was thinking about something a bit like Pjotr's list, though
with a bit more detail. (E.g. things like using `git format-patch' to
create patches for submission.) 

But in retrospect, maybe this would be better as a separate post.

cheers,
   —Ben



On 2018-10-08T17:05:54-0600, Pierre Neidhardt <mail@ambrevar.xyz> wrote:

 > Hi Ben!

 > I've sent two versions so far and I'm going to send the final version just now.

 > As for the checklists: hmm... Which steps do you have in mind?  The trick is
 > that the is little to repeat once the Git checkout has been set up.
 > Pjotr has a checklist however, let me know what you think:

 > From https://gitlab.com/pjotrp/guix-notes/blob/master/HACKING.org:

 > --8<---------------cut here---------------start------------->8---
 > Workflow for packaging

 > The general workflow for adding and maintaining packages is a bit
 >   complex. Everything goes via de guix-dev mailing list and includes a
 >   review process which can be discouraging and is more geared towards
 >   reviewers than towards newbies. This should not discourage you because
 >   GNU Guix is great. Note that the reviewers do this work voluntarily
 >   and most ‘rules’ have been agreed by the community. In the end your
 >   labours will get rewarded. So, how to start?

 >  1 - Work on a recent git checkout of guix
 >  2 - Use ‘guix import’ if you can (e.g. for python and R modules)
 >  3 - Build the package yourself
 >  4 - If tests are there, make sure they pass
 >  5 - Test your work with ‘guix lint’
 >  6 - Create a patch as described above
 >  7 - Send it to debbugs as described above
 >  8 - Submit one patch at a time and submit the next one when it goes in
 >  9 - Be patient, review can take a while - monitor debbugs
 > --8<---------------cut here---------------end--------------->8---

 > I'm not sure those are exactly steps, while some items actually belong the
 > manual's check list "(guix) Submitting Patches".

 > Opinions?

 > --
 > Pierre Neidhardt
 > https://ambrevar.xyz/


--
Benjamin Slade - https://babbagefiles.xyz
  `(pgp_fp: ,(21BA 2AE1 28F6 DF36 110A 0E9C A320 BBE8 2B52 EE19))
    '(sent by mu4e on Emacs running under GNU/Linux . https://gnu.org )
       `(Choose Linux ,(Choose Freedom) . https://linux.com )

Re: Blog: Guix packaging tutorial

[Ludovic Courtès]

Hi Benjamin,

Benjamin Slade <slade@jnanam.net> skribis:

> ...But I was thinking about something a bit like Pjotr's list, though
> with a bit more detail. (E.g. things like using `git format-patch' to
> create patches for submission.) 

Does the list at
<https://www.gnu.org/software/guix/manual/en/html_node/Submitting-Patches.html>
corresponds to what you’re looking for?

If not, we should probably change it (see doc/guix.texi in the repo.)

Thanks,
Ludo’.

Re: Blog: Guix packaging tutorial

[Ludovic Courtès]

Hello Pierre,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> And here is the last version, in Markdown (hopefully the syntax is right).
>
> title: A packaging tutorial for Guix
> date: 2018-10-09
> author: Pierre Neidhardt
> tags: Software development, Programming interfaces, Scheme API

This LGTM so if there are no objections, I’ll go ahead and publish it
later today when I get around to it.

Thank you, and thanks Ricardo and everyone who provided feedback!

Ludo’.

Re: Blog: Guix packaging tutorial

[George Clemmer]

This LGTM except ... the first paragraphs (quoted below) are an
advertisement. This is out of place in a tutorial. So, IMO, they should
simply be removed.

> GNU Guix stands out as the *hackable* package manager, mostly because
> it uses [GNU Guile](https://www.gnu.org/software/guile/), a powerful
> high-level programming language, one of the
> [Scheme](https://en.wikipedia.org/wiki/Scheme_(programming_language))
> dialects from the [Lisp
> family](https://en.wikipedia.org/wiki/Lisp_(programming_language)).
>
> Package definitions are also written in Scheme, which empowers Guix in
> some very unique ways, unlike most other package managers that use
> shell scripts or simple languages.
>
> - Use functions, structures, macros and all of Scheme expressiveness
> for your package definitions.
>
> - Inheritance makes it easy to customize a package by inheriting from
> it and modifying only what is needed.
>
> - Batch processing: the whole package collection can be parsed,
> filtered and processed.  Building a headless server with all graphical
> interfaces stripped out?  It's possible.  Want to rebuild everything
> from source using specific compiler optimization flags?  Pass the
> `#:make-flags "..."` argument to the list of packages.  It wouldn't be
> a stretch to think [Gentoo USE
> flags](https://wiki.gentoo.org/wiki/USE_flag) here, but this goes even
> further: the changes don't have to be thought out beforehand by the
> packager, they can be *programmed* by the user!

If you don't remove them, then at least move the paragraph quoted below
to the top of the piece ...

> The following tutorial covers all the basics around package creation
> with Guix.  It does not assume much knowledge of the Guix system nor
> of the Lisp language.  The reader is only expected to be familiar with
> the command line and to have some basic programming knowledge.

... and preface the above mentioned paragraphs with something like ...

# Guix is a unique approach to package management

HTH - George

Re: Blog: Guix packaging tutorial

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 569 bytes --]


> This LGTM except ... the first paragraphs (quoted below) are an
> advertisement. This is out of place in a tutorial. So, IMO, they should
> simply be removed.

It's a tutorial addressed to newcomers, so I believe it's important that we
mention (or remind?) what Guix can do, because this is precisely what we are
tackling in this tutorial.  I don't think it's a problem, actually I think it's
necessary as part of the introduction.

Opinions, anyone?

I'm not opposed to swapping both paragraphs however.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Guix packaging tutorial is on-line!

[Ludovic Courtès]

Hello!

It took us a while, but it’s on-line now!

  https://gnu.org/software/guix/blog/2018/a-packaging-tutorial-for-guix/

I took the liberty to add a paragraph about tarball authentication (it
hadn’t occurred to me before), I hope that’s fine.

Thanks a lot for your work and for your patience, Pierre!  I’m sure
it’ll be very useful to newcomers.

Ludo’.

Re: Guix packaging tutorial is on-line!

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 555 bytes --]

Thank you Ludo!

A few issues I've spot so far;

- The level-2 section formatting is wrong, starting from "git-fetch method", any
  clue what's up?

- This implementation of markdown does not automatically remove the extra
margin indentation in code snippets,
I've got to fix that.

- I was a bit tired when I rewrote the conclusion and I'm realizing now that the
  last paragraph is poorly phrased.  Gotta fix that too.

- https://guix.info/blog is not up to date and is missing 3 articles.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Blog: Guix packaging tutorial

[Ricardo Wurmus]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

>> This LGTM except ... the first paragraphs (quoted below) are an
>> advertisement. This is out of place in a tutorial. So, IMO, they should
>> simply be removed.
>
> It's a tutorial addressed to newcomers, so I believe it's important that we
> mention (or remind?) what Guix can do, because this is precisely what we are
> tackling in this tutorial.  I don't think it's a problem, actually I think it's
> necessary as part of the introduction.

I agree.  It works very well as an introduction.

-- 
Ricardo

Re: Guix packaging tutorial is on-line!

[Ricardo Wurmus]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> - https://guix.info/blog is not up to date and is missing 3 articles.

That’s on me.  I’ll regenerate the website soon.

--
Ricardo

Re: Guix packaging tutorial is on-line!

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 403 bytes --]


> * I don’t know if there should be a section “Common pitfalls and best
>   practices” other than to point to the checklist in the manual.  If the
>   list in the manual misses one of the points I’d rather add them there
>   and only link to the list.

I've added the URL recommendations to the "Submitting patches" section of the manual.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Blog: Guix packaging tutorial

[Benjamin Slade]

Hi Ludo',

So for me, the list at
<https://www.gnu.org/software/guix/manual/en/html_node/Submitting-Patches.html>
seems more like a list of `desired output conditions' and the numbering
doesn't really correspond directly to the temporal ordering of steps
(with some of the numbered items probably being temporally independent
but perhaps some not, which is somewhat confusing). For instance,
references to documentation (gender-neutral pronouns) occur lower in the
list than linting, and so if one tried to follow the checklist
temporally, one could end up doing things which invalidated the
linting. Should formatting occur before or after linting? Or is it
irrelevant?

For another instance, actually producing the patch (via `git
format-patch') is mentioned in the `preamble' but is not part of the
list (where reasonably it might occur near the end). And then there are
other things which are like steps (the actual sending of the patch)
which occur below the checklist.

It might make sense if there were:

A. a list of temporally-ordered steps for creating and submitting a package
 - where one of big steps could refer to Pierre's build tutorial
 - and where another step could refer to a list of things to check (B)

B. a non-temporal list of things to check before submitting
  (e.g. using gender-neutral pronouns in documentation isn't really a
  step, but a `desired output' condition)

 —Ben


On 2018-10-10T03:02:42-0600, Ludovic Courtès <ludo@gnu.org> wrote:

 > Hi Benjamin,

 > Benjamin Slade <slade@jnanam.net> skribis:

 > > ...But I was thinking about something a bit like Pjotr's list, though
 > > with a bit more detail. (E.g. things like using `git format-patch' to
 > > create patches for submission.)

 > Does the list at
 > <https://www.gnu.org/software/guix/manual/en/html_node/Submitting-Patches.html>
 > corresponds to what you’re looking for?

 > If not, we should probably change it (see doc/guix.texi in the repo.)

 > Thanks,
 > Ludo’.


--
Benjamin Slade - https://babbagefiles.xyz
  `(pgp_fp: ,(21BA 2AE1 28F6 DF36 110A 0E9C A320 BBE8 2B52 EE19))
    '(sent by mu4e on Emacs running under GNU/Linux . https://gnu.org )
       `(Choose Linux ,(Choose Freedom) . https://linux.com )

Re: Blog: Guix packaging tutorial

[Gábor Boskovits]

Benjamin Slade <slade@jnanam.net> ezt írta (időpont: 2018. okt. 11., Cs, 3:38):
>
> Hi Ludo',
>
> So for me, the list at
> <https://www.gnu.org/software/guix/manual/en/html_node/Submitting-Patches.html>
> seems more like a list of `desired output conditions' and the numbering
> doesn't really correspond directly to the temporal ordering of steps
> (with some of the numbered items probably being temporally independent
> but perhaps some not, which is somewhat confusing). For instance,
> references to documentation (gender-neutral pronouns) occur lower in the
> list than linting, and so if one tried to follow the checklist
> temporally, one could end up doing things which invalidated the
> linting. Should formatting occur before or after linting? Or is it
> irrelevant?

Not answering all your questions, but formatting should come before linting.
I suggest linting to be done when there are no more changes.

>
> For another instance, actually producing the patch (via `git
> format-patch') is mentioned in the `preamble' but is not part of the
> list (where reasonably it might occur near the end). And then there are
> other things which are like steps (the actual sending of the patch)
> which occur below the checklist.
>
> It might make sense if there were:
>
> A. a list of temporally-ordered steps for creating and submitting a package
>  - where one of big steps could refer to Pierre's build tutorial
>  - and where another step could refer to a list of things to check (B)
>
> B. a non-temporal list of things to check before submitting
>   (e.g. using gender-neutral pronouns in documentation isn't really a
>   step, but a `desired output' condition)
>
>  —Ben
>
>
> On 2018-10-10T03:02:42-0600, Ludovic Courtès <ludo@gnu.org> wrote:
>
>  > Hi Benjamin,
>
>  > Benjamin Slade <slade@jnanam.net> skribis:
>
>  > > ...But I was thinking about something a bit like Pjotr's list, though
>  > > with a bit more detail. (E.g. things like using `git format-patch' to
>  > > create patches for submission.)
>
>  > Does the list at
>  > <https://www.gnu.org/software/guix/manual/en/html_node/Submitting-Patches.html>
>  > corresponds to what you’re looking for?
>
>  > If not, we should probably change it (see doc/guix.texi in the repo.)
>
>  > Thanks,
>  > Ludo’.
>
>
> --
> Benjamin Slade - https://babbagefiles.xyz
>   `(pgp_fp: ,(21BA 2AE1 28F6 DF36 110A 0E9C A320 BBE8 2B52 EE19))
>     '(sent by mu4e on Emacs running under GNU/Linux . https://gnu.org )
>        `(Choose Linux ,(Choose Freedom) . https://linux.com )
>

Re: Blog: Guix packaging tutorial

[Ludovic Courtès]

Hello,

Benjamin Slade <slade@jnanam.net> skribis:

> So for me, the list at
> <https://www.gnu.org/software/guix/manual/en/html_node/Submitting-Patches.html>
> seems more like a list of `desired output conditions' and the numbering
> doesn't really correspond directly to the temporal ordering of steps
> (with some of the numbered items probably being temporally independent
> but perhaps some not, which is somewhat confusing). For instance,
> references to documentation (gender-neutral pronouns) occur lower in the
> list than linting, and so if one tried to follow the checklist
> temporally, one could end up doing things which invalidated the
> linting. Should formatting occur before or after linting? Or is it
> irrelevant?
>
> For another instance, actually producing the patch (via `git
> format-patch') is mentioned in the `preamble' but is not part of the
> list (where reasonably it might occur near the end). And then there are
> other things which are like steps (the actual sending of the patch)
> which occur below the checklist.
>
> It might make sense if there were:
>
> A. a list of temporally-ordered steps for creating and submitting a package
>  - where one of big steps could refer to Pierre's build tutorial
>  - and where another step could refer to a list of things to check (B)
>
> B. a non-temporal list of things to check before submitting
>   (e.g. using gender-neutral pronouns in documentation isn't really a
>   step, but a `desired output' condition)

Indeed, I agree (I think it used to be a list of temporally-ordered
steps and then grew into that thing we have now…).

Perhaps one option would be to have a specific “Contributing Packages”
sub-section, where it’s simple to have an ordered list, and then another
subsection (“Contribution Guidelines” maybe?) with other items such as
gender-neutral wording.

Thoughts?  Would you like to give it a try?

Thanks,
Ludo’.

Re: Guix packaging tutorial is on-line!

[Ludovic Courtès]

Hi!

Sorry for the late reply.

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> A few issues I've spot so far;
>
> - The level-2 section formatting is wrong, starting from "git-fetch method", any
>   clue what's up?
>
> - This implementation of markdown does not automatically remove the extra
> margin indentation in code snippets,
> I've got to fix that.
>
> - I was a bit tired when I rewrote the conclusion and I'm realizing now that the
>   last paragraph is poorly phrased.  Gotta fix that too.

If you’d like to make cosmetic changes (I think it would be wrong to
change the text significantly at this point), please feel free to push
to guix-artwork.git, after which I can update the on-line copy.

Thanks!

Ludo’.

Re: Guix packaging tutorial is on-line!

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 463 bytes --]


> - The level-2 section formatting is wrong, starting from "git-fetch method", any
>   clue what's up?

OK, I figured:
- First I've inadvertently used level-3 headers instead of level-2 in some sections.

- Then there is Guix' CSS I think: level-2 headers are bigger than level-1
headers.  It looks odd in my opinion.

So for now I'll change all my level-2 headers to level-3.

Should we fix the CSS?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Guix packaging tutorial is on-line!

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 130 bytes --]

I've pushed some changes.  Let me know if everything is in order, Ludo.
Thanks!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Blog: Guix packaging tutorial

[Benjamin Slade]

On 2018-10-11T07:39:41-0600, Ludovic Courtès <ludo@gnu.org> wrote:
 > Perhaps one option would be to have a specific “Contributing
 > Packages” sub-section, where it’s simple to have an ordered list, and
 > then another subsection (“Contribution Guidelines” maybe?) with other
 > items such as gender-neutral wording.

 > Thoughts?  Would you like to give it a try?

Thanks, Ludo'. I think would you suggest makes sense.

I'm happy to work with someone on it - I'd be hesitant
to try to take it on single-handed as I'm still a neophyte Guix packager.

 —Ben

--
Benjamin Slade - https://babbagefiles.xyz
  `(pgp_fp: ,(21BA 2AE1 28F6 DF36 110A 0E9C A320 BBE8 2B52 EE19))
    '(sent by mu4e on Emacs running under GNU/Linux . https://gnu.org )
       `(Choose Linux ,(Choose Freedom) . https://linux.com )

Re: Guix packaging tutorial is on-line!

[Ludovic Courtès]

Hello Pierre,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> I've pushed some changes.  Let me know if everything is in order, Ludo.

I’ve published the changes (sorry for the delay!) and everything seems
to be alright.

Thanks for those fixes!

Ludo’.

Re: Guix packaging tutorial is on-line!

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 57 bytes --]

Thanks!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Blog: Guix packaging tutorial

[Divan]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> And here is the last version, in Markdown (hopefully the syntax is
> right).

Off topic, but how did you convert this? Guessing pandoc, but it seems
converted better then the standard, =pandoc index.org -t gfm -o
/tmp/index.md= would do.

Re: Blog: Guix packaging tutorial

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1091 bytes --]


Marco van Hulten <marco@hulten.org> writes:
> I guess this will be part of the Guix manual (or maybe the above URL is
> solid enough), so improvements are welcome?

Improvements are always welcome!  That said, the tutorial will not become part
of the manual.  We've discussed this before, and the consensus was that this
tutorial is mostly about putting together some parts of the manual in an
accessible way to get newcomers started.

It's completely redundant with the manual, so there is no need to move it there.

> I have minor things (not useful for — so I didn't post to — the list):
>
>   specific case that the output tells → specific case the output tells
>   if none at all → if at all  [or remove]

Thanks for spotting this!
I've committed the changes.  Ludo, can you update the blog?

> Please, let me know if I should report changes differently.

This is the right way to do it for blog posts.
The blog is hosted there:

	https://git.savannah.gnu.org/git/guix/guix-artwork.git

Thanks again!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Blog: Guix packaging tutorial

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 776 bytes --]


Divan <divan@santanas.co.za> writes:
> Off topic, but how did you convert this? Guessing pandoc, but it seems
> converted better then the standard, =pandoc index.org -t gfm -o
> /tmp/index.md= would do.

This is a very good question.  Indeed, Org support in Pandoc is sub-par, so I
did not use that.  Instead, I've used Emacs directly and its
~(org-md-export-to-markdown)~ function.  But even then, the export result lacked
a few elements, such as fenced code language tag or the header.
So I wrote a wrapper script to fix that automatically for me.
It's not very generic but it's a starting point.

Here is the implementation:

	https://gitlab.com/ambrevar/ambrevar.gitlab.io/tree/master/source/guix-packaging

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Blog: Guix packaging tutorial

[Divan Santana]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> Divan <divan@santanas.co.za> writes:
>> Off topic, but how did you convert this? Guessing pandoc, but it seems
>> converted better then the standard, =pandoc index.org -t gfm -o
>> /tmp/index.md= would do.
>
> This is a very good question.  Indeed, Org support in Pandoc is sub-par, so I
> did not use that.  Instead, I've used Emacs directly and its
> ~(org-md-export-to-markdown)~ function.  But even then, the export result lacked
> a few elements, such as fenced code language tag or the header.
> So I wrote a wrapper script to fix that automatically for me.
> It's not very generic but it's a starting point.
>
> Here is the implementation:
>
> 	https://gitlab.com/ambrevar/ambrevar.gitlab.io/tree/master/source/guix-packaging

This is really great and handy. I'll certainly be using this.

Would be great to package something like in emacs community. A blog
about converting back and forth between org and markdown would be
welcome to many in the Emacs community. :)

I'm slow to get around to completing your guix packaging tutorial. As a
complete noob I'm very interested in this and keen to submit a few basic
packages there after.

Re: Blog: Guix packaging tutorial

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1076 bytes --]


> Would be great to package something like in emacs community. A blog
> about converting back and forth between org and markdown would be
> welcome to many in the Emacs community. :)

Well, technically what I did was just fixing some nits, really.  But you are
right, those nits should be fixed upstream.  If I have time, I'll try to work
with upstream to polish the Org->markdown exporter.

As for the Markdown->Org conversion, the only reasonable option I know at the
moment is Pandoc, which is just as limited.  I'm less interested in this however
since I mostly use Org.

I wrote about my "Org-blog workflow":
https://ambrevar.xyz/blog-architecture/index.html.

> I'm slow to get around to completing your guix packaging tutorial. As a
> complete noob I'm very interested in this and keen to submit a few basic
> packages there after.

Take your time, there is a lot to digest in there.  Don't hesitate to ask for
help on the mailing list or on IRC.  Looking forward to your contributions!

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]