From mboxrd@z Thu Jan 1 00:00:00 1970 From: Pierre Neidhardt Subject: Re: Blog: Guix packaging tutorial Date: Wed, 03 Oct 2018 20:16:46 +0200 Message-ID: <874le2j3ox.fsf@ambrevar.xyz> References: <87in397jsd.fsf@ambrevar.xyz> <20180913191151.GA1865@jurong> <87woro5ocf.fsf@ambrevar.xyz> <20180914113302.elqrk3tvdkln2cde@thebird.nl> <87o9cmj0fc.fsf@ambrevar.xyz> <87mus6iypf.fsf@ambrevar.xyz> <87zhw02ea9.fsf@mdc-berlin.de> <87o9cex112.fsf@ambrevar.xyz> <87efdau5x2.fsf@gnu.org> <87murywuvn.fsf@ambrevar.xyz> <87lg7go8cv.fsf@gnu.org> <87va6kuyk4.fsf@ambrevar.xyz> <87sh1ot9m5.fsf@elephly.net> <87bm8bv4ao.fsf@ambrevar.xyz> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="==-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:57598) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g7lhm-0000Vg-Lh for guix-devel@gnu.org; Wed, 03 Oct 2018 14:17:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1g7lhi-00074Q-Go for guix-devel@gnu.org; Wed, 03 Oct 2018 14:17:02 -0400 In-reply-to: <87bm8bv4ao.fsf@ambrevar.xyz> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Ricardo Wurmus Cc: Ricardo Wurmus , guix-devel , guix-blog@gnu.org --==-=-= Content-Type: multipart/mixed; boundary="=-=-=" --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Find the last revision attached. I've taken all comments into consideration and I'm quite happy with the result. I've found a much better advanced example: libgit2. The only thing is... it does not pass the tests. In guix, libgit2 uses an archive upstream. I've changed the package to check out the git repository. Only then the test fail: =2D-8<---------------cut here---------------start------------->8--- starting phase `check' Loaded 341 suites:=20 Started apply::fromdiff................... apply::fromfile.......................................... attr::file.... attr::flags... attr::ignoreF 1) Failure: attr::ignore::honor_temporary_rules [/tmp/guix-build-my-libgit2-0.26.6-1.e9= 8d0a3.drv-0/my-libgit2-0.26.6-1.e98d0a3-checkout/tests/clar_libgit2.c:47] Expression is not true: (fd =3D p_open(path, flags, mode)) >=3D 0 =2D-8<---------------cut here---------------end--------------->8--- Any clue? I'll keep investigating tomorrow. Once this is fixed, it should be ready for publication. =2D-=20 Pierre Neidhardt https://ambrevar.xyz/ --=-=-= Content-Type: text/x-org; charset=utf-8 Content-Disposition: attachment; filename=index.org Content-Transfer-Encoding: quoted-printable #+TITLE: A packaging tutorial for Guix #+AUTHOR: Pierre Neidhardt #+date: <2018-10-03 Wed> #+tags: Software development, Programming interfaces, Scheme API * Introduction GNU Guix stands out as the /hackable/ package manager, mostly because it us= es [[https://www.gnu.org/software/guile/][GNU Guile]], a powerful high-level p= rogramming language, one of the [[https://en.wikipedia.org/wiki/Scheme_(pro= gramming_language)][Scheme]] dialects from the [[https://en.wikipedia.org/wiki/Lisp_(programming_languag= e)][Lisp family]]. Package definitions are also written in Scheme, which empowers Guix in some= very unique ways, unlike most other package managers that use shell scripts or simple languages. =2D Use functions, structures, macros and all of Scheme expressiveness for = your package definitions. =2D Inheritance makes it easy to customize a package by inheriting from it = and modifying only what is needed. =2D Batch processing: the whole package collection can be parsed, filtered = and processed. Building a headless server with all graphical interfaces stri= pped out? It's possible. Want to rebuild everything from source using specif= ic compiler optimization flags? Pass the ~#:make-flags "..."~ argument to t= he list of packages. It wouldn't be a stretch to think [[https://wiki.gento= o.org/wiki/USE_flag][Gentoo USE flags]] here, but this goes even further: the changes don't have to be thought out beforehand by the packager, they can be /programmed/ by the user! The following tutorial covers all the basics around package creation with G= uix. It does not assume much knowledge of the Guix system nor of the Lisp langua= ge. The reader is only expected to be familiar with the commandline and to have= some basic programming knowledge. * A "Hello World" package The [[https://www.gnu.org/software/guix/manual/en/html_node/Defining-Packag= es.html][Defining Packages section of the manual]] introduces the basics of= Guix packaging. In the following section, we will partly go over those basics a= gain. =3DGNU hello=3D is a dummy project that serves as an idiomatic example for packaging. It uses the GNU build system (~./configure && make && make inst= all~). Guix already provides a package definition which is a perfect example to st= art with. You can look up its declaration with ~guix edit hello~ from the commandline. Let's see how it looks: #+BEGIN_SRC scheme (define-public hello (package (name "hello") (version "2.10") (source (origin (method url-fetch) (uri (string-append "mirror://gnu/hello/hello-" version ".tar.gz")) (sha256 (base32 "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i")))) (build-system gnu-build-system) (synopsis "Hello, GNU world: An example GNU package") (description "GNU Hello prints the message \"Hello, world!\" and then exits. It serves as an example of standard GNU coding practices. As such, it supports command-line arguments, multiple languages, and so on.") (home-page "https://www.gnu.org/software/hello/") (license gpl3+))) #+END_SRC As you can see, most of it is rather straightforward. But let's review the fields together: =2D name :: The project name. Using Scheme conventions, we prefer to keep = it lower case, without underscore and using dash-separated words. =2D source :: This field contains a description of the source code origin. = The ~origin~ record contains these fields: 1. The method, here ~url-fetch~ to download via HTTP/FTP, but other metho= ds exist, such as ~git-fetch~ for Git repositories. 2. The URI, which is typically some =3Dhttps://=3D location for ~url-fetc= h~. Here the special =3Dmirror://gnu=3D refers to a set of well known locations= , all of which can be used by Guix to fetch the source, should some of them fai= l. 3. The ~sha256~ checksum of the requested file. This is essential to ens= ure the source is not corrupted. Note that Guix works with base32 strings, hence the call to the ~base32~ function. =2D build-system :: This is where the power of abstraction provided by the = Scheme language really shines: in this case, the ~gnu-build-syst= em~ abstracts away the famous ~./configure && make && make install~ shell invocations. Other build systems include = the ~trivial-build-system~ which does not do anything and req= uires from the packager to program all the build steps, the ~python-build-system~, the ~emacs-build-system~, [[https:= //www.gnu.org/software/guix/manual/en/html_node/Build-Systems.html][and many more]]. =2D synopsis :: It should be a concise summary of what the package does. For many packages a tagline from the project's home page can be used as the synopsis. =2D description :: Same as for the synopsis, it's fine to re-use the project description from the homepage. Note that Guix uses Texinfo syntax. =2D home-page :: Use HTTPS if available. =2D license :: See =3D$GUIX_CHECKOUT/guix/licenses.scm=3D for a full list. Time to build our first package! Nothing fancy here for now: we will stick= to a dummy "my-hello", a copy of the above declaration. As with the ritualistic "Hello World" taught with most programming language, this will possibly be the most "manual" approach. We will work out an ideal setup later; for now we will go the simplest route. Save the following to a file =3Dmy-hello.scm=3D. #+BEGIN_SRC scheme (use-modules (guix packages) (guix download) (guix build-system gnu) (guix licenses)) (package (name "my-hello") (version "2.10") (source (origin (method url-fetch) (uri (string-append "mirror://gnu/hello/hello-" version ".tar.gz")) (sha256 (base32 "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i")))) (build-system gnu-build-system) (synopsis "Hello, Guix world: An example custom Guix package") (description "GNU Hello prints the message \"Hello, world!\" and then exits. It serves as an example of standard GNU coding practices. As such, it supports command-line arguments, multiple languages, and so on.") (home-page "https://www.gnu.org/software/hello/") (license gpl3+)) #+END_SRC We will explain the extra code in a moment. Feel free to play with the different values of the various fields. If you change the source, you'll need to update the checksum. Indeed, Guix refuse= s to build anything if the given checksum does not match the computed checksum o= f the source code. To obtain the correct checksum of the package declaration, we need to download the source, compute the sha256 checksum and convert it to base32. Thankfully, Guix can automate this task for us; all we need is to provide t= he URI: #+BEGIN_SRC sh $ guix download mirror://gnu/hello/hello-2.10.tar.gz Starting download of /tmp/guix-file.JLYgL7 From=20https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz... following redirection to `https://mirror.ibcp.fr/pub/gnu/hello/hello-2.10.t= ar.gz'... =E2=80=A610.tar.gz 709KiB 2.5MiB/s 00:00 = [##################] 100.0% /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz 0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i #+END_SRC Note in this specific case that the output tells us which mirror was chosen. If the result of the above command is not the same as in the above snippet, update your =3Dmy-hello=3D declaration accordingly. Now you can happily run #+BEGIN_SRC sh $ guix package --install-from-file=3Dmy-hello.scm #+END_SRC You should now have =3Dmy-hello=3D in your profile! #+BEGIN_SRC sh $ guix package --list-installed=3Dmy-hello my-hello 2.10 out /gnu/store/f1db2mfm8syb8qvc357c53slbvf1g9m9-my-hello-2.10 #+END_SRC We've gone as far as we could without any knowledge of Scheme. Now is the = right time to introduce the minimum we need from the language before we can proce= ed. * A Scheme crash-course As we've seen above, basic packages don't require much Scheme knowledge, if= none at all. But as you progress and your desire to write more and more complex packages grows, it will become both necessary and empowering to hone your L= isper skills. Since an extensive Lisp course is very much out of the scope of this tutori= al, we will only cover some basics here. Guix uses the Guile implementation of Scheme. To start playing with the language, install it with ~guix package --install guile~ and start a [[http= s://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop][REPL]] by running ~guile~ from the commandline. Alternatively you can also run ~guix environment --ad-hoc guile -- guile~ if you'd rather not have Guile installed in your user profile. In the following examples we use the =3D>=3D symbol to denote the REPL prom= pt, that is, the line reserved for user input. See [[https://www.gnu.org/software/g= uile/manual/html_node/Using-Guile-Interactively.html][the Guile manual]] fo= r more details on the REPL. =2D Scheme syntax boils down to a tree of expressions (or /s-expression/ in= Lisp lingo) It can be a literal such numbers and strings, or a compound which = is a parenthesized list of compounds and literals. ~#t~ and ~#f~ stand for the booleans "true" and "false", respectively. Examples of valid expressions: #+BEGIN_SRC scheme > "Hello World!" "Hello World!" > 17 17 > (display (string-append "Hello " "Guix" "\n")) "Hello Guix!" #+END_SRC =2D This last example is a function call embedded in another function call.= When a parenthesized expression is evaluated, the first term is the function a= nd the rest are the arguments passed to the function. Every function return= s the last evaluated expression as value. =2D Anonymous functions are declared with the ~lambda~ term: #+BEGIN_SRC scheme > (lambda (x) (* x x)) #:24:0 (x)> #+END_SRC The above lambda returns the square of its argument. Since everything is= an expression, the ~lambda~ expression returns an anonymous function, which = can in turn be applied to an argument: #+BEGIN_SRC scheme > ((lambda (x) (* x x)) 3) 9 #+END_SRC =2D Anything can be assigned a global name with ~define~: #+BEGIN_SRC scheme > (define a 3) > (define square (lambda (x) (* x x))) > (square a) 9 #+END_SRC =2D Procedures can be defined more concisely with the following syntax: #+BEGIN_SRC scheme (define (square x) (* x x)) #+END_SRC =2D A list structure can be created with the ~list~ procedure: #+BEGIN_SRC scheme > (list 2 a 5 7) (2 3 5 7) #+END_SRC =2D The /quote/ disables evaluation of a parenthesized expression: the firs= t term is not called over the other terms. Thus it effectively returns a list of terms. #+BEGIN_SRC scheme > '(display (string-append "Hello " "Guix" "\n")) (display (string-append "Hello " "Guix" "\n")) > '(2 a 5 7) (2 a 5 7) #+END_SRC =2D The /quasiquote/ disables evaluation of a parenthesized expression unti= l a colon re-enables it. Thus it provides us with fine-grained control over = what is evaluated and what is not. #+BEGIN_SRC scheme > `(2 a 5 7 (2 ,a 5 ,(+ a 4))) (2 a 5 7 (2 3 5 7)) #+END_SRC Note that the above result is a list of mixed elements: numbers, symbols = (here ~a~) and the last element is a list itself. =2D Multiple variables can be named locally with ~let~: #+BEGIN_SRC scheme > (define x 10) > (let ((x 2) (y 3)) (list x y)) (2 3) > x 10 > y ERROR: In procedure module-lookup: Unbound variable: y #+END_SRC Use ~let*~ to re-allows the initializers of later variables to refer to t= he earlier variables. #+BEGIN_SRC scheme > (let* ((x 2) (y (* x 3))) (list x y)) (2 6) #+END_SRC =2D The keyword syntax is ~#:~, it is used to create unique identifiers. S= ee also the [[https://www.gnu.org/software/guile/manual/html_node/Keywords.html][= Keywords section in the Guile manual]]. =2D The percentage ~%~ is typically used for read-only global variables in = the build stage. Note that it is merely a convention, like ~_~ in C. Scheme= Lisp treats ~%~ exactly the same as any other letter. =2D Modules are created with ~define-module~. For instance #+BEGIN_SRC scheme (define-module (guix build-system ruby) #:use-module (guix store) #:export (ruby-build ruby-build-system)) #+END_SRC defines the module ~ruby~ which must be located in ~guix/build-system/ruby.scm~ somewhere in =3DGUILE_LOAD_PATH=3D. It depe= nds on the ~(guix store)~ module and it exports two symbols, ~ruby-build~ and ~ruby-build-system~. For a more detailed introduction, check out [[http://www.troubleshooters.co= m/codecorn/scheme_guile/hello.htm][Scheme at a Glance]], by Steve Litt. One of the reference Scheme books is the seminal /Structure and Interpretat= ion of Computer Programs/, by Harold Abelson and Gerald Jay Sussman, with Julie Sussman. You'll find a free copy [[https://mitpress.mit.edu/sites/default/= files/sicp/index.html][online]], together with [[https://ocw.mit.edu/course= s/electrical-engineering-and-computer-science/6-001-structure-and-interpret= ation-of-computer-programs-spring-2005/video-lectures/][videos of the lectu= res by the authors]]. The book is available in Texinfo format as the =3Dsicp= =3D Guix package. Go ahead, run ~guix package --install sicp~ and start reading with ~info sicp~ (or with the Emacs Info reader). An unofficial ebook [[https:/= /sarabander.github.io/sicp/][is also available]]. You'll find more books, tutorials and other resources at https://schemers.o= rg/. * Setup Now that we know some Scheme basics we can detail the different possible se= tups for working on Guix packages. There are several ways to set up a Guix packaging environment. We recommend you work directly on the Guix source checkout since it makes it easier for everyone to contribute to the project. But first, let's look at other possibilities. *** Local file This is what we previously did with =3Dmy-hello=3D. Now that we know more = Scheme, let's explain the leading chunks. As stated in ~guix package --help~: #+BEGIN_SRC sh -f, --install-from-file=3DFILE install the package that the code within FILE evaluates to #+END_SRC Thus the last expression /must/ return a package, which is the case in our earlier example. The ~use-modules~ expression tells which of the modules we need in the file. Modules are a collection of values and procedures. They are commonly called "libraries" or "packages" in other programming languages. *** GUIX_PACKAGE_PATH /Note: Starting from Guix 0.16, the more flexible Guix "channels" are the preferred way and supersede =3DGUIX_PACKAGE_PATH=3D. See below./ It can be tedious to specify the file from the commandline instead of simply calling ~guix package --install my-hello~ as you would do with the official packages. Guix makes it possible to streamline the process by adding as many "package declaration paths" as you want. Create a folder, say =3D~./guix-packages=3D and add it to the =3DGUIX_PACKA= GE_PATH=3D environment variable: #+BEGIN_SRC sh $ mkdir ~/guix-packages $ export GUIX_PACKAGE_PATH=3D~/guix-packages #+END_SRC To add several folders, separate them with a colon (~:~). Our previous =3Dmy-hello=3D needs some adjustments though: #+BEGIN_SRC scheme (define-module (my-hello) #:use-module (guix licenses) #:use-module (guix packages) #:use-module (guix build-system gnu) #:use-module (guix download)) (define-public my-hello (package (name "my-hello") (version "2.10") (source (origin (method url-fetch) (uri (string-append "mirror://gnu/hello/hello-" version ".tar.gz")) (sha256 (base32 "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i")))) (build-system gnu-build-system) (synopsis "Hello, Guix world: An example custom Guix package") (description "GNU Hello prints the message \"Hello, world!\" and then exits. It serves as an example of standard GNU coding practices. As such, it supports command-line arguments, multiple languages, and so on.") (home-page "https://www.gnu.org/software/hello/") (license gpl3+))) #+END_SRC Notice that we have wrapped the package into a ~define-public~ record. Thi= s is effectively assigning the package to the ~my-hello~ variable so that it can= be referenced, among other as dependency of other packages. If you use ~guix package --install-from-file=3Dmy-hello.scm~ on the above f= ile, it will fail because the last expression, ~define-public~, does not return a package. If you want to use ~define-public~ in this use-case nonetheless, = make sure the file ends with an evaluation of ~my-hello~: #+BEGIN_SRC scheme ; ... (define-public my-hello ; ... ) my-hello #+END_SRC This last example is not very typical. Now =3Dmy-hello=3D should be part of the package collection like all other = official packages. You can verify this with: #+BEGIN_SRC sh $ guix package --show=3Dmy-hello #+END_SRC *** Guix channels Guix 0.16 features channels, which is very similar to =3DGUIX_PACKAGE_PATH= =3D but provides better integration and provenance tracking. Channels are not necessarily local, they can be maintained as a public Git repository for instance. Of course, several channels can be used at the same time. See the [[http://guix.info/manual/en/Channels.html][Channel secion in the m= anual]] for setup details. *** Direct checkout hacking Working directly on the Guix project is recommended: it reduces the friction when the time comes to submit your changes upstream to let the community be= nefit from your hard work! Unlike most software distributions, the Guix repository holds in one place = both the tooling (including the package manager) and the package definitions. T= his choice was made so that it would give developers the flexibility to modify = the API without breakage by updating all packages at the same time. This reduc= es development inertia. Check out the official [[https://git-scm.com/][Git]] repository: #+BEGIN_SRC sh $ git clone https://git.savannah.gnu.org/git/guix.git #+END_SRC Follow the instruction from the [[https://www.gnu.org/software/guix/manual/= en/html_node/Contributing.html]["Contributing" chapter]] in the manual to s= et up the repository environment. Once ready, you should be able to use the package definitions from the repository environment. Feel free to edit package definitions found in =3D$GUIX_CHECKOUT/gnu/packag= es=3D. The =3D$GUIX_CHECKOUT/pre-inst-env=3D script lets you use =3Dguix=3D over t= he package collection of the repository. =2D Search packages: #+BEGIN_SRC sh $ cd $GUIX_CHECKOUT $ ./pre-inst-env guix package --list-available=3Druby ruby 1.8.7-p374 out gnu/packages/ruby.scm:119:2 ruby 2.1.6 out gnu/packages/ruby.scm:91:2 ruby 2.2.2 out gnu/packages/ruby.scm:39:2 #+END_SRC =2D Build a package: #+BEGIN_SRC sh $ ./pre-inst-env guix build --keep-failed ruby@2.1 /gnu/store/c13v73jxmj2nir2xjqaz5259zywsa9zi-ruby-2.1.6 #+END_SRC =2D Install it to your user profile: #+BEGIN_SRC sh $ ./pre-inst-env guix package --install ruby@2.1 #+END_SRC =2D Check for common mistakes: #+BEGIN_SRC sh $ ./pre-inst-env guix lint ruby@2.1 #+END_SRC Once you are happy with the result, you are welcome to send your contributi= on to make it part of Guix. This process is also detailed in the [[https://www.g= nu.org/software/guix/manual/en/html_node/Contributing.html][manual]]. It's a community effort so the more join in, the better Guix becomes! * Extended example The above "Hello World" example is as simple as it goes. Packages can be m= ore complex than that and Guix can handle more advanced scenarios. Let's looks= at another, more sophisticated package (slightly modified from the source): # REVIEW: Find a better example? # qpdf: lacks input. # qtbase: too long. # bigloo: no configure, no make flag. # libpng-apng misses snippets and git. # minetest (65): needs outputs. # pulseaudio (80): needs outputs. # libgit2 (54): need git + makeflags # TODO: The following does not pass the tests: # # starting phase `check' # Loaded 341 suites: # Started # # apply::fromdiff................... # apply::fromfile.......................................... # attr::file.... # attr::flags... # attr::ignoreF # # 1) Failure: # attr::ignore::honor_temporary_rules [/tmp/guix-build-my-libgit2-0.26.6-1.= e98d0a3.drv-0/my-libgit2-0.26.6-1.e98d0a3-checkout/tests/clar_libgit2.c:47] # Expression is not true: (fd =3D p_open(path, flags, mode)) >=3D 0 #+BEGIN_SRC scheme (define-module (gnu packages version-control) #:use-module ((guix licenses) #:prefix license:) #:use-module (guix utils) #:use-module (guix packages) #:use-module (guix git-download) #:use-module (guix build-system cmake) #:use-module (gnu packages ssh) #:use-module (gnu packages web) #:use-module (gnu packages pkg-config) #:use-module (gnu packages python) #:use-module (gnu packages compression) #:use-module (gnu packages tls)) (define-public my-libgit2 (let ((commit "e98d0a37c93574d2c6107bf7f31140b548c6a7bf") (revision "1")) (package (name "my-libgit2") (version (git-version "0.26.6" revision commit)) (source (origin (method git-fetch) (uri (git-reference (url "https://github.com/libgit2/libgit2/") (commit commit))) (file-name (git-file-name name version)) (sha256 (base32 "17pjvprmdrx4h6bb1hhc98w9qi6ki7yl57f090n9kbhswxqfs7s3")) (patches (search-patches "libgit2-mtime-0.patch")) (modules '((guix build utils))) (snippet '(begin ;; Remove bundled software. (delete-file-recursively "deps") #t)))) (build-system cmake-build-system) (outputs '("out" "debug")) (arguments `(#:tests? #t ; Run the test suite (this = is the default) #:configure-flags '("-DUSE_SHA1DC=3DON") ; SHA-1 collision detecti= on #:phases (modify-phases %standard-phases (add-after 'unpack 'fix-hardcoded-paths (lambda _ (substitute* "tests/repo/init.c" (("#!/bin/sh") (string-append "#!" (which "sh")))) (substitute* "tests/clar/fs.h" (("/bin/cp") (which "cp")) (("/bin/rm") (which "rm"))) #t)) ;; Run checks more verbosely. (replace 'check (lambda _ (invoke "./libgit2_clar" "-v" "-Q")))))) (inputs `(("libssh2" ,libssh2) ("http-parser" ,http-parser) ("python" ,python-wrapper))) (native-inputs `(("pkg-config" ,pkg-config))) (propagated-inputs ;; These two libraries are in 'Requires.private' in libgit2.pc. `(("openssl" ,openssl) ("zlib" ,zlib))) (home-page "https://libgit2.github.com/") (synopsis "Library providing Git core methods") (description "Libgit2 is a portable, pure C implementation of the Git core methods provided as a re-entrant linkable library with a solid API, allowing you to write native speed custom Git applications in any language with bindings.") ;; GPLv2 with linking exception (license license:gpl2)))) #+END_SRC (In those cases were you only want to tweak a few fields from a package definition, you should rely on inheritance instead of copy-pasting everythi= ng. See below.) Let's discuss those fields in depth. ** ~git-fetch~ method Unlike the ~url-fetch~ method, ~git-fetch~ expects a ~git-reference~ which = takes a Git repository and a commit. The commit can be any Git reference such as tags, so if the ~version~ is tagged, then it can be used directly. Sometim= es the tag is prefixed with a ~v~, in which case you'd use ~(commit (string-ap= pend "v" version))~. When downloaded to the store, the source should be different from version to version. Since Git repositories don't contain a specific version in their = name, we've got to force the download folder using ~(file-name (git-file-name name version))~. Note that there is also a ~git-version~ procedure that can be used to deriv= e the version when packaging programs for a specific commit. ** Snippets Snippets are quoted (i.e. non-evaluated) Scheme code that are a mean of pat= ching the source. They are a Guix-y alternative to the traditional =3D.patch=3D = files. Because of the quote, the code in only evaluated when passed to the Guix da= emon for building. There can be as many snippet as needed. Snippets might need additional Guile modules which can be imported from the ~modules~ field. ** Inputs First, a syntactic comment: See the quasi-quote / comma syntax? #+BEGIN_SRC scheme (native-inputs `(("pkg-config" ,pkg-config))) #+END_SRC is equivalent to #+BEGIN_SRC scheme (native-inputs (list (list "pkg-config" pkg-config))) #+END_SRC You'll mostly see the former because it's shorter. There are 3 different input types. In short: =2D native-inputs :: Required for building but not runtime -- installing a = package through a substitute won't install these inputs. =2D inputs :: Installed in the store but not in the profile, as well as bei= ng present at build time. =2D propagated-inputs :: Installed in the store and in the profile, as well= as being present at build time. See [[https://www.gnu.org/software/guix/manual/en/html_node/package-Referen= ce.html][the package reference in the manual]] for more details. The distinction between the various inputs is important: if a dependency ca= n be handled as an /input/ instead of a /propagated input/, it should be done so= , or else it "pollutes" the user profile for no good reason. For instance, a user installing a graphical program that depends on a commandline tool might only be interested in the graphical part, so there i= s no need to force the commandline tool into the user profile. The dependency i= s a concern to the package, no to the user. /Inputs/ make it possible to handle dependencies without bugging the user by adding undesired executable files = (or libraries) to their profile. Same goes for /native-inputs/: once the program is installed, build-time dependencies can be safely garbage-collected. It also matters when a substitute is available, in which case only the /inp= uts/ and /propagated inputs/ will be fetched: the /native inputs/ are not requir= ed to install a package from a substitute. ** Outputs Just like how a package can have multiple inputs, it can also produce multi= ple outputs. Each output corresponds to a separate folder in the store. The end user can choose which output to install; this is useful to save spa= ce or to avoid polluting the user profile with unwanted executables or libraries. Output separation is optional. When the ~outputs~ field is left out, the default and only output (the complete package) is referred to as ~"out"~. Typical separate output names include ~debug~ and ~doc~. It's advised to separate outputs only when you've shown it worth: if the ou= tput size is significant (compare with ~guix size~) or in case the package is modular. ** Build system arguments The ~arguments~ is a keyword-value list used to configure the build process. The simplest argument ~#:tests?~ can be used to disable the test suite when building the package. This is mostly useful when the package does not feat= ure any test suite. It's strongly recommended to keep the test suite on if the= re is one. Another common argument is ~:make-flags~, which specifies a list of flags = to append when running make, as you would from the commandline. For instance,= the following flags #+BEGIN_SRC scheme #:make-flags (list (string-append "prefix=3D" (assoc-ref %outputs = "out")) "CC=3Dgcc") #+END_SRC translate into #+BEGIN_SRC sh $ make CC=3Dgcc prefix=3D/gnu/store/...- #+END_SRC This sets the C compiler to ~gcc~ and the ~prefix~ variable (the installati= on folder in Make parlance) to ~(assoc-ref %outputs "out")~, which is a build-= stage global variable pointing to the destination folder in the store (something = like =3D/gnu/store/...-my-mg-20180408=3D). Similarly, it's possible to set the "configure" flags. In the above exampl= e, we set the extensions to the store download folder of our fictitious mg-extens= ions. #+BEGIN_SRC scheme #:configure-flags `(,(string-append "--with-ext=3D" (assoc-ref %build-inputs "mg-e= xtensions"))) #+END_SRC The ~%build-inputs~ variable is also generated in scope. It's an associati= on table that maps the input names to their store folders. The ~phases~ keyword lists the sequential steps of the build system. Typic= ally phases include ~unpack~, ~configure~, ~build~, ~install~ and ~check~. To k= now more about those phases, you need to work out the appropriate build system definition in =3D$GUIX_CHECKOUT/guix/build/gnu-build-system.scm=3D: #+BEGIN_SRC scheme (define %standard-phases ;; Standard build phases, as a list of symbol/procedure pairs. (let-syntax ((phases (syntax-rules () ((_ p ...) `((p . ,p) ...))))) (phases set-SOURCE-DATE-EPOCH set-paths install-locale unpack bootstrap patch-usr-bin-file patch-source-shebangs configure patch-generated-file-shebangs build check install patch-shebangs strip validate-runpath validate-documentation-location delete-info-dir-file patch-dot-desktop-files install-license-files reset-gzip-timestamps compress-documentation))) #+END_SRC Or from the REPL (assuming the Guix source is in your Guile load path): #+BEGIN_SRC scheme > ,module (guix build gnu-build-system) > (map first %standard-phases) (set-SOURCE-DATE-EPOCH set-paths install-locale unpack bootstrap patch-usr-= bin-file patch-source-shebangs configure patch-generated-file-shebangs buil= d check install patch-shebangs strip validate-runpath validate-documentatio= n-location delete-info-dir-file patch-dot-desktop-files install-license-fil= es reset-gzip-timestamps compress-documentation) #+END_SRC If you want to know more about what happens during those phases, consult the associated functions. For instance, as of this writing the definition of ~unpack~ for the GNU bui= ld system is #+BEGIN_SRC scheme (define* (unpack #:key source #:allow-other-keys) "Unpack SOURCE in the working directory, and change directory within the source. When SOURCE is a directory, copy it in a sub-directory of the curr= ent working directory." (if (file-is-directory? source) (begin (mkdir "source") (chdir "source") ;; Preserve timestamps (set to the Epoch) on the copied tree so that ;; things work deterministically. (copy-recursively source "." #:keep-mtime? #t)) (begin (if (string-suffix? ".zip" source) (invoke "unzip" source) (invoke "tar" "xvf" source)) (chdir (first-subdirectory ".")))) #t) #+END_SRC Note the ~chdir~ call: it changes the working directory to where the source= was unpacked. Thus every phase following the ~unpack~ will use the source as a working directory, which is why we can directly work on the source files. That is to say, unless a later phase changes the working directory to somet= hing else. To manipulate the phases, =2D ~add-before PHASE NEW-PHASE PROCEDURE~: Run ~PROCEDURE~ named ~NEW-PHAS= E~ before ~PHASE~. =2D ~add-after PHASE NEW-PHASE PROCEDURE~: Same, but afterwards. =2D ~replace PHASE PROCEDURE~. =2D ~delete PHASE~. The ~PROCEDURE~ supports the keyword arguments ~inputs~ and ~outputs~. Each input (whether /native/, /propagated/ or not) and output directory is refer= enced by their name in those variables. Thus ~(assoc-ref outputs "out")~ is the = store directory of the main output of the package. A phase procedure may look li= ke this: #+BEGIN_SRC scheme (lambda* (#:key inputs outputs #:allow-other-keys) (let (((bash-folder (assoc-ref inputs "bash")) (output-folder (assoc-ref outputs "out")) (doc-folder (assoc-ref outputs "doc")) ; ... #t) #+END_SRC The procedure must return ~#t~ on success. It's brittle to rely on the ret= urn value of the last expression used to tweak the phase because there is no guarantee it would be a ~#t~. Hence the trailing ~#t~ to ensure the right = value is returned on success. ** Code staging The astute reader may have noticed the quasi-quote and comma syntax in the argument field. Indeed, the build code in the package declaration should n= ot be evaluated on the client side, but only when passed to the Guix daemon. This mechanism of passing code around two running processes is called [[https://= arxiv.org/abs/1709.00833][code staging]]. ** "Utils" functions When customizing ~phases~, we often need to write code that mimics the equivalent system invocations (~make~, ~mkdir~, ~cp~, etc.) commonly used d= uring regular "Unix-style" installations. Some like ~chmod~ are native to Guile. See the [[https://www.gnu.org/softw= are/guile/manual/guile.html][Guile reference manual]] for a complete list. Guix provides additional helper functions which prove especially handy in t= he context of package management. Some of those functions can be found in =3D$GUIX_CHECKOUT/guix/guix/build/utils.scm=3D. Most of them mirror the be= haviour of the traditional Unix system commands: =2D which :: Like the =3Dwhich=3D system command. =2D find-files :: Akin to the =3Dfind=3D system command. =2D mkdir-p :: Like =3Dmkdir -p=3D, which creates all parents as needed. =2D install-file :: Similar to =3Dinstall=3D when installing a file to a (p= ossibly non-existing) directory. Guile has ~copy-file~ which wor= ks like =3Dcp=3D. =2D copy-recursively :: Like =3Dcp -r=3D. =2D delete-file-recursively :: Like =3Drm -rf=3D. =2D invoke :: Run an executable. This should be used instead of ~system*~. =2D with-directory-excursion :: Run the body in a different working directo= ry, then restore the previous working directory. =2D substitute* :: A "sed-like" function. ** Module prefix The license in our last example needs a prefix: this is because of how the ~license~ module was imported in the package, as ~#:use-module ((guix licen= ses) #:prefix license:)~. The [[https://www.gnu.org/software/guile/manual/html_= node/Using-Guile-Modules.html][Guile module import mechanism]] gives the us= er full control over namespacing: this is needed to avoid clashes between, say, the =3Dzlib=3D variable from =3Dlicenses.scm=3D and the =3Dzlib=3D variable from =3Dcompression.scm=3D. * Other build systems What we've seen so far covers the majority of packages using a build system other than the ~trivial-build-system~. The latter does not automate anythi= ng and leaves you to build everything manually. This can be more demanding an= d we won't cover it here for now, but thankfully it is rarely necessary to fall = back on this system. For the other build systems, such as ASDF, Emacs, Perl, Ruby and many more,= the process is very similar to the GNU build system but for a few specialized arguments. Find more about build systems in =2D [[https://www.gnu.org/software/guix/manual/en/html_node/Build-Systems.h= tml#Build-Systems][the manual, section 4.2 Build systems]], =2D the source code in the =3D$GUIX_CHECKOUT/guix/build=3D and =3D$GUIX_CHECKOUT/guix/build-system=3D folders. * Common pitfalls and best practices The following check list is important to maintain a high packaging standard= , in particular when contributing to the Guix project. =2D Use the /linter/: ~guix lint $PACKAGES~. =2D Use mirrors when possible in the source URL. =2D Use reliable URLs, not generated ones (e.g. GitHub archives). Don't use the ~name~ field in the URL: it is not very useful and if the n= ame changes, the URL will probably be wrong. =2D Sort inputs alphabetically. =2D Synopsis should be as concise as possible, don't start with "The" or "A= ", don't repeat the package name. =2D Description uses Texinfo syntax with two spaces at the end of sentences. =2D Reproducibility: use ~guix build --check~ and ~guix build --rounds=3DN~. =2D Follow the [[https://www.gnu.org/software/guix/manual/en/html_node/Codi= ng-Style.html][coding style]] from the manual. =2D Review the [[https://www.gnu.org/software/guix/manual/en/html_node/Subm= itting-Patches.html][check list]] from the manual. * Programmable and automated package definition We can't repeat it enough: having a full-fledged programming language at ha= nd empowers us in ways that reach far beyond traditional package management. Let's illustrate this with some awesome features of Guix! ** Recursive importers You might find some build systems good enough that there is little to do at= all to write a package, to the point that it becomes repetitive and tedious aft= er a while. A /raison d'=C3=AAtre/ of computers is to replace human beings at t= hose boring tasks. So let's tell Guix to do this for us and create the package definition of an ELPA package: #+BEGIN_SRC sh $ guix import elpa exwm (package (name "emacs-exwm") (version "0.19") (source (origin (method url-fetch) (uri (string-append "https://elpa.gnu.org/packages/exwm-" version ".tar")) (sha256 (base32 "11xd2w4h3zdwkdxypvmcz8s7q72cn76lfr9js77jbizyj6b04lr0")))) (build-system emacs-build-system) (propagated-inputs `(("emacs-xelb" ,emacs-xelb))) (home-page "https://github.com/ch11ng/exwm") (synopsis "Emacs X Window Manager") (description "EXWM (Emacs X Window Manager) is a full-featured tiling X window manag= er for Emacs built on top of [XELB](https://github.com/ch11ng/xelb). It features: + Fully keyboard-driven operations + Hybrid layout modes (tiling & stacking) + Dynamic workspace support + ICCCM/EWMH compliance + (Optional) RandR (multi-monitor) support + (Optional) Built-in system tray") (license license:gpl3+)) #+END_SRC Not all applications can be packaged this way, only those relying on a sele= ct number of supported systems. Read about the full list of importers in the = [[https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-impor= t.html][guix import section]] of the manual. ** Automatic update Guix can be smart enough to check for updates on systems it knows. It can report outdated package definitions with #+BEGIN_SRC sh $ guix refresh hello #+END_SRC In most cases, updating a package to a newer version requires little more t= han changing the version number and the checksum. Guix can do that automatical= ly as well: #+BEGIN_SRC $ guix refresh hello --update #+END_SRC ** Inheritance If you've started browsing the existing package definitions, you might have noticed that a significant number of them have a ~inherit~ field: #+BEGIN_SRC scheme (define-public adwaita-icon-theme (package (inherit gnome-icon-theme) (name "adwaita-icon-theme") (version "3.26.1") (source (origin (method url-fetch) (uri (string-append "mirror://gnome/sources/" name "/" (version-major+minor version) "/" name "-" version ".tar.xz")) (sha256 (base32 "17fpahgh5dyckgz7rwqvzgnhx53cx9kr2xw0szprc6bnqy977fi8")))) (native-inputs `(("gtk-encode-symbolic-svg" ,gtk+ "bin"))))) #+END_SRC All unspecified fields inherit from the parent package. This is very conve= nient to create alternative packages, for instance with different source, version= or compilation options. * Getting help Sadly some applications can be tough to package. Sometimes they need a pat= ch to work with the non-standard filesystem hierarchy enforced by the store. Sometimes the tests won't run properly. (They can be skipped but this is n= ot recommended.) Other times the resulting package won't be reproducible. Should you be stuck, unable to figure out how to fix any sort of packaging issue, don't hesitate to ask the community for help. See the [[https://www.gnu.org/software/guix/contact/][Guix homepage]] for i= nformation on the mailing lists, IRC, etc. * Conclusion This tutorial is an introductory showcase of how Guix can bring automation = to the field of packaging. The GNU build system is the flagship of the high-l= evel abstraction layers Guix makes possible to build. Now where do we go from here? Next we ought to dissect the innards of the = build system by removing all abstractions, using the ~trivial-build-system~: this should give us a thorough understanding of the process before investigating= some more advanced packaging techniques and edge cases. Other features worth exploring are the interactive editing and debugging capabilities of Guix provided by the Guile REPL. * References =2D The [[https://www.gnu.org/software/guix/manual/en/html_node/Defining-Pa= ckages.html][package reference in the manual]] =2D [[https://gitlab.com/pjotrp/guix-notes/blob/master/HACKING.org][Pjotr= =E2=80=99s hacking guide to GNU Guix]] =2D "Guix Guix: Package without a scheme!", by Andreas Enge (in =3Dguix-maintenance.git/talks/ghm-2013/andreas/slides.pdf=3D) * About GNU Guix [[https://www.gnu.org/software/guix][GNU=C2=A0Guix]] is a transactional pac= kage manager for the GNU system. The Guix System Distribution or GuixSD is an advanced distribution of the GNU system that r= elies on GNU Guix and [[https://www.gnu.org/distros/free-system-distribution-guid= elines.html][respects the user's freedom]]. In addition to standard package management features, Guix supports transact= ional upgrades and roll-backs, unprivileged package management, per-user profiles= , and garbage collection. Guix uses low-level mechanisms from the Nix package manager, except that packages are defined as native [[https://www.gnu.org/s= oftware/guile][Guile]] modules, using extensions to the [[http://schemers.org][Scheme]] language. GuixSD offers = a declarative approach to operating system configuration management, and is highly customizable and hackable. GuixSD can be used on an i686, x86_64 and armv7 machines. It is also possi= ble to use Guix on top of an already installed GNU/Linux system, including on mips64el and aarch64. --=-=-=-- --==-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAlu1B44ACgkQm9z0l6S7 zH+6cQgAnLvqunCLqV2nOvhqtxA87SyVIRbOXbhJ1VEwFAA7/EEYRPn3R6Z88l1Q SPGQWyMkHO349p7b2wfV3JPa/JQMjI3y7hrSAsSaWYClplZsbCVxgCCapSGjAQtG HlB6nD3J10o6vKBVEGtUILkY8pGYdt2jYv7HevZz5WRhLuuDMx4LLNa1bD4vqNrz Yvzln9cyp4DF0jcER5H5qqwZ96C5/FlWWNI1V2sRGBoDCD3l4asiqlDJXeHkk2rv L+UswHp1NSCj/WKm2LDXQi6YQlzG5AzEmJKDPZ6KufZ0fGug2W7FoOdzQ2DP7L7a kp9jCbtbgFgFKXuQdtY0vulEJSCHeg== =Eugf -----END PGP SIGNATURE----- --==-=-=--