Re: WIP: zig-build-system

[Ekaitz Zarraga <ekaitz@elenq.tech>]

[-- Attachment #1: Type: text/plain, Size: 6310 bytes --]

Hi Efraim and everyone,

> cross-compilation can come later. It's not uncommon when writing new
> build systems.

Great!

> > - The `zig` command uses a compiler to find the libc it needs to use
> > we should be able to remove the `gcc-toolchain` from the
> > dependencies if we just add the libc to the build system properly
> > (this point is mixed with the point above)
>
>
> Are you sure about removing gcc-toolchain? I don't have a problem with
> it if it is possible, especially since zig does provide a 'zig cc'
> command. We'd still likely need to use something like the
> make-gcc-toolchain to properly use ld-wrapper, unless we somehow
> wouldn't need that either.

I managed to remove it from the package descriptions because, as you said, it
comes with the gnu-build-system that I used as a base, but we still need to
(setenv "CC" "gcc") and I don't really like it because I think this will cause
problems in other usecases... Like cross-compilation.

I have to ask the Zig people (hello? Anyone?) about how important having a GCC
around is, maybe we can substitute that GCC call with something else that we
add to the build system. Simply finding the correct libc should be something we
can do.

I've found here and there that the Zig compiler supports a `--libc` argument
which receives a text file as an input, and they probably build that input from
the GCC command. I'm pretty sure that we can build that from our
zig-build-system directly, using scheme code. That way we could tweak it in the
future with no reliance on GCC, which is way more flexible.
Also, I don't like relying in GCC's output a lot because it's a supercomplex
package and it's hard to maintain/understand. If we can stay away from it,
better for everyone.

> Isn't gcc-toolchain part of %standard-packages in a build-system? Did I
> miss it not being there or does it also need to be added here for the
> reason you listed below?

True. Removed from the package as the build-system includes it.

> by default glibc is named 'libc' in the build system, and when
> cross-compiling 'libc' is generally the correct libc that you'd be
> looking for. Also I'd be curious about reusing the libc vendored in zig
> itself or if that's not actually possible/only for embedded uses.
>
> Also if you just need libc you could import glibc-final for the
> non-cross-build as a standard native-input (input?) and then you'd have
> it available.

I'm not sure if I understood this (which probably means I didn't). Can you give
me an example about how to use the `libc` in the build system?
I'm not used to the gexps yet and I don't find the best way to do this :S

The libc that comes with Zig we can use, but in that case we have to check the
system and all that so maybe it's better to leave it for the cross-compilation
step. Also, I'm not sure if it will affect reproducibility somehow?

> > + (add-before 'build 'set-env
> > + (lambda _
> > + (setenv "CC" "gcc"))))))
>
>
> If you really don't need gcc by default you could consider something
> like what we have in the cargo-build-system:
>
> (when (assoc-ref inputs "openssl")
> (setenv "OPENSSL_DIR" (assoc-ref inputs "openssl")))
>
> could be:
>
> (cond
> ((assoc-ref inputs "gcc") (setenv "CC" "gcc"))
> ((assoc-ref inputs "clang") (setenv "CC" "llvm"))
> (_ (setenv "CC" "zig cc")))

That's an accurate suggestion, but we need to solve first if we need to
actually use the external compiler or not. I need a Zig mentor for this :D

> > +(define (default-zig)
> > + "Return the default zig package."
> > + ;; Lazily resolve the binding to avoid a circular dependency.
> > + (let ((zig (resolve-interface '(gnu packages zig))))
> > + (module-ref zig 'zig)))
> > +
> > +(define %zig-build-system-modules
> > + ;; Build-side modules imported by default.
> > + `((guix build zig-build-system)
> > + (guix build syscalls)
> > + ,@%gnu-build-system-modules))
> > +
> > +(define* (zig-build name inputs
> > + #:key
> > + source
> > + (tests? #t)
> > + (test-target #f)
> > + (zig-build-flags ''())
>
>
> Is it possible to pass flags during the test phase also? If it is then
> having a zig-test-flags keyword would also be good.

I think it is! Good catch.

> I would also suggest a 'zig' flag, to allow choosing a different zig
> binary to be used for a build.

Done.

> I would define a global-cache-dir and use ZIG_GLOBAL_CACHE_DIR to refer
> to it and not pass it around to all the phases.

I don't know if I understood what you mean, check the new patch set please and
confirm.

> > + "-p" "" ;; Don't add /usr
>
> If '-p' is the short form of '--prefix' or something then use the long
> form.

Done

> cmake-build-system has an option to determine Release or Debug (or to
> leave it as RelWithDebInfo) named 'build-type'. I think that should be
> added here.

I did something similar, check please if you like it.

> I assume this is to prevent overwriting the actual binary with the tests
> enabled binary? I would consider unsetting it at the end of the phase.
> Or storing its previous value and setting it back to that, we have seen
> packages which use DESTDIR and I wouldn't want any surprises.

Good, applied.

> > + (apply invoke
> > + `("zig" "build" "test"
> > + "--global-cache-dir" "zig-cache"))))
>
> What's the difference between 'zig test build.zig' and 'zig build test'?

I'll give you a really naïve answer: in my computer the second works and the
first doesn't.

> Beware of copy-pasting :). You don't use inputs, don't need let* vs let,
> and could probably directly (copy-recursively #$output out). Or rely on
> 'make install'. 'make install' was available for ncdu2, not sure about
> other zig projects.

Make install won't be available generally, but yeah, I got you.
Should this install phase be something like:
`(map (lambda (out) (copy-recursively (car out) (cdr out))) outputs)`

Maybe this allows an easier way to generate outputs? Or should we just rely on
the packages themselves to define a better install phase than this?
I mean, if we have just this phase, any package that needs to install docs,
just adding a `doc` output and leaving its contents in a `/doc` folder should
work... IDK

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-WIP-Add-a-zig-build-system.patch --]
[-- Type: text/x-patch; name=0001-WIP-Add-a-zig-build-system.patch, Size: 10503 bytes --]

From b868123d95429025f67b879985514083510f6acb Mon Sep 17 00:00:00 2001
From: Ekaitz Zarraga <ekaitz@elenq.tech>
Date: Thu, 15 Dec 2022 13:01:23 +0100
Subject: [PATCH 1/2] WIP: Add a zig-build-system

---
 Makefile.am                               |   2 +
 etc/snippets/yas/scheme-mode/guix-package |   5 +-
 guix/build-system/zig.scm                 | 119 ++++++++++++++++++++++
 guix/build/zig-build-system.scm           |  94 +++++++++++++++++
 4 files changed, 218 insertions(+), 2 deletions(-)
 create mode 100644 guix/build-system/zig.scm
 create mode 100644 guix/build/zig-build-system.scm

diff --git a/Makefile.am b/Makefile.am
index b54288c0fc..4cc7c2e5cf 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -179,6 +179,7 @@ MODULES =					\
   guix/build-system/scons.scm			\
   guix/build-system/texlive.scm			\
   guix/build-system/trivial.scm			\
+  guix/build-system/zig.scm			\
   guix/ftp-client.scm				\
   guix/http-client.scm				\
   guix/gnupg.scm				\
@@ -263,6 +264,7 @@ MODULES =					\
   guix/build/graft.scm				\
   guix/build/bournish.scm			\
   guix/build/qt-utils.scm			\
+  guix/build/zig-build-system.scm		\
   guix/build/make-bootstrap.scm			\
   guix/search-paths.scm				\
   guix/packages.scm				\
diff --git a/etc/snippets/yas/scheme-mode/guix-package b/etc/snippets/yas/scheme-mode/guix-package
index 724a392f81..3bb6307659 100644
--- a/etc/snippets/yas/scheme-mode/guix-package
+++ b/etc/snippets/yas/scheme-mode/guix-package
@@ -43,8 +43,9 @@
                                           "scons-build-system"
                                           "texlive-build-system"
                                           "trivial-build-system"
-                                          "waf-build-system")})
+                                          "waf-build-system"
+                                          "zig-build-system")})
     (home-page "$4")
     (synopsis "$5")
     (description "$6")
-    (license $7)))
\ No newline at end of file
+    (license $7)))
diff --git a/guix/build-system/zig.scm b/guix/build-system/zig.scm
new file mode 100644
index 0000000000..77010f1bed
--- /dev/null
+++ b/guix/build-system/zig.scm
@@ -0,0 +1,119 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Ekaitz Zarraga <ekaitz@elenq.tech>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix build-system zig)
+  #:use-module (guix search-paths)
+  #:use-module (guix store)
+  #:use-module (guix utils)
+  #:use-module (guix gexp)
+  #:use-module (guix monads)
+  #:use-module (guix packages)
+  #:use-module (guix build-system)
+  #:use-module (guix build-system gnu)
+  #:use-module (ice-9 match)
+  #:use-module (srfi srfi-26)
+  #:export (zig-build-system))
+
+
+(define (default-zig)
+  "Return the default zig package."
+  ;; Lazily resolve the binding to avoid a circular dependency.
+  (let ((zig (resolve-interface '(gnu packages zig))))
+    (module-ref zig 'zig)))
+
+(define %zig-build-system-modules
+  ;; Build-side modules imported by default.
+  `((guix build zig-build-system)
+    (guix build syscalls)
+    ,@%gnu-build-system-modules))
+
+(define* (zig-build name inputs
+                    #:key
+                    source
+                    (tests? #t)
+                    (test-target #f)
+                    (zig-build-flags ''())
+                    (phases '%standard-phases)
+                    (outputs '("out"))
+                    (search-paths '())
+                    (system (%current-system))
+                    (guile #f)
+                    (imported-modules %zig-build-system-modules)
+                    (modules '((guix build zig-build-system)
+                               (guix build utils))))
+  "Build SOURCE using Zig, and with INPUTS."
+  (define builder
+    (with-imported-modules imported-modules
+      #~(begin
+          (use-modules #$@(sexp->gexp modules))
+          (zig-build #:name #$name
+                     #:source #+source
+                     #:system #$system
+                     #:test-target #$test-target
+                     #:zig-build-flags #$zig-build-flags
+                     #:tests? #$tests?
+                     #:phases #$phases
+                     #:outputs #$(outputs->gexp outputs)
+                     #:search-paths '#$(sexp->gexp
+                                        (map search-path-specification->sexp
+                                             search-paths))
+                     #:inputs #$(input-tuples->gexp inputs)))))
+
+  (mlet %store-monad ((guile (package->derivation (or guile (default-guile))
+                                                  system #:graft? #f)))
+    (gexp->derivation name builder
+                      #:system system
+                      #:guile-for-build guile)))
+
+(define* (lower name
+                #:key source inputs native-inputs outputs system target
+                (zig (default-zig))
+                #:allow-other-keys
+                #:rest arguments)
+  "Return a bag for NAME."
+
+  (define private-keywords
+    '(#:target #:zig #:inputs #:native-inputs #:outputs))
+
+  ;; TODO: support cross-compilation
+  ;; It's as simple as adding some build flags to `zig-build-flags`
+  (and (not target)
+       (bag
+         (name name)
+         (system system)
+         (target target)
+         (host-inputs `(,@(if source
+                              `(("source" ,source))
+                              '())
+                        ,@inputs
+
+                        ;; Keep the standard inputs of 'gnu-build-system'
+                        ;; TODO: do we need this?
+                        ,@(standard-packages)))
+         (build-inputs `(("zig" ,zig)
+                         ,@native-inputs))
+         (outputs outputs)
+         (build zig-build)
+         (arguments (strip-keyword-arguments private-keywords arguments)))))
+
+(define zig-build-system
+  (build-system
+    (name 'zig)
+    (description
+     "Zig build system, to build Zig packages")
+    (lower lower)))
diff --git a/guix/build/zig-build-system.scm b/guix/build/zig-build-system.scm
new file mode 100644
index 0000000000..75a42543cf
--- /dev/null
+++ b/guix/build/zig-build-system.scm
@@ -0,0 +1,94 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Ekaitz Zarraga <ekaitz@elenq.tech>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix build zig-build-system)
+  #:use-module ((guix build gnu-build-system) #:prefix gnu:)
+  #:use-module (guix build utils)
+  #:use-module (ice-9 popen)
+  #:use-module (ice-9 rdelim)
+  #:use-module (ice-9 ftw)
+  #:use-module (ice-9 format)
+  #:use-module (ice-9 match)
+  #:use-module (rnrs io ports)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-26)
+  #:export (%standard-phases
+            zig-build))
+
+;; Interesting guide here:
+;; https://github.com/riverwm/river/blob/master/PACKAGING.md
+(define global-cache-dir "zig-cache")
+
+(define* (set-zig-global-cache-dir #:rest args)
+  (setenv "ZIG_GLOBAL_CACHE_DIR" global-cache-dir))
+
+(define* (build #:key
+                (zig "zig")
+                (zig-build-flags '())
+                zig-release-type      ;; "safe", "fast" or "small" empty for a
+                                      ;; debug build"
+                #:allow-other-keys)
+  "Build a given Zig package."
+  (setenv "DESTDIR" "out")
+  (let ((call `(,zig "build"
+                     "--prefix"             ""            ;; Don't add /usr
+                     "--prefix-lib-dir"     "lib"
+                     "--prefix-exe-dir"     "bin"
+                     "--prefix-include-dir" "include"
+                     ,@(if zig-release-type
+                         (list (string-append "-Drelease-" zig-release-type))
+                         '())
+                     ,@zig-build-flags)))
+  (format #t "running: ~s~%" call)
+  (apply invoke call)))
+
+(define* (check #:key tests?
+                (zig-test-flags '())
+                (zig "zig")
+                #:allow-other-keys)
+  "Run all the tests"
+  (when tests?
+    (let ((old-destdir (getenv "DESTDIR")))
+      (setenv "DESTDIR" "test-out") ;; Avoid colisions with the build output
+      (let ((call `(,zig "build" "test"
+                    ,@zig-test-flags)))
+        (format #t "running: ~s~%" call)
+        (apply invoke call))
+      (if old-destdir
+        (setenv "DESTDIR" old-destdir)
+        (unsetenv "DESTDIR")))))
+
+(define* (install #:key inputs outputs #:allow-other-keys)
+  "Install a given Zig package."
+  (let ((out (assoc-ref outputs "out")))
+    (copy-recursively "out" out)))
+
+(define %standard-phases
+  (modify-phases gnu:%standard-phases
+    (delete 'bootstrap)
+    (delete 'configure)
+    (add-before 'build 'set-zig-global-cache-dir set-zig-global-cache-dir)
+    (replace 'build build)
+    (replace 'check check)
+    (replace 'install install)))
+
+
+(define* (zig-build #:key inputs (phases %standard-phases)
+                      #:allow-other-keys #:rest args)
+  "Build the given Zig package, applying all of PHASES in order."
+  (apply gnu:gnu-build #:inputs inputs #:phases phases args))
-- 
2.38.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-Add-a-sample-package-for-testing-the-zig-build-syste.patch --]
[-- Type: text/x-patch; name=0002-Add-a-sample-package-for-testing-the-zig-build-syste.patch, Size: 2500 bytes --]

From 06e420c301b46a0db29546aa757c75abcd6e69bf Mon Sep 17 00:00:00 2001
From: Ekaitz Zarraga <ekaitz@elenq.tech>
Date: Thu, 15 Dec 2022 13:02:25 +0100
Subject: [PATCH 2/2] Add a sample package for testing the zig build system

---
 gnu/packages/zig.scm | 42 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 41 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/zig.scm b/gnu/packages/zig.scm
index cda93bed2e..43b2f60722 100644
--- a/gnu/packages/zig.scm
+++ b/gnu/packages/zig.scm
@@ -25,7 +25,9 @@ (define-module (gnu packages zig)
   #:use-module ((guix licenses) #:prefix license:)
   #:use-module (guix build-system cmake)
   #:use-module (gnu packages)
-  #:use-module (gnu packages llvm))
+  #:use-module (gnu packages llvm)
+  #:use-module (guix build-system zig)
+  #:use-module (gnu packages commencement))
 
 (define-public zig
   (package
@@ -104,3 +106,41 @@ (define-public zig
     ;; https://github.com/ziglang/zig/issues/6485
     (supported-systems %64bit-supported-systems)
     (license license:expat)))
+
+
+(define-public tigerbeetle
+  (let ((commit-id "2022-12-12-weekly")
+        (revision "0"))
+    (package
+      (name "tigerbeetle")
+      (version (string-append revision "-" commit-id))
+      (source
+        (origin
+          (method git-fetch)
+          (uri (git-reference
+                 (url "https://github.com/tigerbeetledb/tigerbeetle.git")
+                 (commit commit-id)))
+          (file-name (git-file-name name version))
+          (sha256
+            (base32 "18rawl8rhyplw8hpa3fzbq9fqg088x0calz688c7zdff6y6f6mcr"))))
+      (build-system zig-build-system)
+
+      (arguments
+         `(#:phases
+           (modify-phases %standard-phases
+          ;; TODO: Zig needs the gcc-toolchain in order to find the libc.
+          ;;       we need to think about how to solve this in the build system
+          ;;       directly: --libc
+                          (add-before 'build 'set-env
+                            (lambda _
+                              (setenv "CC" "gcc"))))))
+
+      (synopsis "Distributed financial accounting database designed for mission
+critical safety and performance")
+
+      (description "Financial accounting database designed for mission critical
+safet y and performance to power the future of financial
+services.")
+      (home-page "https://github.com/tigerbeetledb/tigerbeetle.git")
+      (supported-systems %64bit-supported-systems)
+      (license license:asl2.0))))
-- 
2.38.0