* On a Guile-based Build-Tool complimenting Guix @ 2024-12-19 20:12 Divya Ranjan 2024-12-19 21:08 ` Sergio Pastor Pérez ` (3 more replies) 0 siblings, 4 replies; 7+ messages in thread From: Divya Ranjan @ 2024-12-19 20:12 UTC (permalink / raw) To: guix-devel; +Cc: pjotr.public12 Hello Guix, The other day, after being frustrated of build systems (auto-tools, meson, maven etc.), I wondered why doesn’t Guix which has such powerful tools within it (`guix build`, `guix pack` etc.) also not have a purely Guile-based build tool? After all, our goal is to make deployment, and building both more declarative and away from the all-too-familiar “dependency hell”. I am not exactly sure how I want to go with this, but I want to extend (if needed) the capabilities of Guix, to allow the developer of a package to use it also to build the package effectively replacing existing build tools. Since we already have build-system, instead of executing make (or whatever other tool) commands from it, we could modify it to itself have all those things that a Makefile would have. The developer would use Guile to declare their build config, I am again not sure what this might exactly look like, but can’t we have it such that we provide the developer with some tools to _declare_ a custom and package-specific build-system in Guile (just like our familiar gnu-build-system), but this is purely in Guile and executes whatever commands, path declarations and other interactions (such as calling gcc) directly from Guile and not by just calling `make`. I haven’t thought through this clearly, but even if this doesn’t work, the main idea I’d like to propose is to fully replace existing build-tools by making a new Guile build tool to work alongside Guix. Ideally, once the developer has a build config ready, one can just wrap it up with a package definition in Guile, just like the ones we create to package something upstream. This package definition can then be used in `guix build -f package.scm` to result in a fully transactional building process that focuses not on getting out of dependency hell, but just declaring your config right. And all of this without having to learn yet another build tool that might disappear, and of course, without leaving the comfortable world of Lisp (Scheme). I was indicated by others that such an idea has previously been conceievd among Guix developers themselves, namely as a GSoC proposal[0]. I couldn’t find if that has progressed towards anything, nor could find anything in the mailing list. I did see Pjotr volunteering to mentor for it, I’ve CC-ed them to see if they’re still interested in such a project. Meanwhile, I’d like input from other Guix core developers on what they think of this, and if they could provide me with some leads on where to go with this. [0]: https://libreplanet.org/wiki/Group:Guix/GSoC-2024 Regards, -- Divya Ranjan, Philosophy, Mathematics, Libre Software. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: On a Guile-based Build-Tool complimenting Guix 2024-12-19 20:12 On a Guile-based Build-Tool complimenting Guix Divya Ranjan @ 2024-12-19 21:08 ` Sergio Pastor Pérez 2024-12-19 21:17 ` Janneke Nieuwenhuizen ` (2 subsequent siblings) 3 siblings, 0 replies; 7+ messages in thread From: Sergio Pastor Pérez @ 2024-12-19 21:08 UTC (permalink / raw) To: Divya Ranjan, guix-devel; +Cc: pjotr.public12, ngraves Hello Divya! I would like to show interest in having a Guile based build-tool. If the community decides this is something worth putting effort into, I would like to participate. While I don’t have a specific starting point in mind, I believe that an essential feature such a tool, is the ability of continuing builds from successful phases. This is better described in this issue[1] where Nicolas made great progress towards allowing Guix to do it. [1]: https://issues.guix.gnu.org/68315 Thanks for showing interest in improving Guix! Regards, Sergio ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: On a Guile-based Build-Tool complimenting Guix 2024-12-19 20:12 On a Guile-based Build-Tool complimenting Guix Divya Ranjan 2024-12-19 21:08 ` Sergio Pastor Pérez @ 2024-12-19 21:17 ` Janneke Nieuwenhuizen 2024-12-20 0:06 ` indieterminacy 2024-12-19 21:33 ` Olivier Dion 2024-12-20 17:31 ` Attila Lendvai 3 siblings, 1 reply; 7+ messages in thread From: Janneke Nieuwenhuizen @ 2024-12-19 21:17 UTC (permalink / raw) To: Divya Ranjan; +Cc: guix-devel, pjotr.public12 Divya Ranjan writes: Hello, > The other day, after being frustrated of build systems (auto-tools, > meson, maven etc.), I wondered why doesn’t Guix which has such > powerful tools within it (`guix build`, `guix pack` etc.) also not > have a purely Guile-based build tool? After all, our goal is to make > deployment, and building both more declarative and away from the > all-too-familiar “dependency hell”. [..] > I was indicated by others that such an idea has previously been > conceievd among Guix developers themselves, namely as a GSoC > proposal[0]. I couldn’t find if that has progressed towards anything, > nor could find anything in the mailing list. From my notes I have * make in Guile: potato-make, bespoke ** https://github.com/spk121/potato-make ** https://codeberg.org/lechner/bespoke but there's probably more. And then there's also the Guix Workflow Language. Greetings, Janneke -- Janneke Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond https://LilyPond.org Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: On a Guile-based Build-Tool complimenting Guix 2024-12-19 21:17 ` Janneke Nieuwenhuizen @ 2024-12-20 0:06 ` indieterminacy 0 siblings, 0 replies; 7+ messages in thread From: indieterminacy @ 2024-12-20 0:06 UTC (permalink / raw) To: Janneke Nieuwenhuizen; +Cc: Divya Ranjan, guix-devel, pjotr.public12 On 2024-12-19 22:17, Janneke Nieuwenhuizen wrote: > Divya Ranjan writes: > > Hello, > >> The other day, after being frustrated of build systems (auto-tools, >> meson, maven etc.), I wondered why doesn’t Guix which has such >> powerful tools within it (`guix build`, `guix pack` etc.) also not >> have a purely Guile-based build tool? After all, our goal is to make >> deployment, and building both more declarative and away from the >> all-too-familiar “dependency hell”. > > [..] > >> I was indicated by others that such an idea has previously been >> conceievd among Guix developers themselves, namely as a GSoC >> proposal[0]. I couldn’t find if that has progressed towards anything, >> nor could find anything in the mailing list. > > From my notes I have > > * make in Guile: potato-make, bespoke > ** https://github.com/spk121/potato-make > ** https://codeberg.org/lechner/bespoke > > but there's probably more. And then there's also the Guix Workflow > Language. > Dont forget CWL. Pjotr Prins did a great writeup on an approach 5 years ago: http://hpc.guix.info/blog/2019/01/creating-a-reproducible-workflow-with-cwl/ Greetings, Jonathan ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: On a Guile-based Build-Tool complimenting Guix 2024-12-19 20:12 On a Guile-based Build-Tool complimenting Guix Divya Ranjan 2024-12-19 21:08 ` Sergio Pastor Pérez 2024-12-19 21:17 ` Janneke Nieuwenhuizen @ 2024-12-19 21:33 ` Olivier Dion 2024-12-21 7:53 ` Pjotr Prins 2024-12-20 17:31 ` Attila Lendvai 3 siblings, 1 reply; 7+ messages in thread From: Olivier Dion @ 2024-12-19 21:33 UTC (permalink / raw) To: Divya Ranjan, guix-devel; +Cc: pjotr.public12 Hi, For what it is worth, I have been using my own build system purely in Guile for a few years now. I use it in my main project libpatch (https://git.sr.ht/~old/libpatch/tree). It is compatible with the gnu-build-system (configure && make && make check && make install), but it does not have to if you want to diverge from this flow. However, it is order of magnitude faster than the autotools stuff. Configuration for example is a single Guile instance that may do some fork, but not that much. Building is also fast, althought it would be optimal if fibers were available (I use par-map but the bottleneck is due to waitpid). I have plan to add guile-parallel to it instead since it is also purely Guile based. My project is a mix of C, C++ and Guile. Thus I've only added support for these, but I it is possible to extend it since it is base on GOOPS. It is tested on Ubuntu, Debian, Fedora, Arch Linux and Guix. Here are some snippets of what it looks like: project/libpatch/build/libraries.scm (C libraries): --8<---------------cut here---------------start------------->8--- (define-module (project libpatch build libraries) #:use-module ((config) #:prefix conf:) #:use-module (ice-9 match) #:use-module (project build c) #:use-module (project utils list) #:use-module (srfi srfi-26) #:export (libpatch libpatch-ftrace)) (define libpatch-common (c-binary (inputs '("src/lib/libpatch-common/io.c" "src/lib/libpatch-common/read.c" "src/lib/libpatch-common/sleep.c" "src/lib/libpatch-common/write.c")) (cppflags (list (string-append "-I" conf:srcdir "/src/lib/libpatch-common"))) (library? #t) (shared? #f) (output "src/lib/libpatch-common.a"))) (define libpatch (c-binary (inputs (cons libpatch-common (cond-list '("src/lib/libpatch/core/address.c" "src/lib/libpatch/core/attr.c" "src/lib/libpatch/core/canon.c" "src/lib/libpatch/core/commit.c" "src/lib/libpatch/core/configure.c" "src/lib/libpatch/core/const.c" "src/lib/libpatch/core/coverage.c" "src/lib/libpatch/core/disable.c" "src/lib/libpatch/core/disasm.c" "src/lib/libpatch/core/drop.c" "src/lib/libpatch/core/dump.c" "src/lib/libpatch/core/dwarf.c" "src/lib/libpatch/core/dynamic.c" "src/lib/libpatch/core/enable.c" "src/lib/libpatch/core/fini.c" "src/lib/libpatch/core/gc.c" "src/lib/libpatch/core/global.c" "src/lib/libpatch/core/init.c" "src/lib/libpatch/core/integrity.c" "src/lib/libpatch/core/last-error.c" "src/lib/libpatch/core/make.c" "src/lib/libpatch/core/olx.c" "src/lib/libpatch/core/panic.c" "src/lib/libpatch/core/patch.c" "src/lib/libpatch/core/resolve.c" "src/lib/libpatch/core/states.c" "src/lib/libpatch/core/thread.c" "src/lib/libpatch/core/tls.c" "src/lib/libpatch/core/trampoline.c" "src/lib/libpatch/core/wxe.c" "src/lib/libpatch/core/page-allocator.c") ((string=? conf:arch "x86_64") "src/lib/libpatch/x86_64/const.c" "src/lib/libpatch/x86_64/disasm.c" "src/lib/libpatch/x86_64/dwarf.c" "src/lib/libpatch/x86_64/dynamic.c" "src/lib/libpatch/x86_64/generic-handlers.S" "src/lib/libpatch/x86_64/init.c" "src/lib/libpatch/x86_64/patch.c" "src/lib/libpatch/x86_64/ret-handlers.S")))) (external-dependencies (list conf:capstone conf:libdw conf:libolx conf:liburcu-bp)) (cppflags (list (string-append "-I" conf:srcdir "/src/lib/libpatch") (string-append "-I" conf:srcdir "/src/lib/libpatch-common") (string-append "-I" conf:srcdir "/src/lib/libpatch/" conf:arch "/include") "-DLIBPATCH_CORE")) (cflags '("-fvisibility=hidden")) (libs '("-pthread")) (shared? #t) (library? #t) (output "src/lib/libpatch.so") (install-location conf:libdir))) (define libpatch-ftrace (c-binary (enabled? conf:build-libpatch-ftrace) (inputs (list libpatch "src/lib/libpatch-ftrace/libpatch-ftrace.c")) (external-dependencies (list conf:lttng-ust)) (cppflags (list (string-append "-I" conf:srcdir "/src/lib/libpatch-ftrace"))) (cflags (list "-fvisibility=hidden")) (shared? #t) (library? #t) (output "src/lib/libpatch-ftrace.so") (install-location conf:libdir))) --8<---------------cut here---------------end--------------->8--- project/libpatch/build/bindings.scm (Guile bindings to Libpatch): --8<---------------cut here---------------start------------->8--- (define-module (project libpatch build bindings) #:use-module ((config) #:prefix conf:) #:use-module (project build) #:use-module (project build guile) #:use-module (project libpatch build libraries) #:export (bindings)) (define guile-go-dir (string-append conf:libdir "/guile/3.0/site-ccache/libpatch")) (define guile-sources '("src/guile/libpatch/ffi.scm" "src/guile/libpatch/patch.scm")) (define guile-modules (map (lambda (source) (guile-module (inputs (list source libpatch)) (load-paths (list (string-append (builddir) "/src/guile") (string-append (srcdir) "/src/guile"))) (install-location guile-go-dir))) guile-sources)) (define bindings guile-modules) --8<---------------cut here---------------end--------------->8--- configure (compatible with configure.ac): --8<---------------cut here---------------start------------->8--- #!/bin/sh #-*-Scheme-*- GUILE="$(command -v guile || command -v guile3.0)" if [ -z "$GUILE" ]; then echo "Missing guile executable." exit 1 fi DIR=$(dirname "$0") THIS_PATH=$(realpath "$DIR") exec $GUILE -L "$THIS_PATH" --no-auto-compile -e main -s "$0" "$@" !# (use-modules (ice-9 exceptions) (ice-9 match) (project configure) (project configure config-file) (project configure guess) (project configure option) (project configure package) (project configure template-file) (project configure toolchain) (project configure variable) (project file-system search) (project progress) (project utils colors) (srfi srfi-1) (srfi srfi-26) (system base target)) (define (make-simple-action name value) (lambda* (this-opt _ #:key variables #:allow-other-keys) (let ((var (assoc-ref variables name))) (if var (set-variable-value! var value) (raise-exception (make-exception (make-programming-error) (make-exception-with-irritants name) (make-exception-with-message "Unknown variable"))))))) (define dyninst-allowed? #t) (define liteinst-allowed? #t) (define lttng-allowed? #t) (define* (has-dyninst? #:key packages #:allow-other-keys) (and dyninst-allowed? (package-exists? (assoc-ref packages "dyninst")))) (define* (has-liteinst? #:key packages #:allow-other-keys) (and liteinst-allowed? (package-exists? (assoc-ref packages "liteinst")))) (define* (has-lttng-ust? #:key packages #:allow-other-keys) (and lttng-allowed? (package-exists? (assoc-ref packages "lttng-ust")))) (define (support-flags? . flags) (lambda* (#:key variables #:allow-other-keys) (apply toolchain-support-flags? (variable-value (assoc-ref variables "CC")) flags))) (define libpatch-configuration (configuration (unique-file "dev-env") (arguments (cdr (program-arguments))) (packages (list (package (name "capstone")) (package (name "libdw") (minimum-version "0.158")) (package (name "libolx")) (package (name "liburcu-bp")) (package (name "lttng-ust") (required? #f)) (package (name "dyninst") (resolver (lambda (this) (let* ((lib (find-library "libdyninstAPI.so")) (header (find-header "BPatch.h")) (exists? (and lib header #t))) (set! (package-exists? this) exists?) (when exists? (set! (package-libs-only-l this) "-ldyninstAPI") (set! (package-libs-only-L this) (string-append "-L" (dirname lib))) (set! (package-cflags-only-I this) (string-append "-I" (dirname header))))))) (required? #f)) (package (name "liteinst") (resolver (lambda (this) (let* ((lib (find-library "libliteinst.so")) (header (find-header "liteinst.hpp")) (exists? (and lib header #t))) (set! (package-exists? this) exists?) (when exists? (set! (package-libs-only-l this) "-lliteinst") (set! (package-libs-only-L this) (string-append "-L" (dirname lib))) (set! (package-cflags-only-I this) (string-append "-I" (dirname header))))))) (required? #f)))) (variables (append (list (variable (name "LIBPATCH_VERSION") (value "1.0.0")) (variable (name "GUILE_BIN") (value (delay (or (find-binary/progress "guile") (find-binary/progress "guile3.0") (push-error! (make-exception (make-external-error) (make-exception-with-irritants "guile") (make-exception-with-message "missing required binary"))))))) (variable (name "GUILD_BIN") (value (delay (or (find-binary/progress "guild") (find-binary/progress "guild3.0") (push-error! (make-exception (make-external-error) (make-exception-with-irritants "guild") (make-exception-with-message "missing required binary"))))))) (variable (name "CC") (value (delay (guess-c-toolchain)))) (variable (name "CXX") (value (delay (guess-c++-toolchain #:required? #f)))) (variable (name "arch") (value (delay (target-cpu)))) (variable (name "build-benchmarks") (value #t)) (variable (name "tsan-native-benchmarks") (value #f)) (variable (name "build-libpatch-ftrace") (value has-lttng-ust?)) (variable (name "build-libpatch-coverage") (value #t)) (variable (name "build-libpatch-integrity") (value #t)) (variable (name "build-manpages") (value (delay (and (find-binary/progress "emacs") #t)))) (variable (name "has-man") (value (delay (and (find-binary/progress "man") #t)))) (variable (name "manpages-source-highlight") (value (delay (and (find-binary/progress "source-highlight") #t)))) (variable (name "has-dyninst") (value has-dyninst?)) (variable (name "has-gdb-inproctrace") (value (delay (and (find-binary/progress "gdb") (find-library "libinproctrace.so"))))) (variable (name "has-liteinst") (value has-liteinst?)) (variable (name "has-csmith") (value (delay (and (find-binary/progress "csmith") #t)))) (variable (name "csmith-include-directory") (value (delay (and=> (find-header "csmith/csmith.h") dirname)))) (variable (name "environ") (value (delay (environ))))) %standard-directory-variables)) (options (append (list (option (switch "disable-ftrace-build") (synopsis "do no build libpatch-ftrace.so") (action (make-simple-action "build-libpatch-ftrace" #f))) (option (switch "disable-patch-coverage-build") (synopsis "do not build libpatch-coverage.so") (action (make-simple-action "build-libpatch-coverage" #f))) (option (switch "disable-patch-integrity-build") (synopsis "do not build libpatch-integrity.so") (action (make-simple-action "build-libpatch-integrity" #f))) (option (switch "enable-tsan-on-native-benchmarks") (synopsis "enable TSAN for native benchmarks") (action (make-simple-action "tsan-native-benchmarks" #t))) (option (switch "without-manpages") (synopsis "disable manpages generation") (action (make-simple-action "build-manpages" #f))) (option (switch "without-lttng") (synopsis "disable any usage of LTTng") (action (lambda _ (set! lttng-allowed? #f)))) (option (switch "without-dyninst") (synopsis "disable any usage of Dyninst for benchmarking") (action (lambda _ (set! dyninst-allowed? #f)))) (option (switch "without-liteinst") (synopsis "disable any usage of LiteInst for benchmarking") (action (lambda _ (set! liteinst-allowed? #f)))) (option (switch "without-benchmarks") (synopsis "do not compile benchmarks") (action (make-simple-action "build-benchmarks" #f)))) %standard-directory-options)) (templates (list (template-file (input "aux/Makefile.in") (output "Makefile")) (template-file (input "aux/project-repl.in") (output "project-repl") (mode #o744)) (template-file (input "src/lib/pkgconfig/libpatch.pc.in")) (template-file (input "src/bin/patch-ftrace.in") (mode #o744)) (template-file (input "src/bin/patch-coverage.in") (mode #o744)) (template-file (input "src/bin/patch-integrity.in") (mode #o744)) (template-file (input "scripts/debug-test.in") (mode #o744)))) (config-files (list (c-config-file (path ".config.h") (namespace "CONFIG_")) (scm-config-file (path "config.scm") (namespace "config")))) (status-file "config.status"))) (define (main args) (with-exception-handler (lambda (exn) (if (eq? 'quit (exception-kind exn)) (raise-exception exn) (begin (print-exception (current-error-port) #f (exception-kind exn) (exception-args exn)) (primitive-exit EXIT_FAILURE)))) (lambda () (call-with-values (lambda () (configure libpatch-configuration)) (lambda (variables _ errors) (with-nest-progress "features summary" (for-each (match-lambda ((feature-name . feature-description) (let ((name+feature (assoc feature-name variables))) (if name+feature (progress "~a ~a" (if (cdr name+feature) (colorize-string "OK" %enabled-color) (colorize-string "NO" %disabled-color)) feature-description) (raise-exception (make-exception (make-exception-with-origin "Features summary") (make-programming-error) (make-exception-with-irritants feature-name) (make-exception-with-message "Unknown feature"))))))) '(("build-benchmarks" . "compile benchmarks") ("build-libpatch-ftrace" . "compile libpatch-ftace.so") ("build-libpatch-coverage" . "compile libpatch-coverage.so") ("build-libpatch-integrity" . "compile libpatch-integrity.so") ("build-manpages" . "generate man pages")))) (exit (if (null? errors) EXIT_SUCCESS EXIT_FAILURE))))))) --8<---------------cut here---------------end--------------->8--- On Thu, 19 Dec 2024, Divya Ranjan <divya@subvertising.org> wrote: > Hello Guix, > > The other day, after being frustrated of build systems (auto-tools, meson, maven etc.), I wondered why doesn’t Guix which has such powerful tools within it (`guix build`, `guix pack` etc.) also not have a purely Guile-based build tool? After all, our goal is to make deployment, and building both more declarative and away from the all-too-familiar “dependency hell”. > > I am not exactly sure how I want to go with this, but I want to extend (if needed) the capabilities of Guix, to allow the developer of a package to use it also to build the package effectively replacing existing build tools. Since we already have build-system, instead of executing make (or whatever other tool) commands from it, we could modify it to itself have all those things that a Makefile would have. > > The developer would use Guile to declare their build config, I am again not sure what this might exactly look like, but can’t we have it such that we provide the developer with some tools to _declare_ a custom and package-specific build-system in Guile (just like our familiar gnu-build-system), but this is purely in Guile and executes whatever commands, path declarations and other interactions (such as calling gcc) directly from Guile and not by just calling `make`. I haven’t thought through this clearly, but even if this doesn’t work, the main idea I’d like to propose is to fully replace existing build-tools by making a new Guile build tool to work alongside Guix. > > Ideally, once the developer has a build config ready, one can just wrap it up with a package definition in Guile, just like the ones we create to package something upstream. This package definition can then be used in `guix build -f package.scm` to result in a fully transactional building process that focuses not on getting out of dependency hell, but just declaring your config right. And all of this without having to learn yet another build tool that might disappear, and of course, without leaving the comfortable world of Lisp (Scheme). > > I was indicated by others that such an idea has previously been conceievd among Guix developers themselves, namely as a GSoC proposal[0]. I couldn’t find if that has progressed towards anything, nor could find anything in the mailing list. I did see Pjotr volunteering to mentor for it, I’ve CC-ed them to see if they’re still interested in such a project. Meanwhile, I’d like input from other Guix core developers on what they think of this, and if they could provide me with some leads on where to go with this. > > > [0]: https://libreplanet.org/wiki/Group:Guix/GSoC-2024 > > Regards, > -- > Divya Ranjan, > Philosophy, Mathematics, Libre Software. > -- Olivier Dion EfficiOS Inc. https://www.efficios.com ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: On a Guile-based Build-Tool complimenting Guix 2024-12-19 21:33 ` Olivier Dion @ 2024-12-21 7:53 ` Pjotr Prins 0 siblings, 0 replies; 7+ messages in thread From: Pjotr Prins @ 2024-12-21 7:53 UTC (permalink / raw) To: Olivier Dion; +Cc: Divya Ranjan, guix-devel, pjotr.public12 This is pretty cool Olivier! My thought to build systems is that autotools and cmake are really ugly and I, for sure, have that itch to scratch. I have proposed GSoC projects in that direction. For a Guix target we can have a much lighter system that, similar to cmake, could generate ninja or make files. Guix solves dependencies and build arch, so you that complexity no longer belongs in the build system. All you need is something that can handle the 'logic' of compilers to provide a nice developer experience. Olivier's example below is going a pretty long way as a DSL and cmake replacement, though I, myself, would farm out the compile tree resolver to make or ninja :) GWL, as Janneke pointed out, could also be interesting for implementation. Though I am not sure how fast that is. You want something really speedy to deal with compilers. That is one reason autotools and cmake are complex: they cache a lot. Where it comes to other targets, such as Debian distros and all, it is probably going to be too painful to handle to resolve all their demands. One possibility is to provide a generated make file in your repo and cross fingers. Debian has a long standing history of dealing with make files. They may contribute a meson build or something to your project (it has happened to mine) and that is cool. As long as I don't have to use it myself. Anyway, that is another reason to generate make. Pj. On Thu, Dec 19, 2024 at 04:33:25PM -0500, Olivier Dion wrote: > Hi, > > For what it is worth, I have been using my own build system purely in > Guile for a few years now. I use it in my main project libpatch > (https://git.sr.ht/~old/libpatch/tree). It is compatible with the > gnu-build-system (configure && make && make check && make install), but > it does not have to if you want to diverge from this flow. > > However, it is order of magnitude faster than the autotools stuff. > Configuration for example is a single Guile instance that may do some > fork, but not that much. Building is also fast, althought it would be > optimal if fibers were available (I use par-map but the bottleneck is > due to waitpid). I have plan to add guile-parallel to it instead since > it is also purely Guile based. > > My project is a mix of C, C++ and Guile. Thus I've only added support > for these, but I it is possible to extend it since it is base on GOOPS. > > It is tested on Ubuntu, Debian, Fedora, Arch Linux and Guix. > > Here are some snippets of what it looks like: > > project/libpatch/build/libraries.scm (C libraries): > --8<---------------cut here---------------start------------->8--- > (define-module (project libpatch build libraries) > #:use-module ((config) #:prefix conf:) > #:use-module (ice-9 match) > #:use-module (project build c) > #:use-module (project utils list) > #:use-module (srfi srfi-26) > #:export (libpatch > libpatch-ftrace)) > > (define libpatch-common > (c-binary > (inputs > '("src/lib/libpatch-common/io.c" > "src/lib/libpatch-common/read.c" > "src/lib/libpatch-common/sleep.c" > "src/lib/libpatch-common/write.c")) > (cppflags > (list > (string-append "-I" conf:srcdir "/src/lib/libpatch-common"))) > (library? #t) > (shared? #f) > (output "src/lib/libpatch-common.a"))) > > (define libpatch > (c-binary > (inputs > (cons > libpatch-common > (cond-list > '("src/lib/libpatch/core/address.c" > "src/lib/libpatch/core/attr.c" > "src/lib/libpatch/core/canon.c" > "src/lib/libpatch/core/commit.c" > "src/lib/libpatch/core/configure.c" > "src/lib/libpatch/core/const.c" > "src/lib/libpatch/core/coverage.c" > "src/lib/libpatch/core/disable.c" > "src/lib/libpatch/core/disasm.c" > "src/lib/libpatch/core/drop.c" > "src/lib/libpatch/core/dump.c" > "src/lib/libpatch/core/dwarf.c" > "src/lib/libpatch/core/dynamic.c" > "src/lib/libpatch/core/enable.c" > "src/lib/libpatch/core/fini.c" > "src/lib/libpatch/core/gc.c" > "src/lib/libpatch/core/global.c" > "src/lib/libpatch/core/init.c" > "src/lib/libpatch/core/integrity.c" > "src/lib/libpatch/core/last-error.c" > "src/lib/libpatch/core/make.c" > "src/lib/libpatch/core/olx.c" > "src/lib/libpatch/core/panic.c" > "src/lib/libpatch/core/patch.c" > "src/lib/libpatch/core/resolve.c" > "src/lib/libpatch/core/states.c" > "src/lib/libpatch/core/thread.c" > "src/lib/libpatch/core/tls.c" > "src/lib/libpatch/core/trampoline.c" > "src/lib/libpatch/core/wxe.c" > "src/lib/libpatch/core/page-allocator.c") > ((string=? conf:arch "x86_64") > "src/lib/libpatch/x86_64/const.c" > "src/lib/libpatch/x86_64/disasm.c" > "src/lib/libpatch/x86_64/dwarf.c" > "src/lib/libpatch/x86_64/dynamic.c" > "src/lib/libpatch/x86_64/generic-handlers.S" > "src/lib/libpatch/x86_64/init.c" > "src/lib/libpatch/x86_64/patch.c" > "src/lib/libpatch/x86_64/ret-handlers.S")))) > (external-dependencies > (list conf:capstone > conf:libdw > conf:libolx > conf:liburcu-bp)) > (cppflags (list > (string-append "-I" conf:srcdir "/src/lib/libpatch") > (string-append "-I" conf:srcdir "/src/lib/libpatch-common") > (string-append "-I" conf:srcdir "/src/lib/libpatch/" conf:arch "/include") > "-DLIBPATCH_CORE")) > (cflags '("-fvisibility=hidden")) > (libs '("-pthread")) > (shared? #t) > (library? #t) > (output "src/lib/libpatch.so") > (install-location conf:libdir))) > > (define libpatch-ftrace > (c-binary > (enabled? conf:build-libpatch-ftrace) > (inputs > (list > libpatch > "src/lib/libpatch-ftrace/libpatch-ftrace.c")) > (external-dependencies > (list conf:lttng-ust)) > (cppflags > (list (string-append "-I" conf:srcdir "/src/lib/libpatch-ftrace"))) > (cflags (list "-fvisibility=hidden")) > (shared? #t) > (library? #t) > (output "src/lib/libpatch-ftrace.so") > (install-location conf:libdir))) > --8<---------------cut here---------------end--------------->8--- > > project/libpatch/build/bindings.scm (Guile bindings to Libpatch): > --8<---------------cut here---------------start------------->8--- > (define-module (project libpatch build bindings) > #:use-module ((config) #:prefix conf:) > #:use-module (project build) > #:use-module (project build guile) > #:use-module (project libpatch build libraries) > #:export (bindings)) > > (define guile-go-dir > (string-append conf:libdir "/guile/3.0/site-ccache/libpatch")) > > (define guile-sources > '("src/guile/libpatch/ffi.scm" > "src/guile/libpatch/patch.scm")) > > (define guile-modules > (map > (lambda (source) > (guile-module > (inputs (list source libpatch)) > (load-paths (list > (string-append > (builddir) "/src/guile") > (string-append > (srcdir) "/src/guile"))) > (install-location guile-go-dir))) > guile-sources)) > > (define bindings > guile-modules) > --8<---------------cut here---------------end--------------->8--- > > > configure (compatible with configure.ac): > --8<---------------cut here---------------start------------->8--- > #!/bin/sh > #-*-Scheme-*- > GUILE="$(command -v guile || command -v guile3.0)" > > if [ -z "$GUILE" ]; then > echo "Missing guile executable." > exit 1 > fi > > DIR=$(dirname "$0") > THIS_PATH=$(realpath "$DIR") > > exec $GUILE -L "$THIS_PATH" --no-auto-compile -e main -s "$0" "$@" > !# > > (use-modules > (ice-9 exceptions) > (ice-9 match) > (project configure) > (project configure config-file) > (project configure guess) > (project configure option) > (project configure package) > (project configure template-file) > (project configure toolchain) > (project configure variable) > (project file-system search) > (project progress) > (project utils colors) > (srfi srfi-1) > (srfi srfi-26) > (system base target)) > > (define (make-simple-action name value) > (lambda* (this-opt _ #:key variables #:allow-other-keys) > (let ((var (assoc-ref variables name))) > (if var > (set-variable-value! var value) > (raise-exception > (make-exception > (make-programming-error) > (make-exception-with-irritants name) > (make-exception-with-message > "Unknown variable"))))))) > > (define dyninst-allowed? #t) > (define liteinst-allowed? #t) > (define lttng-allowed? #t) > > (define* (has-dyninst? #:key packages #:allow-other-keys) > (and > dyninst-allowed? > (package-exists? > (assoc-ref packages "dyninst")))) > > (define* (has-liteinst? #:key packages #:allow-other-keys) > (and > liteinst-allowed? > (package-exists? > (assoc-ref packages "liteinst")))) > > (define* (has-lttng-ust? #:key packages #:allow-other-keys) > (and > lttng-allowed? > (package-exists? > (assoc-ref packages "lttng-ust")))) > > (define (support-flags? . flags) > (lambda* (#:key variables #:allow-other-keys) > (apply > toolchain-support-flags? > (variable-value (assoc-ref variables "CC")) > flags))) > > (define libpatch-configuration > (configuration > (unique-file "dev-env") > (arguments (cdr (program-arguments))) > (packages > (list > (package > (name "capstone")) > (package > (name "libdw") > (minimum-version "0.158")) > (package > (name "libolx")) > (package > (name "liburcu-bp")) > (package > (name "lttng-ust") > (required? #f)) > (package > (name "dyninst") > (resolver > (lambda (this) > (let* ((lib (find-library "libdyninstAPI.so")) > (header (find-header "BPatch.h")) > (exists? (and lib header #t))) > (set! (package-exists? this) exists?) > (when exists? > (set! (package-libs-only-l this) "-ldyninstAPI") > (set! (package-libs-only-L this) > (string-append "-L" (dirname lib))) > (set! (package-cflags-only-I this) > (string-append "-I" (dirname header))))))) > (required? #f)) > (package > (name "liteinst") > (resolver > (lambda (this) > (let* ((lib (find-library "libliteinst.so")) > (header (find-header "liteinst.hpp")) > (exists? (and lib header #t))) > (set! (package-exists? this) exists?) > (when exists? > (set! (package-libs-only-l this) "-lliteinst") > (set! (package-libs-only-L this) > (string-append "-L" (dirname lib))) > (set! (package-cflags-only-I this) > (string-append "-I" (dirname header))))))) > (required? #f)))) > (variables > (append > (list > (variable > (name "LIBPATCH_VERSION") > (value "1.0.0")) > (variable > (name "GUILE_BIN") > (value > (delay > (or > (find-binary/progress "guile") > (find-binary/progress "guile3.0") > (push-error! > (make-exception > (make-external-error) > (make-exception-with-irritants "guile") > (make-exception-with-message > "missing required binary"))))))) > (variable > (name "GUILD_BIN") > (value > (delay > (or > (find-binary/progress "guild") > (find-binary/progress "guild3.0") > (push-error! > (make-exception > (make-external-error) > (make-exception-with-irritants "guild") > (make-exception-with-message > "missing required binary"))))))) > (variable > (name "CC") > (value (delay (guess-c-toolchain)))) > (variable > (name "CXX") > (value (delay (guess-c++-toolchain #:required? #f)))) > (variable > (name "arch") > (value (delay (target-cpu)))) > (variable > (name "build-benchmarks") > (value #t)) > (variable > (name "tsan-native-benchmarks") > (value #f)) > (variable > (name "build-libpatch-ftrace") > (value has-lttng-ust?)) > (variable > (name "build-libpatch-coverage") > (value #t)) > (variable > (name "build-libpatch-integrity") > (value #t)) > (variable > (name "build-manpages") > (value > (delay (and (find-binary/progress "emacs") #t)))) > (variable > (name "has-man") > (value > (delay (and (find-binary/progress "man") #t)))) > (variable > (name "manpages-source-highlight") > (value > (delay (and (find-binary/progress "source-highlight") #t)))) > (variable > (name "has-dyninst") > (value has-dyninst?)) > (variable > (name "has-gdb-inproctrace") > (value (delay > (and (find-binary/progress "gdb") > (find-library "libinproctrace.so"))))) > (variable > (name "has-liteinst") > (value has-liteinst?)) > (variable > (name "has-csmith") > (value > (delay (and (find-binary/progress "csmith") #t)))) > (variable > (name "csmith-include-directory") > (value > (delay > (and=> (find-header "csmith/csmith.h") dirname)))) > (variable > (name "environ") > (value (delay (environ))))) > %standard-directory-variables)) > (options > (append > (list > (option > (switch "disable-ftrace-build") > (synopsis "do no build libpatch-ftrace.so") > (action (make-simple-action "build-libpatch-ftrace" #f))) > (option > (switch "disable-patch-coverage-build") > (synopsis "do not build libpatch-coverage.so") > (action (make-simple-action "build-libpatch-coverage" #f))) > (option > (switch "disable-patch-integrity-build") > (synopsis "do not build libpatch-integrity.so") > (action (make-simple-action "build-libpatch-integrity" #f))) > (option > (switch "enable-tsan-on-native-benchmarks") > (synopsis "enable TSAN for native benchmarks") > (action (make-simple-action "tsan-native-benchmarks" #t))) > (option > (switch "without-manpages") > (synopsis "disable manpages generation") > (action (make-simple-action "build-manpages" #f))) > (option > (switch "without-lttng") > (synopsis "disable any usage of LTTng") > (action (lambda _ (set! lttng-allowed? #f)))) > (option > (switch "without-dyninst") > (synopsis "disable any usage of Dyninst for benchmarking") > (action (lambda _ (set! dyninst-allowed? #f)))) > (option > (switch "without-liteinst") > (synopsis "disable any usage of LiteInst for benchmarking") > (action (lambda _ (set! liteinst-allowed? #f)))) > (option > (switch "without-benchmarks") > (synopsis "do not compile benchmarks") > (action (make-simple-action "build-benchmarks" #f)))) > %standard-directory-options)) > (templates > (list > (template-file > (input "aux/Makefile.in") > (output "Makefile")) > (template-file > (input "aux/project-repl.in") > (output "project-repl") > (mode #o744)) > (template-file > (input "src/lib/pkgconfig/libpatch.pc.in")) > (template-file > (input "src/bin/patch-ftrace.in") > (mode #o744)) > (template-file > (input "src/bin/patch-coverage.in") > (mode #o744)) > (template-file > (input "src/bin/patch-integrity.in") > (mode #o744)) > (template-file > (input "scripts/debug-test.in") > (mode #o744)))) > (config-files > (list > (c-config-file > (path ".config.h") > (namespace "CONFIG_")) > (scm-config-file > (path "config.scm") > (namespace "config")))) > (status-file > "config.status"))) > > (define (main args) > (with-exception-handler > (lambda (exn) > (if (eq? 'quit (exception-kind exn)) > (raise-exception exn) > (begin > (print-exception (current-error-port) > #f > (exception-kind exn) > (exception-args exn)) > (primitive-exit EXIT_FAILURE)))) > (lambda () > (call-with-values > (lambda () > (configure libpatch-configuration)) > (lambda (variables _ errors) > (with-nest-progress > "features summary" > (for-each > (match-lambda > ((feature-name . feature-description) > (let ((name+feature (assoc feature-name variables))) > (if name+feature > (progress "~a ~a" > (if (cdr name+feature) > (colorize-string "OK" %enabled-color) > (colorize-string "NO" %disabled-color)) > feature-description) > (raise-exception > (make-exception > (make-exception-with-origin > "Features summary") > (make-programming-error) > (make-exception-with-irritants feature-name) > (make-exception-with-message > "Unknown feature"))))))) > '(("build-benchmarks" . "compile benchmarks") > ("build-libpatch-ftrace" . "compile libpatch-ftace.so") > ("build-libpatch-coverage" . "compile libpatch-coverage.so") > ("build-libpatch-integrity" . "compile libpatch-integrity.so") > ("build-manpages" . "generate man pages")))) > (exit (if (null? errors) > EXIT_SUCCESS > EXIT_FAILURE))))))) > --8<---------------cut here---------------end--------------->8--- > > > On Thu, 19 Dec 2024, Divya Ranjan <divya@subvertising.org> wrote: > > Hello Guix, > > > > The other day, after being frustrated of build systems (auto-tools, meson, maven etc.), I wondered why doesn’t Guix which has such powerful tools within it (`guix build`, `guix pack` etc.) also not have a purely Guile-based build tool? After all, our goal is to make deployment, and building both more declarative and away from the all-too-familiar “dependency hell”. > > > > I am not exactly sure how I want to go with this, but I want to extend (if needed) the capabilities of Guix, to allow the developer of a package to use it also to build the package effectively replacing existing build tools. Since we already have build-system, instead of executing make (or whatever other tool) commands from it, we could modify it to itself have all those things that a Makefile would have. > > > > The developer would use Guile to declare their build config, I am again not sure what this might exactly look like, but can’t we have it such that we provide the developer with some tools to _declare_ a custom and package-specific build-system in Guile (just like our familiar gnu-build-system), but this is purely in Guile and executes whatever commands, path declarations and other interactions (such as calling gcc) directly from Guile and not by just calling `make`. I haven’t thought through this clearly, but even if this doesn’t work, the main idea I’d like to propose is to fully replace existing build-tools by making a new Guile build tool to work alongside Guix. > > > > Ideally, once the developer has a build config ready, one can just wrap it up with a package definition in Guile, just like the ones we create to package something upstream. This package definition can then be used in `guix build -f package.scm` to result in a fully transactional building process that focuses not on getting out of dependency hell, but just declaring your config right. And all of this without having to learn yet another build tool that might disappear, and of course, without leaving the comfortable world of Lisp (Scheme). > > > > I was indicated by others that such an idea has previously been conceievd among Guix developers themselves, namely as a GSoC proposal[0]. I couldn’t find if that has progressed towards anything, nor could find anything in the mailing list. I did see Pjotr volunteering to mentor for it, I’ve CC-ed them to see if they’re still interested in such a project. Meanwhile, I’d like input from other Guix core developers on what they think of this, and if they could provide me with some leads on where to go with this. > > > > > > [0]: https://libreplanet.org/wiki/Group:Guix/GSoC-2024 > > > > Regards, > > -- > > Divya Ranjan, > > Philosophy, Mathematics, Libre Software. > > > -- > Olivier Dion > EfficiOS Inc. > https://www.efficios.com -- ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: On a Guile-based Build-Tool complimenting Guix 2024-12-19 20:12 On a Guile-based Build-Tool complimenting Guix Divya Ranjan ` (2 preceding siblings ...) 2024-12-19 21:33 ` Olivier Dion @ 2024-12-20 17:31 ` Attila Lendvai 3 siblings, 0 replies; 7+ messages in thread From: Attila Lendvai @ 2024-12-20 17:31 UTC (permalink / raw) To: Divya Ranjan; +Cc: guix-devel, pjotr.public12 > The other day, after being frustrated of build systems (auto-tools, > meson, maven etc.), I wondered why doesn’t Guix which has such > powerful tools within it (`guix build`, `guix pack` etc.) also not > have a purely Guile-based build tool? After all, our goal is to make > deployment, and building both more declarative and away from the > all-too-familiar “dependency hell”. just a word of warning: proper build systems are surprisingly complicated beasts. even when they are old and not too well designed, like makefiles, they still have lots of moving parts with strange peculiarities. if you want to be compatible with e.g. makefiles written by upstream, then it will take a lot of effort. although, i'm not sure whether that is part of your goals. maybe you just want to provide an alternative build system, and merely inspire upstream authors to migrate to it? (as opposed to providing a compatible guile-based build script in parallel to upstream's makefiles) nevertheless, here's a random pick from the relatively recent history of making ASDF (common lisp) usable; i.e. not great, just usable: fasl files needed to be split in two: compile time and runtime. the compilers had to be patched for to emit compile-time side effects into a different fasl file (defining something is also a side effect). then ASDF needed to be extended to differentiate between compile-time and runtime dependencies. this was a necessary, but not sufficient step to make ASDF reproducible. then lots of packages had to be patched to e.g. adjust dependency specifications so that e.g. macro definitions have a proper lifetime. AFAIU guile modules are much better in this regard than the CL model, but they may also have toplevel expressions that have side effects that are external to the module that is being compiled/loaded. this is not common, and not good style, but i think it's not forbidden either. if so, then the same treatment would be needed for such a hypothethical build system and the guile ecosystem in general. in short: beware that your proposal may in fact easily be something like: "let's rewrite e.g. bazel in guile!" and as a final note: below is a nice bird's eye view on build systems. it's part of an enlightening series of chapters where a martian programmer crashes their ship on earth and enters into a dialog with a human programmer while trying to rebuild his ship: https://ngnghm.github.io/blog/2016/04/26/chapter-9-build-systems-and-modularity/ -- • attila lendvai • PGP: 963F 5D5F 45C7 DFCD 0A39 -- “Forces beyond your control can take away everything you possess except one thing, your freedom to choose how you will respond to the situation.” — Viktor E. Frankl (1905–1997), 'Man's Search for Meaning' (1946) ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2024-12-22 19:21 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-12-19 20:12 On a Guile-based Build-Tool complimenting Guix Divya Ranjan 2024-12-19 21:08 ` Sergio Pastor Pérez 2024-12-19 21:17 ` Janneke Nieuwenhuizen 2024-12-20 0:06 ` indieterminacy 2024-12-19 21:33 ` Olivier Dion 2024-12-21 7:53 ` Pjotr Prins 2024-12-20 17:31 ` Attila Lendvai
Code repositories for project(s) associated with this external index https://git.savannah.gnu.org/cgit/guix.git This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.