* [PATCH 00/48] Extend bag-build to gexps.
@ 2024-01-08 7:51 Nicolas Graves via Development of GNU Guix and the GNU System distribution.
2024-01-10 21:50 ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
0 siblings, 1 reply; 2+ messages in thread
From: Nicolas Graves via Development of GNU Guix and the GNU System distribution. @ 2024-01-08 7:51 UTC (permalink / raw)
To: guix-devel; +Cc: ngraves
Rationale:
Almost all build-systems are defined with gexpressions in functions
that return derivations. Derivations are not easily extensible while
gexps are. An example usage is given below.
This is a pretty big rewrite that should recompile almost all packages,
but a lot of grafting happens such as I could rebuild my system quickly.
I was trying to get the build-phases of an existing package to apply to
a local repository, because guix as a development tool for heavy packages
(emacs, ungoogled-chromium) is tedious, and there are precious info in
build-phases that can be applied in a local repository. I'm not aware of
prior work on this particular issue.
These patches allow to do extensions such as:
(build-system
(name 'local-gnu)
(description "GNU Build System applied in the current directory")
(lower
(lambda* args
(let ((old-bag (apply
(build-system-lower
(package-build-system emacs-pgtk))
args)))
(bag
(inherit old-bag)
(build
(lambda* build-args
(mlet %store-monad
((builder (apply (bag-build old-bag) build-args)))
(return (with-imported-modules '((guix build utils))
#~(begin
(use-modules (guix build utils))
(with-directory-excursion #$(getcwd)
#$builder))))))))))))
Of course this type of build-system isn't directly applicable because of
the chroot of the builder, but this other trick makes it happen :
;; We can't use package->derivation directly because we want the user rather
;; than the daemon to build the derivation.
(with-store store
(run-with-store store
(mlet* %store-monad ((bag -> (package->bag pkg))
(drv (bag->derivation bag pkg)))
;; ensure inputs are in the store.
(built-derivations (derivation-inputs drv))
(with-environment-excursion
(apply invoke (derivation-builder (pk 'd drv))
(derivation-builder-arguments drv))))))
This isn't polished yet, but could serve as an handy way to develop
heavy packages locally while taking advantage of the code that's
already in guix build phases.
Nicolas Graves (48):
guix: packages: Extend bag-build to support gexp.
build-system: gnu: Improve gnu-cross-build style.
build-system: gnu: Redefine gnu-build and gnu-cross-build.
build-system: agda: Redefine agda-build.
build-system: android-ndk: Redefine gnu-build.
build-system: ant: Redefine ant-build.
build-system: asdf: Redefine asdf-build.
build-system: cargo: Redefine cargo-build and cargo-cross-build.
build-system: chicken: Redefine chicken-build.
build-system: clojure: Redefine clojure-build.
build-system: cmake: Redefine cmake-build and cmake-cross-build.
build-system: composer: Redefine composer-build.
build-system: copy: Redefine copy-build.
build-system: dub: Redefine dub-build.
build-system: dune: Redefine dune-build.
build-system: elm: Redefine elm-build.
build-system: emacs: Redefine emacs-build.
build-system: font: Redefine font-build.
build-system: glib-or-gtk: Improve glib-or-gtk-cross-build style.
build-system: glib-or-gtk: Redefine glib-or-gtk-build functions.
build-system: go: Redefine go-build and go-cross-build.
build-system: guile: Redefine guile-build and guile-cross-build.
build-system: haskell: Redefine haskell-build.
build-system: julia: Redefine julia-build.
build-system: linux-module: Redefine linux-module-build functions.
build-system: maven: Redefine maven-build.
build-system: meson: Redefine meson-build and meson-cross-build.
build-system: minify: Redefine minify-build.
build-system: mix: Redefine mix-build.
build-system: node: Redefine node-build.
build-system: ocaml: Redefine ocaml-build.
build-system: perl: Redefine perl-build and perl-cross-build.
build-system: pyproject: Redefine pyproject-build.
build-system: python: Redefine python-build.
build-system: qt: Redefine qt-build and qt-cross-build.
build-system: r: Redefine r-build.
build-system: rakudo: Redefine rakudo-build.
build-system: rebar: Redefine rebar-build.
build-system: renpy: Redefine renpy-build.
build-system: ruby: Improve ruby-cross-build style.
build-system: ruby: Redefine ruby-build.
build-system: scons: Redefine scons-build.
build-system: texlive: Redefine texlive-build.
build-system: tree-sitter: Redefine tree-sitter-build functions.
build-system: vim: Redefine vim-build.
build-system: waf: Improve waf-build style.
build-system: zig: Redefine zig-build.
build-system: trivial: Redefine trivial-build functions.
guix/build-system.scm | 2 +-
guix/build-system/agda.scm | 8 +-
guix/build-system/android-ndk.scm | 8 +-
guix/build-system/ant.scm | 8 +-
guix/build-system/asdf.scm | 8 +-
guix/build-system/cargo.scm | 19 ++---
guix/build-system/chicken.scm | 8 +-
guix/build-system/clojure.scm | 8 +-
guix/build-system/cmake.scm | 24 ++----
guix/build-system/composer.scm | 9 +--
guix/build-system/copy.scm | 11 +--
guix/build-system/dub.scm | 8 +-
guix/build-system/dune.scm | 9 +--
guix/build-system/elm.scm | 8 +-
guix/build-system/emacs.scm | 8 +-
guix/build-system/font.scm | 10 +--
guix/build-system/glib-or-gtk.scm | 115 ++++++++++++----------------
guix/build-system/gnu.scm | 119 +++++++++++++----------------
guix/build-system/go.scm | 20 ++---
guix/build-system/guile.scm | 21 ++---
guix/build-system/haskell.scm | 8 +-
guix/build-system/julia.scm | 8 +-
guix/build-system/linux-module.scm | 17 ++---
guix/build-system/maven.scm | 8 +-
guix/build-system/meson.scm | 25 ++----
guix/build-system/minify.scm | 8 +-
guix/build-system/mix.scm | 12 +--
guix/build-system/node.scm | 8 +-
guix/build-system/ocaml.scm | 9 +--
guix/build-system/perl.scm | 22 ++----
guix/build-system/pyproject.scm | 13 +---
guix/build-system/python.scm | 12 +--
guix/build-system/qt.scm | 17 ++---
guix/build-system/r.scm | 9 +--
guix/build-system/rakudo.scm | 8 +-
guix/build-system/rebar.scm | 12 +--
guix/build-system/renpy.scm | 8 +-
guix/build-system/ruby.scm | 48 ++++++------
guix/build-system/scons.scm | 9 +--
guix/build-system/texlive.scm | 11 +--
guix/build-system/tree-sitter.scm | 16 ++--
guix/build-system/trivial.scm | 41 ++++------
guix/build-system/vim.scm | 15 +---
guix/build-system/waf.scm | 32 ++++----
guix/build-system/zig.scm | 8 +-
guix/packages.scm | 53 ++++++++++---
46 files changed, 348 insertions(+), 520 deletions(-)
--
2.41.0
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: [PATCH 00/48] Extend bag-build to gexps.
2024-01-08 7:51 [PATCH 00/48] Extend bag-build to gexps Nicolas Graves via Development of GNU Guix and the GNU System distribution.
@ 2024-01-10 21:50 ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
0 siblings, 0 replies; 2+ messages in thread
From: Nicolas Graves via Development of GNU Guix and the GNU System distribution. @ 2024-01-10 21:50 UTC (permalink / raw)
To: guix-devel; +Cc: Andrew Tropin
[-- Attachment #1: Type: text/plain, Size: 726 bytes --]
Here's a more complete proof of concept:
The attached file guix.scm, when run a checkout of emacs branch emacs-29
with guix build -f guix.scm will :
- compile everything as if run locally but with the patches provided by
guix sources.
- a local edit and the rerun of guix build -f guix.scm will only compile
new / changed code.
Now I can try developping emacs without worring about huge compilation
times and without the developping issues that would happen if I ran the
build-system by hand without the patches.
We could even imagine adding an option to guix shell -D that could drop
such a file for any package, since most of the code is reproducible
(although applying a derivation this way is still quite hacky).
[-- Attachment #2: guix.scm --]
[-- Type: application/octet-stream, Size: 10485 bytes --]
(use-modules (guix git)
(guix git-download)
(guix gexp)
(guix scripts)
(guix packages)
(guix derivations)
(guix store)
(guix utils)
(guix monads)
(guix search-paths)
(guix build utils)
(srfi srfi-1)
(srfi srfi-26)
(ice-9 match)
(ice-9 optargs)
(gnu packages)
(gnu packages emacs)
(gnu packages base)
(gnu packages glib)
(gnu packages version-control)
(guix build-system)
(guix build-system copy)
(guix build-system glib-or-gtk)
(guix build-system gnu))
(define %srcdir (dirname (current-filename)))
;; GNU Guix is phenomenal in terms of extensibility and software
;; reproducibility. Some recent blog articles summed up how to use
;; Guix for local package development, see:
;; https://guix.gnu.org/blog/2023/from-development-environments-to\
;; -continuous-integrationthe-ultimate-guide-to-software-development-with-guix
;; One drawback of local development with Guix is the inability to
;; reuse compiled binary files for rapid software development: Guix
;; systematically rebuilds the whole package using all build phases.
;; This makes developping / hacking on heavy packages quite tedious.
;; In the absence of a better alternative, this hack/script allows to
;; develop locally by creating an equivalent store output from a local
;; repository using build phases from Guix source.
;; Important : We need to go through the store and derivations, since
;; we want to get the phases from Guix source. However, the derivation
;; builder can only affect the store. Thus the code needs to be
;; executed by the user. (I've also tried wide directory permissions,
;; which aren't enough. Maybe there's a way to build this using the
;; build daemon with the --disable-chroot option. But we already have
;; a build daemon which manages the store. Starting a new daemon for
;; this seems a bit overkill, but may be worth it with several
;; development environments.)
;; We separate phases that are only needed to be applied once and phases
;; that need to be repeated each time the source is modified.
;; XXX: adapted from guix/profiles.scm
(define-syntax-rule (with-environment-excursion exp ...)
(let ((env (environ)))
(dynamic-wind
(lambda () (environ '()))
(lambda () exp ...)
(lambda () (environ env)))))
;; XXX: copied from guix/packages.scm
(define instantiate-patch
(match-lambda
((? string? patch) ;deprecated
(local-file patch #:recursive? #t))
((? struct? patch) ;origin, local-file, etc.
patch)))
(with-store store
(let* ((flags #~("-p1"))
(patches (map instantiate-patch
(origin-patches (package-source emacs-pgtk))))
(phases-ignored-when-cached
'(;; set-SOURCE-DATE-EPOCH
;; set-paths
;; set-libgccjit-path
;; install-locale
;; unpack ; Ignored in both cases.
patch-compilation-driver
patch-program-file-names
enable-elogind
;; generate-gdk-pixbuf-loaders-cache-file
bootstrap
patch-usr-bin-file
patch-source-shebangs
fix-/bin/pwd
configure
patch-generated-file-shebangs
;; build
;; check
;; install
;; wrap-emacs-paths
;; undo-double-wrap
;; install-site-start
;; glib-or-gtk-wrap
;; restore-emacs-pdmp
;; glib-or-gtk-compile-schemas
;; patch-shebangs
;; strip
;; validate-runpath
;; validate-documentation-location
;; delete-info-dir-file
;; patch-dot-desktop-files
;; make-dynamic-linker-cache
;; install-license-files ; FIXME strip-store-file-name breaks it.
;; reset-gzip-timestamps
;; compress-documentation
))
(local-build-system
(build-system
(name 'local)
(description "Inherited Build System applied in the current directory")
(lower
(lambda* args
(let ((old-bag (apply
(build-system-lower
(package-build-system emacs-pgtk))
args)))
(bag
(inherit old-bag)
(build
(lambda* (name inputs #:key (outputs '("out"))
#:allow-other-keys #:rest rest)
(mlet %store-monad
((builder (apply (bag-build old-bag)
name inputs #:outputs outputs rest)))
(return
(with-imported-modules '((guix build utils))
#~(begin
(use-modules (guix build utils))
(with-directory-excursion #$(getcwd)
(for-each
(lambda (out)
(setenv out (string-append #$(getcwd) "/" out)))
'#$outputs)
#$builder)))))))))))))
(emacs-source (package-source emacs-pgtk))
(pkg
(package/inherit emacs-pgtk
(source #f)
(build-system local-build-system)
(native-inputs
(modify-inputs (package-native-inputs emacs-pgtk)
(append patch git-minimal)))
(arguments
(substitute-keyword-arguments (package-arguments emacs-pgtk)
((#:substitutable? _) #f)
((#:phases phases)
(let ((filtered-phases
(if (file-exists? "guix.cached")
(with-imported-modules '((srfi srfi-1))
;; This fold is a simple opposite filter-alist based on key.
#~(fold
(lambda (key result)
(if (member (car key) '#$phases-ignored-when-cached)
result
(cons key result)))
'()
(reverse #$phases)))
phases)))
#~(modify-phases #$filtered-phases
;; The source is the current working directory.
(delete 'unpack)
;; FIXME strip-store-file-name breaks it.
(delete 'install-license-files)
;; The next phases are also applied with the copy-build-system.
;; No need to repeat them several times.
(delete 'strip)
(delete 'validate-runpath)
(delete 'validate-documentation-location)
(delete 'delete-info-dir-file)
;; We need to apply patches and snippets in the source.
(add-after 'install-locale 'patch-source
(lambda _
;; XXX: copied from guix/packages.scm
(define (apply-patch patch)
(format (current-error-port) "applying '~a'...~%" patch)
;; Use '--force' so that patches that do not apply perfectly are
;; rejected. Use '--no-backup-if-mismatch' to prevent making
;; "*.orig" file if a patch is applied with offset.
(invoke (string-append #$(this-package-native-input "patch")
"/bin/patch")
"--force" "--no-backup-if-mismatch"
#+@flags "--input" patch))
(when (not (file-exists? "guix.cached"))
(for-each apply-patch '#$patches)
;; XXX: copied from guix/packages.scm
;; Works but there's no log yet.
#+(let ((snippet (origin-snippet emacs-source)))
(if snippet
#~(let ((module (make-fresh-user-module)))
(module-use-interfaces!
module
(map resolve-interface '#+(origin-modules emacs-source)))
((@ (system base compile) compile)
'#+(if (pair? snippet)
(sexp->gexp snippet)
snippet)
#:to 'value
#:opts %auto-compilation-options
#:env module))
#~#t)))))
(add-before 'install-locale 'delete-former-output
(lambda _
(when (file-exists? "out")
(delete-file-recursively "out"))))
(add-before 'build 'flag-as-cached
(lambda _
(call-with-output-file "guix.cached" (const #t)))))))))))
;; We can't use package->derivation directly because we want the
;; user rather than the daemon to build the derivation.
(bag (package->bag pkg))
(drv ((@@ (guix packages) bag->derivation*) store bag pkg)))
(build-derivations store (derivation-inputs drv))
(with-environment-excursion
(apply invoke (derivation-builder drv)
(derivation-builder-arguments drv)))))
(package/inherit emacs-pgtk
(source
(local-file "out" (string-append "local-" (package-name emacs-pgtk))
#:recursive? #t
#:select? (const #t)))
(build-system copy-build-system)
(arguments '()))
[-- Attachment #3: Type: text/plain, Size: 7559 bytes --]
Cheers!
Nicolas
On 2024-01-08 08:51, Nicolas Graves wrote:
> Rationale:
> Almost all build-systems are defined with gexpressions in functions
> that return derivations. Derivations are not easily extensible while
> gexps are. An example usage is given below.
>
> This is a pretty big rewrite that should recompile almost all packages,
> but a lot of grafting happens such as I could rebuild my system quickly.
>
> I was trying to get the build-phases of an existing package to apply to
> a local repository, because guix as a development tool for heavy packages
> (emacs, ungoogled-chromium) is tedious, and there are precious info in
> build-phases that can be applied in a local repository. I'm not aware of
> prior work on this particular issue.
>
> These patches allow to do extensions such as:
>
> (build-system
> (name 'local-gnu)
> (description "GNU Build System applied in the current directory")
> (lower
> (lambda* args
> (let ((old-bag (apply
> (build-system-lower
> (package-build-system emacs-pgtk))
> args)))
> (bag
> (inherit old-bag)
> (build
> (lambda* build-args
> (mlet %store-monad
> ((builder (apply (bag-build old-bag) build-args)))
> (return (with-imported-modules '((guix build utils))
> #~(begin
> (use-modules (guix build utils))
> (with-directory-excursion #$(getcwd)
> #$builder))))))))))))
>
> Of course this type of build-system isn't directly applicable because of
> the chroot of the builder, but this other trick makes it happen :
>
> ;; We can't use package->derivation directly because we want the user rather
> ;; than the daemon to build the derivation.
> (with-store store
> (run-with-store store
> (mlet* %store-monad ((bag -> (package->bag pkg))
> (drv (bag->derivation bag pkg)))
> ;; ensure inputs are in the store.
> (built-derivations (derivation-inputs drv))
> (with-environment-excursion
> (apply invoke (derivation-builder (pk 'd drv))
> (derivation-builder-arguments drv))))))
>
> This isn't polished yet, but could serve as an handy way to develop
> heavy packages locally while taking advantage of the code that's
> already in guix build phases.
>
>
> Nicolas Graves (48):
> guix: packages: Extend bag-build to support gexp.
> build-system: gnu: Improve gnu-cross-build style.
> build-system: gnu: Redefine gnu-build and gnu-cross-build.
> build-system: agda: Redefine agda-build.
> build-system: android-ndk: Redefine gnu-build.
> build-system: ant: Redefine ant-build.
> build-system: asdf: Redefine asdf-build.
> build-system: cargo: Redefine cargo-build and cargo-cross-build.
> build-system: chicken: Redefine chicken-build.
> build-system: clojure: Redefine clojure-build.
> build-system: cmake: Redefine cmake-build and cmake-cross-build.
> build-system: composer: Redefine composer-build.
> build-system: copy: Redefine copy-build.
> build-system: dub: Redefine dub-build.
> build-system: dune: Redefine dune-build.
> build-system: elm: Redefine elm-build.
> build-system: emacs: Redefine emacs-build.
> build-system: font: Redefine font-build.
> build-system: glib-or-gtk: Improve glib-or-gtk-cross-build style.
> build-system: glib-or-gtk: Redefine glib-or-gtk-build functions.
> build-system: go: Redefine go-build and go-cross-build.
> build-system: guile: Redefine guile-build and guile-cross-build.
> build-system: haskell: Redefine haskell-build.
> build-system: julia: Redefine julia-build.
> build-system: linux-module: Redefine linux-module-build functions.
> build-system: maven: Redefine maven-build.
> build-system: meson: Redefine meson-build and meson-cross-build.
> build-system: minify: Redefine minify-build.
> build-system: mix: Redefine mix-build.
> build-system: node: Redefine node-build.
> build-system: ocaml: Redefine ocaml-build.
> build-system: perl: Redefine perl-build and perl-cross-build.
> build-system: pyproject: Redefine pyproject-build.
> build-system: python: Redefine python-build.
> build-system: qt: Redefine qt-build and qt-cross-build.
> build-system: r: Redefine r-build.
> build-system: rakudo: Redefine rakudo-build.
> build-system: rebar: Redefine rebar-build.
> build-system: renpy: Redefine renpy-build.
> build-system: ruby: Improve ruby-cross-build style.
> build-system: ruby: Redefine ruby-build.
> build-system: scons: Redefine scons-build.
> build-system: texlive: Redefine texlive-build.
> build-system: tree-sitter: Redefine tree-sitter-build functions.
> build-system: vim: Redefine vim-build.
> build-system: waf: Improve waf-build style.
> build-system: zig: Redefine zig-build.
> build-system: trivial: Redefine trivial-build functions.
>
> guix/build-system.scm | 2 +-
> guix/build-system/agda.scm | 8 +-
> guix/build-system/android-ndk.scm | 8 +-
> guix/build-system/ant.scm | 8 +-
> guix/build-system/asdf.scm | 8 +-
> guix/build-system/cargo.scm | 19 ++---
> guix/build-system/chicken.scm | 8 +-
> guix/build-system/clojure.scm | 8 +-
> guix/build-system/cmake.scm | 24 ++----
> guix/build-system/composer.scm | 9 +--
> guix/build-system/copy.scm | 11 +--
> guix/build-system/dub.scm | 8 +-
> guix/build-system/dune.scm | 9 +--
> guix/build-system/elm.scm | 8 +-
> guix/build-system/emacs.scm | 8 +-
> guix/build-system/font.scm | 10 +--
> guix/build-system/glib-or-gtk.scm | 115 ++++++++++++----------------
> guix/build-system/gnu.scm | 119 +++++++++++++----------------
> guix/build-system/go.scm | 20 ++---
> guix/build-system/guile.scm | 21 ++---
> guix/build-system/haskell.scm | 8 +-
> guix/build-system/julia.scm | 8 +-
> guix/build-system/linux-module.scm | 17 ++---
> guix/build-system/maven.scm | 8 +-
> guix/build-system/meson.scm | 25 ++----
> guix/build-system/minify.scm | 8 +-
> guix/build-system/mix.scm | 12 +--
> guix/build-system/node.scm | 8 +-
> guix/build-system/ocaml.scm | 9 +--
> guix/build-system/perl.scm | 22 ++----
> guix/build-system/pyproject.scm | 13 +---
> guix/build-system/python.scm | 12 +--
> guix/build-system/qt.scm | 17 ++---
> guix/build-system/r.scm | 9 +--
> guix/build-system/rakudo.scm | 8 +-
> guix/build-system/rebar.scm | 12 +--
> guix/build-system/renpy.scm | 8 +-
> guix/build-system/ruby.scm | 48 ++++++------
> guix/build-system/scons.scm | 9 +--
> guix/build-system/texlive.scm | 11 +--
> guix/build-system/tree-sitter.scm | 16 ++--
> guix/build-system/trivial.scm | 41 ++++------
> guix/build-system/vim.scm | 15 +---
> guix/build-system/waf.scm | 32 ++++----
> guix/build-system/zig.scm | 8 +-
> guix/packages.scm | 53 ++++++++++---
> 46 files changed, 348 insertions(+), 520 deletions(-)
--
Best regards,
Nicolas Graves
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2024-01-10 21:50 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-01-08 7:51 [PATCH 00/48] Extend bag-build to gexps Nicolas Graves via Development of GNU Guix and the GNU System distribution.
2024-01-10 21:50 ` Nicolas Graves via Development of GNU Guix and the GNU System distribution.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).