* [bug#58235] [PATCH] rakdo-build-system
@ 2022-10-01 22:01 Daniel Sockwell via Guix-patches via
2022-10-02 4:27 ` [bug#58235] [PATCH] rakdo-build-system (2 of 3) Daniel Sockwell via Guix-patches via
0 siblings, 1 reply; 2+ messages in thread
From: Daniel Sockwell via Guix-patches via @ 2022-10-01 22:01 UTC (permalink / raw)
To: 58235; +Cc: brown121407, Maxime Devos, paul
I previously submitted patch #46352, which was an attempt to update
the Raku ecosystem and received very helpful feedback from Maxime
Devos. While revising my patch based on that feedback, I realized
that the issue preventing updates to the Raku ecosystem were larger
than I had realized.
In fact, I discovered that I wasn't the first person to try to solve
them: patch #55179 by Paul A. Patience was another attempt in April.
And Paul _also_ wasn't the first attempt; his patch mentioned that
he didn't know about patch #46352 by Alexandru-Sergiu Marton from
February 2021.
Given that two previous patches had attempted to update the Raku
ecosystem without success and that my patch was running into some
of the same issues that had been a problem for Paul, I decided to
take a step back and figure out what was posing such a problem.
I realized that the problem was that the rakudo-build-system didn't
have a complete solution for handling Raku precompilation. To fix
this, I ended up rewiting the rakudo-build-system to add that support;
doing so seems to have resolved the issue.
This patch is part 1 of three, and contains only the changes to the
build system. The second patch will have the changes to the core Raku
packages that the rest of the ecosystem depend on (Rakudo, NQP, Zef,
etc). Finally, the third will adapt the remaining ecosystem packages
to the new version of the build system.
I hope that this 4th patch series will result in Guix being able to
package an up-to-date version of Rakudo!
Best regards,
Daniel
From f03ba44cdab7c3a200cd8f71624848490fdd1099 Mon Sep 17 00:00:00 2001
Message-Id: <f03ba44cdab7c3a200cd8f71624848490fdd1099.1664660088.git.daniel@codesections.com>
From: Daniel Sockwell <daniel@codesections.com>
Date: Sat, 1 Oct 2022 17:24:20 -0400
Subject: [PATCH] build-system/rakudo: rewrite to support precomp
Significantly rewrite the Rakudo build system to support precompiling
Raku programs and to work with modern versions of Rakudo. This has
previously been a significant blocker to updating Raku programs, see
<https://issues.guix.gnu.org/46352>,
<https://issues.guix.gnu.org/55179>, and
<https://issues.guix.gnu.org/57491>.
* guix/build/rakudo-build-system.scm: significant changes
* guix/build-system/rakudo.scm: minor corresponding changes
---
guix/build-system/rakudo.scm | 31 +--
guix/build/rakudo-build-system.scm | 392 +++++++++++++++++++++--------
2 files changed, 309 insertions(+), 114 deletions(-)
diff --git a/guix/build-system/rakudo.scm b/guix/build-system/rakudo.scm
index 05a4d9c2ad..b9c9d3c612 100644
--- a/guix/build-system/rakudo.scm
+++ b/guix/build-system/rakudo.scm
@@ -54,7 +54,7 @@ (define (default-rakudo)
(define (default-prove6)
"Return the default perl6-tap-harness package for tests."
(let ((module (resolve-interface '(gnu packages perl6))))
- (module-ref module 'perl6-tap-harness)))
+ (module-ref module 'perl6-prove6)))
(define (default-zef)
"Return the default perl6-zef package."
@@ -62,19 +62,20 @@ (define (default-zef)
(module-ref module 'perl6-zef)))
(define* (lower name
- #:key source inputs native-inputs outputs
+ #:key source inputs native-inputs
+ outputs
system target
(rakudo (default-rakudo))
(prove6 (default-prove6))
(zef (default-zef))
- (with-prove6? #t)
- (with-zef? #t)
+ (tests? #t)
+ (test-runner "zef")
#:allow-other-keys
#:rest arguments)
"Return a bag for NAME."
(define private-keywords
'(#:target #:rakudo #:prove6 #:zef #:inputs #:native-inputs))
-
+
(and (not target) ;XXX: no cross-compilation
(bag
(name name)
@@ -87,12 +88,14 @@ (define private-keywords
;; Keep the standard inputs of 'gnu-build-system'.
,@(standard-packages)))
(build-inputs `(("rakudo" ,rakudo)
- ,@(if with-prove6?
- `(("perl6-tap-harness" ,prove6)
- ,@(if with-zef?
- `(("perl6-zef" ,zef))
- '()))
- '())
+ ,@(cond ((not tests?)
+ '())
+ ((and tests? (eq? test-runner "zef"))
+ `(("test-runner-zef" ,zef)))
+ ((and tests? (eq? test-runner "prove6"))
+ `(("test-runner-prove6" ,prove6)))
+ (else '()))
+
,@native-inputs))
(outputs outputs)
(build rakudo-build)
@@ -104,11 +107,10 @@ (define* (rakudo-build name inputs
(search-paths '())
(tests? #t)
(phases '%standard-phases)
- (outputs '("out"))
+ outputs
(system (%current-system))
(guile #f)
- (with-zef? #t)
- (with-prove6? #t)
+ (test-runner "zef")
(imported-modules %rakudo-build-system-modules)
(modules '((guix build rakudo-build-system)
(guix build utils))))
@@ -125,6 +127,7 @@ (define builder
#:phases #$phases
#:system #$system
#:tests? #$tests?
+ #:test-runner #$test-runner
#:outputs #$(outputs->gexp outputs)
#:inputs #$(input-tuples->gexp inputs)))))
diff --git a/guix/build/rakudo-build-system.scm b/guix/build/rakudo-build-system.scm
index 5cf1cc55bc..82f08f56c3 100644
--- a/guix/build/rakudo-build-system.scm
+++ b/guix/build/rakudo-build-system.scm
@@ -22,132 +22,324 @@ (define-module (guix build rakudo-build-system)
#:use-module (guix build utils)
#:use-module (ice-9 ftw)
#:use-module (ice-9 match)
+ #:use-module (ice-9 regex)
+ #:use-module (ice-9 rdelim)
#:use-module (srfi srfi-1)
#:use-module (srfi srfi-26)
+ #:use-module (srfi srfi-34)
+ #:use-module (srfi srfi-35)
#:export (%standard-phases
rakudo-build))
;; Commentary:
;;
-;; Builder-side code of the standard Rakudo package build procedure.
+;; Builder-side code of the Rakudo build procedure for Raku packages.
+;;
+;;
+;; The rakudo-build-system uses the Rakudo compiler to compile and install Raku
+;; programs, and then uses a Raku test runner (e.g., Zef) to test those
+;; programs. The Rakudo build system is an extension of the GNU build system
+;; that is customized ways to support Raku packages, especially for
+;; precompilation. The Rakudo build system accepts one non-standard argument:
+;;
+;; #:test-runner - The Test Anything Protocol program used to run tests.
+;; (ignored if #:tests? #f)
+;; Valid values: "zef" - [default] Use perl6-zef, which is closest to how
+;; tests are normally run in the Raku ecosystem.
+;; "prove6" - Use perl6-prove6, a Raku TAP test runner
+;; "rakudo" - Run tests directly via Rakudo (no TAP test runner)
+;;
+;; This build system configures Raku programs to write a metadata cache to
+;; $XDG_DATA_HOME/raku/repo
+;;
+;; The above should be all you need to know to package Raku programs with the
+;; Rakudo build system; the rest of this commentary provides background on the
+;; build system and Raku precompilation generally.
+;;
+;; Raku's strategy for dependency management is remarkably similar to Guix's:
+;; Raku compiles each program into an immutable output; then, whenever that
+;; program's (recursive) inputs change, Raku generates a new output and
+;; redirects the relevant references to that new output. This gives Raku many
+;; of the same advantages that Guix enjoys, including allowing simultaneous
+;; installation of multiple package versions.
+;;
+;; But the similarity between Guix and Raku creates an issue for packaging Raku
+;; programs: Raku expects to be able to manage its equivalent to /gnu/store/ by
+;; recompiling Raku programs in their install location. This doesn't work when
+;; those programs are installed in Guix's actual, read-only /gnu/store/. Thus,
+;; integrating Raku and Guix requires some care.
+;;
+;; Specifically, because Raku can't update precomp files in /gnu/store/, Guix
+;; needs replace Raku as the director of Raku's precomp process, including by
+;; guaranteeing that every Raku package is fully precompiled at install time.
+;; This is different from how Raku normally handles precomp (partly lazily, with
+;; some precomp delayed until a module is first loaded).
+;;
+;; Accordingly, the Rakudo build system installs all precomp files to the store.
+;; The only non-store data a Raku package should generate is a metadata index
+;; consisting of *.repo-id files and *.lock files that gets written to
+;; $XDG_DATA_HOME/raku/repo. Raku reads this index to avoid the need to check
+;; input integrity, which is purely a performance optimization – Raku
+;; automatically rebuilds the index if it's missing.
+;;
+;; The Rakudo build system supports using Guix-installed packages in combination
+;; with Zef-installed ones. (Zef is the LPM (language package manager) that
+;; Raku developers use, i.e., Raku's equivalent to Rust's Cargo or JavaScript's
+;; NPM/Yarn/PNPM.) Programs installed via Zef have all their data installed to
+;; the same ./raku/repo directory used for Guix-installed packages' metadata.
+;; After installation, Zef-installed packages be used interchangeably with Raku
+;; packages installed via Guix – though, of course, they lack access to
+;; rollbacks, build transformations, and the rest of Guix's superpowers.
+;;
+;; Including Raku precomp files in the Guix store creates one minor issue: it
+;; decreases the number of Raku packages that pass "guix build --check". This
+;; shouldn't be the case: In theory, Raku precomp files are bit-for-bit
+;; reproducible but, in practice, a few precomp files show (extremely minor)
+;; differences. Until that's fixed, excluding precomp files from Guix's store
+;; would let some Raku packages appear to be fully reproducible. But doing that
+;; wouldn't *actually* help reproduciblity: those slightly non-reproducible
+;; precomp files would still be generated and executed outside the store.
+;;
+;; The Rakudo build system does not yet have an importer, but creating one is
+;; conceptually simple and is planned.
+;;
;;
;; Code:
-(define* (check #:key tests? inputs with-prove6? #:allow-other-keys)
- (if (and tests? (assoc-ref inputs "perl6-tap-harness"))
- ;(if (and tests? with-prove6?)
- (invoke "prove6" "-I=lib" "t/")
- (format #t "test suite not run~%"))
- #t)
-
-(define* (install #:key inputs outputs with-zef? #:allow-other-keys)
- "Install a given Perl6 package."
- (let* ((out (assoc-ref outputs "out"))
- (perl6 (string-append out "/share/perl6")))
- (if (assoc-ref inputs "perl6-zef")
- ;(if with-zef?
- (begin
- (let ((zef (string-append (assoc-ref inputs "perl6-zef")
- "/bin/zef")))
- (setenv "HOME" (getcwd))
- (mkdir-p perl6)
- (invoke zef "install" "--verbose" "."
- ;; Don't install any of the following:
- "--/depends" "--/build-depends" "--/test-depends"
- (string-append "--install-to=" perl6))
- (delete-file (string-append perl6 "/repo.lock")))
- #t)
- (begin
- (let ((inst (string-append (assoc-ref inputs "rakudo")
- "/share/perl6/tools/install-dist.p6")))
- (setenv "RAKUDO_RERESOLVE_DEPENDENCIES" "0")
- (setenv "RAKUDO_MODULE_DEBUG" "1") ; be verbose while building
- (invoke inst (string-append "--to=" perl6) "--for=site"))))))
-
-(define* (install-libs #:key outputs #:allow-other-keys)
- (let ((out (assoc-ref outputs "out"))
- (lock "lib/.precomp/.lock"))
- (when (file-exists? lock)
- (delete-file "lib/.precomp/.lock"))
- (copy-recursively "lib" (string-append out "/share/perl6/lib"))
- #t))
-(define* (install-bins #:key outputs #:allow-other-keys)
- (let ((out (assoc-ref outputs "out")))
- (when (file-exists? "bin")
- (for-each (lambda (file)
- (install-file file (string-append out "/bin"))
- (chmod (string-append out "/" file) #o555))
- (find-files "bin" ".*")))
- (when (file-exists? "sbin")
- (for-each (lambda (file)
- (install-file file (string-append out "/sbin"))
- (chmod (string-append out "/" file) #o555))
- (find-files "sbin" ".*")))
- #t))
+(define (raku-input? input)
+ "Test if an input appears to be a Rakudo program based on its name."
+ (or (string-prefix? "raku-" (car input))
+ (string-prefix? "perl6-" (car input))))
+
+(define (with-vendor-path dir)
+ "Append the /share/perl6/vendor path to a directory path"
+ (string-append dir "/share/perl6/vendor"))
+
+(define (inputs->raku-vendor-paths inputs)
+ "Map a list of Guix inputs into a list of Raku vendor paths."
+ (let* ((raku-inputs (filter raku-input? inputs))
+ (raku-input-paths (map cdr raku-inputs)))
+ (map with-vendor-path raku-input-paths)))
+
+(define (dir->inst dir-path)
+ "Produce a CompUnit::Repository::Installation path from a directory path.
+Raku provides several types of Repositories, most relevantly 'FileSystem' repos
+and 'Installation' repos. Guix should use Installation repos, which support
+installation of multiple versions. To specify an Installation repo, the path
+should be prefixed with 'inst#'"
+ (string-append "inst#" dir-path))
+
+(define (copy-raku-dependencies inputs out-dir)
+ "Recursively copy files from each Raku dependency to the out directory. This
+lets Rakudo find the precomp files without recursively checking the integrity of
+each one (which is prohibitively slow). Copying the files doesn't increase disk
+usage thanks to Guix's deduplication via hard links."
+ (let* ((raku-inputs (filter raku-input? inputs))
+ (raku-input-paths (map (lambda (input)
+ (with-vendor-path (cdr input)))
+ raku-inputs)))
+ (mkdir-p out-dir)
+ (for-each
+ (lambda (source-dir)
+ (for-each make-file-writable (find-files out-dir))
+ (copy-recursively source-dir out-dir))
+ raku-input-paths)))
+
+(define (set-repository-version repository-dir version)
+ "Set the repository format version for a Raku repository. Raku repositories
+have used incompatible formats and an unspecified version defaults to v1; thus,
+Raku packages must set the repo version. NOTE: the 'install-dist.raku' script
+cleans up the version marker, so you may need to set the version again.
+Forgetting to set the version currently triggers a cryptic error that includes
+the text `cannot do '.open' on a directory'."
+ ;; see https://github.com/Raku/old-issue-tracker/issues/6422
+ ;; TODO: Consider upstreaming a patch to provide a less cryptic error msg.
+ (let ((version-file (open-output-file "version")))
+ (format version-file "~a\n" version)
+ (close-port version-file)
+ (install-file "version" repository-dir)))
+
+(define* (validate-keyword-arg keyword allowed-values
+ #:key (keyword-name "a keyword argument"))
+ "Check that the keyword argument has one of the allowed values."
+ (unless (member keyword allowed-values)
+ (let ((allowed-strings (map (cut format #f "~s" <>) allowed-values)))
+ (raise
+ (condition
+ (&message (message
+ (format #f "Invalid value for ~a: '~s'"
+ keyword-name keyword)))
+ (&message (message
+ (format #f "Valid values: '~a'."
+ (string-join allowed-strings "', '")))))))))
+
+(define (extract-provided-modules meta6)
+ "Extract the list of provided modules from a packages META6.json file."
+ ;; Could alternatively use (json->scm), at the cost of a build dependency
+ (let* ((q-mark "\"")
+ (non-quote "[^\"]+")
+ (json-key (string-append q-mark "(" non-quote ")" q-mark ))
+ (json-value (string-append q-mark non-quote q-mark))
+ (json-key-value (make-regexp (string-append non-quote json-key
+ non-quote json-value)))
+ (json-provides-field "[,{]\\s*\"provides\"")
+ (provides-value-regexp
+ (make-regexp
+ (string-append json-provides-field
+ "\\s*:\\s*"
+ "\\{([^}]+)\\}")))
+ (provides-value (regexp-exec provides-value-regexp meta6)))
+ (map (cut match:substring <> 1)
+ (list-matches json-key-value
+ (match:substring provides-value 1)))))
+
+;;; Phases
+
+(define* (setup-rakudo-env #:key inputs outputs #:allow-other-keys)
+ "Set various RAKUDO* environment variables."
+ (let* ((out (assoc-ref outputs "out"))
+ (dest (with-vendor-path out))
+ (rakudo-dir (assoc-ref inputs "rakudo"))
+ (rakudo-home (string-append rakudo-dir "/share/perl6"))
+ (raku-vendor-paths (inputs->raku-vendor-paths inputs))
+ (raku-installations (map dir->inst raku-vendor-paths)))
+
+ (setenv "HOME" (getcwd))
+ (setenv "RAKUDO_HOME" rakudo-home)
+ (setenv "RAKUDO_RESOLVE_DEPENDENCIES" "0")
+ (setenv "RAKUDO_LOG_PRECOMP" "1")
+ (setenv "RAKUDO_MODULE_DEBUG" "1")
+ (setenv "RAKULIB"
+ (string-append (dir->inst dest) "," "home,"
+ (string-join raku-installations ",")))))
+
+
+(define* (check #:key tests? test-runner inputs #:allow-other-keys)
+ "Run the tests in ./t with the supplied #:test-runner (default: zef) unless
+#:tests? is #f."
+ (validate-keyword-arg test-runner '("zef" "prove6" "rakudo")
+ #:keyword-name "test-runner")
+ (when tests?
+ (cond
+ ((eq? test-runner "zef")
+ (let* ((zef-dir (assoc-ref inputs "test-runner-zef"))
+ (zef (string-append zef-dir "/bin/zef")))
+ (invoke zef "test" ".")))
+ ((eq? test-runner "prove6")
+ (let* ((prove6-dir (assoc-ref inputs "test-runner-prove6"))
+ (prove6 (string-append prove6-dir "/bin/prove6")))
+ (invoke prove6 "-Ilib" "t/")))
+ ((eq? test-runner "rakudo")
+ (let* ((rakudo-dir (assoc-ref inputs "rakudo"))
+ (rakudo (string-append rakudo-dir "/bin/rakudo")))
+ (for-each (cut invoke rakudo "-Ilib" <>)
+ (find-files "t/")))))))
+
+(define* (install #:key inputs outputs #:allow-other-keys)
+ "Install the code of a given Raku package."
+ (let* ((out (assoc-ref outputs "out"))
+ (vendor-dir (with-vendor-path out))
+ (rakudo-dir (assoc-ref inputs "rakudo"))
+ (install-dist-script (string-append rakudo-dir
+ "/share/perl6/tools/install-dist.raku")))
+
+ (copy-raku-dependencies inputs vendor-dir)
+ (set-repository-version vendor-dir 2)
+
+ (invoke install-dist-script
+ "--from=."
+ (string-append "--to=" vendor-dir)
+ "--for=vendor"
+ "--build"
+ "--precompile")))
(define* (install-resources #:key outputs #:allow-other-keys)
- (let ((out (assoc-ref outputs "out")))
- (when (file-exists? "resources")
- (copy-recursively "resources"
- (string-append out "/share/perl6/resources")))
- #t))
-
-(define* (wrap #:key inputs outputs #:allow-other-keys)
- (define (list-of-files dir)
- (map (cut string-append dir "/" <>)
- (or (scandir dir (lambda (f)
- (let ((s (stat (string-append dir "/" f))))
- (and (eq? 'regular (stat:type s))
- (not (wrapped-program? f))))))
- '())))
-
- (define bindirs
- (append-map (match-lambda
- ((_ . dir)
- (list (string-append dir "/bin")
- (string-append dir "/sbin"))))
- outputs))
-
- ;; Do not require bash to be present in the package inputs
- ;; even when there is nothing to wrap.
- ;; Also, calculate (sh) only once to prevent some I/O.
- (define %sh (delay (search-input-file inputs "bin/bash")))
- (define (sh) (force %sh))
-
- (let* ((out (assoc-ref outputs "out"))
- (var `("PERL6LIB" "," prefix
- ,(cons (string-append out "/share/perl6/lib,"
- out "/share/perl6/site/lib,"
- out "/share/perl6/vendor/lib")
- (search-path-as-string->list
- (or (getenv "PERL6LIB") "") #\,)))))
- (for-each (lambda (dir)
- (let ((files (list-of-files dir)))
- (for-each (cut wrap-program <> #:sh (sh) var)
- files)))
- bindirs)
- #t))
+ "Install the resources supplied by a Raku package. Resources are typically
+items such as configuration or data files; for details, see
+https://docs.raku.org/language/variables#%?RESOURCES"
+ (when (file-exists? "resources")
+ (let* ((out (assoc-ref outputs "out"))
+ (resources-dir (string-append out "/share/perl6/resources")))
+ (copy-recursively "resources" resources-dir))))
+
+(define* (install-bins #:key outputs #:allow-other-keys)
+ "Install any binary programs supplied by a Raku package."
+
+ (define (install-file-list file-list target-dir)
+ (for-each (cut install-file <> target-dir) file-list))
+
+ (let* ((binary-dirs (filter file-exists? '("bin" "sbin")))
+ (out (assoc-ref outputs "out")))
+ (when binary-dirs
+ (for-each (lambda (binary-dir)
+ (install-file-list (find-files binary-dir)
+ (string-append out "/" binary-dir)))
+ binary-dirs))))
+
+(define* (wrap-raku-programs #:key inputs outputs #:allow-other-keys)
+ "Wrap executable Raku programs to ensure they always uses the correct inputs
+(including the correct Rakudo version) even if other versions are installed."
+
+ (define (with-rakulib program rakulib)
+ (wrap-program program #:sh (search-input-file inputs "bin/sh")
+ `("RAKULIB" "," prefix ,rakulib)))
+
+ (let ((out (assoc-ref outputs "out")))
+ (define bin-dirs
+ (filter directory-exists?
+ (list (string-append out "/bin")
+ (string-append out "/sbin"))))
+ (unless (eq? bin-dirs '())
+ (let* ((bin-files (fold append '() (map find-files bin-dirs)))
+ (wrappable-bin-files (filter (negate wrapped-program?) bin-files))
+ (raku-vendor-paths (inputs->raku-vendor-paths inputs))
+ (raku-installations (map dir->inst raku-vendor-paths))
+ (current-installation (dir->inst (with-vendor-path out)))
+ (xdg-data-home-inst "inst#${XDG_DATA_HOME:-$HOME/.local/share}")
+ (guix-pkg-index-file (string-append xdg-data-home-inst
+ "/raku/repo")))
+ (for-each (cut with-rakulib
+ <>
+ (append (list current-installation)
+ raku-installations
+ (list guix-pkg-index-file)))
+ wrappable-bin-files)))))
+
+(define* (precompile #:key inputs outputs #:allow-other-keys)
+ "Fully precompile all Raku code."
+ (let ((provides (extract-provided-modules (read-string (open-input-file "META6.json")))))
+ (let* ((out (assoc-ref outputs "out"))
+ (rakudo-dir (assoc-ref inputs "rakudo"))
+ (rakudo (string-append rakudo-dir "/bin/.rakudo-real")))
+ (setenv "RAKULIB" (dir->inst (with-vendor-path out)))
+ (for-each (lambda (module)
+ (invoke rakudo
+ "-c" ; compile w/o running
+ "-e" (string-append "use " module)))
+ provides))))
(define %standard-phases
- ;; No need for 'bootstrap, 'configure or 'build.
(modify-phases gnu:%standard-phases
(delete 'bootstrap)
(delete 'configure)
(delete 'build)
+ (add-after 'unpack 'setup-rakudo-environment setup-rakudo-env)
(replace 'check check)
(replace 'install install)
- (add-before 'install 'install-lib-dir install-libs)
- (add-after 'install-lib-dir 'install-resources install-resources)
+ (add-after 'install 'install-resources install-resources)
(add-after 'install-resources 'install-binaries install-bins)
- ;; needs to be after 'install-binaries and all 'install phases
- (add-after 'install 'wrap wrap)))
+ ;; needs to be after 'install-binaries
+ (add-after 'install-binaries 'wrap-raku-programs wrap-raku-programs)
+ (add-after 'wrap-raku-programs 'precompile-raku-modules precompile)))
(define* (rakudo-build #:key inputs (phases %standard-phases)
#:allow-other-keys #:rest args)
- "Build the given Perl6 package, applying all of PHASES in order."
+ "Build the given Raku package, applying all of PHASES in order."
(apply gnu:gnu-build
- #:inputs inputs #:phases phases
+ #:inputs inputs
+ #:phases phases
args))
+
;;; rakudo-build-system.scm ends here
--
2.37.3
^ permalink raw reply related [flat|nested] 2+ messages in thread
* [bug#58235] [PATCH] rakdo-build-system (2 of 3)
2022-10-01 22:01 [bug#58235] [PATCH] rakdo-build-system Daniel Sockwell via Guix-patches via
@ 2022-10-02 4:27 ` Daniel Sockwell via Guix-patches via
0 siblings, 0 replies; 2+ messages in thread
From: Daniel Sockwell via Guix-patches via @ 2022-10-02 4:27 UTC (permalink / raw)
To: 58235
As previously indicated, this is the second of three patches to update
the Raku ecosystem. This patch contains updates to the core Raku packages
that are used to build/test the rest of the ecosystem; these updates align
the core packages with the changes to rakudo-build-system in the previous
patch.
For clarity, I split changes to different packages into separate commits.
The final patch in this series will complete the update process for the
Raku ecosystem.
Best,
Daniel
From 8a76462dfb478f0d7290e1facecf51153257be8f Mon Sep 17 00:00:00 2001
Message-Id: <8a76462dfb478f0d7290e1facecf51153257be8f.1664684430.git.daniel@codesections.com>
From: Daniel Sockwell <daniel@codesections.com>
Date: Sat, 1 Oct 2022 19:10:04 -0400
Subject: [PATCH 1/5] gnu: moarvm: Update 2019.03 -> 2022.07
* gnu/packages/perl6.scm Update MoarVM to 2022.07
Change build system from perl-build-system to gnu-build-system
Update synopsis and description
---
gnu/packages/perl6.scm | 97 ++++++++++++++++++++++--------------------
1 file changed, 51 insertions(+), 46 deletions(-)
diff --git a/gnu/packages/perl6.scm b/gnu/packages/perl6.scm
index dab9dd477b..b4e1ac8c27 100644
--- a/gnu/packages/perl6.scm
+++ b/gnu/packages/perl6.scm
@@ -22,11 +22,14 @@ (define-module (gnu packages perl6)
#:use-module (guix download)
#:use-module (guix git-download)
#:use-module (guix packages)
+ #:use-module (guix build-system gnu)
#:use-module (guix build-system perl)
#:use-module (guix build-system rakudo)
+ #:use-module (gnu packages)
#:use-module (gnu packages bdw-gc)
#:use-module (gnu packages libevent)
#:use-module (gnu packages libffi)
+ #:use-module (gnu packages perl)
#:use-module (gnu packages multiprecision)
#:use-module (gnu packages pkg-config)
#:use-module (gnu packages tls))
@@ -34,65 +37,67 @@ (define-module (gnu packages perl6)
(define-public moarvm
(package
(name "moarvm")
- (version "2019.03")
+ (version "2022.07")
(source
- (origin
- (method url-fetch)
- (uri (string-append "https://moarvm.org/releases/MoarVM-"
- version ".tar.gz"))
- (sha256
- (base32
- "017w1zvr6yl0cgjfc1b3ddlc6vjw9q8p7alw1vvsckw95190xc14"))
- (modules '((guix build utils)))
- (snippet
- '(begin
- ;(delete-file-recursively "3rdparty/dynasm") ; JIT
- (delete-file-recursively "3rdparty/dyncall")
- (delete-file-recursively "3rdparty/freebsd")
- (delete-file-recursively "3rdparty/libatomicops")
- (delete-file-recursively "3rdparty/libuv")
- (delete-file-recursively "3rdparty/libtommath")
- (delete-file-recursively "3rdparty/msinttypes")
- #t))))
- (build-system perl-build-system)
+ (origin
+ (method url-fetch)
+ (uri (string-append "https://moarvm.org/releases/MoarVM-" version ".tar.gz"))
+ (sha256 (base32 "0ijvdd7jyq990zyxrlv8pqhf53q20qhbjly6cnagj9pq2r6z0zik"))
+ (modules '((guix build utils)))
+ (snippet
+ '(begin (for-each
+ delete-file-recursively
+ '(;"3rdparty/dyncall" ; upstream prefers dyncall to libffi
+ ;"3rdparty/dynasm" ; needed for JIT
+ ;"3rdparty/freebsd" ; freebsd/memmem.o is used on linux
+ "3rdparty/libatomicops"
+ "3rdparty/libtommath"
+ "3rdparty/libuv"
+ "3rdparty/msinttypes"))))))
+ (build-system gnu-build-system)
(arguments
- '(#:phases
+ `(#:test-target "test"
+ #:phases
(modify-phases %standard-phases
(replace 'configure
(lambda* (#:key inputs outputs #:allow-other-keys)
- (let ((out (assoc-ref outputs "out"))
- (pkg-config (assoc-ref inputs "pkg-config")))
+ (let* ((out (assoc-ref outputs "out"))
+ (perl (string-append (assoc-ref inputs "perl") "/bin/perl"))
+ (dest (string-append out "/moar"))
+ (pkg-config (assoc-ref inputs "pkg-config")))
(setenv "CFLAGS" "-fcommon")
(setenv "LDFLAGS" (string-append "-Wl,-rpath=" out "/lib"))
- (invoke "perl" "Configure.pl"
- "--prefix" out
- "--pkgconfig" (string-append pkg-config "/bin/pkg-config")
- "--has-libtommath"
+ (invoke perl "Configure.pl"
+ "--mimalloc"
"--has-libatomic_ops"
- "--has-libffi"
- "--has-libuv")))))))
+ "--has-libtommath"
+ "--has-libuv"
+ ;"--has-dyncall" ; TODO: package dyncall for guix
+ "--toolchain" "gnu"
+ "--os" "linux"
+ "--pkgconfig" (string-append pkg-config "/bin/pkg-config")
+ "--prefix" out)))))))
(home-page "https://moarvm.org/")
;; These should be inputs but moar.h can't find them when building rakudo
- (propagated-inputs
- (list libatomic-ops libffi libtommath-1.0 libuv))
- (native-inputs
- (list pkg-config))
- (synopsis "VM for NQP And Rakudo Perl 6")
+ (propagated-inputs (list libatomic-ops libtommath libuv))
+ (native-inputs (list pkg-config perl))
+ (synopsis "Virtual machine for the Raku programming language")
(description
- "Short for \"Metamodel On A Runtime\", MoarVM is a modern virtual machine
-built for the Rakudo Perl 6 compiler and the NQP Compiler Toolchain. Highlights
-include:
+ "MoarVM (\"Metamodel On A Runtime\") is a modern virtual machine built for
+the Raku programming language, NQP, and the Rakudo Raku compiler. MoarVM's
+features include:
@itemize
-@item Great Unicode support, with strings represented at grapheme level
-@item Dynamic analysis of running code to identify hot functions and loops, and
-perform a range of optimizations, including type specialization and inlining
-@item Support for threads, a range of concurrency control constructs, and
-asynchronous sockets, timers, processes, and more
-@item Generational, parallel, garbage collection
-@item Support for numerous language features, including first class functions,
-exceptions, continuations, runtime loading of code, big integers and interfacing
-with native libraries.
+@item Just-in-time compilation and optimization of running code hot functions/loops.
+Optimizations include type specialization, code inlining/elimination, and on-stack
+replacement.
+@item Unusually strong Unicode support due to representing strings as
+graphemes and the compiled-in data from the Unicode Character Database
+@item Support various Raku language features, such as delimited continuations,
+first-class functions, resumable exceptions, runtime code loading, bounded
+serialization, and native calling/pointer manipulation.
+@item The low-level concurrency primitives required for Raku's concurrency
+system.
@end itemize")
(license license:artistic2.0)))
--
2.37.3
From e5416f42bd130a1306bf44c1628188a32f0222da Mon Sep 17 00:00:00 2001
Message-Id: <e5416f42bd130a1306bf44c1628188a32f0222da.1664684430.git.daniel@codesections.com>
In-Reply-To: <8a76462dfb478f0d7290e1facecf51153257be8f.1664684430.git.daniel@codesections.com>
References: <8a76462dfb478f0d7290e1facecf51153257be8f.1664684430.git.daniel@codesections.com>
From: Daniel Sockwell <daniel@codesections.com>
Date: Sat, 1 Oct 2022 19:21:20 -0400
Subject: [PATCH 2/5] gnu: nqp-configure: add package
* gnu/packages/perl6.scm: add nqp-configure
Add package needed as input to NQP and Rakudo
---
gnu/packages/perl6.scm | 25 +++++++++++++++++++++++++
1 file changed, 25 insertions(+)
diff --git a/gnu/packages/perl6.scm b/gnu/packages/perl6.scm
index b4e1ac8c27..1c125c02ae 100644
--- a/gnu/packages/perl6.scm
+++ b/gnu/packages/perl6.scm
@@ -23,6 +23,7 @@ (define-module (gnu packages perl6)
#:use-module (guix git-download)
#:use-module (guix packages)
#:use-module (guix build-system gnu)
+ #:use-module (guix build-system copy)
#:use-module (guix build-system perl)
#:use-module (guix build-system rakudo)
#:use-module (gnu packages)
@@ -101,6 +102,30 @@ (define-public moarvm
@end itemize")
(license license:artistic2.0)))
+(define-public nqp-configure
+ (let ((commit "9b98931e0bfb8c4aac61590edf5074e63aa8ea4b" )
+ (revision "0"))
+ (package
+ (name "nqp-configure")
+ (version (git-version "0.1" revision commit))
+ (source
+ (origin
+ (method git-fetch)
+ (uri (git-reference (url "https://github.com/Raku/nqp-configure")
+ (commit commit)))
+ (sha256 (base32 "1vc1q11kjb964jal9dhgf5vwp371a3rfw7gj987n33kzli7a10n0"))
+ (file-name (git-file-name name version))))
+ (build-system copy-build-system)
+ (arguments
+ '(#:install-plan
+ '(("lib" "lib")
+ ("bin" "bin"))))
+ (synopsis "Configuration utility for NQP and Rakudo installations")
+ (description "A utility that simplifies the process of compiling and
+installing NQP (Not Quite Perl) and the Rakudo Raku compiler.")
+ (license license:artistic2.0)
+ (home-page "https://github.com/Raku/nqp-configure" ))))
+
(define-public nqp
(package
(name "nqp")
--
2.37.3
From 1652e35754b06a54ff1d90fc3372f10bfb149898 Mon Sep 17 00:00:00 2001
Message-Id: <1652e35754b06a54ff1d90fc3372f10bfb149898.1664684430.git.daniel@codesections.com>
In-Reply-To: <8a76462dfb478f0d7290e1facecf51153257be8f.1664684430.git.daniel@codesections.com>
References: <8a76462dfb478f0d7290e1facecf51153257be8f.1664684430.git.daniel@codesections.com>
From: Daniel Sockwell <daniel@codesections.com>
Date: Sat, 1 Oct 2022 19:26:30 -0400
Subject: [PATCH 3/5] gnu: nqp: Update 2019.03 -> 2022.07
* gnu/packages/perl6.scm: Update nqp to 2022.07
Add patch to use nqp-configure as input (in place of fetching from GitHub)
Update synopsis and description
---
.../nqp.do-not-pull-files-via-git.patch | 26 +++++++
gnu/packages/perl6.scm | 69 +++++++++----------
2 files changed, 57 insertions(+), 38 deletions(-)
create mode 100644 gnu/packages/patches/nqp.do-not-pull-files-via-git.patch
diff --git a/gnu/packages/patches/nqp.do-not-pull-files-via-git.patch b/gnu/packages/patches/nqp.do-not-pull-files-via-git.patch
new file mode 100644
index 0000000000..b99919286f
--- /dev/null
+++ b/gnu/packages/patches/nqp.do-not-pull-files-via-git.patch
@@ -0,0 +1,26 @@
+# Don't fetch nqp-configure from git (Guix supplies it as an input)
+diff --git a/Configure.pl b/Configure.patched.pl
+index ded227d..1fd59c8 100755
+--- a/Configure.pl
++++ b/Configure.patched.pl
+@@ -11,20 +11,6 @@ use File::Spec;
+ use File::Path;
+ use FindBin;
+
+-BEGIN {
+- # Download / Update submodules
+- my $set_config = !qx{git config nqp.initialized};
+- if ( !-e '3rdparty/nqp-configure/LICENSE' ) {
+- my $code = system($^X, 'tools/build/update-submodules.pl', Cwd::cwd(), @ARGV);
+- exit 1 if $code >> 8 != 0;
+- $set_config = 1;
+- }
+- if ($set_config) {
+- system("git config submodule.recurse true");
+- system("git config nqp.initialized 1");
+- }
+-}
+-
+ use lib ( "$FindBin::Bin/tools/lib",
+ "$FindBin::Bin/3rdparty/nqp-configure/lib", );
+ use NQP::Config qw<system_or_die>;
diff --git a/gnu/packages/perl6.scm b/gnu/packages/perl6.scm
index 1c125c02ae..af85df0aca 100644
--- a/gnu/packages/perl6.scm
+++ b/gnu/packages/perl6.scm
@@ -129,19 +129,19 @@ (define-public nqp-configure
(define-public nqp
(package
(name "nqp")
- (version "2019.03")
+ (version "2022.07")
(source
- (origin
- (method url-fetch)
- (uri (string-append "https://rakudo.perl6.org/downloads/nqp/nqp-"
- version ".tar.gz"))
- (sha256
- (base32
- "183zhll13fx416s3hkg4bkvib77kyr857h0nydgrl643fpacxp83"))
- (modules '((guix build utils)))
+ (origin
+ (method url-fetch)
+ (uri (string-append
+ "https://github.com/Raku/nqp/releases/download/2022.07/nqp-"
+ version ".tar.gz"))
+ (sha256 (base32 "0s124fl2kjahbhr52n92pw9gv1flr099kmlg04358ak7dl81q22q"))
+ (patches (search-patches "nqp.do-not-pull-files-via-git.patch"))
+ (modules '((guix build utils)))
(snippet
'(begin
- (delete-file-recursively "3rdparty") #t))))
+ (delete-file-recursively "3rdparty")))))
(build-system perl-build-system)
(arguments
'(#:phases
@@ -152,43 +152,36 @@ (define-public nqp
"tools/build/gen-js-cross-runner.pl"
"tools/build/gen-js-runner.pl"
"tools/build/install-js-runner.pl"
- "tools/build/install-moar-runner.pl"
- "tools/build/gen-moar-runner.pl"
"t/nqp/111-spawnprocasync.t"
"t/nqp/113-run-command.t")
- (("/bin/sh") (which "sh")))
- #t))
- (add-after 'unpack 'patch-source-date
- (lambda _
- (substitute* "tools/build/gen-version.pl"
- (("gmtime") "gmtime(0)"))
- #t))
- (add-after 'unpack 'remove-failing-test
- ;; One subtest fails for unknown reasons
- (lambda _
- (delete-file "t/nqp/019-file-ops.t")
- #t))
+ (("/bin/sh") (which "sh")))))
(replace 'configure
(lambda* (#:key inputs outputs #:allow-other-keys)
- (let ((out (assoc-ref outputs "out"))
- (moar (assoc-ref inputs "moarvm")))
+ (let ((out (assoc-ref outputs "out"))
+ (moar (assoc-ref inputs "moarvm"))
+ (nqp-config (assoc-ref inputs "nqp-configure")))
+ (setenv "PERL5LIB" (string-append
+ nqp-config "/lib:" (or (getenv "PERL5LIB")
+ "")))
+ ; MoarVM has some M_AST files that NQP expects to share
+ (copy-recursively (string-append moar "/share/nqp/lib")
+ (string-append out "/share/nqp/lib"))
+ (mkdir-p (string-append out "/share/nqp/lib/short"))
(invoke "perl" "Configure.pl"
"--backends=moar"
"--with-moar" (string-append moar "/bin/moar")
"--prefix" out)))))))
- (inputs
- (list moarvm))
- (home-page "https://github.com/perl6/nqp")
- (synopsis "Not Quite Perl")
- (description "This is \"Not Quite Perl\" -- a lightweight Perl 6-like
-environment for virtual machines. The key feature of NQP is that it's designed
-to be a very small environment (as compared with, say, perl6 or Rakudo) and is
-focused on being a high-level way to create compilers and libraries for virtual
-machines like MoarVM, the JVM, and others.
+ (inputs (list moarvm))
+ (native-inputs (list nqp-configure))
+ (home-page "https://github.com/Raku/nqp")
+ (synopsis "Small programming language built for writing compilers and VMs")
+ (description "NQP is \"Not Quite Perl\", a lightweight Raku-like
+programming language designed for compilers and VM creation. NQP is used to
+implement the Rakudo compiler for Raku and to integrate it with VMs such as
+MoarVM and the Java Virtual Machine.
-Unlike a full-fledged implementation of Perl 6, NQP strives to have as small a
-runtime footprint as it can, while still providing a Perl 6 object model and
-regular expression engine for the virtual machine.")
+NQP attempts to minimize its runtime footprint while providing Raku's object
+model and regular expression engine.")
(license license:artistic2.0)))
(define-public rakudo
--
2.37.3
From 9f9cd47ce2293f7dfdab07f024a246af000ed03a Mon Sep 17 00:00:00 2001
Message-Id: <9f9cd47ce2293f7dfdab07f024a246af000ed03a.1664684430.git.daniel@codesections.com>
In-Reply-To: <8a76462dfb478f0d7290e1facecf51153257be8f.1664684430.git.daniel@codesections.com>
References: <8a76462dfb478f0d7290e1facecf51153257be8f.1664684430.git.daniel@codesections.com>
From: Daniel Sockwell <daniel@codesections.com>
Date: Sun, 2 Oct 2022 00:01:18 -0400
Subject: [PATCH 5/5] gnu: perl6-zef: Update 0.6.7 -> 0.14.2
* gnu/packages/perl6.scm: Update perl6-zef to 0.14.2
Update synopsis and description
Set install directory to integrate with Guix-installed Raku packages
---
.../perl6-zef.config-interpolation.patch | 42 ++++++++++++++++++
gnu/packages/perl6.scm | 44 +++++++++++--------
2 files changed, 67 insertions(+), 19 deletions(-)
create mode 100644 gnu/packages/patches/perl6-zef.config-interpolation.patch
diff --git a/gnu/packages/patches/perl6-zef.config-interpolation.patch b/gnu/packages/patches/perl6-zef.config-interpolation.patch
new file mode 100644
index 0000000000..357dfd18e6
--- /dev/null
+++ b/gnu/packages/patches/perl6-zef.config-interpolation.patch
@@ -0,0 +1,42 @@
+From f9e4306cba875999a44286bd8c585ef9e84a7a88 Mon Sep 17 00:00:00 2001
+From: Daniel Sockwell <daniel@codesections.com>
+Date: Fri, 30 Sep 2022 17:43:04 -0400
+Subject: [PATCH 4/4] Improve support for env vars in config.json
+
+The "StoreDir" and "TempDir" previously allowed for limited
+interpolation (limited to $*HOME and $*TMPDIR). This commit adds
+interpolation of $*XDG_DATA_HOME, $*XDG_CONFIG_HOME, and
+$*XDG_STATE_HOME with default values from the XDG spec, see
+https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html
+
+It also extends the interpolation for the same values to "DefaultCUR",
+which also specifies a path.
+---
+ lib/Zef/Config.rakumod | 11 +++++++++--
+ 1 file changed, 9 insertions(+), 2 deletions(-)
+
+diff --git a/lib/Zef/Config.rakumod b/lib/Zef/Config.rakumod
+index f77c6be..4362b17 100644
+--- a/lib/Zef/Config.rakumod
++++ b/lib/Zef/Config.rakumod
+@@ -4,8 +4,15 @@ module Zef::Config {
+ our sub parse-file($path) {
+ my %config = %(Zef::from-json( $path.IO.slurp ));
+
+- for %config.grep(*.key.ends-with('Dir')) {
+- %config{$_.key} = $_.value.subst(/'{$*HOME}' || '$*HOME'/, $*HOME // $*TMPDIR, :g);
++ for %config.grep({.key.ends-with('Dir') || .key eq 'DefaultCUR'}) {
++ my $home = $*HOME // $*TMPDIR;
++ %config{$_.key} = $_.value.subst(/'{$*XDG_DATA_HOME}' || '$*XDG_DATA_HOME'/,
++ %*ENV<XDG_DATA_HOME> // "$home/.local/share", :g);
++ %config{$_.key} = $_.value.subst(/'{$*XDG_CONFIG_HOME}' || '$*XDG_CONFIG_HOME'/,
++ %*ENV<XDG_CONFIG_HOME> // "$home/.config", :g);
++ %config{$_.key} = $_.value.subst(/'{$*XDG_STATE_HOME}' || '$*XDG_STATE_HOME'/,
++ %*ENV<XDG_STATE_HOME> // "$home/.local/bin", :g);
++ %config{$_.key} = $_.value.subst(/'{$*HOME}' || '$*HOME'/, $home, :g);
+ %config{$_.key} = $_.value.subst(/'{$*TMPDIR}' || '$*TMPDIR'/, $*TMPDIR, :g);
+ }
+
+--
+2.37.3
+
diff --git a/gnu/packages/perl6.scm b/gnu/packages/perl6.scm
index e571920e53..a47d55a901 100644
--- a/gnu/packages/perl6.scm
+++ b/gnu/packages/perl6.scm
@@ -803,30 +803,36 @@ (define-public perl6-xml-writer
(define-public perl6-zef
(package
(name "perl6-zef")
- (version "0.6.7")
+ (version "0.14.2")
(source
- (origin
- (method git-fetch)
- (uri (git-reference
- (url "https://github.com/ugexe/zef")
- (commit (string-append "v" version))))
- (file-name (git-file-name name version))
- (sha256
- (base32
- "07n7g1xw2c4g860rs890gx85vyhdq0ysgwbrnzw6q905jph2bkv7"))))
+ (origin
+ (method git-fetch)
+ (uri (git-reference (url "https://github.com/ugexe/zef")
+ (commit (string-append "v" version))))
+ (file-name (git-file-name name version))
+ (sha256 (base32 "09lgpkz93w5g9bsw44rag763fa8857bvjj9r4wwvnnawykl4lkzr"))
+ ;; patch submitted upstream, https://github.com/ugexe/zef/pull/470
+ (patches (search-patches "perl6-zef.config-interpolation.patch"))))
(build-system rakudo-build-system)
(arguments
- '(#:with-zef? #f
+ '(#:test-runner "rakudo" ; to avoid bootstrapping issues
#:phases
(modify-phases %standard-phases
- (replace 'check
+ (add-after 'unpack 'set-config-values
(lambda _
- (setenv "HOME" "/tmp")
- (invoke "perl6" "-I." "bin/zef" "--debug"
- "--tap-harness" "test" "."))))))
+ ;; Upstream recommends editing config.json to set system defaults
+ (substitute* "resources/config.json"
+ (("DefaultCUR.+$")
+ "DefaultCUR\" : [\"$*XDG_DATA_HOME/raku/repo\"],\n")))))))
(home-page "https://github.com/ugexe/zef")
- (synopsis "Perl6 Module Management")
- (description "Zef is a Perl 6 package (module) manager. It can be used to
-download and install Perl 6 modules in your home directory or as a system-wide
-module.")
+ (synopsis "Raku Language Package Management")
+ (description "Zef is a language package manager for the Raku programming
+language. It can be used to search for, download, install, and manage Raku
+modules to your home directory.
+
+As an end user, you should typically prefer to install executable Raku
+packages via Guix when possible. However, Zef provides particular support for
+using Raku packages while developing other Raku programs, so developers may
+prefer Zef in that situation. Zef and Guix packages are installed such that
+they should work smoothly together.")
(license license:artistic2.0)))
--
2.37.3
^ permalink raw reply related [flat|nested] 2+ messages in thread
end of thread, other threads:[~2022-10-02 4:28 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-10-01 22:01 [bug#58235] [PATCH] rakdo-build-system Daniel Sockwell via Guix-patches via
2022-10-02 4:27 ` [bug#58235] [PATCH] rakdo-build-system (2 of 3) Daniel Sockwell via Guix-patches via
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/guix.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.