[-- Attachment #1: Type: text/plain, Size: 960 bytes --] Hi., these day I had spent some more hours struggling with rust ans cargo, trying to get "pre-built" crates. Summery; Cargo is cruft, no solution found yet. I tried reusing a crate from the very same place it was built (see enclosed script). Anyhow, this does not work since cargo uses a different "metadata" value, even if noting changed. Based in the verbose output (cargo build -v …) I assume that some parameters of the "destination" build get included into this value. This meets another observation; when building the sequoia suite, several crates are build several times - even if all builds are performed in the same environment. Rust's build system is such a cruft - I really would like to throw it where it belongs: into the trash. -- Regards Hartmut Goebel | Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible | [-- Attachment #2: rlib-test.scm --] [-- Type: text/x-scheme, Size: 4300 bytes --] (use-modules (guix download) (guix packages) (guix build-system cargo) (gnu packages crates-io) (srfi srfi-1) (srfi srfi-26) (gnu packages crates-graphics) (gnu packages rust-apps)) (define-public rust-pretty-assertions-0.5 (package (inherit rust-pretty-assertions-0.6) (name "rust-pretty-assertions") (version "0.5.1") (source (origin (method url-fetch) (uri (crate-uri "pretty_assertions" version)) (file-name (string-append name "-" version ".tar.gz")) (sha256 (base32 "1ins6swkpxmrh8q5h96h8nv0497d3dclsiyx2lyvqi6py0q980is")))) (build-system cargo-build-system) (arguments `(#:tests? #f #:cargo-inputs (("rust-ansi-term" ,rust-ansi-term-0.11) ("rust-difference" ,rust-difference-2)))))) ;; /gnu/store/wknzymkfbfjbxwfd3djrn4hk9zdfgs56-rust-xmlparser-0.13.3 -- original ;; ;; libxmlparser-f82b201ea4144ed3.rlib (define-public myrust-xmlparser (package (inherit rust-xmlparser-0.13) (outputs '("out" "rlib")) (arguments `(#:skip-build? #f #:tests? #f #:cargo-build-flags (list "--release" "-vv") #:phases (modify-phases %standard-phases (add-after 'install 'install-rlib (lambda* (#:key outputs #:allow-other-keys) (let* ((rout (assoc-ref outputs "rlib")) (dest (string-append rout "/rlib"))) ;;(mkdir dest) ;;(for-each (cut install-file <> (string-append rout "/rlib")) (for-each (lambda (fn) (install-file fn (string-append rout "/rlib"))) (find-files "target/release/deps" "\\.(rlib|rmeta)$")) ))) ;; (add-after 'install 'fail ;; (lambda _ #f)) ))))) (define-public myrust-roxmltree (package (inherit rust-roxmltree-0.14) ;;(outputs '("out" "crate")) (inputs `(("rust-xmlparser" ,myrust-xmlparser "rlib"))) (arguments `(#:skip-build? #f #:tests? #f ;;#:vendor-dir "/tmp/src" #:cargo-build-flags (list "--release" "-vv") #:cargo-inputs (("rust-xmlparser:src" ,rust-xmlparser-0.13) ("rust-pretty-assertions" ,rust-pretty-assertions-0.5)) #:phases (modify-phases %standard-phases (add-after 'patch-cargo-checksums 'bring-in-rlib (lambda* (#:key inputs #:allow-other-keys) (let* ((rin (assoc-ref inputs "rust-xmlparser")) (src (assoc-ref inputs "rust-xmlparser")) (rlib (string-append rin "/rlib"))) (mkdir "/tmp/guix-build-rust-xmlparser-0.13.3.drv-0/") (copy-recursively "guix-vendor/rust-xmlparser-0.13.3.tar.gz" "/tmp/guix-build-rust-xmlparser-0.13.3.drv-0/xmlparser-0.13.3") (rename-file "guix-vendor/rust-xmlparser-0.13.3.tar.gz" "../rust-xmlparser-0.13.3.tar.gz") (symlink "/tmp/guix-build-rust-xmlparser-0.13.3.drv-0/xmlparser-0.13.3" "guix-vendor/rust-xmlparser-0.13.3") ;; (let ((port (open-file ".cargo/config" "w" #:encoding "utf-8"))) ;; (display " ;; #paths = [\"/tmp/guix-build-rust-xmlparser-0.13.3.drv-0/xmlparser-0.13.3\"] ;; [source.crates-io] ;; replace-with = 'vendored-sources' ;; #[patch.crates-io] ;; #xmlparser = { path = '/tmp/guix-build-rust-xmlparser-0.13.3.drv-0/xmlparser-0.13.3' } ;; [source.vendored-sources] ;; directory = '" port) ;; (display (string-append (getcwd) "/guix-vendor") port) ;; (display "' ;; " port) ;; (close-port port)) ;; (substitute* ".cargo/config" ;; (("\\.vendored-sources\\]" _) ;; (string-append _ "\n[paths]")) (for-each (lambda (fn) (install-file fn "target/release/deps")) (pk (find-files rlib "\\.(rlib|rmeta)$"))) #t ))) (add-before 'install 'fail (lambda _ #f))) )))) ;;myrust-xmlparser myrust-roxmltree
Am 06.06.21 um 20:38 schrieb Pjotr Prins:
> Since that community is about not invented here - maybe we can incense
> someone to pick it up. Needs a mature programmer though.
One solution that came to my mind is to not use Cargo, but instead parse
Cargo.toml and issue the appropriate "rustc" commands ourself.
--
Regards
Hartmut Goebel
| Hartmut Goebel | h.goebel@crazy-compilers.com |
| www.crazy-compilers.com | compilers which you thought are impossible |
On Mon, Jun 07, 2021 at 09:10:48AM +0200, Hartmut Goebel wrote:
> Am 06.06.21 um 20:38 schrieb Pjotr Prins:
> > Since that community is about not invented here - maybe we can incense
> > someone to pick it up. Needs a mature programmer though.
>
> One solution that came to my mind is to not use Cargo, but instead parse
> Cargo.toml and issue the appropriate "rustc" commands ourself.
Exactly my idea. One challenge will be that the source of dependencies
need to be available - think of it as include files. One thing we
could do as ship them as part of the Guix package. Or have a separate
one for sources. We do that for include files already.
Pj.
Am 07.06.21 um 10:28 schrieb Pjotr Prins:
> Exactly my idea. One challenge will be that the source of dependencies
> need to be available - think of it as include files. One thing we
> could do as ship them as part of the Guix package. Or have a separate
> one for sources. We do that for include files already.
Well, the current cargo-build-system already handles the source
dependencies.
We need to aim towards pre-built libraries (rlib, much like .a files in
C, I assume)
When cargo calls rustc, the command looks like:
LD_LIBRARY_PATH='$PWD/target/release/deps:/gnu/store/…-rust-1.45.2/lib' \
rustc … src/lib.rs --crate-type lib \
-L dependency=$PWD/target/release/deps \
--extern
xmlparser=$PWD/target/release/deps/libxmlparser-53596ac1828b1c97.rmeta
Thus I assume one could pass the rlib's of all dependencies in -L and
the respective mata-data in --extern
--
Regards
Hartmut Goebel
| Hartmut Goebel | h.goebel@crazy-compilers.com |
| www.crazy-compilers.com | compilers which you thought are impossible |
[-- Attachment #1: Type: text/plain, Size: 2283 bytes --] Hi Hartmut and Pjotr, My feeling on this is that we should partner with the Rust community to make shared library support from cargo a priority. Specifying an output directory is currently a nightly feature, that could be helpful. In general Rust tooling does not compose with existing tools. I believe they will be amenable to the thought that it should. If Rust wants to be used in the linux kernel, for instance, it should be easy to use with Make. Rust has a very well documented rfc process and we can at least bring it up that way. I brought up the possibility of collaboration between rust and functional package managers on the rust Zulip, even. They seemed to like the idea. Another path we should checkout is to see what Debian does. My understanding is that they figured something out. Worth a shot, but I’d rather the problem be fixed upstream. It will just take collaboration. Kindly, John > On Monday, Jun 07, 2021 at 5:04 AM, Hartmut Goebel <h.goebel@crazy-compilers.com (mailto:h.goebel@crazy-compilers.com)> wrote: > Am 07.06.21 um 10:28 schrieb Pjotr Prins: > > Exactly my idea. One challenge will be that the source of dependencies > > need to be available - think of it as include files. One thing we > > could do as ship them as part of the Guix package. Or have a separate > > one for sources. We do that for include files already. > > Well, the current cargo-build-system already handles the source > dependencies. > > We need to aim towards pre-built libraries (rlib, much like .a files in > C, I assume) > > When cargo calls rustc, the command looks like: > > LD_LIBRARY_PATH='$PWD/target/release/deps:/gnu/store/…-rust-1.45.2/lib' \ > rustc … src/lib.rs --crate-type lib \ > -L dependency=$PWD/target/release/deps \ > --extern > xmlparser=$PWD/target/release/deps/libxmlparser-53596ac1828b1c97.rmeta > > Thus I assume one could pass the rlib's of all dependencies in -L and > the respective mata-data in --extern > > -- > Regards > Hartmut Goebel > > | Hartmut Goebel | h.goebel@crazy-compilers.com | > | https://urldefense.com/v3/__http://www.crazy-compilers.com__;!!IKRxdwAv5BmarQ!KvLeCSlBvF9faFS6qJxYA0Rl0AuuhxYx7oxMtZXdbILpmv_Rz4n8swX7_p74sQ$ | compilers which you thought are impossible | > > [-- Attachment #2: Type: text/html, Size: 3029 bytes --]
[-- Attachment #1: Type: text/plain, Size: 49 bytes --] Oh my goodness, I’m so sorry for the top quote. [-- Attachment #2: Type: text/html, Size: 234 bytes --]
Hi John. Am 07.06.21 um 17:13 schrieb John Soo: > Rust has a very well documented rfc process and we can at least bring > it up that way. I brought up the possibility of collaboration between > rust and functional package managers on the rust Zulip, even. They > seemed to like the idea. > I'd be more than happy if you could start a RFC process then. This issue bugs us and other distros since many months. (I have not contact to the rust community and will not sign up at another proprietary communication platform (Zulip)). > My feeling on this is that we should partner with the Rust community > to make shared library support from cargo a priority. Our issue is a different one: Its about being able to reuse already compiled binaries - keeping current behavior of rust binaries being statically linked. While this looks like being the same as dynamic library support, it is not: While for dynamic libraries you meet to ensure the very correct version of a "crate" is loaded, for static linking with pre-build binaries you only need to ensure this at build-time. (For guix, of course, both would not be a problem, but I doubt we can make rust people understand this. And other distros will still have the problem.) rustc already has a notion for "static libraries", just cargo fu** it up. (Sorry for hard wording, but I'm actually quite angry about cargos' narrow-minded behavior). So our task is much, much easier and doesn't require changed to rustc, only to cargo. > Specifying an output directory is currently a nightly feature, that > could be helpful. Not exactly sure what you mean. But what breaks build with cargo are the *input* directories - and other magic which gets included into the "meta-data" for no reason. > In general Rust tooling does not compose with existing tools. I > believe they will be amenable to the thought that it should. If Rust > wants to be used in the linux kernel, for instance, it should be easy > to use with Make. From your lips to God's ears. From what I've seen an read, I'm not convident they will change anything. I'd like to be proofed wrong, though :-) > Another path we should checkout is to see what Debian does. My > understanding is that they figured something out. Worth a shot, but > I’d rather the problem be fixed upstream. It will just take collaboration. I did not check their tollchain lately, but package-earch still does not show many packages <https://packages.debian.org/search?suite=experimental&searchon=names&keywords=rust>. Last time I check, they basically do what guix does: compile everything from source - again and again. -- Regards Hartmut Goebel | Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible |
Am 07.06.21 um 18:26 schrieb Hartmut Goebel: >> Another path we should checkout is to see what Debian does. My >> understanding is that they figured something out. Worth a shot, but >> I’d rather the problem be fixed upstream. It will just take >> collaboration. > > I did not check their tollchain lately, but package-earch still does > not show many packages > <https://packages.debian.org/search?suite=experimental&searchon=names&keywords=rust>. > Last time I check, they basically do what guix does: compile > everything from source - again and again. > Just checked: Debian has quite some source packages [1], which even list "binary" packages (which are development packages). Anyhow it seems as if these are just the source again [2] [1] <https://packages.debian.org/search?keywords=rust&searchon=sourcenames&suite=stable§ion=all> [2] <https://packages.debian.org/buster/amd64/librust-build-const-dev/filelist> -- Regards Hartmut Goebel | Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible |
> Our issue is a different one: Its about being able to reuse already
> compiled binaries - keeping current behavior of rust binaries being
> statically linked.
>
> While this looks like being the same as dynamic library support, it
> is not: While for dynamic libraries you meet to ensure the very
> correct version of a "crate" is loaded, for static linking with pre-
> build binaries you only need to ensure this at build-time. (For guix,
> of course, both would not be a problem, but I doubt we can make rust
> people understand this. And other distros will still have the
> problem.)
That plus static libraries can't be grafted, which can be a problem
when your programming language despite being touted as secure still
allows large arrays of buggy behaviour. So while "statically linked,
but reusable and I swear, it really is reproducible" might work for the
time being, it is not a long-term solution.
One might be tempted to compare this to Go, which we also can't
dynamically link, but the important difference here is that Rust is
already being pulled in by core-updatesy stuff like GTK, so a bug in
certain Rust software would affect *a lot* of Guix users whereas Go is
mostly an ecosystem onto itself. (Note, though, that a Go update would
need to go through staging – not an ideal situation either.)
Regards,
Leo
[-- Attachment #1: Type: text/plain, Size: 1394 bytes --] On Mon, Jun 07, 2021 at 06:41:41PM +0200, Hartmut Goebel wrote: > Am 07.06.21 um 18:26 schrieb Hartmut Goebel: > > > Another path we should checkout is to see what Debian does. My > > > understanding is that they figured something out. Worth a shot, but > > > I’d rather the problem be fixed upstream. It will just take > > > collaboration. > > > > I did not check their tollchain lately, but package-earch still does not > > show many packages <https://packages.debian.org/search?suite=experimental&searchon=names&keywords=rust>. > > Last time I check, they basically do what guix does: compile everything > > from source - again and again. > > > Just checked: Debian has quite some source packages [1], which even list > "binary" packages (which are development packages). Anyhow it seems as if > these are just the source again [2] > > [1] <https://packages.debian.org/search?keywords=rust&searchon=sourcenames&suite=stable§ion=all> > [2] > <https://packages.debian.org/buster/amd64/librust-build-const-dev/filelist> > My understanding is that they pool together all the sources that they have and then build all the rust packages in one go. -- Efraim Flashner <efraim@flashner.co.il> אפרים פלשנר GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --]
Am 08.06.21 um 11:15 schrieb Efraim Flashner (referring to Debian)
> My understanding is that they pool together all the sources that they
> have and then build all the rust packages in one go.
Maybe, I have no clue. Anyhow given my experience with sequoia, I doubt
this will reduce build-times:
When building sequoia, several crates are build up to 4 times - even
sequoia-openpgp itself.
--
Regards
Hartmut Goebel
| Hartmut Goebel | h.goebel@crazy-compilers.com |
| www.crazy-compilers.com | compilers which you thought are impossible |
On Mon, Jun 07, 2021 at 08:48:11PM +0200, Leo Prikler wrote:
> (Note, though, that a Go update would need to go through staging – not
> an ideal situation either.)
I have "cheated" regarding this in the past. Although changing the Go
package should qualify for staging, it's so cheap to build Go packages
that I have sometimes decided to "just do it" on the master branch.
[-- Attachment #1: Type: text/plain, Size: 1433 bytes --] Hi Hartmut, Hartmut Goebel <h.goebel@crazy-compilers.com> writes: > Hi., > > these day I had spent some more hours struggling with rust ans cargo, > trying to get "pre-built" crates. > > Summery; Cargo is cruft, no solution found yet. > > I tried reusing a crate from the very same place it was built (see > enclosed script). Anyhow, this does not work since cargo uses a > different "metadata" value, even if noting changed. Based in the > verbose output (cargo build -v …) I assume that some parameters of the > "destination" build get included into this value. > > This meets another observation; when building the sequoia suite, > several crates are build several times - even if all builds are > performed in the same environment. > > Rust's build system is such a cruft - I really would like to throw it > where it belongs: into the trash. I've also tried to naively copy prebuilt rlibs at their expected cached location to see if rust would reuse them, but they didn't seem to; see the patch below if you are curious. There was a 'build-plan' feature in cargo that made it output Makefiles but it's not been under active development and will probably be removed. [0] There's an issue demanding support in Cargo to allow reusing pre-built libraries (last comment is mine) [1]. [0] https://github.com/rust-lang/cargo/issues/7614 [1] https://github.com/rust-lang/cargo/issues/1139 [-- Attachment #2: 0001-tentatively-reuse-rlib-for-cargo-build-system.patch --] [-- Type: text/x-patch, Size: 7213 bytes --] From 0807a912db3faaa1ac2a652c7570ecf63ebed8a5 Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer <maxim.cournoyer@gmail.com> Date: Thu, 3 Jun 2021 12:58:15 -0400 Subject: [PATCH] tentatively reuse rlib for cargo-build-system --- guix/build-system/cargo.scm | 3 +- guix/build/cargo-build-system.scm | 78 ++++++++++++++++++++++++++----- 2 files changed, 68 insertions(+), 13 deletions(-) diff --git a/guix/build-system/cargo.scm b/guix/build-system/cargo.scm index e53d2a7523..9ef9f6b149 100644 --- a/guix/build-system/cargo.scm +++ b/guix/build-system/cargo.scm @@ -271,7 +271,8 @@ any dependent crates. This can be a benefits: (build-inputs `(("cargo" ,rust "cargo") ("rustc" ,rust) ,@(expand-crate-sources cargo-inputs cargo-development-inputs) - ,@native-inputs)) + ,@native-inputs + ,@(if target '() inputs))) (outputs outputs) (build cargo-build) (arguments (strip-keyword-arguments private-keywords arguments))))) diff --git a/guix/build/cargo-build-system.scm b/guix/build/cargo-build-system.scm index 0a95672b00..e68f20e463 100644 --- a/guix/build/cargo-build-system.scm +++ b/guix/build/cargo-build-system.scm @@ -5,6 +5,7 @@ ;;; Copyright © 2019, 2020, 2021 Efraim Flashner <efraim@flashner.co.il> ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net> ;;; Copyright © 2020 Marius Bakke <marius@gnu.org> +;;; Copyright © 2021 Maxim Cournoyer <maxim.cournoyer@gmail.com> ;;; ;;; This file is part of GNU Guix. ;;; @@ -24,7 +25,7 @@ (define-module (guix build cargo-build-system) #:use-module ((guix build gnu-build-system) #:prefix gnu:) #:use-module (guix build json) - #:use-module (guix build utils) + #:use-module ((guix build utils) #:hide (delete)) #:use-module (guix build cargo-utils) #:use-module (ice-9 popen) #:use-module (ice-9 rdelim) @@ -34,7 +35,10 @@ #:use-module (srfi srfi-1) #:use-module (srfi srfi-26) #:export (%standard-phases - cargo-build)) + cargo-build + + rust-version + rust-library-prefix)) ;; Commentary: ;; @@ -42,6 +46,25 @@ ;; ;; Code: +(define (rust-version rust) + "Return the version triplet (major.minor.patch) as a string, given RUST, a +store file name." + (let* ((version (last (string-split rust #\-))) + (components (string-split version #\.)) + (major+minor+patch (take components 3))) + (string-join major+minor+patch "."))) + +(define (rust-library-prefix/relative inputs) + "Return the relative versioned Rust library prefix where Rust libraries are +to be installed." + (string-append "lib/rust/" (rust-version (assoc-ref inputs "rustc")))) + +(define (rust-library-prefix inputs outputs) + "Return the absolute versioned Rust library prefix where Rust libraries are +to be installed." + (string-append (assoc-ref outputs "out") "/" + (rust-library-prefix/relative inputs))) + (define (manifest-targets) "Extract all targets from the Cargo.toml manifest" (let* ((port (open-input-pipe "cargo read-manifest")) @@ -73,6 +96,16 @@ Cargo.toml file present at its root." " | cut -d/ -f2" " | grep -q '^Cargo.toml$'"))))) +(define (rlib? file) + "Check if FILE has the .rlib extension." + (string-suffix? ".rlib" file)) + +(define (inputs->directories inputs) + "Extract the directory part from INPUTS." + (match inputs + (((names . directories) ...) + directories))) + (define* (unpack-rust-crates #:key inputs vendor-dir #:allow-other-keys) (define (inputs->rust-inputs inputs) "Filter using the label part from INPUTS." @@ -80,11 +113,6 @@ Cargo.toml file present at its root." (match input ((name . _) (rust-package? name)))) inputs)) - (define (inputs->directories inputs) - "Extract the directory part from INPUTS." - (match inputs - (((names . directories) ...) - directories))) (let ((rust-inputs (inputs->directories (inputs->rust-inputs inputs)))) (unless (null? rust-inputs) @@ -185,6 +213,22 @@ directory = '" port) (generate-all-checksums vendor-dir) #t) +(define* (populate-cargo-cache #:key inputs outputs #:allow-other-keys) + "Populate the 'target/release' directory with any pre-built Rust libraries, +to avoid rebuilding them from sources when possible." + (let* ((rust-lib-prefix (rust-library-prefix/relative inputs)) + (input-dirs (inputs->directories inputs)) + (rust-lib-dirs (filter (lambda (f) + (file-exists? (string-append + f "/" rust-lib-prefix))) + input-dirs)) + (rlibs (delete-duplicates (append-map (cut find-files <> "\\.rlib$") + rust-lib-dirs)))) + (pk 'rust-lib-dirs rust-lib-dirs) + (pk 'rlibs rlibs) + (for-each (cut install-file <> "target/release") rlibs) + (invoke "find" "target"))) + (define* (build #:key skip-build? (features '()) @@ -228,7 +272,9 @@ directory = '" port) "Install a given Cargo package." (let* ((out (assoc-ref outputs "out")) (registry (string-append out "/share/cargo/registry")) - (sources (string-append out "/share/cargo/src"))) + (sources (string-append out "/share/cargo/src")) + (libdir (rust-library-prefix inputs outputs)) + (release-dir "target/release")) (mkdir-p out) ;; Make cargo reuse all the artifacts we just built instead @@ -237,10 +283,17 @@ directory = '" port) ;; Only install crates which include binary targets, ;; otherwise cargo will raise an error. - (or skip-build? - (not (has-executable-target?)) - (invoke "cargo" "install" "--no-track" "--path" "." "--root" out - "--features" (string-join features))) + (unless skip-build? + ;; Install binaries. + (when (has-executable-target?) + (apply invoke "cargo" "install" "--no-track" "--path" "." "--root" out + (if (not (null? features)) + (list "--features" (string-join features)) + '()))) + ;; Install static libraries. + (for-each (lambda (file) + (install-file (string-append release-dir "/" file) libdir)) + (scandir release-dir (cut string-suffix? ".rlib" <>)))) (when install-source? ;; Install crate tarballs and unpacked sources for later use. @@ -260,6 +313,7 @@ directory = '" port) (modify-phases gnu:%standard-phases (delete 'bootstrap) (replace 'configure configure) + (add-before 'build 'populate-cargo-cache populate-cargo-cache) (replace 'build build) (replace 'check check) (replace 'install install) -- 2.31.1