From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id sE64NIBrImCnAQAA0tVLHw (envelope-from ) for ; Tue, 09 Feb 2021 11:01:20 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id 6DhjMIBrImDtXwAA1q6Kng (envelope-from ) for ; Tue, 09 Feb 2021 11:01:20 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 318DD9403C9 for ; Tue, 9 Feb 2021 11:01:20 +0000 (UTC) Received: from localhost ([::1]:56078 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l9Qlh-0003m3-IL for larch@yhetil.org; Tue, 09 Feb 2021 06:01:19 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34042) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l9QlS-0003lu-RW for guix-patches@gnu.org; Tue, 09 Feb 2021 06:01:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:41222) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l9QlS-0008Eh-4B for guix-patches@gnu.org; Tue, 09 Feb 2021 06:01:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l9QlS-00011j-12 for guix-patches@gnu.org; Tue, 09 Feb 2021 06:01:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#46399] [PATCH] build-system/cargo: Propagate crates across builds. Resent-From: Efraim Flashner Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 09 Feb 2021 11:01:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 46399 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 46399@debbugs.gnu.org Cc: Efraim Flashner X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.16128684293899 (code B ref -1); Tue, 09 Feb 2021 11:01:01 +0000 Received: (at submit) by debbugs.gnu.org; 9 Feb 2021 11:00:29 +0000 Received: from localhost ([127.0.0.1]:52768 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l9Qkq-00010k-AI for submit@debbugs.gnu.org; Tue, 09 Feb 2021 06:00:29 -0500 Received: from lists.gnu.org ([209.51.188.17]:57682) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l9Qkn-00010c-Pv for submit@debbugs.gnu.org; Tue, 09 Feb 2021 06:00:22 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:33694) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l9Qkn-0003iC-Hm for guix-patches@gnu.org; Tue, 09 Feb 2021 06:00:21 -0500 Received: from flashner.co.il ([178.62.234.194]:50368) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l9Qkb-0007e0-EP for guix-patches@gnu.org; Tue, 09 Feb 2021 06:00:19 -0500 Received: from localhost (unknown [31.210.181.184]) by flashner.co.il (Postfix) with ESMTPSA id 9C66D401B2; Tue, 9 Feb 2021 10:59:36 +0000 (UTC) From: Efraim Flashner Date: Tue, 9 Feb 2021 12:58:43 +0200 Message-Id: <20210209105843.13230-1-efraim@flashner.co.il> X-Mailer: git-send-email 2.30.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=178.62.234.194; envelope-from=efraim@flashner.co.il; helo=flashner.co.il X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -1.36 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 318DD9403C9 X-Spam-Score: -1.36 X-Migadu-Scanner: scn1.migadu.com X-TUID: kco9tOjjq1Nb * guix/build-system/cargo.scm (cargo-build): Add cargo-package-flags, install-source flags. * guix/build/cargo-build-system.scm (unpack-rust-crates, package): New procedures. (install): Also install crate sources. (%standard-phases): Add new phases. * doc/guix.texi (Packaging-guidelines)[Rust Crates]: Adjust to changes in the cargo-build-system. --- With this patch, each current crate will also install its sources, both in a .crate (tar.gz) and unpacked. (ins)efraim@3900XT ~$ tree -d /gnu/store/anklck1x25qk43dk5p442iarfcpknwwp-rust-bencher-0.1.5 /gnu/store/anklck1x25qk43dk5p442iarfcpknwwp-rust-bencher-0.1.5 └── share ├── cargo │   ├── registry │   └── src │   └── bencher-0.1.5 │   └── benches └── doc └── rust-bencher-0.1.5 8 directories A new phase after 'unpack will find the .crate files in share/cargo/registry, copy them to target/registry and unpack them into vendor-dir for use with the next build. At the end of that build all the .crate tarballs are copied to that package's %out/share/cargo/registry, and also unpacked into %out/share/cargo/src. This means that with this patch and some modifications to existing packages to use the new features it should be possible to run 'guix environment rust-foo' and then in the .cargo/config set $GUIX_ENVIRONMENT/share/cargo/src as a 'directory source' for cargo. I have to look more into source-replacement¹ to see if I can organize the .crate files in %out/share/cargo/registry into a local-registry format for use as a 'cache' for crates.io. ¹ https://doc.rust-lang.org/cargo/reference/source-replacement.html doc/guix.texi | 15 ++++--- guix/build-system/cargo.scm | 4 ++ guix/build/cargo-build-system.scm | 66 +++++++++++++++++++++++++++++-- 3 files changed, 77 insertions(+), 8 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index 7d18703283..0d153a1470 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -32,7 +32,7 @@ Copyright @copyright{} 2015, 2016, 2017, 2019, 2020 Leo Famulari@* Copyright @copyright{} 2015, 2016, 2017, 2018, 2019, 2020 Ricardo Wurmus@* Copyright @copyright{} 2016 Ben Woodcroft@* Copyright @copyright{} 2016, 2017, 2018 Chris Marusich@* -Copyright @copyright{} 2016, 2017, 2018, 2019, 2020 Efraim Flashner@* +Copyright @copyright{} 2016, 2017, 2018, 2019, 2020, 2021 Efraim Flashner@* Copyright @copyright{} 2016 John Darrington@* Copyright @copyright{} 2016, 2017 Nikita Gillmann@* Copyright @copyright{} 2016, 2017, 2018, 2019, 2020 Jan Nieuwenhuizen@* @@ -7437,8 +7437,10 @@ supports builds of packages using Cargo, the build tool of the It adds @code{rustc} and @code{cargo} to the set of inputs. A different Rust package can be specified with the @code{#:rust} parameter. -Regular cargo dependencies should be added to the package definition via the -@code{#:cargo-inputs} parameter as a list of name and spec pairs, where the +Regular cargo dependencies should be added to the package definition similarly +to other packages; those needed only at build time to native-inputs, others to +inputs. If you need to add source-only crates then you should add them to via +the @code{#:cargo-inputs} parameter as a list of name and spec pairs, where the spec can be a package or a source definition. Note that the spec must evaluate to a path to a gzipped tarball which includes a @code{Cargo.toml} file at its root, or it will be ignored. Similarly, cargo dev-dependencies @@ -7449,8 +7451,11 @@ In its @code{configure} phase, this build system will make any source inputs specified in the @code{#:cargo-inputs} and @code{#:cargo-development-inputs} parameters available to cargo. It will also remove an included @code{Cargo.lock} file to be recreated by @code{cargo} during the -@code{build} phase. The @code{install} phase installs the binaries -defined by the crate. +@code{build} phase. The @code{package} phase will run @code{cargo package} +to create a source crate for future use. The @code{install} phase installs +the binaries defined by the crate. Unless @code{install-source? #f} is +defined it will also install a source crate repository of itself and unpacked +sources, to ease in future hacking on rust packages. @end defvr @defvr {Scheme Variable} chicken-build-system diff --git a/guix/build-system/cargo.scm b/guix/build-system/cargo.scm index 6c8edf6bac..01667ef045 100644 --- a/guix/build-system/cargo.scm +++ b/guix/build-system/cargo.scm @@ -5,6 +5,7 @@ ;;; Copyright © 2016 David Craven ;;; Copyright © 2019 Ivan Petkov ;;; Copyright © 2020 Jakub Kądziołka +;;; Copyright © 2021 Efraim Flashner ;;; ;;; This file is part of GNU Guix. ;;; @@ -77,8 +78,10 @@ to NAME and VERSION." (vendor-dir "guix-vendor") (cargo-build-flags ''("--release")) (cargo-test-flags ''("--release")) + (cargo-package-flags ''("--no-metadata" "--no-verify")) (features ''()) (skip-build? #f) + (install-source? #t) (phases '(@ (guix build cargo-build-system) %standard-phases)) (outputs '("out")) @@ -106,6 +109,7 @@ to NAME and VERSION." #:vendor-dir ,vendor-dir #:cargo-build-flags ,cargo-build-flags #:cargo-test-flags ,cargo-test-flags + #:cargo-package-flags ,cargo-package-flags #:features ,features #:skip-build? ,skip-build? #:tests? ,(and tests? (not skip-build?)) diff --git a/guix/build/cargo-build-system.scm b/guix/build/cargo-build-system.scm index 1d21b33895..0328ede4cb 100644 --- a/guix/build/cargo-build-system.scm +++ b/guix/build/cargo-build-system.scm @@ -2,7 +2,7 @@ ;;; Copyright © 2016 David Craven ;;; Copyright © 2017 Mathieu Othacehe ;;; Copyright © 2019 Ivan Petkov -;;; Copyright © 2019, 2020 Efraim Flashner +;;; Copyright © 2019, 2020, 2021 Efraim Flashner ;;; Copyright © 2020 Jakub Kądziołka ;;; Copyright © 2020 Marius Bakke ;;; @@ -73,6 +73,38 @@ Cargo.toml file present at its root." " | cut -d/ -f2" " | grep -q '^Cargo.toml$'"))))) +(define* (unpack-rust-crates #:key inputs vendor-dir #:allow-other-keys) + (define (inputs->rust-inputs inputs) + "Filter using the label part from INPUTS." + (filter (lambda (input) + (match input + ((name . _) (rust-package? name)))) + inputs)) + (define (inputs->directories inputs) + "Extract the directory part from INPUTS." + (match inputs + (((names . directories) ...) + directories))) + + (let ((rust-inputs (inputs->directories (inputs->rust-inputs inputs)))) + (unless (null? rust-inputs) + (mkdir-p "target/package") + (mkdir-p vendor-dir) + ;; TODO: copy only regular inputs to target/package, not native-inputs. + (for-each (lambda (input-crate) + (copy-recursively (string-append input-crate + "/share/cargo/registry") + "target/package")) + (delete-duplicates rust-inputs)) + + (for-each (lambda (crate) + (invoke "tar" "xzf" crate "-C" vendor-dir)) + (find-files "target/package" "\\.crate$")))) + #t) + +(define (rust-package? name) + (string-prefix? "rust-" name)) + (define* (configure #:key inputs (vendor-dir "guix-vendor") #:allow-other-keys) @@ -170,9 +202,23 @@ directory = '" port) (apply invoke "cargo" "test" cargo-test-flags) #t)) -(define* (install #:key inputs outputs skip-build? features #:allow-other-keys) +(define* (package #:key + (cargo-package-flags '("--no-metadata" "--no-verify")) + #:allow-other-keys) + "Run 'cargo-package' for a given Cargo package." + (apply invoke `("cargo" "package" ,@cargo-package-flags))) + +(define* (install #:key + inputs + outputs + skip-build? + (install-source? #t) + features + #:allow-other-keys) "Install a given Cargo package." - (let* ((out (assoc-ref outputs "out"))) + (let* ((out (assoc-ref outputs "out")) + (registry (string-append out "/share/cargo/registry")) + (sources (string-append out "/share/cargo/src"))) (mkdir-p out) ;; Make cargo reuse all the artifacts we just built instead @@ -186,6 +232,18 @@ directory = '" port) (invoke "cargo" "install" "--no-track" "--path" "." "--root" out "--features" (string-join features))) + (when install-source? + ;; Install crate tarballs and unpacked sources for later use. + ;; TODO: Is there a better format/directory for these files? + (mkdir-p sources) + (for-each (lambda (crate) + (install-file crate registry)) + (find-files "target/package" "\\.crate$")) + + (for-each (lambda (crate) + (invoke "tar" "xzf" crate "-C" sources)) + (find-files registry "\\.crate$"))) + #t)) (define %standard-phases @@ -195,6 +253,8 @@ directory = '" port) (replace 'build build) (replace 'check check) (replace 'install install) + (add-after 'build 'package package) + (add-after 'unpack 'unpack-rust-crates unpack-rust-crates) (add-after 'patch-generated-file-shebangs 'patch-cargo-checksums patch-cargo-checksums))) (define* (cargo-build #:key inputs (phases %standard-phases) -- 2.30.0