[bug#46399] [PATCH] build-system/cargo: Propagate crates across builds.

[bug#46399] [PATCH] build-system/cargo: Propagate crates across builds.

[Efraim Flashner]

* guix/build-system/cargo.scm (cargo-build): Add cargo-package-flags,
install-source flags.
* guix/build/cargo-build-system.scm (unpack-rust-crates, package): New
procedures.
(install): Also install crate sources.
(%standard-phases): Add new phases.
* doc/guix.texi (Packaging-guidelines)[Rust Crates]: Adjust to changes
in the cargo-build-system.
---

With this patch, each current crate will also install its sources, both
in a .crate (tar.gz) and unpacked.

(ins)efraim@3900XT ~$ tree -d /gnu/store/anklck1x25qk43dk5p442iarfcpknwwp-rust-bencher-0.1.5
/gnu/store/anklck1x25qk43dk5p442iarfcpknwwp-rust-bencher-0.1.5
└── share
    ├── cargo
    │   ├── registry
    │   └── src
    │       └── bencher-0.1.5
    │           └── benches
    └── doc
        └── rust-bencher-0.1.5

8 directories

A new phase after 'unpack will find the .crate files in
share/cargo/registry, copy them to target/registry and unpack them into
vendor-dir for use with the next build. At the end of that build all the
.crate tarballs are copied to that package's %out/share/cargo/registry,
and also unpacked into %out/share/cargo/src.

This means that with this patch and some modifications to existing
packages to use the new features it should be possible to run 'guix
environment rust-foo' and then in the .cargo/config set
$GUIX_ENVIRONMENT/share/cargo/src as a 'directory source' for cargo.

I have to look more into source-replacement¹ to see if I can organize
the .crate files in %out/share/cargo/registry into a local-registry
format for use as a 'cache' for crates.io.

¹ https://doc.rust-lang.org/cargo/reference/source-replacement.html


 doc/guix.texi                     | 15 ++++---
 guix/build-system/cargo.scm       |  4 ++
 guix/build/cargo-build-system.scm | 66 +++++++++++++++++++++++++++++--
 3 files changed, 77 insertions(+), 8 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 7d18703283..0d153a1470 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -32,7 +32,7 @@ Copyright @copyright{} 2015, 2016, 2017, 2019, 2020 Leo Famulari@*
 Copyright @copyright{} 2015, 2016, 2017, 2018, 2019, 2020 Ricardo Wurmus@*
 Copyright @copyright{} 2016 Ben Woodcroft@*
 Copyright @copyright{} 2016, 2017, 2018 Chris Marusich@*
-Copyright @copyright{} 2016, 2017, 2018, 2019, 2020 Efraim Flashner@*
+Copyright @copyright{} 2016, 2017, 2018, 2019, 2020, 2021 Efraim Flashner@*
 Copyright @copyright{} 2016 John Darrington@*
 Copyright @copyright{} 2016, 2017 Nikita Gillmann@*
 Copyright @copyright{} 2016, 2017, 2018, 2019, 2020 Jan Nieuwenhuizen@*
@@ -7437,8 +7437,10 @@ supports builds of packages using Cargo, the build tool of the
 It adds @code{rustc} and @code{cargo} to the set of inputs.
 A different Rust package can be specified with the @code{#:rust} parameter.
 
-Regular cargo dependencies should be added to the package definition via the
-@code{#:cargo-inputs} parameter as a list of name and spec pairs, where the
+Regular cargo dependencies should be added to the package definition similarly
+to other packages; those needed only at build time to native-inputs, others to
+inputs.  If you need to add source-only crates then you should add them to via
+the @code{#:cargo-inputs} parameter as a list of name and spec pairs, where the
 spec can be a package or a source definition.  Note that the spec must
 evaluate to a path to a gzipped tarball which includes a @code{Cargo.toml}
 file at its root, or it will be ignored.  Similarly, cargo dev-dependencies
@@ -7449,8 +7451,11 @@ In its @code{configure} phase, this build system will make any source inputs
 specified in the @code{#:cargo-inputs} and @code{#:cargo-development-inputs}
 parameters available to cargo.  It will also remove an included
 @code{Cargo.lock} file to be recreated by @code{cargo} during the
-@code{build} phase.  The @code{install} phase installs the binaries
-defined by the crate.
+@code{build} phase.  The @code{package} phase will run @code{cargo package}
+to create a source crate for future use.  The @code{install} phase installs
+the binaries defined by the crate.  Unless @code{install-source? #f} is
+defined it will also install a source crate repository of itself and unpacked
+sources, to ease in future hacking on rust packages.
 @end defvr
 
 @defvr {Scheme Variable} chicken-build-system
diff --git a/guix/build-system/cargo.scm b/guix/build-system/cargo.scm
index 6c8edf6bac..01667ef045 100644
--- a/guix/build-system/cargo.scm
+++ b/guix/build-system/cargo.scm
@@ -5,6 +5,7 @@
 ;;; Copyright © 2016 David Craven <david@craven.ch>
 ;;; Copyright © 2019 Ivan Petkov <ivanppetkov@gmail.com>
 ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net>
+;;; Copyright © 2021 Efraim Flashner <efraim@flashner.co.il>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -77,8 +78,10 @@ to NAME and VERSION."
                       (vendor-dir "guix-vendor")
                       (cargo-build-flags ''("--release"))
                       (cargo-test-flags ''("--release"))
+                      (cargo-package-flags ''("--no-metadata" "--no-verify"))
                       (features ''())
                       (skip-build? #f)
+                      (install-source? #t)
                       (phases '(@ (guix build cargo-build-system)
                                   %standard-phases))
                       (outputs '("out"))
@@ -106,6 +109,7 @@ to NAME and VERSION."
                     #:vendor-dir ,vendor-dir
                     #:cargo-build-flags ,cargo-build-flags
                     #:cargo-test-flags ,cargo-test-flags
+                    #:cargo-package-flags ,cargo-package-flags
                     #:features ,features
                     #:skip-build? ,skip-build?
                     #:tests? ,(and tests? (not skip-build?))
diff --git a/guix/build/cargo-build-system.scm b/guix/build/cargo-build-system.scm
index 1d21b33895..0328ede4cb 100644
--- a/guix/build/cargo-build-system.scm
+++ b/guix/build/cargo-build-system.scm
@@ -2,7 +2,7 @@
 ;;; Copyright © 2016 David Craven <david@craven.ch>
 ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com>
 ;;; Copyright © 2019 Ivan Petkov <ivanppetkov@gmail.com>
-;;; Copyright © 2019, 2020 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2019, 2020, 2021 Efraim Flashner <efraim@flashner.co.il>
 ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net>
 ;;; Copyright © 2020 Marius Bakke <marius@gnu.org>
 ;;;
@@ -73,6 +73,38 @@ Cargo.toml file present at its root."
                                             " | cut -d/ -f2"
                                             " | grep -q '^Cargo.toml$'")))))
 
+(define* (unpack-rust-crates #:key inputs vendor-dir #:allow-other-keys)
+  (define (inputs->rust-inputs inputs)
+    "Filter using the label part from INPUTS."
+    (filter (lambda (input)
+              (match input
+                ((name . _) (rust-package? name))))
+            inputs))
+  (define (inputs->directories inputs)
+    "Extract the directory part from INPUTS."
+    (match inputs
+      (((names . directories) ...)
+       directories)))
+
+  (let ((rust-inputs (inputs->directories (inputs->rust-inputs inputs))))
+    (unless (null? rust-inputs)
+      (mkdir-p "target/package")
+      (mkdir-p vendor-dir)
+      ;; TODO: copy only regular inputs to target/package, not native-inputs.
+      (for-each (lambda (input-crate)
+                  (copy-recursively (string-append input-crate
+                                                   "/share/cargo/registry")
+                                    "target/package"))
+                (delete-duplicates rust-inputs))
+
+      (for-each (lambda (crate)
+                  (invoke "tar" "xzf" crate "-C" vendor-dir))
+                (find-files "target/package" "\\.crate$"))))
+  #t)
+
+(define (rust-package? name)
+  (string-prefix? "rust-" name))
+
 (define* (configure #:key inputs
                     (vendor-dir "guix-vendor")
                     #:allow-other-keys)
@@ -170,9 +202,23 @@ directory = '" port)
       (apply invoke "cargo" "test" cargo-test-flags)
       #t))
 
-(define* (install #:key inputs outputs skip-build? features #:allow-other-keys)
+(define* (package #:key
+                  (cargo-package-flags '("--no-metadata" "--no-verify"))
+                  #:allow-other-keys)
+  "Run 'cargo-package' for a given Cargo package."
+  (apply invoke `("cargo" "package" ,@cargo-package-flags)))
+
+(define* (install #:key
+                  inputs
+                  outputs
+                  skip-build?
+                  (install-source? #t)
+                  features
+                  #:allow-other-keys)
   "Install a given Cargo package."
-  (let* ((out (assoc-ref outputs "out")))
+  (let* ((out      (assoc-ref outputs "out"))
+         (registry (string-append out "/share/cargo/registry"))
+         (sources  (string-append out "/share/cargo/src")))
     (mkdir-p out)
 
     ;; Make cargo reuse all the artifacts we just built instead
@@ -186,6 +232,18 @@ directory = '" port)
         (invoke "cargo" "install" "--no-track" "--path" "." "--root" out
                 "--features" (string-join features)))
 
+    (when install-source?
+      ;; Install crate tarballs and unpacked sources for later use.
+      ;; TODO: Is there a better format/directory for these files?
+      (mkdir-p sources)
+      (for-each (lambda (crate)
+                  (install-file crate registry))
+                (find-files "target/package" "\\.crate$"))
+
+      (for-each (lambda (crate)
+                  (invoke "tar" "xzf" crate "-C" sources))
+                (find-files registry "\\.crate$")))
+
     #t))
 
 (define %standard-phases
@@ -195,6 +253,8 @@ directory = '" port)
     (replace 'build build)
     (replace 'check check)
     (replace 'install install)
+    (add-after 'build 'package package)
+    (add-after 'unpack 'unpack-rust-crates unpack-rust-crates)
     (add-after 'patch-generated-file-shebangs 'patch-cargo-checksums patch-cargo-checksums)))
 
 (define* (cargo-build #:key inputs (phases %standard-phases)
-- 
2.30.0

bug#46399: Acknowledgement ([PATCH] build-system/cargo: Propagate crates across builds.)

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 336 bytes --]

Pushed with minor revisions. Also patched a few packages which needed
#:install-source #? in order to continue to build.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bug#46399] [PATCH] build-system/cargo: Propagate crates across builds.

[pelzflorian (Florian Pelz)]

Hello Efraim!

Thank you for improving Rust development with Guix.  I have a question
about this change to doc/guix.texi when translating.  Sorry for
noticing so late just before the string freeze.

On Tue, Feb 09, 2021 at 12:58:43PM +0200, Efraim Flashner wrote:
> diff --git a/doc/guix.texi b/doc/guix.texi
> index 7d18703283..0d153a1470 100644
> --- a/doc/guix.texi
> +++ b/doc/guix.texi
> […]
> @@ -7437,8 +7437,10 @@ supports builds of packages using Cargo, the build tool of the
>  It adds @code{rustc} and @code{cargo} to the set of inputs.
>  A different Rust package can be specified with the @code{#:rust} parameter.
>  
> -Regular cargo dependencies should be added to the package definition via the
> -@code{#:cargo-inputs} parameter as a list of name and spec pairs, where the
> +Regular cargo dependencies should be added to the package definition similarly
> +to other packages; those needed only at build time to native-inputs, others to
> +inputs.  If you need to add source-only crates then you should add them to via
> +the @code{#:cargo-inputs} parameter as a list of name and spec pairs, where the
>  spec can be a package or a source definition.  Note that the spec must
>  evaluate to a path to a gzipped tarball which includes a @code{Cargo.toml}
>  file at its root, or it will be ignored.  Similarly, cargo dev-dependencies

I have no experience in Rust.  Do I understand correctly that
source-only crates among the inputs should be added both to inputs and
to #:cargo-inputs?  Is the

> +inputs.  If you need to add source-only crates then you should add them to via

a typo which should read “add them too” instead of “add them to”?

I look at the subsequent change
f35ab79e033aa8e4955e9c9100c74ecbb3ce4d0b to rust-wayland-commons

> diff --git a/gnu/packages/crates-graphics.scm b/gnu/packages/crates-graphics.scm
> index d67e9dc02f..7dde00ffdb 100644
> --- a/gnu/packages/crates-graphics.scm
> +++ b/gnu/packages/crates-graphics.scm
> @@ -2415,8 +2415,11 @@ the wayland protocol, client side.")
>      (arguments
> -     `(#:skip-build? #t
> -       #:cargo-inputs
> +     `(#:cargo-inputs
>         (("rust-nix" ,rust-nix-0.18)
>          ("rust-once-cell" ,rust-once-cell-1)
> -        ("rust-smallvec" ,rust-smallvec-1)
> -        ("rust-wayland-sys" ,rust-wayland-sys-0.28))))
> +        ("rust-smallvec" ,rust-smallvec-1))))
> +    (inputs
> +     `(("rust-nix" ,rust-nix-0.18)
> +       ("rust-once-cell" ,rust-once-cell-1)
> +       ("rust-smallvec" ,rust-smallvec-1)
> +       ("rust-wayland-sys" ,rust-wayland-sys-0.28)))
>      (home-page "https://github.com/smithay/wayland-rs")

and do not understand why rust-wayland-sys is only part of inputs but
rust-once-cell is also among the cargo-inputs.  Also the inputs are
empty for older versions of rust-wayland-commons.

Could you clarify what is written in the doc/guix.texi?

Regards,
Florian

[bug#46399] [PATCH] build-system/cargo: Propagate crates across builds.

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 4776 bytes --]

On Sun, Apr 11, 2021 at 05:11:21PM +0200, pelzflorian (Florian Pelz) wrote:
> Hello Efraim!
> 
> Thank you for improving Rust development with Guix.  I have a question
> about this change to doc/guix.texi when translating.  Sorry for
> noticing so late just before the string freeze.
> 
> On Tue, Feb 09, 2021 at 12:58:43PM +0200, Efraim Flashner wrote:
> > diff --git a/doc/guix.texi b/doc/guix.texi
> > index 7d18703283..0d153a1470 100644
> > --- a/doc/guix.texi
> > +++ b/doc/guix.texi
> > […]
> > @@ -7437,8 +7437,10 @@ supports builds of packages using Cargo, the build tool of the
> >  It adds @code{rustc} and @code{cargo} to the set of inputs.
> >  A different Rust package can be specified with the @code{#:rust} parameter.
> >  
> > -Regular cargo dependencies should be added to the package definition via the
> > -@code{#:cargo-inputs} parameter as a list of name and spec pairs, where the
> > +Regular cargo dependencies should be added to the package definition similarly
> > +to other packages; those needed only at build time to native-inputs, others to
> > +inputs.  If you need to add source-only crates then you should add them to via
> > +the @code{#:cargo-inputs} parameter as a list of name and spec pairs, where the
> >  spec can be a package or a source definition.  Note that the spec must
> >  evaluate to a path to a gzipped tarball which includes a @code{Cargo.toml}
> >  file at its root, or it will be ignored.  Similarly, cargo dev-dependencies
> 
> I have no experience in Rust.  Do I understand correctly that
> source-only crates among the inputs should be added both to inputs and
> to #:cargo-inputs?  Is the
> 
> > +inputs.  If you need to add source-only crates then you should add them to via
> 
> a typo which should read “add them too” instead of “add them to”?

I went back and refreshed myself with what I wrote there. source-only
crates should be added to both regular inputs and to #:cargo-inputs.
Actually converting the ~1000 rust packages to actually make good use of
the changes will allow us to drop nearly all the #:cargo-inputs and just
use regular inputs, but until then we still need the propagated source
magic that comes from #:cargo-inputs.

Yes, definitely a typo, it should be "add them too".

> I look at the subsequent change
> f35ab79e033aa8e4955e9c9100c74ecbb3ce4d0b to rust-wayland-commons
> 
> > diff --git a/gnu/packages/crates-graphics.scm b/gnu/packages/crates-graphics.scm
> > index d67e9dc02f..7dde00ffdb 100644
> > --- a/gnu/packages/crates-graphics.scm
> > +++ b/gnu/packages/crates-graphics.scm
> > @@ -2415,8 +2415,11 @@ the wayland protocol, client side.")
> >      (arguments
> > -     `(#:skip-build? #t
> > -       #:cargo-inputs
> > +     `(#:cargo-inputs
> >         (("rust-nix" ,rust-nix-0.18)
> >          ("rust-once-cell" ,rust-once-cell-1)
> > -        ("rust-smallvec" ,rust-smallvec-1)
> > -        ("rust-wayland-sys" ,rust-wayland-sys-0.28))))
> > +        ("rust-smallvec" ,rust-smallvec-1))))
> > +    (inputs
> > +     `(("rust-nix" ,rust-nix-0.18)
> > +       ("rust-once-cell" ,rust-once-cell-1)
> > +       ("rust-smallvec" ,rust-smallvec-1)
> > +       ("rust-wayland-sys" ,rust-wayland-sys-0.28)))
> >      (home-page "https://github.com/smithay/wayland-rs")
> 
> and do not understand why rust-wayland-sys is only part of inputs but
> rust-once-cell is also among the cargo-inputs.  Also the inputs are
> empty for older versions of rust-wayland-commons.

#:cargo-inputs are "magic" in that they propagate their own #:cargo-inputs
to future builds. The changes to the cargo-build-system mimic this so we
can use regular inputs and not #:cargo-inputs. rust-wayland-sys-0.28 has
enough of its dependencies converted that it can be used as just an
input but rust-once-cell-1 hasn't yet so it still needs to be in
#:cargo-inputs. As far as the other versions of rust-wayland-commons, I
started by focusing on alacritty so we could remove some of the build
phases and move them to the crates which actually needed patching.

> Could you clarify what is written in the doc/guix.texi?
> 
> Regards,
> Florian

I think for now the best option is to add the rust inputs of all types
to both inputs and #:cargo-inputs (or native-inputs and
#:cargo-development-inputs) and then remove them from #:cargo-inputs
when it's possible to do so. Ultimately Someone™ should revive the
wip-rust branch and convert all the crates over in one go and adjust the
importer and manual.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bug#46399] [PATCH] build-system/cargo: Propagate crates across builds.

[pelzflorian (Florian Pelz)]

On Mon, Apr 12, 2021 at 09:40:17AM +0300, Efraim Flashner wrote:
> I went back and refreshed myself with what I wrote there. source-only
> crates should be added to both regular inputs and to #:cargo-inputs.

Thank you for clarifying.  I will translate the manual accordingly.

> #:cargo-inputs are "magic" in that they propagate their own #:cargo-inputs
> to future builds.

I had not known about this difference.

> I think for now the best option is to add the rust inputs of all types
> to both inputs and #:cargo-inputs (or native-inputs and
> #:cargo-development-inputs) and then remove them from #:cargo-inputs
> when it's possible to do so. Ultimately Someone™ should revive the
> wip-rust branch and convert all the crates over in one go and adjust the
> importer and manual.

I suppose this paragraph of your reply should be added to the manual.

Regards,
Florian

[bug#46399] [PATCH] build-system/cargo: Propagate crates across builds.

[Leo Famulari]

On Tue, Apr 13, 2021 at 02:31:39PM +0200, pelzflorian (Florian Pelz) wrote:
> On Mon, Apr 12, 2021 at 09:40:17AM +0300, Efraim Flashner wrote:
> > I went back and refreshed myself with what I wrote there. source-only
> > crates should be added to both regular inputs and to #:cargo-inputs.
> 
> Thank you for clarifying.  I will translate the manual accordingly.

We are in "string freeze" for the upcoming release, so please delay your
changes to the manual until the release is completed.