rust-team branch merged

rust-team branch merged

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2163 bytes --]

The rust team is pleased to announce that the rust-team branch has been
merged back into master.  There are 570 commits across the branch.
Cross-compiling support for the cargo-build-system was added, including
for librsvg.  Cross-compiling was tested for (nearly) all architectures
supported in `guix build --list-targets`.  Upstream rust added
'i686-unknown-hurd-gnu' as a target in rust-1.74, so that will need to
wait until another time.  In the meantime, x86_64-w64-mingw32 at least
compiles correctly and tests well under wine.

Notable notes:

* New rust version is 1.73.
* 4001 packages use the cargo-build-system
* Rust crates produced in the 'package' phase of the cargo-build-system
should now be reproducible (for real this time)
* Packages added: eza, kibi, libgit2-1.6, libgit2-1.7, spotifyd, stgit-2
* Packages updated: alfis, rust-analyzer, rust-cargo-c
* Rust-analyzer is built from the rust sources now, updating it to 1.73.
It can be installed as 'rust-analyzer' or as 'rust:tools'
* Cross-compiling support for the cargo-build-system was added

Additional notes:

* crates.io no longer accepts '-' instead of '_' in crate lookups, so
there are a (probably large) number of packages which need to have their
source adjusted.

* Compiled rust packages currently have a 'package' phase, which runs
the command used to crate a 'crate tarball', and is installed in
%output/share/cargo/registry, with unpacked sources in
%output/share/cargo/src.  In theory it should be possible to use these
for local rust development.  The benefits include everything that comes
with being a guix package, including pre-patched shebangs.  Currently no
index file is created in $GUIX_ENVIRONMENT/share/cargo/registry/index,
which is likely necessary to actually make use of this.  Additionally, I
am unsure how to use '$GUIX_ENVIRONMENT' in ~/.cargo/config so that it
is expanded and not taken as a literal string.

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: rust-team branch merged

[Maxim Cournoyer]

Hi Efraim,

Efraim Flashner <efraim@flashner.co.il> writes:

> The rust team is pleased to announce that the rust-team branch has been
> merged back into master.  There are 570 commits across the branch.
> Cross-compiling support for the cargo-build-system was added, including
> for librsvg.  Cross-compiling was tested for (nearly) all architectures
> supported in `guix build --list-targets`.  Upstream rust added
> 'i686-unknown-hurd-gnu' as a target in rust-1.74, so that will need to
> wait until another time.  In the meantime, x86_64-w64-mingw32 at least
> compiles correctly and tests well under wine.
>
> Notable notes:
>
> * New rust version is 1.73.
> * 4001 packages use the cargo-build-system
> * Rust crates produced in the 'package' phase of the cargo-build-system
> should now be reproducible (for real this time)
> * Packages added: eza, kibi, libgit2-1.6, libgit2-1.7, spotifyd, stgit-2
> * Packages updated: alfis, rust-analyzer, rust-cargo-c
> * Rust-analyzer is built from the rust sources now, updating it to 1.73.
> It can be installed as 'rust-analyzer' or as 'rust:tools'
> * Cross-compiling support for the cargo-build-system was added

Sounds good!  Many thanks for working toward a better Rust in Guix.

-- 
Thanks,
Maxim

Re: rust-team branch merged

[Efraim Flashner]

[-- Attachment #1.1: Type: text/plain, Size: 1878 bytes --]

On Wed, Dec 13, 2023 at 10:34:11AM +0200, Efraim Flashner wrote:
> * Compiled rust packages currently have a 'package' phase, which runs
> the command used to crate a 'crate tarball', and is installed in
> %output/share/cargo/registry, with unpacked sources in
> %output/share/cargo/src.  In theory it should be possible to use these
> for local rust development.  The benefits include everything that comes
> with being a guix package, including pre-patched shebangs.  Currently no
> index file is created in $GUIX_ENVIRONMENT/share/cargo/registry/index,
> which is likely necessary to actually make use of this.  Additionally, I
> am unsure how to use '$GUIX_ENVIRONMENT' in ~/.cargo/config so that it
> is expanded and not taken as a literal string.

In the Guix London meetup someone mentioned that they were interested in
playing around with using Guix for rust development.  I've adjusted the
cargo-build-system to produce the registry index files and I added a
profile hook to generate the config.json to locate the packaged crates.

toml files can't process environment variables (which is probably a good
thing ...) but that means its a little harder to test out.

with the two patches applied create an environment with the crates you
want and get the location of GUIX_ENVIRONMENT:
`env | grep GUIX_ENVIRONMENT | cut -f2 -d=`

in ~/.cargo/config:
[source.crates-io]
local-registry = '<location of GUIX_ENVIRONMENT>/share/cargo/registry'

'cargo build' should pull from the local crates in the GUIX_ENVIRONMENT.
I'm not sure what happens if it doesn't have those crates available and
would need to get them from crates.io.


-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #1.2: 0001-build-cargo-build-system-Produce-registry-index-file.patch --]
[-- Type: text/plain, Size: 8648 bytes --]

From 84d2f06e9facec9a1646afd53adcbf2b95da4da0 Mon Sep 17 00:00:00 2001
Message-ID: <84d2f06e9facec9a1646afd53adcbf2b95da4da0.1702565894.git.efraim@flashner.co.il>
In-Reply-To: <cover.1702565894.git.efraim@flashner.co.il>
References: <cover.1702565894.git.efraim@flashner.co.il>
From: Efraim Flashner <efraim@flashner.co.il>
Date: Thu, 14 Dec 2023 16:49:58 +0200
Subject: [PATCH 1/2] build: cargo-build-system: Produce registry index files.

* guix/build/cargo-build-system.scm (rewrite-deps): New procedure.
(package): Obtain output from 'cargo manifest' of the current package
when the package will be installed.
(install): Don't install cargo crates and source.
(install-source): New phase.  Install cargo crates.  Generate cargo
registry index files and install them in a known location.
(%standard-phases): Add 'install-source phase after 'install.

Change-Id: I6ce6c5b33fe3eb7667f86964daef798320724a25
---
 guix/build/cargo-build-system.scm | 102 +++++++++++++++++++++++++-----
 1 file changed, 87 insertions(+), 15 deletions(-)

diff --git a/guix/build/cargo-build-system.scm b/guix/build/cargo-build-system.scm
index ffb2ec898e..f1e35a5c65 100644
--- a/guix/build/cargo-build-system.scm
+++ b/guix/build/cargo-build-system.scm
@@ -73,6 +73,21 @@ (define (crate-src? path)
                                             " | cut -d/ -f2"
                                             " | grep -q '^Cargo.toml$'")))))
 
+(define (rewrite-deps cargo-deps)
+  (map (lambda (dependency)
+         `(@ ("name" . ,(assoc-ref dependency "name"))
+             ("req" . ,(assoc-ref dependency "req"))
+             ("features" . ,(assoc-ref dependency "features"))
+             ("optional" . ,(assoc-ref dependency "optional"))
+             ("default_features" . ,(assoc-ref dependency
+                                               "uses_default_features"))
+             ("target" . ,(assoc-ref dependency "target"))
+             ("kind" . ,(match (assoc-ref dependency "kind")
+                               (null? "normal")
+                               (kind kind)))
+             ("registry" . ,(assoc-ref dependency "registry"))))
+       cargo-deps))
+
 (define* (unpack-rust-crates #:key inputs vendor-dir #:allow-other-keys)
   (define (inputs->rust-inputs inputs)
     "Filter using the label part from INPUTS."
@@ -271,7 +286,8 @@ (define* (package #:key
                   install-source?
                   (cargo-package-flags '("--no-metadata" "--no-verify"))
                   #:allow-other-keys)
-  "Run 'cargo-package' for a given Cargo package."
+  "Run 'cargo-package' for a given Cargo package.  Also generate metadata so we can
+create a package index for the crates."
   (if install-source?
     (if skip-build?
       (begin
@@ -322,19 +338,24 @@ (define* (package #:key
                 (delete-file-recursively dir)))
             (find-files "." "\\.crate$")))))
     (format #t "Not installing cargo sources, skipping `cargo package`.~%"))
-  #t)
+
+  (when install-source?
+    ;; First generate the metadata so we can create the index file.
+    ;; https://doc.rust-lang.org/cargo/commands/cargo-metadata.html#json-format
+    (with-output-to-file "cargo-metadata.json"
+      (lambda _
+        (invoke "cargo" "metadata"
+                "--manifest-path" "Cargo.toml"
+                "--format-version" "1"
+                "--no-deps")))))
 
 (define* (install #:key
-                  inputs
                   outputs
                   skip-build?
-                  install-source?
                   features
                   #:allow-other-keys)
   "Install a given Cargo package."
-  (let* ((out      (assoc-ref outputs "out"))
-         (registry (string-append out "/share/cargo/registry"))
-         (sources  (string-append out "/share/cargo/src")))
+  (let ((out (assoc-ref outputs "out")))
     (mkdir-p out)
 
     ;; Make cargo reuse all the artifacts we just built instead
@@ -346,21 +367,71 @@ (define* (install #:key
     (or skip-build?
         (not (has-executable-target?))
         (invoke "cargo" "install" "--no-track" "--path" "." "--root" out
-                "--features" (string-join features)))
+                "--features" (string-join features)))))
+
+(define* (install-sources #:key
+                          name
+                          outputs
+                          install-source?
+                          #:allow-other-keys)
+  "Install a given Cargo package."
+  (let* ((out      (assoc-ref outputs "out"))
+         (registry (string-append out "/share/cargo/registry/"))
+         (name+ver (if (string-prefix? "rust-" name)
+                       (string-drop name 5)
+                       name))
+         ;; Strip the version if it exists.
+         (pkgname  (if (char-set-every
+                         (lambda (item)
+                           (char-set-contains?
+                             (list->char-set (list #\- #\.) char-set:digit) item))
+                         (string->char-set
+                           (string-drop name+ver (string-index-right name+ver #\-))))
+                     (string-take name+ver (string-index-right name+ver #\-))
+                     name+ver)))
 
     (when install-source?
       ;; Install crate tarballs and unpacked sources for later use.
-      ;; TODO: Is there a better format/directory for these files?
-      (mkdir-p sources)
       (for-each (lambda (crate)
                   (install-file crate registry))
                 (find-files "target/package" "\\.crate$"))
 
-      (for-each (lambda (crate)
-                  (invoke "tar" "xzf" crate "-C" sources))
-                (find-files registry "\\.crate$")))
+      (let ((path (match (string-length pkgname)
+                         (1 "1")
+                         (2 "2")
+                         (3 (string-append "3/" (string-take pkgname 1)))
+                         (else (string-append (substring pkgname 0 2) "/"
+                                              (substring pkgname 2 4)))))
+            (cargo-metadata (match (call-with-input-file
+                                     "cargo-metadata.json" read-json)
+                                   (('@ . alist) alist)))
+            (sha256sum (read-delimited
+                         " "
+                         (open-pipe* OPEN_READ
+                                     "sha256sum" "--"
+                                     (first (find-files registry pkgname))))))
 
-    #t))
+        ;; Now it's time to generate the actual index file:
+        ;; https://doc.rust-lang.org/cargo/reference/registry-index.html#json-schema
+        (call-with-output-file pkgname
+          (lambda (out)
+            (write-json
+              `(@ ("name" . ,(assoc-ref
+                               (first (assoc-ref cargo-metadata "packages"))
+                               "name"))
+                  ("vers" . ,(assoc-ref
+                               (first (assoc-ref cargo-metadata "packages"))
+                               "version"))
+                  ("deps" . ,(rewrite-deps (assoc-ref
+                               (first (assoc-ref cargo-metadata "packages"))
+                               "dependencies")))
+                  ("cksum" . ,sha256sum)
+                  ("features" . ,(assoc-ref
+                                   (first (assoc-ref cargo-metadata "packages"))
+                                   "features"))
+                  ("yanked" . #f))
+              out)))
+        (install-file pkgname (string-append registry "/index/" path))))))
 
 (define %standard-phases
   (modify-phases gnu:%standard-phases
@@ -372,7 +443,8 @@ (define %standard-phases
     (add-after 'build 'package package)
     (add-after 'unpack 'check-for-pregenerated-files check-for-pregenerated-files)
     (add-after 'check-for-pregenerated-files 'unpack-rust-crates unpack-rust-crates)
-    (add-after 'patch-generated-file-shebangs 'patch-cargo-checksums patch-cargo-checksums)))
+    (add-after 'patch-generated-file-shebangs 'patch-cargo-checksums patch-cargo-checksums)
+    (add-after 'install 'install-sources install-sources)))
 
 (define* (cargo-build #:key inputs (phases %standard-phases)
                       #:allow-other-keys #:rest args)
-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted


[-- Attachment #1.3: 0002-guix-profiles-Add-cargo-registry-profile-hook.patch --]
[-- Type: text/plain, Size: 2913 bytes --]

From 1dca2772f81df7d9a66f7a97668c3f9acca141a8 Mon Sep 17 00:00:00 2001
Message-ID: <1dca2772f81df7d9a66f7a97668c3f9acca141a8.1702565894.git.efraim@flashner.co.il>
In-Reply-To: <cover.1702565894.git.efraim@flashner.co.il>
References: <cover.1702565894.git.efraim@flashner.co.il>
From: Efraim Flashner <efraim@flashner.co.il>
Date: Thu, 14 Dec 2023 16:54:11 +0200
Subject: [PATCH 2/2] guix: profiles: Add cargo-registry profile hook.

* guix/profiles.scm (cargo-registry): New profile-hook.
(%default-profile-hooks): Add cargo-registry.

Change-Id: I8642e116e7ff7df2ae2dde77c98d5cfeed85f99d
---
 guix/profiles.scm | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/guix/profiles.scm b/guix/profiles.scm
index ce2f8337bf..129448b2b1 100644
--- a/guix/profiles.scm
+++ b/guix/profiles.scm
@@ -13,6 +13,7 @@
 ;;; Copyright © 2020 Danny Milosavljevic <dannym@scratchpost.org>
 ;;; Copyright © 2014 David Thompson <davet@gnu.org>
 ;;; Copyright © 2022 Arun Isaac <arunisaac@systemreboot.net>
+;;; Copyright © 2023 Efraim Flashner <efraim@flashner.co.il>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -1193,6 +1194,29 @@ (define* (ca-certificate-bundle manifest #:optional system)
                     `((type . profile-hook)
                       (hook . ca-certificate-bundle))))
 
+(define* (cargo-registry manifest #:optional system)
+  (define build
+    (with-imported-modules '((guix build utils))
+      #~(begin
+          (use-modules (guix build utils))
+          (let ((registry (string-append #$output "/share/cargo/registry")))
+            (mkdir-p registry)
+            ;; https://doc.rust-lang.org/cargo/reference/registry-index.html#index-configuration
+            (with-output-to-file (string-append registry "/config.json")
+              (lambda _
+                (format #t "{~@
+                        \"dl\": \"~a/{crate}-{version}.crate\",~@
+                        }~%"
+                        registry)))))))
+
+  (gexp->derivation "cargo-registry" build
+                    #:system system
+                    #:local-build? #t
+                    #:substitutable? #f
+                    #:properties
+                    `((type . profile-hook)
+                      (hook . cargo-registry))))
+
 (define* (emacs-subdirs manifest #:optional system)
   (define build
     (with-imported-modules (source-module-closure
@@ -1932,6 +1956,7 @@ (define %default-profile-hooks
         fonts-dir-file
         ghc-package-cache-file
         ca-certificate-bundle
+        cargo-registry
         emacs-subdirs
         gdk-pixbuf-loaders-cache-file
         glib-schemas
-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: rust-team branch merged

[Jason Conroy]

[-- Attachment #1: Type: text/plain, Size: 3149 bytes --]

Hello Efraim,

Thanks for investigating this - a Rust development workflow using only
Guix-native crates is something I've been waiting for!

I was experimenting with your patches and it seems that they do pull in the
source crates for requested packages, but not their dependencies (example
below). Is there something I'm missing?

Cheers,
Jason

$ guix shell --pure bash findutils rust-rand -- bash -c 'find -L
$GUIX_ENVIRONMENT/share/cargo'
/gnu/store/zf88v65rbg2di4qhgdbvhfcjf31rdzby-profile/share/cargo
/gnu/store/zf88v65rbg2di4qhgdbvhfcjf31rdzby-profile/share/cargo/registry
/gnu/store/zf88v65rbg2di4qhgdbvhfcjf31rdzby-profile/share/cargo/registry/index
/gnu/store/zf88v65rbg2di4qhgdbvhfcjf31rdzby-profile/share/cargo/registry/index/ra
/gnu/store/zf88v65rbg2di4qhgdbvhfcjf31rdzby-profile/share/cargo/registry/index/ra/nd
/gnu/store/zf88v65rbg2di4qhgdbvhfcjf31rdzby-profile/share/cargo/registry/index/ra/nd/rand
/gnu/store/zf88v65rbg2di4qhgdbvhfcjf31rdzby-profile/share/cargo/registry/rand-0.8.5.crate
/gnu/store/zf88v65rbg2di4qhgdbvhfcjf31rdzby-profile/share/cargo/registry/config.json

On Thu, Dec 14, 2023 at 10:10 AM Efraim Flashner <efraim@flashner.co.il>
wrote:

> On Wed, Dec 13, 2023 at 10:34:11AM +0200, Efraim Flashner wrote:
> > * Compiled rust packages currently have a 'package' phase, which runs
> > the command used to crate a 'crate tarball', and is installed in
> > %output/share/cargo/registry, with unpacked sources in
> > %output/share/cargo/src.  In theory it should be possible to use these
> > for local rust development.  The benefits include everything that comes
> > with being a guix package, including pre-patched shebangs.  Currently no
> > index file is created in $GUIX_ENVIRONMENT/share/cargo/registry/index,
> > which is likely necessary to actually make use of this.  Additionally, I
> > am unsure how to use '$GUIX_ENVIRONMENT' in ~/.cargo/config so that it
> > is expanded and not taken as a literal string.
>
> In the Guix London meetup someone mentioned that they were interested in
> playing around with using Guix for rust development.  I've adjusted the
> cargo-build-system to produce the registry index files and I added a
> profile hook to generate the config.json to locate the packaged crates.
>
> toml files can't process environment variables (which is probably a good
> thing ...) but that means its a little harder to test out.
>
> with the two patches applied create an environment with the crates you
> want and get the location of GUIX_ENVIRONMENT:
> `env | grep GUIX_ENVIRONMENT | cut -f2 -d=`
>
> in ~/.cargo/config:
> [source.crates-io]
> local-registry = '<location of GUIX_ENVIRONMENT>/share/cargo/registry'
>
> 'cargo build' should pull from the local crates in the GUIX_ENVIRONMENT.
> I'm not sure what happens if it doesn't have those crates available and
> would need to get them from crates.io.
>
>
> --
> Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
> GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
> Confidentiality cannot be guaranteed on emails sent or received unencrypted
>

[-- Attachment #2: Type: text/html, Size: 3895 bytes --]

Re: rust-team branch merged

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 4153 bytes --]

On Mon, Feb 26, 2024 at 09:24:29PM -0500, Jason Conroy wrote:
> Hello Efraim,
> 
> Thanks for investigating this - a Rust development workflow using only
> Guix-native crates is something I've been waiting for!
> 
> I was experimenting with your patches and it seems that they do pull in the
> source crates for requested packages, but not their dependencies (example
> below). Is there something I'm missing?

When you say they pull in the source crates do you mean the sources of
the other rust packages needed by rust-rand? I didn't test that, but I
assumed it wouldn't.

Currently if you were to pull in rust-rand-0.8 and rust-rand-0.7 then
you'd have both rand-0.*.crate files in the registry but only one of
them would be listed in share/cargo/registry/index/ra/nd/rand. I need to
adjust the generation of that file to combine multiple sources if they
exist, and sort them (I'm not sure it's necessary, but wouldn't be
surprised if we hit undefined behaviour if they were listed multiple
times or out of order).  I also need to figure out something with a
config.toml to see if it's possible to generate one that could be
included from another one, since you can't add 'local-registry =
$GUIX_PROFILE/...' in a toml file.

> Cheers,
> Jason
> 
> $ guix shell --pure bash findutils rust-rand -- bash -c 'find -L
> $GUIX_ENVIRONMENT/share/cargo'
> /gnu/store/zf88v65rbg2di4qhgdbvhfcjf31rdzby-profile/share/cargo
> /gnu/store/zf88v65rbg2di4qhgdbvhfcjf31rdzby-profile/share/cargo/registry
> /gnu/store/zf88v65rbg2di4qhgdbvhfcjf31rdzby-profile/share/cargo/registry/index
> /gnu/store/zf88v65rbg2di4qhgdbvhfcjf31rdzby-profile/share/cargo/registry/index/ra
> /gnu/store/zf88v65rbg2di4qhgdbvhfcjf31rdzby-profile/share/cargo/registry/index/ra/nd
> /gnu/store/zf88v65rbg2di4qhgdbvhfcjf31rdzby-profile/share/cargo/registry/index/ra/nd/rand
> /gnu/store/zf88v65rbg2di4qhgdbvhfcjf31rdzby-profile/share/cargo/registry/rand-0.8.5.crate
> /gnu/store/zf88v65rbg2di4qhgdbvhfcjf31rdzby-profile/share/cargo/registry/config.json
> 
> On Thu, Dec 14, 2023 at 10:10 AM Efraim Flashner <efraim@flashner.co.il>
> wrote:
> 
> > On Wed, Dec 13, 2023 at 10:34:11AM +0200, Efraim Flashner wrote:
> > > * Compiled rust packages currently have a 'package' phase, which runs
> > > the command used to crate a 'crate tarball', and is installed in
> > > %output/share/cargo/registry, with unpacked sources in
> > > %output/share/cargo/src.  In theory it should be possible to use these
> > > for local rust development.  The benefits include everything that comes
> > > with being a guix package, including pre-patched shebangs.  Currently no
> > > index file is created in $GUIX_ENVIRONMENT/share/cargo/registry/index,
> > > which is likely necessary to actually make use of this.  Additionally, I
> > > am unsure how to use '$GUIX_ENVIRONMENT' in ~/.cargo/config so that it
> > > is expanded and not taken as a literal string.
> >
> > In the Guix London meetup someone mentioned that they were interested in
> > playing around with using Guix for rust development.  I've adjusted the
> > cargo-build-system to produce the registry index files and I added a
> > profile hook to generate the config.json to locate the packaged crates.
> >
> > toml files can't process environment variables (which is probably a good
> > thing ...) but that means its a little harder to test out.
> >
> > with the two patches applied create an environment with the crates you
> > want and get the location of GUIX_ENVIRONMENT:
> > `env | grep GUIX_ENVIRONMENT | cut -f2 -d=`
> >
> > in ~/.cargo/config:
> > [source.crates-io]
> > local-registry = '<location of GUIX_ENVIRONMENT>/share/cargo/registry'
> >
> > 'cargo build' should pull from the local crates in the GUIX_ENVIRONMENT.
> > I'm not sure what happens if it doesn't have those crates available and
> > would need to get them from crates.io.
> >
> >

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: rust-team branch merged

[Jason Conroy]

[-- Attachment #1: Type: text/plain, Size: 3802 bytes --]

On Wed, Mar 6, 2024 at 4:06 AM Efraim Flashner <efraim@flashner.co.il>
wrote:

> On Mon, Feb 26, 2024 at 09:24:29PM -0500, Jason Conroy wrote:
> > Hello Efraim,
> >
> > Thanks for investigating this - a Rust development workflow using only
> > Guix-native crates is something I've been waiting for!
> >
> > I was experimenting with your patches and it seems that they do pull in
> the
> > source crates for requested packages, but not their dependencies (example
> > below). Is there something I'm missing?
>
> When you say they pull in the source crates do you mean the sources of
> the other rust packages needed by rust-rand? I didn't test that, but I
> assumed it wouldn't.
>

That's right, I'm interested in having `guix shell` populate the full set
of transitive dependencies in the registry, similar to what happens when
using `cargo build` in the conventional way (but instead using
Guix-packaged crates). I interpreted your message up-thread to mean that
you were trying to accomplish the same, but please correct me if not.

I took rust-rand as an example only because it does have some dependencies:

$ cd $CARGO_PROJECT
$ cat Cargo.toml
[package]
name = "test_prog"
...
[dependencies]
rand = "0.8.5"

$ cargo build
    Updating crates.io index
  Downloaded cfg-if v1.0.0
  Downloaded rand_chacha v0.3.1
  Downloaded rand v0.8.5
  Downloaded ppv-lite86 v0.2.17
  Downloaded rand_core v0.6.4
  Downloaded getrandom v0.2.12
  Downloaded libc v0.2.153
  Downloaded 7 crates (932.0 KB) in 0.48s
   Compiling libc v0.2.153
   Compiling cfg-if v1.0.0
   Compiling ppv-lite86 v0.2.17
   Compiling getrandom v0.2.12
   Compiling rand_core v0.6.4
   Compiling rand_chacha v0.3.1
   Compiling rand v0.8.5
   Compiling test_prog v0.1.0 (/home/...)


> Currently if you were to pull in rust-rand-0.8 and rust-rand-0.7 then
> you'd have both rand-0.*.crate files in the registry but only one of
> them would be listed in share/cargo/registry/index/ra/nd/rand. I need to
> adjust the generation of that file to combine multiple sources if they
> exist, and sort them (I'm not sure it's necessary, but wouldn't be
> surprised if we hit undefined behaviour if they were listed multiple
> times or out of order).


I'm somewhat new to rust, but it appears that outside of Guix, the
local-only development workflow looks like this:

$ cd $CARGO_PROJECT
$ mkdir $VENDOR
$ cargo vendor $VENDOR

After downloading and unpacking all of the crates into $VENDOR, this last
command instructs me to add the following in ~/cargo/config.toml.
Then, after opening a new guix shell without network access, I can confirm
that `cargo build` works fine with the vendored crates.

[source.crates-io]
replace-with = "vendored-sources"

[source.vendored-sources]
directory = "<VENDOR>"

Getting back to your patch set: would it make sense to emulate this vendor
workflow instead of trying to construct a registry directly? Even assuming
that all details of the registry structure are stable and documented, the
layout of the vendor directory appears much simpler. And IIUC the code for
setting up vendored libraries already exists in cargo-build-system.

I also need to figure out something with a
> config.toml to see if it's possible to generate one that could be
> included from another one, since you can't add 'local-registry =
> $GUIX_PROFILE/...' in a toml file.
>

You've probably researched this more than I have, but it seems that this
use case is explicitly unsupported in the TOML language spec:
https://github.com/toml-lang/toml/issues/397

With that option off the table, I can't think of any elegant solutions.
Maybe a wrapper for the cargo binary that pre-processes cargo.toml and then
calls the real cargo?

[-- Attachment #2: Type: text/html, Size: 5087 bytes --]

Re: rust-team branch merged

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 5761 bytes --]

On Wed, Mar 06, 2024 at 12:53:44PM -0500, Jason Conroy wrote:
> On Wed, Mar 6, 2024 at 4:06 AM Efraim Flashner <efraim@flashner.co.il>
> wrote:
> 
> > On Mon, Feb 26, 2024 at 09:24:29PM -0500, Jason Conroy wrote:
> > > Hello Efraim,
> > >
> > > Thanks for investigating this - a Rust development workflow using only
> > > Guix-native crates is something I've been waiting for!
> > >
> > > I was experimenting with your patches and it seems that they do pull in
> > the
> > > source crates for requested packages, but not their dependencies (example
> > > below). Is there something I'm missing?
> >
> > When you say they pull in the source crates do you mean the sources of
> > the other rust packages needed by rust-rand? I didn't test that, but I
> > assumed it wouldn't.
> >
> 
> That's right, I'm interested in having `guix shell` populate the full set
> of transitive dependencies in the registry, similar to what happens when
> using `cargo build` in the conventional way (but instead using
> Guix-packaged crates). I interpreted your message up-thread to mean that
> you were trying to accomplish the same, but please correct me if not.

The transitive dependencies getting pulled in automatically should work
automatically if we ever finish the antioxidant-build-system. Until then
I've been experimenting by manually listing the other crates I've needed
but in theory we could try to make `guix shell --development` pull in
the needed crates.

I misread what you wrote as it was working. It's definitely something I
want but wasn't ready to work on yet.

> I took rust-rand as an example only because it does have some dependencies:
> 
> $ cd $CARGO_PROJECT
> $ cat Cargo.toml
> [package]
> name = "test_prog"
> ...
> [dependencies]
> rand = "0.8.5"
> 
> $ cargo build
>     Updating crates.io index
>   Downloaded cfg-if v1.0.0
>   Downloaded rand_chacha v0.3.1
>   Downloaded rand v0.8.5
>   Downloaded ppv-lite86 v0.2.17
>   Downloaded rand_core v0.6.4
>   Downloaded getrandom v0.2.12
>   Downloaded libc v0.2.153
>   Downloaded 7 crates (932.0 KB) in 0.48s
>    Compiling libc v0.2.153
>    Compiling cfg-if v1.0.0
>    Compiling ppv-lite86 v0.2.17
>    Compiling getrandom v0.2.12
>    Compiling rand_core v0.6.4
>    Compiling rand_chacha v0.3.1
>    Compiling rand v0.8.5
>    Compiling test_prog v0.1.0 (/home/...)
> 
> 
> > Currently if you were to pull in rust-rand-0.8 and rust-rand-0.7 then
> > you'd have both rand-0.*.crate files in the registry but only one of
> > them would be listed in share/cargo/registry/index/ra/nd/rand. I need to
> > adjust the generation of that file to combine multiple sources if they
> > exist, and sort them (I'm not sure it's necessary, but wouldn't be
> > surprised if we hit undefined behaviour if they were listed multiple
> > times or out of order).
> 
> 
> I'm somewhat new to rust, but it appears that outside of Guix, the
> local-only development workflow looks like this:
> 
> $ cd $CARGO_PROJECT
> $ mkdir $VENDOR
> $ cargo vendor $VENDOR
> 
> After downloading and unpacking all of the crates into $VENDOR, this last
> command instructs me to add the following in ~/cargo/config.toml.
> Then, after opening a new guix shell without network access, I can confirm
> that `cargo build` works fine with the vendored crates.
> 
> [source.crates-io]
> replace-with = "vendored-sources"
> 
> [source.vendored-sources]
> directory = "<VENDOR>"

I wanted local-registry over replace-with because IIRC replace-with
won't fall back to downloading from crates.io if there's missing crates,
while local-registry will check there first and then download any
missing crates.  The use-case I was looking at for that was adding a new
dependency to a project and then not needing to re-create a shell or
package the new crates before continuing on.

The registry as setup in the patches is actually mostly correct, minus
the multiple versions of a crate, it just needs to be fed the crates.

> Getting back to your patch set: would it make sense to emulate this vendor
> workflow instead of trying to construct a registry directly? Even assuming
> that all details of the registry structure are stable and documented, the
> layout of the vendor directory appears much simpler. And IIUC the code for
> setting up vendored libraries already exists in cargo-build-system.
> 
> I also need to figure out something with a
> > config.toml to see if it's possible to generate one that could be
> > included from another one, since you can't add 'local-registry =
> > $GUIX_PROFILE/...' in a toml file.
> >
> 
> You've probably researched this more than I have, but it seems that this
> use case is explicitly unsupported in the TOML language spec:
> https://github.com/toml-lang/toml/issues/397
>
> With that option off the table, I can't think of any elegant solutions.
> Maybe a wrapper for the cargo binary that pre-processes cargo.toml and then
> calls the real cargo?

Thanks, I didn't see that one.  So it looks like environment variables
and INCLUDEs are off the table.  An autoconf style macro that take a
config.toml.in and spits out a config.toml with the correct directory
would work, but from my understanding that's not really in line with any
bit of how cargo and the rust ecosystem works.

Another option would be to symlink
$GUIX_ENVIRONMENT/share/cargo/registry to ./local-registry and then add
the line 'local-registry = 'local-registry' in a config.toml.

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: rust-team branch merged

[Jason Conroy]

[-- Attachment #1: Type: text/plain, Size: 5973 bytes --]

On Thu, Mar 7, 2024 at 3:08 AM Efraim Flashner <efraim@flashner.co.il>
wrote:

> The transitive dependencies getting pulled in automatically should work
> automatically if we ever finish the antioxidant-build-system.


Since you bring up antioxidant, I'm kind of curious whether that stalled
mainly due to shifts in contributor priorities, or due to significant
technical issues.


> Until then
> I've been experimenting by manually listing the other crates I've needed
> but in theory we could try to make `guix shell --development` pull in
> the needed crates.
>

I was considering another option that falls somewhere in between: since I'm
already building shells from manifest files, it should suffice to have a
Scheme utility function that calculates the transitive dependencies for a
given list of library packages. Similar logic seems to exist already in
`(guix build-system cargo)` but it's not exposed publicly. As interim
solutions go, what do you think about this one versus modifying `guix
shell`?


>
> I misread what you wrote as it was working. It's definitely something I
> want but wasn't ready to work on yet.
>
> > I took rust-rand as an example only because it does have some
> dependencies:
> >
> > $ cd $CARGO_PROJECT
> > $ cat Cargo.toml
> > [package]
> > name = "test_prog"
> > ...
> > [dependencies]
> > rand = "0.8.5"
> >
> > $ cargo build
> >     Updating crates.io index
> >   Downloaded cfg-if v1.0.0
> >   Downloaded rand_chacha v0.3.1
> >   Downloaded rand v0.8.5
> >   Downloaded ppv-lite86 v0.2.17
> >   Downloaded rand_core v0.6.4
> >   Downloaded getrandom v0.2.12
> >   Downloaded libc v0.2.153
> >   Downloaded 7 crates (932.0 KB) in 0.48s
> >    Compiling libc v0.2.153
> >    Compiling cfg-if v1.0.0
> >    Compiling ppv-lite86 v0.2.17
> >    Compiling getrandom v0.2.12
> >    Compiling rand_core v0.6.4
> >    Compiling rand_chacha v0.3.1
> >    Compiling rand v0.8.5
> >    Compiling test_prog v0.1.0 (/home/...)
> >
> >
> > > Currently if you were to pull in rust-rand-0.8 and rust-rand-0.7 then
> > > you'd have both rand-0.*.crate files in the registry but only one of
> > > them would be listed in share/cargo/registry/index/ra/nd/rand. I need
> to
> > > adjust the generation of that file to combine multiple sources if they
> > > exist, and sort them (I'm not sure it's necessary, but wouldn't be
> > > surprised if we hit undefined behaviour if they were listed multiple
> > > times or out of order).
> >
> >
> > I'm somewhat new to rust, but it appears that outside of Guix, the
> > local-only development workflow looks like this:
> >
> > $ cd $CARGO_PROJECT
> > $ mkdir $VENDOR
> > $ cargo vendor $VENDOR
> >
> > After downloading and unpacking all of the crates into $VENDOR, this last
> > command instructs me to add the following in ~/cargo/config.toml.
> > Then, after opening a new guix shell without network access, I can
> confirm
> > that `cargo build` works fine with the vendored crates.
> >
> > [source.crates-io]
> > replace-with = "vendored-sources"
> >
> > [source.vendored-sources]
> > directory = "<VENDOR>"
>
> I wanted local-registry over replace-with because IIRC replace-with
> won't fall back to downloading from crates.io if there's missing crates,
> while local-registry will check there first and then download any
> missing crates.  The use-case I was looking at for that was adding a new
> dependency to a project and then not needing to re-create a shell or
> package the new crates before continuing on.
>

I see, thanks. My preferred workflow is different but I acknowledge that
use case.

The link below claims that one can update the vendor directory in a similar
way by re-running `cargo vendor` after adding a dependency to Cargo.toml,
but for your use case I agree that it's nicer if `cargo build` can pull to
the registry automatically.
https://old.reddit.com/r/rust/comments/uvvmjy/how_to_include_vendored_crates_into_a_project/


> The registry as setup in the patches is actually mostly correct, minus
> the multiple versions of a crate, it just needs to be fed the crates.
>
> > Getting back to your patch set: would it make sense to emulate this
> vendor
> > workflow instead of trying to construct a registry directly? Even
> assuming
> > that all details of the registry structure are stable and documented, the
> > layout of the vendor directory appears much simpler. And IIUC the code
> for
> > setting up vendored libraries already exists in cargo-build-system.
> >
> > I also need to figure out something with a
> > > config.toml to see if it's possible to generate one that could be
> > > included from another one, since you can't add 'local-registry =
> > > $GUIX_PROFILE/...' in a toml file.
> > >
> >
> > You've probably researched this more than I have, but it seems that this
> > use case is explicitly unsupported in the TOML language spec:
> > https://github.com/toml-lang/toml/issues/397
> >
> > With that option off the table, I can't think of any elegant solutions.
> > Maybe a wrapper for the cargo binary that pre-processes cargo.toml and
> then
> > calls the real cargo?
>
> Thanks, I didn't see that one.  So it looks like environment variables
> and INCLUDEs are off the table.  An autoconf style macro that take a
> config.toml.in and spits out a config.toml with the correct directory
> would work, but from my understanding that's not really in line with any
> bit of how cargo and the rust ecosystem works.
>
> Another option would be to symlink
> $GUIX_ENVIRONMENT/share/cargo/registry to ./local-registry and then add
> the line 'local-registry = 'local-registry' in a config.toml.
>
> --
> Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
> GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
> Confidentiality cannot be guaranteed on emails sent or received unencrypted
>

[-- Attachment #2: Type: text/html, Size: 7930 bytes --]

Re: rust-team branch merged

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 6672 bytes --]

On Thu, Mar 07, 2024 at 11:48:53AM -0500, Jason Conroy wrote:
> On Thu, Mar 7, 2024 at 3:08 AM Efraim Flashner <efraim@flashner.co.il>
> wrote:
> 
> > The transitive dependencies getting pulled in automatically should work
> > automatically if we ever finish the antioxidant-build-system.
> 
> 
> Since you bring up antioxidant, I'm kind of curious whether that stalled
> mainly due to shifts in contributor priorities, or due to significant
> technical issues.

More the first.  I found that the actual build time was slightly faster
using the antioxidant-build-system over the cargo-build-system but I
wasn't in a position to take it up when the original author moved on to
other things.

> > Until then
> > I've been experimenting by manually listing the other crates I've needed
> > but in theory we could try to make `guix shell --development` pull in
> > the needed crates.
> >
> 
> I was considering another option that falls somewhere in between: since I'm
> already building shells from manifest files, it should suffice to have a
> Scheme utility function that calculates the transitive dependencies for a
> given list of library packages. Similar logic seems to exist already in
> `(guix build-system cargo)` but it's not exposed publicly. As interim
> solutions go, what do you think about this one versus modifying `guix
> shell`?

Once you have something like that working it shouldn't be too hard to
merge that into `guix shell`, assuming we go that route.

> >
> > I misread what you wrote as it was working. It's definitely something I
> > want but wasn't ready to work on yet.
> >
> > > I took rust-rand as an example only because it does have some
> > dependencies:
> > >
> > > $ cd $CARGO_PROJECT
> > > $ cat Cargo.toml
> > > [package]
> > > name = "test_prog"
> > > ...
> > > [dependencies]
> > > rand = "0.8.5"
> > >
> > > $ cargo build
> > >     Updating crates.io index
> > >   Downloaded cfg-if v1.0.0
> > >   Downloaded rand_chacha v0.3.1
> > >   Downloaded rand v0.8.5
> > >   Downloaded ppv-lite86 v0.2.17
> > >   Downloaded rand_core v0.6.4
> > >   Downloaded getrandom v0.2.12
> > >   Downloaded libc v0.2.153
> > >   Downloaded 7 crates (932.0 KB) in 0.48s
> > >    Compiling libc v0.2.153
> > >    Compiling cfg-if v1.0.0
> > >    Compiling ppv-lite86 v0.2.17
> > >    Compiling getrandom v0.2.12
> > >    Compiling rand_core v0.6.4
> > >    Compiling rand_chacha v0.3.1
> > >    Compiling rand v0.8.5
> > >    Compiling test_prog v0.1.0 (/home/...)
> > >
> > >
> > > > Currently if you were to pull in rust-rand-0.8 and rust-rand-0.7 then
> > > > you'd have both rand-0.*.crate files in the registry but only one of
> > > > them would be listed in share/cargo/registry/index/ra/nd/rand. I need
> > to
> > > > adjust the generation of that file to combine multiple sources if they
> > > > exist, and sort them (I'm not sure it's necessary, but wouldn't be
> > > > surprised if we hit undefined behaviour if they were listed multiple
> > > > times or out of order).
> > >
> > >
> > > I'm somewhat new to rust, but it appears that outside of Guix, the
> > > local-only development workflow looks like this:
> > >
> > > $ cd $CARGO_PROJECT
> > > $ mkdir $VENDOR
> > > $ cargo vendor $VENDOR
> > >
> > > After downloading and unpacking all of the crates into $VENDOR, this last
> > > command instructs me to add the following in ~/cargo/config.toml.
> > > Then, after opening a new guix shell without network access, I can
> > confirm
> > > that `cargo build` works fine with the vendored crates.
> > >
> > > [source.crates-io]
> > > replace-with = "vendored-sources"
> > >
> > > [source.vendored-sources]
> > > directory = "<VENDOR>"
> >
> > I wanted local-registry over replace-with because IIRC replace-with
> > won't fall back to downloading from crates.io if there's missing crates,
> > while local-registry will check there first and then download any
> > missing crates.  The use-case I was looking at for that was adding a new
> > dependency to a project and then not needing to re-create a shell or
> > package the new crates before continuing on.
> >
> 
> I see, thanks. My preferred workflow is different but I acknowledge that
> use case.
> 
> The link below claims that one can update the vendor directory in a similar
> way by re-running `cargo vendor` after adding a dependency to Cargo.toml,
> but for your use case I agree that it's nicer if `cargo build` can pull to
> the registry automatically.
> https://old.reddit.com/r/rust/comments/uvvmjy/how_to_include_vendored_crates_into_a_project/
> 
> 
> > The registry as setup in the patches is actually mostly correct, minus
> > the multiple versions of a crate, it just needs to be fed the crates.
> >
> > > Getting back to your patch set: would it make sense to emulate this
> > vendor
> > > workflow instead of trying to construct a registry directly? Even
> > assuming
> > > that all details of the registry structure are stable and documented, the
> > > layout of the vendor directory appears much simpler. And IIUC the code
> > for
> > > setting up vendored libraries already exists in cargo-build-system.
> > >
> > > I also need to figure out something with a
> > > > config.toml to see if it's possible to generate one that could be
> > > > included from another one, since you can't add 'local-registry =
> > > > $GUIX_PROFILE/...' in a toml file.
> > > >
> > >
> > > You've probably researched this more than I have, but it seems that this
> > > use case is explicitly unsupported in the TOML language spec:
> > > https://github.com/toml-lang/toml/issues/397
> > >
> > > With that option off the table, I can't think of any elegant solutions.
> > > Maybe a wrapper for the cargo binary that pre-processes cargo.toml and
> > then
> > > calls the real cargo?
> >
> > Thanks, I didn't see that one.  So it looks like environment variables
> > and INCLUDEs are off the table.  An autoconf style macro that take a
> > config.toml.in and spits out a config.toml with the correct directory
> > would work, but from my understanding that's not really in line with any
> > bit of how cargo and the rust ecosystem works.
> >
> > Another option would be to symlink
> > $GUIX_ENVIRONMENT/share/cargo/registry to ./local-registry and then add
> > the line 'local-registry = 'local-registry' in a config.toml.
> >

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: rust-team branch merged

[Jason Conroy]

[-- Attachment #1: Type: text/plain, Size: 1177 bytes --]

On Tue, Mar 12, 2024 at 11:12 AM Efraim Flashner <efraim@flashner.co.il>
wrote:

> On Thu, Mar 07, 2024 at 11:48:53AM -0500, Jason Conroy wrote:
> > On Thu, Mar 7, 2024 at 3:08 AM Efraim Flashner <efraim@flashner.co.il>
> > wrote:
> >
> > > Until then
> > > I've been experimenting by manually listing the other crates I've
> needed
> > > but in theory we could try to make `guix shell --development` pull in
> > > the needed crates.
> > >
> >
> > I was considering another option that falls somewhere in between: since
> I'm
> > already building shells from manifest files, it should suffice to have a
> > Scheme utility function that calculates the transitive dependencies for a
> > given list of library packages. Similar logic seems to exist already in
> > `(guix build-system cargo)` but it's not exposed publicly. As interim
> > solutions go, what do you think about this one versus modifying `guix
> > shell`?
>
> Once you have something like that working it shouldn't be too hard to
> merge that into `guix shell`, assuming we go that route.
>

Sounds good, I'll follow up if my exercise produces anything useful. Thanks
for the input!

[-- Attachment #2: Type: text/html, Size: 1689 bytes --]

Re: rust-team branch merged

[Jason Conroy]

Efraim Flashner <efraim@flashner.co.il> writes:

> Currently if you were to pull in rust-rand-0.8 and rust-rand-0.7 
> then
> you'd have both rand-0.*.crate files in the registry but only 
> one of
> them would be listed in share/cargo/registry/index/ra/nd/rand. I 
> need to
> adjust the generation of that file to combine multiple sources 
> if they
> exist, and sort them (I'm not sure it's necessary, but wouldn't 
> be
> surprised if we hit undefined behaviour if they were listed 
> multiple
> times or out of order).

Hi Efraim,

I'm currently investigating this limitation of your proposed 
patch.

Did you have a strategy in mind for how to fix it? I see that the 
index files are currently generated during a phase of 
cargo-build-system, rather than as a profile hook. So, to build an 
index that properly reflects the contents of a profile, it would 
seem that the two simplest options are: a) keep your existing 
index-generation logic during the build, and merge these 
per-package index files when building the profile; or b) move your 
patch's index-generating code out of the build phase and into a 
profile hook, so that we build each index file in a single pass 
(for all versions of a package) rather than merging the files from 
each package output.

On the surface option (b) seems cleaner, but maybe you had a 
reason for generating the index contents during the build?

Jason

Re: rust-team branch merged

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2485 bytes --]

On Thu, Apr 18, 2024 at 12:54:02PM -0400, Jason Conroy wrote:
> 
> Efraim Flashner <efraim@flashner.co.il> writes:
> 
> > Currently if you were to pull in rust-rand-0.8 and rust-rand-0.7 then
> > you'd have both rand-0.*.crate files in the registry but only one of
> > them would be listed in share/cargo/registry/index/ra/nd/rand. I need to
> > adjust the generation of that file to combine multiple sources if they
> > exist, and sort them (I'm not sure it's necessary, but wouldn't be
> > surprised if we hit undefined behaviour if they were listed multiple
> > times or out of order).
> 
> Hi Efraim,
> 
> I'm currently investigating this limitation of your proposed patch.
> 
> Did you have a strategy in mind for how to fix it? I see that the index
> files are currently generated during a phase of cargo-build-system, rather
> than as a profile hook. So, to build an index that properly reflects the
> contents of a profile, it would seem that the two simplest options are: a)
> keep your existing index-generation logic during the build, and merge these
> per-package index files when building the profile; or b) move your patch's
> index-generating code out of the build phase and into a profile hook, so
> that we build each index file in a single pass (for all versions of a
> package) rather than merging the files from each package output.
> 
> On the surface option (b) seems cleaner, but maybe you had a reason for
> generating the index contents during the build?

I like the idea of moving the code into a profile hook much better than
what I have now.  What's there was more of a proof-of-concept of how it
might work.

On the other hand, by generating it during the build of each package we
make sure to pull in all the crates which exist in the build, so we
could add into a profile/manifest just the crates listed in a Cargo.toml
and then each crate would pull in its own dependencies, and then the
profile hook could combine them all together.  The down side is we'd
have to have the logic to combine the overlapping rand-0.7/rand-0.8
paths both in the build system and in the profile.  Overall it doesn't
seem that bad though, since we could then add packages either as a
regular input or as a cargo-input.

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: rust-team branch merged

[Jason Conroy]

Efraim Flashner <efraim@flashner.co.il> writes:
> On the other hand, by generating it during the build of each 
> package we
> make sure to pull in all the crates which exist in the build, so 
> we
> could add into a profile/manifest just the crates listed in a 
> Cargo.toml
> and then each crate would pull in its own dependencies, and then 
> the
> profile hook could combine them all together.

Thanks. Just to make sure I understand: it sounds like you're 
saying that by creating the JSON index files up front, we'd be 
preserving some knowledge about a package's dependency graph that 
isn't easily recovered later by recursively walking through inputs 
and cargo-inputs for the package specs in a manifest/profile?

Jason

Re: rust-team branch merged

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1375 bytes --]

On Wed, Apr 24, 2024 at 11:58:22AM -0400, Jason Conroy wrote:
> 
> Efraim Flashner <efraim@flashner.co.il> writes:
> > On the other hand, by generating it during the build of each package we
> > make sure to pull in all the crates which exist in the build, so we
> > could add into a profile/manifest just the crates listed in a Cargo.toml
> > and then each crate would pull in its own dependencies, and then the
> > profile hook could combine them all together.
> 
> Thanks. Just to make sure I understand: it sounds like you're saying that by
> creating the JSON index files up front, we'd be preserving some knowledge
> about a package's dependency graph that isn't easily recovered later by
> recursively walking through inputs and cargo-inputs for the package specs in
> a manifest/profile?

If we create it upfront it may be easier to keep track of the
cargo-dependencies, but I suppose it would mostly depend on the
implementation.  I'm not sure that'd actually be the case, so don't
worry about it too much.  Whatever we end up with will be better than
what we have now, and we can always make it better later if it needs to
change.

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: rust-team branch merged

[Jason Conroy]

Efraim Flashner <efraim@flashner.co.il> writes:

> [[PGP Signed Part:Undecided]]
> On Wed, Apr 24, 2024 at 11:58:22AM -0400, Jason Conroy wrote:
>> 
>> Efraim Flashner <efraim@flashner.co.il> writes:
>> > On the other hand, by generating it during the build of each 
>> > package we
>> > make sure to pull in all the crates which exist in the build, 
>> > so we
>> > could add into a profile/manifest just the crates listed in a 
>> > Cargo.toml
>> > and then each crate would pull in its own dependencies, and 
>> > then the
>> > profile hook could combine them all together.
>> 
>> Thanks. Just to make sure I understand: it sounds like you're 
>> saying that by
>> creating the JSON index files up front, we'd be preserving some 
>> knowledge
>> about a package's dependency graph that isn't easily recovered 
>> later by
>> recursively walking through inputs and cargo-inputs for the 
>> package specs in
>> a manifest/profile?
>
> If we create it upfront it may be easier to keep track of the
> cargo-dependencies, but I suppose it would mostly depend on the
> implementation.  I'm not sure that'd actually be the case, so 
> don't
> worry about it too much.  Whatever we end up with will be better 
> than
> what we have now, and we can always make it better later if it 
> needs to
> change.

Hi Efraim, I spent some time on this and have a few thoughts 
below. In short, I made progress but also hit some roadblocks that 
give me second thoughts about the approach we discussed up-thread.

As you suggested, I kept the metadata generation code in 
cargo-build-system and used the cargo-registry profile hook to 
merge all of those package artifacts into index files. I also 
modified the hook to lift the full closure of source dependencies 
into the profile for any rust packages in the manifest, borrowing 
logic from `(guix build-system cargo)`.

However, testing revealed some issues with this approach:

1) Formerly, for any cargo package with `#:skip-build? #t` the 
crate was treated as an opaque file: it was copied to a location 
where cargo could access it, but otherwise unprocessed. But to 
generate the index file we need to unpack the crate and call 
`cargo metadata`, which performs some sanity checks on the package 
structure. The implication is that if #:skip-build? is used for 
reasons other than performance - e.g., for a package that works 
fine as a dependency but somehow fails during a standalone build - 
then we might have trouble including such a package in Cargo's 
local registry. I don't know how big an issue this is in practice, 
but I think I encountered one package that fits this scenario: 
custom_derive-0.1.7. (We can discuss the details in a separate 
thread if you're interested.)

2) When building against a local-registry, it seems like Cargo 
expects to find crates that it wouldn't for a vendor tree. As 
mentioned above, my current patch pulls source crates into the 
profile using the same logic as cargo-build-system currently does: 
for each manifest package and its cargo-development-inputs, 
recursively walk through all of the cargo-inputs. But after 
building a profile this way, I encountered the following:

  $ cargo build
  error: no matching package named `automod` found
  location searched: registry `crates-io`
  required by package `serde_json v1.0.111`
      ... which satisfies dependency `serde_json = "^1.0"` of 
      package `rand_chacha v0.3.1`
          ... which satisfies dependency `rand_chacha = "^0.3.0"` 
          of package `rand v0.8.5`
              ... which satisfies dependency `rand = "^0.8.5"` of 
              package `mytest v0.1.0 (/home/guix/mytest)`

Note that serde_json is actually a dev-dependency of rand_chacha, 
and automod is a dev-dependency of serde_json, but our algorithm 
above assumes that dev-dependencies of dependencies can be 
omitted. So in practice, it seems like building a local-registry 
that cargo will accept is even more expensive than building a 
vendor tree.

3) The build error above is interesting for another reason: cargo 
simply gives up rather than fetching the missing dependencies. In 
our earlier discussion, the motivation for using a local-registry 
over a vendor tree in a profile was to support a sort of hybrid 
workflow: the profile seeds the registry with some packages, but 
cargo can still download others on its own during `cargo 
build`. But in this test and several others, I haven't observed a 
case where cargo falls back to crates.io for missing 
packages. Efraim, could you share details on how you accomplished 
this in the past? I wonder if our discussion might have conflated 
cargo's registry cache ($CARGO_HOME/registry) with a true 
local-registry; I haven't found any good documentation on the 
former, but it has a different directory layout and index file 
format from the local-registry's, so it wouldn't surprise me if it 
had different semantics as well.

Cheers,

Jason

Re: rust-team branch merged

[Csepp]

[-- Attachment #1: Type: text/plain, Size: 2150 bytes --]

Efraim Flashner <efraim@flashner.co.il> writes:

> The rust team is pleased to announce that the rust-team branch has been
> merged back into master.  There are 570 commits across the branch.
> Cross-compiling support for the cargo-build-system was added, including
> for librsvg.  Cross-compiling was tested for (nearly) all architectures
> supported in `guix build –list-targets`.  Upstream rust added
> ’i686-unknown-hurd-gnu’ as a target in rust-1.74, so that will need to
> wait until another time.  In the meantime, x86_64-w64-mingw32 at least
> compiles correctly and tests well under wine.
>
> Notable notes:
>
> * New rust version is 1.73.
> * 4001 packages use the cargo-build-system
> * Rust crates produced in the ’package’ phase of the cargo-build-system
> should now be reproducible (for real this time)
> * Packages added: eza, kibi, libgit2-1.6, libgit2-1.7, spotifyd, stgit-2
> * Packages updated: alfis, rust-analyzer, rust-cargo-c
> * Rust-analyzer is built from the rust sources now, updating it to 1.73.
> It can be installed as ’rust-analyzer’ or as ’rust:tools’
> * Cross-compiling support for the cargo-build-system was added
>
> Additional notes:
>
> * crates.io no longer accepts ’-’ instead of ’_’ in crate lookups, so
> there are a (probably large) number of packages which need to have their
> source adjusted.
>
> * Compiled rust packages currently have a ’package’ phase, which runs
> the command used to crate a ’crate tarball’, and is installed in
> %output/share/cargo/registry, with unpacked sources in
> %output/share/cargo/src.  In theory it should be possible to use these
> for local rust development.  The benefits include everything that comes
> with being a guix package, including pre-patched shebangs.  Currently no
> index file is created in $GUIX_ENVIRONMENT/share/cargo/registry/index,
> which is likely necessary to actually make use of this.  Additionally, I
> am unsure how to use ’$GUIX_ENVIRONMENT’ in ~/.cargo/config so that it
> is expanded and not taken as a literal string.

Congrats!  That is some excellent news.

rust-team branch merged

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 493 bytes --]

I've merged the rust-team branch into master. I've dropped rust for
aarch64-linux again for now until I can get it built on cuirass so it's
not holding up the branch any longer.

Unfortunately I forgot about the mesa update earlier so there's more
rebuilds than I expected.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: rust-team branch merged

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 613 bytes --]


Efraim Flashner <efraim@flashner.co.il> writes:

> I've merged the rust-team branch into master. I've dropped rust for
> aarch64-linux again for now until I can get it built on cuirass so it's
> not holding up the branch any longer.

Do you know if it actually doesn't build, or whether it's just a
berlin/Cuirass issue?

While I appreciate the waiting for the build farm to catch up, I think
that in the case of aarch64-linux substitutes from ci.guix.gnu.org,
whatever problems are causing the substitute availability to be so low
aren't really going to be helped by waiting, something more is probably
needed.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: rust-team branch merged

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1305 bytes --]

On Tue, May 09, 2023 at 09:33:46AM +0100, Christopher Baines wrote:
> 
> Efraim Flashner <efraim@flashner.co.il> writes:
> 
> > I've merged the rust-team branch into master. I've dropped rust for
> > aarch64-linux again for now until I can get it built on cuirass so it's
> > not holding up the branch any longer.
> 
> Do you know if it actually doesn't build, or whether it's just a
> berlin/Cuirass issue?

I've managed to build it myself on my own pinebook pro and used that to
build librsvg and python-cryptography. riscv64-linux actually needs more
work to fix so it's building again.

> While I appreciate the waiting for the build farm to catch up, I think
> that in the case of aarch64-linux substitutes from ci.guix.gnu.org,
> whatever problems are causing the substitute availability to be so low
> aren't really going to be helped by waiting, something more is probably
> needed.

The way its currently setup all we need to do is re-add aarch64-linux to
the supported-systems of rust-bootstrap and it'll be enabled again, and
build successfully eventually.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: rust-team branch merged

[Andreas Enge]

Hello,

Am Tue, May 09, 2023 at 11:54:00AM +0300 schrieb Efraim Flashner:
> The way its currently setup all we need to do is re-add aarch64-linux to
> the supported-systems of rust-bootstrap and it'll be enabled again, and
> build successfully eventually.

I am confused by what happened; did you disable rust on aarch64 for everyone,
or just building on the farm? If the first, that sounds like a pity, since
at worst people can still compile packages by themselves.

Andreas

Re: rust-team branch merged

[Andreas Enge]

PS: Congratulations for getting the first team branch through!
    And thanks for waiting until the core-updates merge :-)

Re: rust-team branch merged

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 714 bytes --]


Andreas Enge <andreas@enge.fr> writes:

> Hello,
>
> Am Tue, May 09, 2023 at 11:54:00AM +0300 schrieb Efraim Flashner:
>> The way its currently setup all we need to do is re-add aarch64-linux to
>> the supported-systems of rust-bootstrap and it'll be enabled again, and
>> build successfully eventually.
>
> I am confused by what happened; did you disable rust on aarch64 for everyone,
> or just building on the farm? If the first, that sounds like a pity, since
> at worst people can still compile packages by themselves.

Yeah, while I appreciate paying attention to what substitutes are
available, I think we should add aarch64-linux back to the supported
systems, regardless of what ci.guix.gnu.org is doing.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: rust-team branch merged

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1135 bytes --]

On Tue, May 09, 2023 at 02:22:36PM +0100, Christopher Baines wrote:
> 
> Andreas Enge <andreas@enge.fr> writes:
> 
> > Hello,
> >
> > Am Tue, May 09, 2023 at 11:54:00AM +0300 schrieb Efraim Flashner:
> >> The way its currently setup all we need to do is re-add aarch64-linux to
> >> the supported-systems of rust-bootstrap and it'll be enabled again, and
> >> build successfully eventually.
> >
> > I am confused by what happened; did you disable rust on aarch64 for everyone,
> > or just building on the farm? If the first, that sounds like a pity, since
> > at worst people can still compile packages by themselves.
> 
> Yeah, while I appreciate paying attention to what substitutes are
> available, I think we should add aarch64-linux back to the supported
> systems, regardless of what ci.guix.gnu.org is doing.

Thank you all for the encouragement and feedback. I've re-enabled
aarch64 for rust.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]