unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / Atom feed
* Some more rust/cargo insights
@ 2021-06-06 14:15 Hartmut Goebel
       [not found] ` <20210606183857.gthvipntymstivh4@thebird.nl>
  2021-06-14  5:22 ` Maxim Cournoyer
  0 siblings, 2 replies; 13+ messages in thread
From: Hartmut Goebel @ 2021-06-06 14:15 UTC (permalink / raw)
  To: guix-devel

[-- Attachment #1: Type: text/plain, Size: 960 bytes --]

Hi.,

these day I had spent some more hours struggling with rust ans cargo, 
trying to get "pre-built" crates.

Summery; Cargo is cruft, no solution found yet.

I tried reusing a crate from the very same place it was built (see 
enclosed script). Anyhow, this does not work since cargo uses a 
different "metadata" value, even if noting changed. Based in the verbose 
output (cargo build -v …) I assume that some parameters of the 
"destination" build get included into this value.

This meets another observation; when building the sequoia suite, several 
crates are build several times - even if all builds are performed in the 
same environment.

Rust's build system is such a cruft - I really would like to throw it 
where it belongs: into the trash.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Attachment #2: rlib-test.scm --]
[-- Type: text/x-scheme, Size: 4300 bytes --]

(use-modules
 (guix download)
 (guix packages)
 (guix build-system cargo)
 (gnu packages crates-io)
 (srfi srfi-1)
 (srfi srfi-26)
 (gnu packages crates-graphics)
 (gnu packages rust-apps))

(define-public rust-pretty-assertions-0.5
  (package
    (inherit rust-pretty-assertions-0.6)
    (name "rust-pretty-assertions")
    (version "0.5.1")
    (source
     (origin
       (method url-fetch)
       (uri (crate-uri "pretty_assertions" version))
       (file-name
        (string-append name "-" version ".tar.gz"))
       (sha256
          (base32 "1ins6swkpxmrh8q5h96h8nv0497d3dclsiyx2lyvqi6py0q980is"))))
    (build-system cargo-build-system)
    (arguments
     `(#:tests? #f
       #:cargo-inputs
       (("rust-ansi-term" ,rust-ansi-term-0.11)
        ("rust-difference" ,rust-difference-2))))))

;; /gnu/store/wknzymkfbfjbxwfd3djrn4hk9zdfgs56-rust-xmlparser-0.13.3 -- original
;; 
;; libxmlparser-f82b201ea4144ed3.rlib

(define-public myrust-xmlparser
  (package
    (inherit rust-xmlparser-0.13)
    (outputs '("out" "rlib"))
    (arguments
     `(#:skip-build? #f
       #:tests? #f
       #:cargo-build-flags (list "--release" "-vv")
       #:phases
       (modify-phases %standard-phases
         (add-after 'install 'install-rlib
           (lambda* (#:key outputs #:allow-other-keys)
             (let* ((rout (assoc-ref outputs "rlib"))
                    (dest (string-append rout "/rlib")))
               ;;(mkdir dest)
               ;;(for-each (cut install-file <> (string-append rout "/rlib"))
               (for-each (lambda (fn)
                           (install-file fn (string-append rout "/rlib")))
                         (find-files "target/release/deps" "\\.(rlib|rmeta)$"))
               )))
          ;; (add-after 'install 'fail
          ;;   (lambda _ #f))
    )))))

(define-public myrust-roxmltree
  (package
    (inherit rust-roxmltree-0.14)
    ;;(outputs '("out" "crate"))
    (inputs
     `(("rust-xmlparser" ,myrust-xmlparser "rlib")))
    (arguments
     `(#:skip-build? #f
       #:tests? #f
       ;;#:vendor-dir "/tmp/src"
       #:cargo-build-flags (list "--release" "-vv")
       #:cargo-inputs
       (("rust-xmlparser:src" ,rust-xmlparser-0.13)
        ("rust-pretty-assertions" ,rust-pretty-assertions-0.5))
       #:phases
       (modify-phases %standard-phases
         (add-after 'patch-cargo-checksums 'bring-in-rlib
           (lambda* (#:key inputs #:allow-other-keys)
             (let* ((rin (assoc-ref inputs "rust-xmlparser"))
                    (src (assoc-ref inputs "rust-xmlparser"))
                    (rlib (string-append rin "/rlib")))
               (mkdir "/tmp/guix-build-rust-xmlparser-0.13.3.drv-0/")
               (copy-recursively
               "guix-vendor/rust-xmlparser-0.13.3.tar.gz"
               "/tmp/guix-build-rust-xmlparser-0.13.3.drv-0/xmlparser-0.13.3")
               (rename-file
                "guix-vendor/rust-xmlparser-0.13.3.tar.gz"
                "../rust-xmlparser-0.13.3.tar.gz")
               (symlink
                "/tmp/guix-build-rust-xmlparser-0.13.3.drv-0/xmlparser-0.13.3"
                "guix-vendor/rust-xmlparser-0.13.3")
;;                (let ((port (open-file ".cargo/config" "w" #:encoding "utf-8")))
;;                  (display "
;; #paths = [\"/tmp/guix-build-rust-xmlparser-0.13.3.drv-0/xmlparser-0.13.3\"]

;; [source.crates-io]
;; replace-with = 'vendored-sources'

;; #[patch.crates-io]
;; #xmlparser = { path = '/tmp/guix-build-rust-xmlparser-0.13.3.drv-0/xmlparser-0.13.3' }

;; [source.vendored-sources]
;; directory = '" port)
;;                  (display (string-append (getcwd) "/guix-vendor") port)
;;                  (display "'
;; " port)
;;     (close-port port))
               ;; (substitute* ".cargo/config"
               ;;   (("\\.vendored-sources\\]" _)
               ;;    (string-append _ "\n[paths]"))
               (for-each (lambda (fn)
                           (install-file fn "target/release/deps"))
                         (pk (find-files rlib "\\.(rlib|rmeta)$")))
               #t
               )))
         (add-before 'install 'fail
           (lambda _ #f)))
    ))))


;;myrust-xmlparser
myrust-roxmltree

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Some more rust/cargo insights
       [not found] ` <20210606183857.gthvipntymstivh4@thebird.nl>
@ 2021-06-07  7:10   ` Hartmut Goebel
  2021-06-07  8:28     ` Pjotr Prins
  0 siblings, 1 reply; 13+ messages in thread
From: Hartmut Goebel @ 2021-06-07  7:10 UTC (permalink / raw)
  To: Pjotr Prins; +Cc: guix-devel

Am 06.06.21 um 20:38 schrieb Pjotr Prins:
> Since that community is about not invented here - maybe we can incense
> someone to pick it up. Needs a mature programmer though.

One solution that came to my mind is to not use Cargo, but instead parse 
Cargo.toml and issue the appropriate "rustc" commands ourself.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |



^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Some more rust/cargo insights
  2021-06-07  7:10   ` Hartmut Goebel
@ 2021-06-07  8:28     ` Pjotr Prins
  2021-06-07 12:04       ` Hartmut Goebel
  0 siblings, 1 reply; 13+ messages in thread
From: Pjotr Prins @ 2021-06-07  8:28 UTC (permalink / raw)
  To: Hartmut Goebel; +Cc: guix-devel

On Mon, Jun 07, 2021 at 09:10:48AM +0200, Hartmut Goebel wrote:
> Am 06.06.21 um 20:38 schrieb Pjotr Prins:
> > Since that community is about not invented here - maybe we can incense
> > someone to pick it up. Needs a mature programmer though.
> 
> One solution that came to my mind is to not use Cargo, but instead parse
> Cargo.toml and issue the appropriate "rustc" commands ourself.

Exactly my idea. One challenge will be that the source of dependencies
need to be available - think of it as include files. One thing we
could do as ship them as part of the Guix package. Or have a separate
one for sources. We do that for include files already.

Pj.


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Some more rust/cargo insights
  2021-06-07  8:28     ` Pjotr Prins
@ 2021-06-07 12:04       ` Hartmut Goebel
  2021-06-07 15:13         ` John Soo
  0 siblings, 1 reply; 13+ messages in thread
From: Hartmut Goebel @ 2021-06-07 12:04 UTC (permalink / raw)
  To: Pjotr Prins; +Cc: guix-devel

Am 07.06.21 um 10:28 schrieb Pjotr Prins:
> Exactly my idea. One challenge will be that the source of dependencies
> need to be available - think of it as include files. One thing we
> could do as ship them as part of the Guix package. Or have a separate
> one for sources. We do that for include files already.

Well, the current cargo-build-system already handles the source 
dependencies.

We need to aim towards pre-built libraries (rlib, much like .a files in 
C, I assume)

When cargo calls rustc, the command looks like:

LD_LIBRARY_PATH='$PWD/target/release/deps:/gnu/store/…-rust-1.45.2/lib' \
rustc … src/lib.rs --crate-type lib \
-L dependency=$PWD/target/release/deps \
--extern 
xmlparser=$PWD/target/release/deps/libxmlparser-53596ac1828b1c97.rmeta

Thus I assume one could pass the rlib's of all dependencies in -L and 
the respective mata-data in --extern

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |



^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Some more rust/cargo insights
  2021-06-07 12:04       ` Hartmut Goebel
@ 2021-06-07 15:13         ` John Soo
  2021-06-07 15:15           ` John Soo
  2021-06-07 16:26           ` Hartmut Goebel
  0 siblings, 2 replies; 13+ messages in thread
From: John Soo @ 2021-06-07 15:13 UTC (permalink / raw)
  To: Pjotr Prins, Hartmut Goebel; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 2283 bytes --]

Hi Hartmut and Pjotr,

My feeling on this is that we should partner with the Rust community to make shared library support from cargo a priority. Specifying an output directory is currently a nightly feature, that could be helpful.

In general Rust tooling does not compose with existing tools. I believe they will be amenable to the thought that it should. If Rust wants to be used in the linux kernel, for instance, it should be easy to use with Make.

Rust has a very well documented rfc process and we can at least bring it up that way. I brought up the possibility of collaboration between rust and functional package managers on the rust Zulip, even. They seemed to like the idea.

Another path we should checkout is to see what Debian does. My understanding is that they figured something out. Worth a shot, but I’d rather the problem be fixed upstream. It will just take collaboration.

Kindly,

John

> On Monday, Jun 07, 2021 at 5:04 AM, Hartmut Goebel <h.goebel@crazy-compilers.com (mailto:h.goebel@crazy-compilers.com)> wrote:
> Am 07.06.21 um 10:28 schrieb Pjotr Prins:
> > Exactly my idea. One challenge will be that the source of dependencies
> > need to be available - think of it as include files. One thing we
> > could do as ship them as part of the Guix package. Or have a separate
> > one for sources. We do that for include files already.
>
> Well, the current cargo-build-system already handles the source
> dependencies.
>
> We need to aim towards pre-built libraries (rlib, much like .a files in
> C, I assume)
>
> When cargo calls rustc, the command looks like:
>
> LD_LIBRARY_PATH='$PWD/target/release/deps:/gnu/store/…-rust-1.45.2/lib' \
> rustc … src/lib.rs --crate-type lib \
> -L dependency=$PWD/target/release/deps \
> --extern
> xmlparser=$PWD/target/release/deps/libxmlparser-53596ac1828b1c97.rmeta
>
> Thus I assume one could pass the rlib's of all dependencies in -L and
> the respective mata-data in --extern
>
> --
> Regards
> Hartmut Goebel
>
> | Hartmut Goebel | h.goebel@crazy-compilers.com |
> | https://urldefense.com/v3/__http://www.crazy-compilers.com__;!!IKRxdwAv5BmarQ!KvLeCSlBvF9faFS6qJxYA0Rl0AuuhxYx7oxMtZXdbILpmv_Rz4n8swX7_p74sQ$ | compilers which you thought are impossible |
>
>

[-- Attachment #2: Type: text/html, Size: 3029 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Some more rust/cargo insights
  2021-06-07 15:13         ` John Soo
@ 2021-06-07 15:15           ` John Soo
  2021-06-07 16:26           ` Hartmut Goebel
  1 sibling, 0 replies; 13+ messages in thread
From: John Soo @ 2021-06-07 15:15 UTC (permalink / raw)
  To: Pjotr Prins, Hartmut Goebel; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 49 bytes --]

Oh my goodness, I’m so sorry for the top quote.

[-- Attachment #2: Type: text/html, Size: 234 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Some more rust/cargo insights
  2021-06-07 15:13         ` John Soo
  2021-06-07 15:15           ` John Soo
@ 2021-06-07 16:26           ` Hartmut Goebel
  2021-06-07 16:41             ` Hartmut Goebel
  1 sibling, 1 reply; 13+ messages in thread
From: Hartmut Goebel @ 2021-06-07 16:26 UTC (permalink / raw)
  To: John Soo; +Cc: guix-devel

Hi John.

Am 07.06.21 um 17:13 schrieb John Soo:
> Rust has a very well documented rfc process and we can at least bring 
> it up that way.  I brought up the possibility of collaboration between 
> rust and functional package managers on the rust Zulip, even.  They 
> seemed to like the idea.
>
I'd be more than happy if you could start a RFC process then. This issue 
bugs us and other distros since many months. (I have not contact to the 
rust community and will not sign up at another proprietary communication 
platform (Zulip)).

> My feeling on this is that we should partner with the Rust community 
> to make shared library support from cargo a priority.

Our issue is a different one: Its about being able to reuse already 
compiled binaries - keeping current behavior of rust binaries being 
statically linked.

While this looks like being the same as dynamic library support, it is 
not: While for dynamic libraries you meet to ensure the very correct 
version of a "crate" is loaded, for static linking with pre-build 
binaries you only need to ensure this at build-time. (For guix, of 
course, both would not be a problem, but I doubt we can make rust people 
understand this. And other distros will still have the problem.)

rustc already has a notion for "static libraries", just cargo fu** it 
up. (Sorry for hard wording, but I'm actually quite angry about cargos' 
narrow-minded behavior).

So our task is much, much easier and doesn't require changed to rustc, 
only to cargo.


>  Specifying an output directory is currently a nightly feature, that 
> could be helpful.

Not exactly sure what you mean. But what breaks build with cargo are the 
*input* directories - and other magic which gets included into the 
"meta-data" for no reason.


> In general Rust tooling does not compose with existing tools.  I 
> believe they will be amenable to the thought that it should. If Rust 
> wants to be used in the linux kernel, for instance, it should be easy 
> to use with Make.

 From your lips to God's ears. From what I've seen an read, I'm not 
convident they will change anything. I'd like to be proofed wrong, 
though :-)


> Another path we should checkout is to see what Debian does. My 
> understanding is that they figured something out.  Worth a shot, but 
> I’d rather the problem be fixed upstream. It will just take collaboration.

I did not check their tollchain lately, but package-earch still does not 
show many packages 
<https://packages.debian.org/search?suite=experimental&searchon=names&keywords=rust>. 
Last time I check, they basically do what guix does: compile everything 
from source - again and again.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |




^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Some more rust/cargo insights
  2021-06-07 16:26           ` Hartmut Goebel
@ 2021-06-07 16:41             ` Hartmut Goebel
  2021-06-08  9:15               ` Efraim Flashner
  0 siblings, 1 reply; 13+ messages in thread
From: Hartmut Goebel @ 2021-06-07 16:41 UTC (permalink / raw)
  To: John Soo; +Cc: guix-devel

Am 07.06.21 um 18:26 schrieb Hartmut Goebel:
>> Another path we should checkout is to see what Debian does. My 
>> understanding is that they figured something out.  Worth a shot, but 
>> I’d rather the problem be fixed upstream. It will just take 
>> collaboration.
>
> I did not check their tollchain lately, but package-earch still does 
> not show many packages 
> <https://packages.debian.org/search?suite=experimental&searchon=names&keywords=rust>. 
> Last time I check, they basically do what guix does: compile 
> everything from source - again and again.
>
Just checked: Debian has quite some source packages [1], which even list 
"binary" packages (which are development packages). Anyhow it seems as 
if these are just the source again [2]

[1] 
<https://packages.debian.org/search?keywords=rust&searchon=sourcenames&suite=stable&section=all>
[2] 
<https://packages.debian.org/buster/amd64/librust-build-const-dev/filelist>

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |



^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Some more rust/cargo insights
  2021-06-07 16:41             ` Hartmut Goebel
@ 2021-06-08  9:15               ` Efraim Flashner
  2021-06-08 15:38                 ` Hartmut Goebel
  0 siblings, 1 reply; 13+ messages in thread
From: Efraim Flashner @ 2021-06-08  9:15 UTC (permalink / raw)
  To: Hartmut Goebel; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 1394 bytes --]

On Mon, Jun 07, 2021 at 06:41:41PM +0200, Hartmut Goebel wrote:
> Am 07.06.21 um 18:26 schrieb Hartmut Goebel:
> > > Another path we should checkout is to see what Debian does. My
> > > understanding is that they figured something out.  Worth a shot, but
> > > I’d rather the problem be fixed upstream. It will just take
> > > collaboration.
> > 
> > I did not check their tollchain lately, but package-earch still does not
> > show many packages <https://packages.debian.org/search?suite=experimental&searchon=names&keywords=rust>.
> > Last time I check, they basically do what guix does: compile everything
> > from source - again and again.
> > 
> Just checked: Debian has quite some source packages [1], which even list
> "binary" packages (which are development packages). Anyhow it seems as if
> these are just the source again [2]
> 
> [1] <https://packages.debian.org/search?keywords=rust&searchon=sourcenames&suite=stable&section=all>
> [2]
> <https://packages.debian.org/buster/amd64/librust-build-const-dev/filelist>
> 

My understanding is that they pool together all the sources that they
have and then build all the rust packages in one go.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Some more rust/cargo insights
  2021-06-08  9:15               ` Efraim Flashner
@ 2021-06-08 15:38                 ` Hartmut Goebel
  0 siblings, 0 replies; 13+ messages in thread
From: Hartmut Goebel @ 2021-06-08 15:38 UTC (permalink / raw)
  To: John Soo, guix-devel, Efraim Flashner

Am 08.06.21 um 11:15 schrieb Efraim Flashner (referring to Debian)
> My understanding is that they pool together all the sources that they
> have and then build all the rust packages in one go.

Maybe, I have no clue. Anyhow given my experience with sequoia, I doubt 
this will reduce build-times:

When building sequoia, several crates are build up to 4 times - even 
sequoia-openpgp itself.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |



^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Some more rust/cargo insights
  2021-06-06 14:15 Some more rust/cargo insights Hartmut Goebel
       [not found] ` <20210606183857.gthvipntymstivh4@thebird.nl>
@ 2021-06-14  5:22 ` Maxim Cournoyer
  1 sibling, 0 replies; 13+ messages in thread
From: Maxim Cournoyer @ 2021-06-14  5:22 UTC (permalink / raw)
  To: Hartmut Goebel; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 1433 bytes --]

Hi Hartmut,

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> Hi.,
>
> these day I had spent some more hours struggling with rust ans cargo,
> trying to get "pre-built" crates.
>
> Summery; Cargo is cruft, no solution found yet.
>
> I tried reusing a crate from the very same place it was built (see
> enclosed script). Anyhow, this does not work since cargo uses a 
> different "metadata" value, even if noting changed. Based in the
> verbose output (cargo build -v …) I assume that some parameters of the 
> "destination" build get included into this value.
>
> This meets another observation; when building the sequoia suite,
> several crates are build several times - even if all builds are
> performed in the same environment.
>
> Rust's build system is such a cruft - I really would like to throw it
> where it belongs: into the trash.

I've also tried to naively copy prebuilt rlibs at their expected cached
location to see if rust would reuse them, but they didn't seem to; see
the patch below if you are curious.

There was a 'build-plan' feature in cargo that made it output Makefiles
but it's not been under active development and will probably be
removed. [0] There's an issue demanding support in Cargo to allow
reusing pre-built libraries (last comment is mine) [1].

[0]  https://github.com/rust-lang/cargo/issues/7614
[1]  https://github.com/rust-lang/cargo/issues/1139


[-- Attachment #2: 0001-tentatively-reuse-rlib-for-cargo-build-system.patch --]
[-- Type: text/x-patch, Size: 7213 bytes --]

From 0807a912db3faaa1ac2a652c7570ecf63ebed8a5 Mon Sep 17 00:00:00 2001
From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
Date: Thu, 3 Jun 2021 12:58:15 -0400
Subject: [PATCH] tentatively reuse rlib for cargo-build-system

---
 guix/build-system/cargo.scm       |  3 +-
 guix/build/cargo-build-system.scm | 78 ++++++++++++++++++++++++++-----
 2 files changed, 68 insertions(+), 13 deletions(-)

diff --git a/guix/build-system/cargo.scm b/guix/build-system/cargo.scm
index e53d2a7523..9ef9f6b149 100644
--- a/guix/build-system/cargo.scm
+++ b/guix/build-system/cargo.scm
@@ -271,7 +271,8 @@ any dependent crates. This can be a benefits:
          (build-inputs `(("cargo" ,rust "cargo")
                          ("rustc" ,rust)
                          ,@(expand-crate-sources cargo-inputs cargo-development-inputs)
-                         ,@native-inputs))
+                         ,@native-inputs
+                        ,@(if target '() inputs)))
          (outputs outputs)
          (build cargo-build)
          (arguments (strip-keyword-arguments private-keywords arguments)))))
diff --git a/guix/build/cargo-build-system.scm b/guix/build/cargo-build-system.scm
index 0a95672b00..e68f20e463 100644
--- a/guix/build/cargo-build-system.scm
+++ b/guix/build/cargo-build-system.scm
@@ -5,6 +5,7 @@
 ;;; Copyright © 2019, 2020, 2021 Efraim Flashner <efraim@flashner.co.il>
 ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net>
 ;;; Copyright © 2020 Marius Bakke <marius@gnu.org>
+;;; Copyright © 2021 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -24,7 +25,7 @@
 (define-module (guix build cargo-build-system)
   #:use-module ((guix build gnu-build-system) #:prefix gnu:)
   #:use-module (guix build json)
-  #:use-module (guix build utils)
+  #:use-module ((guix build utils) #:hide (delete))
   #:use-module (guix build cargo-utils)
   #:use-module (ice-9 popen)
   #:use-module (ice-9 rdelim)
@@ -34,7 +35,10 @@
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-26)
   #:export (%standard-phases
-            cargo-build))
+            cargo-build
+
+            rust-version
+            rust-library-prefix))
 
 ;; Commentary:
 ;;
@@ -42,6 +46,25 @@
 ;;
 ;; Code:
 
+(define (rust-version rust)
+  "Return the version triplet (major.minor.patch) as a string, given RUST, a
+store file name."
+  (let* ((version     (last (string-split rust #\-)))
+         (components  (string-split version #\.))
+         (major+minor+patch (take components 3)))
+    (string-join major+minor+patch ".")))
+
+(define (rust-library-prefix/relative inputs)
+  "Return the relative versioned Rust library prefix where Rust libraries are
+to be installed."
+  (string-append "lib/rust/" (rust-version (assoc-ref inputs "rustc"))))
+
+(define (rust-library-prefix inputs outputs)
+  "Return the absolute versioned Rust library prefix where Rust libraries are
+to be installed."
+  (string-append (assoc-ref outputs "out") "/"
+                 (rust-library-prefix/relative inputs)))
+
 (define (manifest-targets)
   "Extract all targets from the Cargo.toml manifest"
   (let* ((port (open-input-pipe "cargo read-manifest"))
@@ -73,6 +96,16 @@ Cargo.toml file present at its root."
                                             " | cut -d/ -f2"
                                             " | grep -q '^Cargo.toml$'")))))
 
+(define (rlib? file)
+  "Check if FILE has the .rlib extension."
+  (string-suffix? ".rlib" file))
+
+(define (inputs->directories inputs)
+  "Extract the directory part from INPUTS."
+  (match inputs
+    (((names . directories) ...)
+     directories)))
+
 (define* (unpack-rust-crates #:key inputs vendor-dir #:allow-other-keys)
   (define (inputs->rust-inputs inputs)
     "Filter using the label part from INPUTS."
@@ -80,11 +113,6 @@ Cargo.toml file present at its root."
               (match input
                 ((name . _) (rust-package? name))))
             inputs))
-  (define (inputs->directories inputs)
-    "Extract the directory part from INPUTS."
-    (match inputs
-      (((names . directories) ...)
-       directories)))
 
   (let ((rust-inputs (inputs->directories (inputs->rust-inputs inputs))))
     (unless (null? rust-inputs)
@@ -185,6 +213,22 @@ directory = '" port)
   (generate-all-checksums vendor-dir)
   #t)
 
+(define* (populate-cargo-cache #:key inputs outputs #:allow-other-keys)
+  "Populate the 'target/release' directory with any pre-built Rust libraries,
+to avoid rebuilding them from sources when possible."
+  (let* ((rust-lib-prefix (rust-library-prefix/relative inputs))
+         (input-dirs (inputs->directories inputs))
+         (rust-lib-dirs (filter (lambda (f)
+                                  (file-exists? (string-append
+                                                 f "/" rust-lib-prefix)))
+                                input-dirs))
+         (rlibs (delete-duplicates (append-map (cut find-files <> "\\.rlib$")
+                                               rust-lib-dirs))))
+    (pk 'rust-lib-dirs rust-lib-dirs)
+    (pk 'rlibs rlibs)
+    (for-each (cut install-file <> "target/release") rlibs)
+    (invoke "find" "target")))
+
 (define* (build #:key
                 skip-build?
                 (features '())
@@ -228,7 +272,9 @@ directory = '" port)
   "Install a given Cargo package."
   (let* ((out      (assoc-ref outputs "out"))
          (registry (string-append out "/share/cargo/registry"))
-         (sources  (string-append out "/share/cargo/src")))
+         (sources  (string-append out "/share/cargo/src"))
+         (libdir   (rust-library-prefix inputs outputs))
+         (release-dir "target/release"))
     (mkdir-p out)
 
     ;; Make cargo reuse all the artifacts we just built instead
@@ -237,10 +283,17 @@ directory = '" port)
 
     ;; Only install crates which include binary targets,
     ;; otherwise cargo will raise an error.
-    (or skip-build?
-        (not (has-executable-target?))
-        (invoke "cargo" "install" "--no-track" "--path" "." "--root" out
-                "--features" (string-join features)))
+    (unless skip-build?
+      ;; Install binaries.
+      (when (has-executable-target?)
+        (apply invoke "cargo" "install" "--no-track" "--path" "." "--root" out
+               (if (not (null? features))
+                   (list "--features" (string-join features))
+                   '())))
+      ;; Install static libraries.
+      (for-each (lambda (file)
+                  (install-file (string-append release-dir "/" file) libdir))
+                (scandir release-dir (cut string-suffix? ".rlib" <>))))
 
     (when install-source?
       ;; Install crate tarballs and unpacked sources for later use.
@@ -260,6 +313,7 @@ directory = '" port)
   (modify-phases gnu:%standard-phases
     (delete 'bootstrap)
     (replace 'configure configure)
+    (add-before 'build 'populate-cargo-cache populate-cargo-cache)
     (replace 'build build)
     (replace 'check check)
     (replace 'install install)
-- 
2.31.1


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Some more rust/cargo insights
  2021-06-07 18:48 Leo Prikler
@ 2021-06-08 17:01 ` Leo Famulari
  0 siblings, 0 replies; 13+ messages in thread
From: Leo Famulari @ 2021-06-08 17:01 UTC (permalink / raw)
  To: Leo Prikler; +Cc: guix-devel

On Mon, Jun 07, 2021 at 08:48:11PM +0200, Leo Prikler wrote:
> (Note, though, that a Go update would need to go through staging – not
> an ideal situation either.)

I have "cheated" regarding this in the past. Although changing the Go
package should qualify for staging, it's so cheap to build Go packages
that I have sometimes decided to "just do it" on the master branch.


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: Some more rust/cargo insights
@ 2021-06-07 18:48 Leo Prikler
  2021-06-08 17:01 ` Leo Famulari
  0 siblings, 1 reply; 13+ messages in thread
From: Leo Prikler @ 2021-06-07 18:48 UTC (permalink / raw)
  To: h.goebel; +Cc: guix-devel

> Our issue is a different one: Its about being able to reuse already
> compiled binaries - keeping current behavior of rust binaries being
> statically linked.
> 
> While this looks like being the same as dynamic library support, it
> is not: While for dynamic libraries you meet to ensure the very
> correct version of a "crate" is loaded, for static linking with pre-
> build binaries you only need to ensure this at build-time. (For guix,
> of course, both would not be a problem, but I doubt we can make rust
> people understand this. And other distros will still have the
> problem.)
That plus static libraries can't be grafted, which can be a problem
when your programming language despite being touted as secure still
allows large arrays of buggy behaviour.  So while "statically linked,
but reusable and I swear, it really is reproducible" might work for the
time being, it is not a long-term solution.

One might be tempted to compare this to Go, which we also can't
dynamically link, but the important difference here is that Rust is
already being pulled in by core-updatesy stuff like GTK, so a bug in
certain Rust software would affect *a lot* of Guix users whereas Go is
mostly an ecosystem onto itself.  (Note, though, that a Go update would
need to go through staging – not an ideal situation either.)

Regards,
Leo



^ permalink raw reply	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2021-06-14  5:22 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-06 14:15 Some more rust/cargo insights Hartmut Goebel
     [not found] ` <20210606183857.gthvipntymstivh4@thebird.nl>
2021-06-07  7:10   ` Hartmut Goebel
2021-06-07  8:28     ` Pjotr Prins
2021-06-07 12:04       ` Hartmut Goebel
2021-06-07 15:13         ` John Soo
2021-06-07 15:15           ` John Soo
2021-06-07 16:26           ` Hartmut Goebel
2021-06-07 16:41             ` Hartmut Goebel
2021-06-08  9:15               ` Efraim Flashner
2021-06-08 15:38                 ` Hartmut Goebel
2021-06-14  5:22 ` Maxim Cournoyer
2021-06-07 18:48 Leo Prikler
2021-06-08 17:01 ` Leo Famulari

unofficial mirror of guix-devel@gnu.org 

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://yhetil.org/guix-devel/0 guix-devel/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 guix-devel guix-devel/ https://yhetil.org/guix-devel \
		guix-devel@gnu.org
	public-inbox-index guix-devel

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.yhetil.org/yhetil.gnu.guix.devel
	nntp://news.gmane.io/gmane.comp.gnu.guix.devel


AGPL code for this site: git clone http://ou63pmih66umazou.onion/public-inbox.git