unofficial mirror of guix-patches@gnu.org 
 help / color / mirror / code / Atom feed
* [bug#35155] [PATCH] build-system/cargo: refactor phases to successfully build
@ 2019-04-05  7:07 Ivan Petkov
  2019-04-06 10:32 ` Danny Milosavljevic
                   ` (2 more replies)
  0 siblings, 3 replies; 10+ messages in thread
From: Ivan Petkov @ 2019-04-05  7:07 UTC (permalink / raw)
  To: 35155; +Cc: Chris Marusich

[-- Attachment #1: Type: text/plain, Size: 1552 bytes --]

This patch refactors the cargo-build-system builder code to correctly build
imported Rust crates. Specifically the changes:

* Do not rely on a Cargo.lock presence to determine installation. Cargo will
automatically create the file at its first invocation, so instead we check
the manifest for any executable targets before attempting an installation
* Do not attempt to modify the Cargo.toml file. There are many system specific
package in crates.io (e.g. for Windows, Redox, Fuschia, WASM, etc.) and
attempting to keep up with what crates must be patched out is futile.
* The build phases will honor a skip-build? flag which allows for
short-circuiting for optional packages which require nightly features or cannot
be built for the current platform.

Changes which still need to be done:
* Update the host-side code to expand transitive inputs: cargo requires that
all transitive crate dependencies are present in its (vendored) index, but
doing so by hand in the package definitions will become unwieldy.
* Update the host-side code to detect any "circular" dependencies which can
result from a naive import

Unfortunately there isn't a good way to test this patch at the moment.
Importing a non-trivial crate requires a lot of manual resolution, especially
with the points above remaining unimplemented.

If someone would really like to see the input package definitions I was using
to test, I'd be happy to share, though I'd advise that it's pretty hacked up
for my own convoluted testing at the moment.

—Ivan



[-- Attachment #2: 0001-build-system-cargo-refactor-phases-to-successfully-b.patch --]
[-- Type: application/octet-stream, Size: 11122 bytes --]

From 09c2c9e76e8c9d12bd230d953f9d197efbfb1079 Mon Sep 17 00:00:00 2001
From: Ivan Petkov <ivanppetkov@gmail.com>
Date: Tue, 2 Apr 2019 03:02:51 -0700
Subject: [PATCH] build-system/cargo: refactor phases to successfully build

* guix/build-system/cargo.scm (%cargo-build-system-modules):
Add (json parser)
(cargo-build):
[vendor-dir]: Define flag and pass it to builder code.
[cargo-test-flags]: Likewise.
[skip-build?]: Likewise.
* guix/build/cargo-build/system.scm (#:use-module): use (json parser)
(package-name->crate-name): Delete it
(manifest-targets): Add it
(has-executable-target?): Add it
(configure): Add #:vendor-dir name and use it.
Don't touch Cargo.toml.
Don't symlink to duplicate inputs.
Remove useless registry line from cargo config.
Define RUSTFLAGS to lift lint restrictions.
(build): Add #:skip-build? flag and use it.
(check): Likewise.
Add #:cargo-tset-flags and pass it to cargo
(install): Factor source logic to install-source.
Define #:skip-build? flag and use it.
Only install if executable targets are present.
(install-source): Copy entire crate directory not just src.
[generate-checksums] pass dummy file for unused second argument
(%standard-phases): Add install-source phase
---
 guix/build-system/cargo.scm       |   7 ++
 guix/build/cargo-build-system.scm | 151 +++++++++++++++++-------------
 2 files changed, 93 insertions(+), 65 deletions(-)

diff --git a/guix/build-system/cargo.scm b/guix/build-system/cargo.scm
index 7ff4e90f71..38290597ee 100644
--- a/guix/build-system/cargo.scm
+++ b/guix/build-system/cargo.scm
@@ -59,13 +59,17 @@ to NAME and VERSION."
 (define %cargo-build-system-modules
   ;; Build-side modules imported by default.
   `((guix build cargo-build-system)
+    (json parser)
     ,@%cargo-utils-modules))
 
 (define* (cargo-build store name inputs
                       #:key
                       (tests? #t)
                       (test-target #f)
+                      (vendor-dir "guix-vendor")
                       (cargo-build-flags ''("--release"))
+                      (cargo-test-flags ''("--release"))
+                      (skip-build? #f)
                       (phases '(@ (guix build cargo-build-system)
                                   %standard-phases))
                       (outputs '("out"))
@@ -90,7 +94,10 @@ to NAME and VERSION."
                                  source))
                     #:system ,system
                     #:test-target ,test-target
+                    #:vendor-dir ,vendor-dir
                     #:cargo-build-flags ,cargo-build-flags
+                    #:cargo-test-flags ,cargo-test-flags
+                    #:skip-build? ,skip-build?
                     #:tests? ,tests?
                     #:phases ,phases
                     #:outputs %outputs
diff --git a/guix/build/cargo-build-system.scm b/guix/build/cargo-build-system.scm
index 20087fa6c4..8616873ce8 100644
--- a/guix/build/cargo-build-system.scm
+++ b/guix/build/cargo-build-system.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2016 David Craven <david@craven.ch>
 ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com>
+;;; Copyright © 2019 Ivan Petkov <ivanppetkov@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -26,6 +27,7 @@
   #:use-module (ice-9 ftw)
   #:use-module (ice-9 format)
   #:use-module (ice-9 match)
+  #:use-module (json parser)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-26)
   #:export (%standard-phases
@@ -37,81 +39,88 @@
 ;;
 ;; Code:
 
-;; FIXME: Needs to be parsed from url not package name.
-(define (package-name->crate-name name)
-  "Return the crate name of NAME."
-  (match (string-split name #\-)
-    (("rust" rest ...)
-     (string-join rest "-"))
-    (_ #f)))
+(define (manifest-targets)
+  "Extract all targets from the Cargo.toml manifest"
+  (let* ((port (open-input-pipe "cargo read-manifest"))
+         (data (json->scm port))
+         (targets (hash-ref data "targets" '())))
+    (close-port port)
+    targets))
 
-(define* (configure #:key inputs #:allow-other-keys)
+(define (has-executable-target?)
+  "Check if the current cargo project declares any binary targets."
+  (let* ((bin? (lambda (kind) (string=? kind "bin")))
+         (get-kinds (lambda (dep) (hash-ref dep "kind")))
+         (bin-dep? (lambda (dep) (find bin? (get-kinds dep)))))
+    (find bin-dep? (manifest-targets))))
+
+(define* (configure #:key inputs
+                    (vendor-dir "guix-vendor")
+                    #:allow-other-keys)
   "Replace Cargo.toml [dependencies] section with guix inputs."
-  ;; Make sure Cargo.toml is writeable when the crate uses git-fetch.
-  (chmod "Cargo.toml" #o644)
   (chmod "." #o755)
-  (if (not (file-exists? "vendor"))
-    (if (not (file-exists? "Cargo.lock"))
-      (begin
-        (substitute* "Cargo.toml"
-          ((".*32-sys.*") "
-")
-          ((".*winapi.*") "
-")
-          ((".*core-foundation.*") "
-"))
-        ;; Prepare one new directory with all the required dependencies.
-        ;; It's necessary to do this (instead of just using /gnu/store as the
-        ;; directory) because we want to hide the libraries in subdirectories
-        ;;   share/rust-source/... instead of polluting the user's profile root.
-        (mkdir "vendor")
-        (for-each
-          (match-lambda
-            ((name . path)
-             (let ((crate (package-name->crate-name name)))
-               (when (and crate path)
-                 (match (string-split (basename path) #\-)
-                   ((_ ... version)
-                    (symlink (string-append path "/share/rust-source")
-                             (string-append "vendor/" (basename path)))))))))
-          inputs)
-        ;; Configure cargo to actually use this new directory.
-        (mkdir-p ".cargo")
-        (let ((port (open-file ".cargo/config" "w" #:encoding "utf-8")))
-          (display "
+  ;; Prepare one new directory with all the required dependencies.
+  ;; It's necessary to do this (instead of just using /gnu/store as the
+  ;; directory) because we want to hide the libraries in subdirectories
+  ;; share/rust-source/... instead of polluting the user's profile root.
+  (mkdir-p vendor-dir)
+  (for-each
+    (match-lambda
+      ((name . path)
+       (let* ((rust-share (string-append path "/share/rust-source"))
+              (basepath (basename path))
+              (link-dir (string-append vendor-dir "/" basepath)))
+         (and (file-exists? rust-share)
+              ;; Gracefully handle duplicate inputs
+              (not (file-exists? link-dir))
+              (symlink rust-share link-dir)))))
+    inputs)
+  ;; Configure cargo to actually use this new directory.
+  (mkdir-p ".cargo")
+  (let ((port (open-file ".cargo/config" "w" #:encoding "utf-8")))
+    (display "
 [source.crates-io]
-registry = 'https://github.com/rust-lang/crates.io-index'
 replace-with = 'vendored-sources'
 
 [source.vendored-sources]
 directory = '" port)
-          (display (getcwd) port)
-          (display "/vendor" port)
-          (display "'
+    (display (string-append (getcwd) "/" vendor-dir) port)
+    (display "'
 " port)
-          (close-port port)))))
-    (setenv "CC" (string-append (assoc-ref inputs "gcc") "/bin/gcc"))
+    (close-port port))
 
-    ;(setenv "CARGO_HOME" "/gnu/store")
-    ; (setenv "CMAKE_C_COMPILER" cc)
+  ;; Lift restriction on any lints: a crate author may have decided to opt
+  ;; into stricter lints (e.g. #![deny(warnings)]) during their own builds
+  ;; but we don't want any build failures that could be caused later by
+  ;; upgrading the compiler for example.
+  (setenv "RUSTFLAGS" "--cap-lints allow")
+  (setenv "CC" (string-append (assoc-ref inputs "gcc") "/bin/gcc"))
   #t)
 
-(define* (build #:key (cargo-build-flags '("--release"))
+(define* (build #:key
+                skip-build?
+                (cargo-build-flags '("--release"))
                 #:allow-other-keys)
   "Build a given Cargo package."
-  (zero? (apply system* `("cargo" "build" ,@cargo-build-flags))))
+  (or skip-build?
+      (zero? (apply system* `("cargo" "build" ,@cargo-build-flags)))))
 
-(define* (check #:key tests? #:allow-other-keys)
+(define* (check #:key
+                skip-build?
+                tests?
+                (cargo-test-flags '("--release"))
+                #:allow-other-keys)
   "Run tests for a given Cargo package."
-  (if (and tests? (file-exists? "Cargo.lock"))
-      (zero? (system* "cargo" "test"))
-      #t))
+  (or skip-build?
+      (and
+        tests?
+        (zero? (apply system* `("cargo" "test" ,@cargo-test-flags))))))
 
 (define (touch file-name)
   (call-with-output-file file-name (const #t)))
 
-(define* (install #:key inputs outputs #:allow-other-keys)
-  "Install a given Cargo package."
+(define* (install-source #:key inputs outputs #:allow-other-keys)
+  "Install the source for a given Cargo package."
   (let* ((out (assoc-ref outputs "out"))
          (src (assoc-ref inputs "source"))
          (rsrc (string-append (assoc-ref outputs "src")
@@ -122,22 +131,34 @@ directory = '" port)
     ;; Until this changes we are working around this by
     ;; distributing crates as source and replacing
     ;; references in Cargo.toml with store paths.
-    (copy-recursively "src" (string-append rsrc "/src"))
+    (copy-recursively "." rsrc)
     (touch (string-append rsrc "/.cargo-ok"))
-    (generate-checksums rsrc src)
+    (generate-checksums rsrc "/dev/null")
     (install-file "Cargo.toml" rsrc)
-    ;; When the package includes executables we install
-    ;; it using cargo install. This fails when the crate
-    ;; doesn't contain an executable.
-    (if (file-exists? "Cargo.lock")
-        (zero? (system* "cargo" "install" "--root" out))
-        (begin
-          (mkdir out)
-          #t))))
+    #t))
+
+(define* (install #:key inputs outputs skip-build? #:allow-other-keys)
+  "Install a given Cargo package."
+  (let* ((out (assoc-ref outputs "out")))
+    (mkdir-p out)
+
+    ;; Make cargo reuse all the artifacts we just built instead
+    ;; of defaulting to making a new temp directory
+    (setenv "CARGO_TARGET_DIR" "./target")
+    ;; Force cargo to honor our .cargo/config definitions
+    ;; https://github.com/rust-lang/cargo/issues/6397
+    (setenv "CARGO_HOME" ".")
+
+    ;; Only install crates which include binary targets,
+    ;; otherwise cargo will raise an error.
+    (or skip-build?
+        (not (has-executable-target?))
+        (zero? (system* "cargo" "install" "--path" "." "--root" out)))))
 
 (define %standard-phases
   (modify-phases gnu:%standard-phases
     (delete 'bootstrap)
+    (add-before 'configure 'install-source install-source)
     (replace 'configure configure)
     (replace 'build build)
     (replace 'check check)
-- 
2.21.0


^ permalink raw reply related	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2019-04-09 16:02 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-04-05  7:07 [bug#35155] [PATCH] build-system/cargo: refactor phases to successfully build Ivan Petkov
2019-04-06 10:32 ` Danny Milosavljevic
2019-04-06 10:41 ` Danny Milosavljevic
2019-04-06 16:12   ` Ivan Petkov
2019-04-06 23:27 ` Chris Marusich
2019-04-07  2:02   ` Ivan Petkov
2019-04-07  3:05     ` Ivan Petkov
2019-04-07  8:40     ` Chris Marusich
2019-04-07 15:49       ` Ivan Petkov
     [not found]         ` <87mukz8p3g.fsf@garuda.local.i-did-not-set--mail-host-address--so-tickle-me>
2019-04-09 16:01           ` bug#35155: " Ivan Petkov

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).