all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* [bug#35318] [PATCH] Update cargo-build-system to expand package inputs
@ 2019-04-19  5:34 Ivan Petkov
  2019-05-04 16:40 ` Ivan Petkov
  2019-05-06  8:00 ` Ludovic Courtès
  0 siblings, 2 replies; 16+ messages in thread
From: Ivan Petkov @ 2019-04-19  5:34 UTC (permalink / raw)
  To: 35318; +Cc: Chris Marusich

[-- Attachment #1: Type: text/plain, Size: 1051 bytes --]

When building, cargo requires the source of all transitive dependencies to be present in its index.
Rather than force package definitions to list out all transitive inputs, the build system will
automatically expand the inputs as necessary.

Because it is possible for crates to have apparent cycles in their native dependencies,
the build system must take some measures to work around potential cycles. This patch series
takes an initial naive stab at resolving the problem, by never building native-inputs.

I plan on revisiting this sometime soon and making the system a bit more intelligent
(namely building native-inputs where possible and breaking any cycles). But for now this should
unblock working on package definitions.

I’ve also included three rust crates as a proof of concept that the system works and it handles
potential native-input cycles. These crates do nothing on their own, but they’re heavily depended
upon by the rest of the crates ecosystem, so they will eventually be useful to have in guix.

—Ivan


[-- Attachment #2: 0001-packages-allow-dynamic-input-closure-computation.patch --]
[-- Type: application/octet-stream, Size: 3435 bytes --]

From ca6dfd9451f22c4d6dc02aa7eceee0c35800dd57 Mon Sep 17 00:00:00 2001
From: Ivan Petkov <ivanppetkov@gmail.com>
Date: Tue, 16 Apr 2019 03:32:44 -0700
Subject: [PATCH 1/4] packages: allow dynamic input closure computation

* guix/packages: (transitive-inputs): Rename to
package-transitive-dependencies.
(package-transitive-dependencies): Add proc parameter and use it.
(transitive-inputs): Add it.
---
 guix/packages.scm | 23 +++++++++++++++++------
 1 file changed, 17 insertions(+), 6 deletions(-)

diff --git a/guix/packages.scm b/guix/packages.scm
index c94a651f27..658b2427ca 100644
--- a/guix/packages.scm
+++ b/guix/packages.scm
@@ -4,6 +4,7 @@
 ;;; Copyright © 2015 Eric Bavier <bavier@member.fsf.org>
 ;;; Copyright © 2016 Alex Kost <alezost@gmail.com>
 ;;; Copyright © 2017 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2019 Ivan Petkov <ivanppetkov@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -95,6 +96,7 @@
             package-direct-sources
             package-transitive-sources
             package-direct-inputs
+            package-transitive-dependencies
             package-transitive-inputs
             package-transitive-target-inputs
             package-transitive-native-inputs
@@ -650,13 +652,13 @@ specifies modules in scope when evaluating SNIPPET."
                         #:deprecation-warnings #t ;to avoid a rebuild
                         #:guile-for-build guile-for-build))))
 
-(define (transitive-inputs inputs)
-  "Return the closure of INPUTS when considering the 'propagated-inputs'
-edges.  Omit duplicate inputs, except for those already present in INPUTS
-itself.
+(define (package-transitive-dependencies inputs proc)
+  "Return the closure of INPUTS when considering the edges extracted
+from each package via PROC.  Omit duplicate inputs, except for those
+already present in INPUTS itself.
 
 This is implemented as a breadth-first traversal such that INPUTS is
-preserved, and only duplicate propagated inputs are removed."
+preserved, and only duplicate extracted inputs are removed."
   (define (seen? seen item outputs)
     ;; FIXME: We're using pointer identity here, which is extremely sensitive
     ;; to memoization in package-producing procedures; see
@@ -680,7 +682,7 @@ preserved, and only duplicate propagated inputs are removed."
            (loop rest result propagated first? seen)
            (loop rest
                  (cons input result)
-                 (cons (package-propagated-inputs package) propagated)
+                 (cons (proc package) propagated)
                  first?
                  (vhash-consq package outputs seen))))
       ((input rest ...)
@@ -696,6 +698,15 @@ PACKAGE's inputs."
                    (_ #f))
                   (package-direct-inputs package))))
 
+(define (transitive-inputs inputs)
+  "Return the closure of INPUTS when considering the 'propagated-inputs'
+edges.  Omit duplicate inputs, except for those already present in INPUTS
+itself.
+
+This is implemented as a breadth-first traversal such that INPUTS is
+preserved, and only duplicate propagated inputs are removed."
+  (package-transitive-dependencies inputs package-propagated-inputs))
+
 (define (package-transitive-sources package)
   "Return PACKAGE's direct sources, and their direct sources, recursively."
   (delete-duplicates
-- 
2.21.0


[-- Attachment #3: 0002-build-system-cargo-expand-transitive-package-inputs.patch --]
[-- Type: application/octet-stream, Size: 4894 bytes --]

From 7b42290f238d5ab50d83232ec6d95db665a98302 Mon Sep 17 00:00:00 2001
From: Ivan Petkov <ivanppetkov@gmail.com>
Date: Tue, 16 Apr 2019 03:37:44 -0700
Subject: [PATCH 2/4] build-system/cargo: expand transitive package inputs

* guix/build/cargo: (package-target-inputs): Add it.
(expand-inputs): Add it.
(expand-native-inputs): Add it.
(lower): Use new variables.
---
 guix/build-system/cargo.scm | 66 +++++++++++++++++++++++++++++++++++--
 1 file changed, 64 insertions(+), 2 deletions(-)

diff --git a/guix/build-system/cargo.scm b/guix/build-system/cargo.scm
index dc137421e9..65dcd5d339 100644
--- a/guix/build-system/cargo.scm
+++ b/guix/build-system/cargo.scm
@@ -121,6 +121,68 @@ to NAME and VERSION."
                                 #:outputs (cons "src" outputs)
                                 #:guile-for-build guile-for-build))
 
+(define package-target-inputs
+  (lambda (pkg)
+    (append (package-inputs pkg)
+            (package-propagated-inputs pkg))))
+
+(define* (expand-inputs inputs)
+  "Return the transitive target inputs for each package in INPUTS.
+
+Cargo requires all transitive crate dependencies to be available in its index,
+even if they are optional (this is so it can generate deterministic
+Cargo.lock files regardless of the target platform or enabled features).
+
+Rather than burden the package definition to expand these, we'll automate
+that process in the build system."
+  (package-transitive-dependencies inputs package-target-inputs))
+
+(define* (expand-native-inputs native-inputs)
+  "Return the transitive target inputs for each package in NATIVE-INPUTS.
+
+We need all transitive crate dependencies for any cargo dev-dependencies,
+but this is only needed when building/testing a crate directly
+(i.e. we will never need transitive dev-dependencies for any dependency
+crates).
+
+In addition, we need to modify any native-inputs to avoid any apparent
+cycles. Cargo does not permit cyclic dependencies between crates, however,
+it permits cycles to occur via dev-dependencies. For example, if crate X
+depends on crate Y, crate Y's tests could pull in crate X to to verify
+everything builds properly (this is a rare scenario, but it it happens for
+example with the `proc-macro2` and `quote` crates). This is allowed by cargo
+because tests are built as a pseudo-crate which happens to depend on the
+X and Y crates, forming an acyclic graph.
+
+Cargo ultimately needs only the crate sources available when doing a build.
+Ultimately, if a cycle is detected, we can break it by replacing the current
+crate we're trying to build with a no-op build which only packages the crate
+source; cargo will then find the source available in its index and build things
+properly."
+  (map
+    (lambda (dep)
+      (match dep
+       ((name input rest ...)
+        ;; TODO: Starting out real simple by ensuring no native-inputs are
+        ;; actually built so we don't accidentally hit a cycle. We can revisit
+        ;; this later and more intelligently replace any dependency on the cycle
+        ;; root with a version of the root with #:skip-build? enabled.
+        ;;
+        ;; Right now this means that if a package shows up in both input
+        ;; and native-input closures, we'll try to build it's src output twice.
+        ;;
+        ;; This also means that we're potentially setting the #:skip-build?
+        ;; flag on non cargo-build-system packages, but this should be okay
+        ;; for now.
+        (let* ((translated-input (package
+                                   (inherit input)
+                                   (inputs '())
+                                   (propagated-inputs '())
+                                   (native-inputs '())
+                                   (arguments '(#:skip-build? #t)))))
+          `(,name ,translated-input ,@rest)))))
+   (package-transitive-dependencies native-inputs package-target-inputs)))
+
 (define* (lower name
                 #:key source inputs native-inputs outputs system target
                 (rust (default-rust))
@@ -139,13 +201,13 @@ to NAME and VERSION."
          (host-inputs `(,@(if source
                               `(("source" ,source))
                               '())
-                        ,@inputs
+                        ,@(expand-inputs inputs)
 
                         ;; Keep the standard inputs of 'gnu-build-system'
                         ,@(standard-packages)))
          (build-inputs `(("cargo" ,rust "cargo")
                          ("rustc" ,rust)
-                         ,@native-inputs))
+                         ,@(expand-native-inputs native-inputs)))
          (outputs outputs)
          (build cargo-build)
          (arguments (strip-keyword-arguments private-keywords arguments)))))
-- 
2.21.0


[-- Attachment #4: 0003-gnu-crate-add-unicode-xid.patch --]
[-- Type: application/octet-stream, Size: 2957 bytes --]

From 922e985c684b4fcafe30f1c25d698396ffb58191 Mon Sep 17 00:00:00 2001
From: Ivan Petkov <ivanppetkov@gmail.com>
Date: Tue, 16 Apr 2019 03:40:14 -0700
Subject: [PATCH 3/4] gnu: crate: add unicode-xid

gnu/local.mk: (GNU_SYSTEM_MODULES): Add new file.
gnu/packages/crates-io.scm: (rust-unicode-xid): New variable.
---
 gnu/local.mk               |  1 +
 gnu/packages/crates-io.scm | 45 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 46 insertions(+)
 create mode 100644 gnu/packages/crates-io.scm

diff --git a/gnu/local.mk b/gnu/local.mk
index 41924a7de5..bf44af76b7 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -121,6 +121,7 @@ GNU_SYSTEM_MODULES =				\
   %D%/packages/cpp.scm 				\
   %D%/packages/cppi.scm				\
   %D%/packages/cran.scm				\
+  %D%/packages/crates-io.scm			\
   %D%/packages/cross-base.scm			\
   %D%/packages/crypto.scm			\
   %D%/packages/cryptsetup.scm			\
diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
new file mode 100644
index 0000000000..533fe0d21e
--- /dev/null
+++ b/gnu/packages/crates-io.scm
@@ -0,0 +1,45 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2019 Ivan Petkov <ivanppetkov@gmail.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu packages crates-io)
+  #:use-module (guix build-system cargo)
+  #:use-module (guix download)
+  #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (guix packages))
+
+(define-public rust-unicode-xid
+  (package
+    (name "rust-unicode-xid")
+    (version "0.1.0")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (crate-uri "unicode-xid" version))
+        (file-name
+          (string-append name "-" version ".tar.gz"))
+        (sha256
+          (base32
+            "1z57lqh4s18rr4x0j4fw4fmp9hf9346h0kmdgqsqx0fhjr3k0wpw"))))
+    (build-system cargo-build-system)
+    (home-page
+      "https://github.com/unicode-rs/unicode-xid")
+    (synopsis "Determine Unicode XID related properties")
+    (description "Determine whether characters have the XID_Start
+or XID_Continue properties according to Unicode Standard Annex #31.")
+    ;; Dual licensed.
+    (license (list license:asl2.0 license:expat))))
-- 
2.21.0


[-- Attachment #5: 0004-gnu-crate-Add-proc-macro2-and-quote.patch --]
[-- Type: application/octet-stream, Size: 2534 bytes --]

From 70f96162e55ac1d1116a5d94134d61aa769f8be7 Mon Sep 17 00:00:00 2001
From: Ivan Petkov <ivanppetkov@gmail.com>
Date: Tue, 16 Apr 2019 03:42:27 -0700
Subject: [PATCH 4/4] gnu: crate: Add proc-macro2 and quote

* gnu/packages/crates-io.scm: (rust-proc-macro2): New variable.
(rust-quote): New variable.
---
 gnu/packages/crates-io.scm | 48 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 48 insertions(+)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index 533fe0d21e..90e08731ea 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -43,3 +43,51 @@
 or XID_Continue properties according to Unicode Standard Annex #31.")
     ;; Dual licensed.
     (license (list license:asl2.0 license:expat))))
+
+(define-public rust-proc-macro2
+  (package
+    (name "rust-proc-macro2")
+    (version "0.4.27")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (crate-uri "proc-macro2" version))
+        (file-name
+          (string-append name "-" version ".tar.gz"))
+        (sha256
+          (base32
+            "05c92v787snyaq4ss16vxc9mdv6zndfgsdq8k3hnnyffmsf7ycad"))))
+    (build-system cargo-build-system)
+    (native-inputs
+      `(("rust-quote" ,rust-quote "src")))
+    (inputs
+      `(("rust-unicode-xid" ,rust-unicode-xid "src")))
+    (home-page "https://github.com/alexcrichton/proc-macro2")
+    (synopsis "Stable implementation of the upcoming new `proc_macro` API")
+    (description "This package provides a stable implementation of the upcoming new
+`proc_macro` API.  Comes with an option, off by default, to also reimplement itself
+in terms of the upstream unstable API.")
+    ;; Dual licensed.
+    (license (list license:asl2.0 license:expat))))
+
+(define-public rust-quote
+  (package
+    (name "rust-quote")
+    (version "0.6.12")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (crate-uri "quote" version))
+        (file-name
+          (string-append name "-" version ".tar.gz"))
+        (sha256
+          (base32
+            "1nw0klza45hf127kfyrpxsxd5jw2l6h21qxalil3hkr7bnf7kx7s"))))
+    (build-system cargo-build-system)
+    (inputs
+      `(("rust-proc-macro2" ,rust-proc-macro2 "src")))
+    (home-page "https://github.com/dtolnay/quote")
+    (synopsis "Quasi-quoting macro quote!(...)")
+    (description "Quasi-quoting macro quote!(...)")
+    ;; Dual licensed.
+    (license (list license:asl2.0 license:expat))))
-- 
2.21.0


[-- Attachment #6: Type: text/plain, Size: 1 bytes --]

 

^ permalink raw reply related	[flat|nested] 16+ messages in thread

end of thread, other threads:[~2019-06-12  1:15 UTC | newest]

Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-04-19  5:34 [bug#35318] [PATCH] Update cargo-build-system to expand package inputs Ivan Petkov
2019-05-04 16:40 ` Ivan Petkov
2019-05-04 18:31   ` Danny Milosavljevic
2019-05-04 21:09     ` Ivan Petkov
2019-05-06  8:00 ` Ludovic Courtès
2019-05-06 16:04   ` Ivan Petkov
2019-05-09 23:17     ` Ivan Petkov
2019-05-15  6:08       ` Ivan Petkov
2019-05-15 12:44         ` Ludovic Courtès
2019-05-20  1:00           ` Ivan Petkov
2019-05-20 19:38             ` Ludovic Courtès
2019-05-22  2:48               ` Ivan Petkov
2019-06-08 18:44             ` Chris Marusich
2019-06-08 23:33               ` Ivan Petkov
2019-06-09 23:53                 ` Ivan Petkov
2019-06-12  1:14                   ` bug#35318: " Chris Marusich

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.