[bug#36841] [PATCH] build/cargo-build-system: Patch cargo checksums.

[bug#36841] [PATCH] build/cargo-build-system: Patch cargo checksums.

[Efraim Flashner]

* guix/build/cargo-build-system.scm (patch-cargo-checksums): New phase.
(%standard-phases): Add 'patch-cargo-checksums after
'patch-generated-file-shebangs.
---
 guix/build/cargo-build-system.scm | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/guix/build/cargo-build-system.scm b/guix/build/cargo-build-system.scm
index f38de16cf7..8e1ee62f65 100644
--- a/guix/build/cargo-build-system.scm
+++ b/guix/build/cargo-build-system.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2016 David Craven <david@craven.ch>
 ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com>
 ;;; Copyright © 2019 Ivan Petkov <ivanppetkov@gmail.com>
+;;; Copyright © 2019 Efraim Flashner <efraim@flashner.co.il>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -121,6 +122,23 @@ directory = '" port)
   (setenv "CC" (string-append (assoc-ref inputs "gcc") "/bin/gcc"))
   #t)
 
+;; After patching the 'patch-generated-file-shebangs phase any vendored crates
+;; will have a mismatch on their checksum.
+(define* (patch-cargo-checksums #:key
+                                (vendor-dir "guix-vendor")
+                                #:allow-other-keys)
+  "Patch the checksums of the vendored crates after patching their shebangs."
+  (for-each
+    (lambda (filename)
+      (delete-file filename)
+      (let* ((dir (dirname filename)))
+        (display (string-append
+                   "patch-cargo-checksums: generate-checksums for "
+                   dir "\n"))
+        (generate-checksums dir)))
+    (find-files "guix-vendor" ".cargo-checksum.json"))
+  #t)
+
 (define* (build #:key
                 skip-build?
                 (cargo-build-flags '("--release"))
@@ -162,7 +180,8 @@ directory = '" port)
     (replace 'configure configure)
     (replace 'build build)
     (replace 'check check)
-    (replace 'install install)))
+    (replace 'install install)
+    (add-after 'patch-generated-file-shebangs 'patch-cargo-checksums patch-cargo-checksums)))
 
 (define* (cargo-build #:key inputs (phases %standard-phases)
                       #:allow-other-keys #:rest args)
-- 
2.22.0

[bug#36841] [PATCH] build/cargo-build-system: Patch cargo checksums.

[Ivan Petkov]

Hi Efraim,

> On Jul 29, 2019, at 12:04 PM, Efraim Flashner <efraim@flashner.co.il> wrote:
> 
> +;; After patching the 'patch-generated-file-shebangs phase any vendored crates
> +;; will have a mismatch on their checksum.
> +(define* (patch-cargo-checksums #:key
> +                                (vendor-dir "guix-vendor")
> +                                #:allow-other-keys)

[snip]

> +    (replace 'install install)
> +    (add-after 'patch-generated-file-shebangs 'patch-cargo-checksums patch-cargo-checksums)))

I can’t quite remember the order the phases run in off the top of my head. Would it be possible to
make the configure/checksum generation phase run after shebang-patching (or ensure the patching
happens first)? It would avoid having to checksum all the files twice that way…

—Ivan

bug#36841: [PATCH] build/cargo-build-system: Patch cargo checksums.

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1661 bytes --]

On Mon, Jul 29, 2019 at 06:44:31PM -0700, Ivan Petkov wrote:
> Hi Efraim,
> 
> > On Jul 29, 2019, at 12:04 PM, Efraim Flashner <efraim@flashner.co.il> wrote:
> > 
> > +;; After patching the 'patch-generated-file-shebangs phase any vendored crates
> > +;; will have a mismatch on their checksum.
> > +(define* (patch-cargo-checksums #:key
> > +                                (vendor-dir "guix-vendor")
> > +                                #:allow-other-keys)
> 
> [snip]
> 
> > +    (replace 'install install)
> > +    (add-after 'patch-generated-file-shebangs 'patch-cargo-checksums patch-cargo-checksums)))
> 
> I can’t quite remember the order the phases run in off the top of my head. Would it be possible to
> make the configure/checksum generation phase run after shebang-patching (or ensure the patching
> happens first)? It would avoid having to checksum all the files twice that way…
> 
> —Ivan

I thought about it a bit more after I sent the patch, and I'm pretty
sure this is only needed when there's a Cargo.lock file in the build
directory. So in actuality it should be more like:

(when (file-exists? "Cargo.lock")
  (begin
    (delete-file "Cargo.lock")
    (invoke "cargo" "generate-lockfile")
    (patch-cargo-checksums ...)))

I'm going to close this bug/patch and re-submit it when I've given it a
bit more work so it doesn't do the expensive compute-checksums
computation on all builds.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bug#36841] [PATCH] build/cargo-build-system: Patch cargo checksums.

[Efraim Flashner]

[-- Attachment #1.1: Type: text/plain, Size: 1678 bytes --]

On Mon, Jul 29, 2019 at 06:44:31PM -0700, Ivan Petkov wrote:
> Hi Efraim,
> 
> > On Jul 29, 2019, at 12:04 PM, Efraim Flashner <efraim@flashner.co.il> wrote:
> > 
> > +;; After patching the 'patch-generated-file-shebangs phase any vendored crates
> > +;; will have a mismatch on their checksum.
> > +(define* (patch-cargo-checksums #:key
> > +                                (vendor-dir "guix-vendor")
> > +                                #:allow-other-keys)
> 
> [snip]
> 
> > +    (replace 'install install)
> > +    (add-after 'patch-generated-file-shebangs 'patch-cargo-checksums patch-cargo-checksums)))
> 
> I can’t quite remember the order the phases run in off the top of my head. Would it be possible to
> make the configure/checksum generation phase run after shebang-patching (or ensure the patching
> happens first)? It would avoid having to checksum all the files twice that way…

The 'configure phase could be renamed the plop-vendored-crates-into-place
phase. It actually can't come after the 'patch-generated-file-shebangs
phase since then there won't be any vendored crates to patch.

If we remove the generate-checksums call from 'configure then there
won't be a .cargo-checksum.json to remove and regenerate during the
'patch-cargo-checksums phase, so I've changed that to search for
"Cargo.toml$" and not delete it. Not as robust as "for each top-level
directory in the 'vendor-dir'", but should be good enough.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #1.2: 0001-build-cargo-build-system-Patch-cargo-checksums.patch --]
[-- Type: text/plain, Size: 3664 bytes --]

From abf262fbe7ed8d9a532a3ce63cb7453c1666d914 Mon Sep 17 00:00:00 2001
From: Efraim Flashner <efraim@flashner.co.il>
Date: Mon, 29 Jul 2019 22:01:05 +0300
Subject: [PATCH] build/cargo-build-system: Patch cargo checksums.

* guix/build/cargo-build-system.scm (update-cargo-lock,
patch-cargo-checksums): New phases.
(%standard-phases): Add 'update=cargo-lock after 'configure and
'patch-cargo-checksums after 'patch-generated-file-shebangs.
---
 guix/build/cargo-build-system.scm | 37 ++++++++++++++++++++++++++++---
 1 file changed, 34 insertions(+), 3 deletions(-)

diff --git a/guix/build/cargo-build-system.scm b/guix/build/cargo-build-system.scm
index f38de16cf7..dfc7923c8d 100644
--- a/guix/build/cargo-build-system.scm
+++ b/guix/build/cargo-build-system.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2016 David Craven <david@craven.ch>
 ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com>
 ;;; Copyright © 2019 Ivan Petkov <ivanppetkov@gmail.com>
+;;; Copyright © 2019 Efraim Flashner <efraim@flashner.co.il>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -94,8 +95,7 @@ Cargo.toml file present at its root."
               ;; so that we can generate any cargo checksums.
               ;; The --strip-components argument is needed to prevent creating
               ;; an extra directory within `crate-dir`.
-              (invoke "tar" "xvf" path "-C" crate-dir "--strip-components" "1")
-              (generate-checksums crate-dir)))))
+              (invoke "tar" "xvf" path "-C" crate-dir "--strip-components" "1")))))
     inputs)
 
   ;; Configure cargo to actually use this new directory.
@@ -121,6 +121,35 @@ directory = '" port)
   (setenv "CC" (string-append (assoc-ref inputs "gcc") "/bin/gcc"))
   #t)
 
+;; The Cargo.lock file tells the build system which crates are required for
+;; building and hardcodes their version and checksum.  In order to build with
+;; the inputs we provide, we need to recreate the file with our inputs.
+(define* (update-cargo-lock #:key
+                            (vendor-dir "guix-vendor")
+                            #:allow-other-keys)
+  "Regenerate the Cargo.lock file with the current build inputs."
+  (when (file-exists? "Cargo.lock")
+    (begin
+      (delete-file "Cargo.lock")
+      (invoke "cargo" "generate-lockfile")))
+  #t)
+
+;; After the 'patch-generated-file-shebangs phase any vendored crates who have
+;; their shebangs patched will have a mismatch on their checksum.
+(define* (patch-cargo-checksums #:key
+                                (vendor-dir "guix-vendor")
+                                #:allow-other-keys)
+  "Patch the checksums of the vendored crates after patching their shebangs."
+  (for-each
+    (lambda (filename)
+      (let* ((dir (dirname filename)))
+        (display (string-append
+                   "patch-cargo-checksums: generate-checksums for "
+                   dir "\n"))
+        (generate-checksums dir)))
+    (find-files "guix-vendor" "Cargo.toml$"))
+  #t)
+
 (define* (build #:key
                 skip-build?
                 (cargo-build-flags '("--release"))
@@ -162,7 +191,9 @@ directory = '" port)
     (replace 'configure configure)
     (replace 'build build)
     (replace 'check check)
-    (replace 'install install)))
+    (replace 'install install)
+    (add-after 'configure 'update-cargo-lock update-cargo-lock)
+    (add-after 'patch-generated-file-shebangs 'patch-cargo-checksums patch-cargo-checksums)))
 
 (define* (cargo-build #:key inputs (phases %standard-phases)
                       #:allow-other-keys #:rest args)
-- 
2.22.0


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bug#36841] [PATCH v3] build/cargo-build-system: Patch cargo checksums.

[Efraim Flashner]

[-- Attachment #1.1: Type: text/plain, Size: 2083 bytes --]

On Tue, Jul 30, 2019 at 11:17:57AM +0300, Efraim Flashner wrote:
> On Mon, Jul 29, 2019 at 06:44:31PM -0700, Ivan Petkov wrote:
> > Hi Efraim,
> > 
> > > On Jul 29, 2019, at 12:04 PM, Efraim Flashner <efraim@flashner.co.il> wrote:
> > > 
> > > +;; After patching the 'patch-generated-file-shebangs phase any vendored crates
> > > +;; will have a mismatch on their checksum.
> > > +(define* (patch-cargo-checksums #:key
> > > +                                (vendor-dir "guix-vendor")
> > > +                                #:allow-other-keys)
> > 
> > [snip]
> > 
> > > +    (replace 'install install)
> > > +    (add-after 'patch-generated-file-shebangs 'patch-cargo-checksums patch-cargo-checksums)))
> > 
> > I can’t quite remember the order the phases run in off the top of my head. Would it be possible to
> > make the configure/checksum generation phase run after shebang-patching (or ensure the patching
> > happens first)? It would avoid having to checksum all the files twice that way…
> 
> The 'configure phase could be renamed the plop-vendored-crates-into-place
> phase. It actually can't come after the 'patch-generated-file-shebangs
> phase since then there won't be any vendored crates to patch.
> 
> If we remove the generate-checksums call from 'configure then there
> won't be a .cargo-checksum.json to remove and regenerate during the
> 'patch-cargo-checksums phase, so I've changed that to search for
> "Cargo.toml$" and not delete it. Not as robust as "for each top-level
> directory in the 'vendor-dir'", but should be good enough.
> 
This one I'm pretty happy with. The checksums are only generated twice
when there's a Cargo.lock file present and I've factored out the
function to generate all the checksums. When that's moved to (guix build
cargo-utils) it can be used by the rust compilers and icecat.


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #1.2: 0001-build-cargo-build-system-Patch-cargo-checksums.patch --]
[-- Type: text/plain, Size: 4436 bytes --]

From d1252887da416b664090bfcee0d84729abb0fbc4 Mon Sep 17 00:00:00 2001
From: Efraim Flashner <efraim@flashner.co.il>
Date: Mon, 29 Jul 2019 22:01:05 +0300
Subject: [PATCH] build/cargo-build-system: Patch cargo checksums.

* guix/build/cargo-build-system.scm (generate-all-checksums): New
procedure.
(update-cargo-lock, patch-cargo-checksums): New phases.
(%standard-phases): Add 'update=cargo-lock after 'configure and
'patch-cargo-checksums after 'patch-generated-file-shebangs.
---
 guix/build/cargo-build-system.scm | 48 +++++++++++++++++++++++++++++--
 1 file changed, 45 insertions(+), 3 deletions(-)

diff --git a/guix/build/cargo-build-system.scm b/guix/build/cargo-build-system.scm
index f38de16cf7..7d363a18a5 100644
--- a/guix/build/cargo-build-system.scm
+++ b/guix/build/cargo-build-system.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2016 David Craven <david@craven.ch>
 ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com>
 ;;; Copyright © 2019 Ivan Petkov <ivanppetkov@gmail.com>
+;;; Copyright © 2019 Efraim Flashner <efraim@flashner.co.il>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -39,6 +40,21 @@
 ;;
 ;; Code:
 
+;; TODO: Move this to (guix build cargo-utils). Will cause a full rebuild
+;; of all rust compilers.
+
+(define (generate-all-checksums dir-name)
+  (for-each
+    (lambda (filename)
+      (let* ((dir (dirname filename))
+             (checksum-file (string-append dir "/.cargo-checksum.json")))
+        (when (file-exists? checksum-file) (delete-file checksum-file))
+        (display (string-append
+                   "patch-cargo-checksums: generate-checksums for "
+                   dir "\n"))
+        (generate-checksums dir)))
+    (find-files dir-name "Cargo.toml$")))
+
 (define (manifest-targets)
   "Extract all targets from the Cargo.toml manifest"
   (let* ((port (open-input-pipe "cargo read-manifest"))
@@ -94,8 +110,7 @@ Cargo.toml file present at its root."
               ;; so that we can generate any cargo checksums.
               ;; The --strip-components argument is needed to prevent creating
               ;; an extra directory within `crate-dir`.
-              (invoke "tar" "xvf" path "-C" crate-dir "--strip-components" "1")
-              (generate-checksums crate-dir)))))
+              (invoke "tar" "xvf" path "-C" crate-dir "--strip-components" "1")))))
     inputs)
 
   ;; Configure cargo to actually use this new directory.
@@ -121,6 +136,31 @@ directory = '" port)
   (setenv "CC" (string-append (assoc-ref inputs "gcc") "/bin/gcc"))
   #t)
 
+;; The Cargo.lock file tells the build system which crates are required for
+;; building and hardcodes their version and checksum.  In order to build with
+;; the inputs we provide, we need to recreate the file with our inputs.
+(define* (update-cargo-lock #:key
+                            (vendor-dir "guix-vendor")
+                            #:allow-other-keys)
+  "Regenerate the Cargo.lock file with the current build inputs."
+  (when (file-exists? "Cargo.lock")
+    (begin
+      ;; Unfortunately we can't generate a Cargo.lock file until the checksums
+      ;; are generated, so we have an extra round of generate-all-checksums here.
+      (generate-all-checksums vendor-dir)
+      (delete-file "Cargo.lock")
+      (invoke "cargo" "generate-lockfile")))
+  #t)
+
+;; After the 'patch-generated-file-shebangs phase any vendored crates who have
+;; their shebangs patched will have a mismatch on their checksum.
+(define* (patch-cargo-checksums #:key
+                                (vendor-dir "guix-vendor")
+                                #:allow-other-keys)
+  "Patch the checksums of the vendored crates after patching their shebangs."
+  (generate-all-checksums vendor-dir)
+  #t)
+
 (define* (build #:key
                 skip-build?
                 (cargo-build-flags '("--release"))
@@ -162,7 +202,9 @@ directory = '" port)
     (replace 'configure configure)
     (replace 'build build)
     (replace 'check check)
-    (replace 'install install)))
+    (replace 'install install)
+    (add-after 'configure 'update-cargo-lock update-cargo-lock)
+    (add-after 'patch-generated-file-shebangs 'patch-cargo-checksums patch-cargo-checksums)))
 
 (define* (cargo-build #:key inputs (phases %standard-phases)
                       #:allow-other-keys #:rest args)
-- 
2.22.0


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bug#36841] [PATCH v3] build/cargo-build-system: Patch cargo checksums.

[Ivan Petkov]

[-- Attachment #1: Type: text/plain, Size: 1221 bytes --]

Hi Efraim,

> On Jul 30, 2019, at 3:46 AM, Efraim Flashner <efraim@flashner.co.il> wrote:
> 
> This one I'm pretty happy with. The checksums are only generated twice
> when there's a Cargo.lock file present and I've factored out the
> function to generate all the checksums. When that's moved to (guix build
> cargo-utils) it can be used by the rust compilers and icecat.

Overall the patch makes sense to me!

However, I am curious what are some of the situations in which you’re encountering
a Cargo.lock file? In a system like guix which maintains all dependencies immutably
and consistently, the Cargo.lock file is virtually useless (in fact it *could* be harmful
if an application is released with a Cargo.lock file pinning to a particular vulnerable
dependency which needs to be updated, requiring patching of the Cargo.lock file).

I’d be willing to go as far as suggest we unconditionally delete any Cargo.lock file
in source tarballs and let cargo generate its own replacement using the vendor
directory we have supplied. (Imports from crates.io <http://crates.io/> also never include a Cargo.lock
file, so this may only pertain if we’re performing a direct source import…)

—Ivan

[-- Attachment #2: Type: text/html, Size: 2016 bytes --]

[bug#36841] [PATCH v3] build/cargo-build-system: Patch cargo checksums.

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2118 bytes --]

On Wed, Jul 31, 2019 at 08:00:00PM -0700, Ivan Petkov wrote:
> Hi Efraim,
> 
> > On Jul 30, 2019, at 3:46 AM, Efraim Flashner <efraim@flashner.co.il> wrote:
> > 
> > This one I'm pretty happy with. The checksums are only generated twice
> > when there's a Cargo.lock file present and I've factored out the
> > function to generate all the checksums. When that's moved to (guix build
> > cargo-utils) it can be used by the rust compilers and icecat.
> 
> Overall the patch makes sense to me!
> 
> However, I am curious what are some of the situations in which you’re encountering
> a Cargo.lock file? In a system like guix which maintains all dependencies immutably
> and consistently, the Cargo.lock file is virtually useless (in fact it *could* be harmful
> if an application is released with a Cargo.lock file pinning to a particular vulnerable
> dependency which needs to be updated, requiring patching of the Cargo.lock file).

One is the package that I'm actually targeting, https://github.com/chfi/rust-qtlreaper/ ,
and three of the others are rust-regex and rust-compiler-builtins and
rust-env-logger. All three of them I got from $(guix import crate foo).
`guix import crate env-logger`, for example, returns this:
https://static.crates.io/crates/env_logger/env_logger-0.6.2.crate

> 
> I’d be willing to go as far as suggest we unconditionally delete any Cargo.lock file
> in source tarballs and let cargo generate its own replacement using the vendor
> directory we have supplied. (Imports from crates.io <http://crates.io/> also never include a Cargo.lock
> file, so this may only pertain if we’re performing a direct source import…)

This is basically what my 'update-cargo-lock phase does. Otherwise we
end up packaging arbitrary versions of crates to satisfy whatever
version they were using when they last updated their Cargo.lock.

> 
> —Ivan

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bug#36841] [PATCH v3] build/cargo-build-system: Patch cargo checksums.

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 259 bytes --]

Pushed with an update to the documentation


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]