Guix build side - including modules (guix hash), (guix gcrypt): How to provide shared library?

Guix build side - including modules (guix hash), (guix gcrypt): How to provide shared library?

[Danny Milosavljevic]

Hi,

how can I use (guix hash) on the build side? It requires the libgcrypt shared library and that's not available on the build side...

Re: Guix build side - including modules (guix hash), (guix gcrypt): How to provide shared library?

[David Craven]

> how can I use (guix hash) on the build side? It requires the libgcrypt shared library and that's not available on the build side...

Depends. For a one off package I'd add it to native-inputs. Otherwise
you can create a new build-system that wraps the gnu-build-system and
extend the gnu-build-system's implicit inputs with libgcrypt. Which is
basically just automatically adding it to native-inputs for you.

Re: Guix build side - including modules (guix hash), (guix gcrypt): How to provide shared library?

[Danny Milosavljevic]

Hi David,

On Mon, 2 Jan 2017 11:06:27 +0100
David Craven <david@craven.ch> wrote:

> > how can I use (guix hash) on the build side? It requires the libgcrypt shared library and that's not available on the build side...  
> 
> Depends. For a one off package I'd add it to native-inputs. Otherwise
> you can create a new build-system that wraps the gnu-build-system and
> extend the gnu-build-system's implicit inputs with libgcrypt. Which is
> basically just automatically adding it to native-inputs for you.

Hmm. I tried both, also in host-inputs and native-inputs but it doesn't seem to work. What I mean is I want to use gcrypt in Guile on the build side. That's a lot more difficult than I thought...

Also, why is sha256 used via FFI via an external library? It's not exactly difficult to implement in Guile :P

Error messages:

@ build-started /gnu/store/3lfanl2xnmx3gr2j11hc51a3m7sp7ljp-module-import-compiled.drv - x86_64-linux /var/log/guix/drvs/3l//fanl2xnmx3gr2j11hc51a3m7sp7ljp-module-import-compiled.drv.bz2
Backtrace:
In ice-9/boot-9.scm:
2404: 19 [save-module-excursion #<procedure 1370870 at ice-9/boot-9.scm:3069:17 ()>]
3088: 18 [#<procedure 1370870 at ice-9/boot-9.scm:3069:17 ()>]
In unknown file:
   ?: 17 [primitive-load-path "guix/hash" ...]
In ice-9/eval.scm:
 505: 16 [#<procedure 79b560 at ice-9/eval.scm:499:4 (exp)> (define-module # # ...)]
In ice-9/psyntax.scm:
1106: 15 [expand-top-sequence ((define-module (guix hash) #:use-module ...)) () ...]
 989: 14 [scan ((define-module (guix hash) #:use-module ...)) () ...]
 279: 13 [scan ((#(syntax-object let # ...) (#) (# #) ...)) () ...]
In ice-9/eval.scm:
 411: 12 [eval # ()]
In ice-9/boot-9.scm:
2954: 11 [define-module* (guix hash) #:filename ...]
2929: 10 [resolve-imports ((#) (#) (#) (#) ...)]
2867: 9 [resolve-interface (guix gcrypt) #:select ...]
2792: 8 [#<procedure 852d00 at ice-9/boot-9.scm:2780:4 (name #:optional autoload version #:key ensure)> # ...]
3068: 7 [try-module-autoload (guix gcrypt) #f]
2404: 6 [save-module-excursion #<procedure 13840c0 at ice-9/boot-9.scm:3069:17 ()>]
3088: 5 [#<procedure 13840c0 at ice-9/boot-9.scm:3069:17 ()>]
In unknown file:
   ?: 4 [primitive-load-path "guix/gcrypt" ...]
In ice-9/eval.scm:
 453: 3 [eval # ()]
 411: 2 [eval # ()]
In unknown file:
   ?: 1 [dynamic-link "/gnu/store/4pnp5scrgmjp21flaihmgbckwrz6z4g3-libgcrypt-1.7.3/lib/libgcrypt"]
In ice-9/boot-9.scm:
 109: 0 [#<procedure 983c80 at ice-9/boot-9.scm:100:6 (thrown-k . args)> misc-error ...]

ice-9/boot-9.scm:109:20: In procedure #<procedure 983c80 at ice-9/boot-9.scm:100:6 (thrown-k . args)>:
ice-9/boot-9.scm:109:20: In procedure dynamic-link: file: "/gnu/store/4pnp5scrgmjp21flaihmgbckwrz6z4g3-libgcrypt-1.7.3/lib/libgcrypt", message: "file not found"
here- 8w6yqqc5f0ny3yq5x5lnx7hhiw60p6l9-gcrypt.scm
note: keeping build directory `/tmp/guix-build-module-import-compiled.drv-0'
builder for `/gnu/store/3lfanl2xnmx3gr2j11hc51a3m7sp7ljp-module-import-compiled.drv' failed with exit code 1
@ build-failed /gnu/store/3lfanl2xnmx3gr2j11hc51a3m7sp7ljp-module-import-compiled.drv - 1 builder for `/gnu/store/3lfanl2xnmx3gr2j11hc51a3m7sp7ljp-module-import-compiled.drv' failed with exit code 1
cannot build derivation `/gnu/store/asr76labrpqq0kcg64yni0sqsxj0d30y-rust-gtk-0.1.1.drv': 1 dependencies couldn't be built
guix build: error: build failed: build of `/gnu/store/asr76labrpqq0kcg64yni0sqsxj0d30y-rust-gtk-0.1.1.drv' failed

My log file doesn't contain any matching files (I added debugging "display" statements).

Source (for important parts search for the text "HERE"):

;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2016 David Craven <david@craven.ch>
;;;
;;; This file is part of GNU Guix.
;;;
;;; GNU Guix is free software; you can redistribute it and/or modify it
;;; under the terms of the GNU General Public License as published by
;;; the Free Software Foundation; either version 3 of the License, or (at
;;; your option) any later version.
;;;
;;; GNU Guix is distributed in the hope that it will be useful, but
;;; WITHOUT ANY WARRANTY; without even the implied warranty of
;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;;; GNU General Public License for more details.
;;;
;;; You should have received a copy of the GNU General Public License
;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.

(define-module (guix build cargo-build-system)
  #:use-module ((guix build gnu-build-system) #:prefix gnu:)
  #:use-module (guix build utils)
  #:use-module (guix hash)
  #:use-module (guix base32)
  #:use-module (ice-9 ftw)
  #:use-module (ice-9 format)
  #:use-module (ice-9 match)
  #:use-module (srfi srfi-1)
  #:use-module (srfi srfi-26)
  #:use-module (rnrs bytevectors)
  #:use-module (ice-9 binary-ports)
  #:use-module (system foreign)
  #:use-module ((guix build utils) #:select (dump-port))
  #:use-module (srfi srfi-11)
  #:use-module (srfi srfi-1)
  #:use-module (srfi srfi-34)
  #:use-module (srfi srfi-35)
  #:use-module (srfi srfi-60)
  #:use-module (rnrs bytevectors)
  #:use-module (ice-9 vlist)
  #:use-module (system foreign)
  #:export (%standard-phases
            cargo-build))

;; Commentary:
;;
;; Builder-side code of the standard Rust package build procedure.
;;
;; Code:

;; FIXME: Needs to be parsed from url not package name.
(define (package-name->crate-name name)
  "Return the crate name of NAME."
  (match (string-split name #\-)
    (("rust" rest ...)
     (string-join rest "-"))
    (_ #f)))

(define* (configure #:key inputs #:allow-other-keys)
  "Replace Cargo.toml [dependencies] section with guix inputs."
  (system* "chmod" "+w" "Cargo.toml")
  (system* "chmod" "+w" ".")
  (let ((port (open-file "Cargo.toml.disable" "a" #:encoding "utf-8")))
    (format port "~%[replace]~%")
    (for-each
     (match-lambda
       ((name . path)
        (let ((crate (package-name->crate-name name)))
          (when (and crate path)
            (match (string-split (basename path) #\-)
              ((_ ... version)
               (format port "\"~a:~a\" = { path = \"~a/rustsrc\" }~%"
                       crate version path)))))))
     inputs)
    (close-port port))
  #t)

(define* (build #:key (cargo-build-flags '("--release")) #:allow-other-keys)
  "Build a given Cargo package."
  (and (zero? (apply system* `("cargo" "build" ,@cargo-build-flags))))
       (zero? (system* "pwd"))
       (zero? (system* "ls")))

(define* (check #:key tests? #:allow-other-keys)
  "Run tests for a given Cargo package."
  (when tests?
    (zero? (system* "cargo" "test"))))

(define (generate-checksums dir-name)
  (let* ((file-names (find-files dir-name "."))
         (dir-prefix-name (string-append dir-name "/"))
         (dir-prefix-name-len (string-length dir-prefix-name))
         (checksums-file-name (string-append dir-name "/.cargo-checksum.json"))
         (checksums-temp-file-name (string-append checksums-file-name ".tmp"))
         (port (open-file checksums-temp-file-name "w")))
    (display "{\"files\":{")
    (for-each (lambda (file-name)
                (let ((file-relative-name (string-drop file-name dir-prefix-name-len)))
                  (write file-relative-name)
                  (display ":")
                  (write (bytevector->nix-base32-string (file-sha256 file-name))) ;   <========================================= HERE
                  (display ",") ; FIXME drop trailing comma?
                  (newline) ; FIXME remove
              )) file-names)
    (display "}, \"package\": ")
    ; FIXME add crate's sha256 checksum here
    (display "}")
    (newline)
    (close-port port)
    (rename-file checksums-temp-file-name checksums-file-name)))

(define* (install #:key inputs outputs #:allow-other-keys)
  "Install a given Cargo package."
  (let* ((out (assoc-ref outputs "out"))
         (src (assoc-ref inputs "source"))
         (bin (string-append out "/bin"))
         (rsrc (string-append out "/rustsrc")))
    (mkdir-p rsrc)
    ;; Rust doesn't have a stable ABI yet. Because of this
    ;; Cargo doesn't have a search path for binaries yet.
    ;; Until this changes we are working around this by
    ;; distributing crates as source and replacing
    ;; references in Cargo.toml with store paths.
    (system* "ls" "/gnu/store")
    (copy-recursively "src" (string-append rsrc "/src"))
    (install-file "Cargo.toml" rsrc)
    (system* "touch" (string-append rsrc "/.cargo-ok"))
    (generate-checksums rsrc)
    ;; When the package includes executables we install
    ;; it using cargo install. This fails when the crate
    ;; doesn't contain an executable.
    (system* "cargo" "install" "--root" out)
    #f))

(define %standard-phases
  ;; 'configure' phase is not needed.
  (modify-phases gnu:%standard-phases
    (replace 'configure configure)
    (replace 'build build)
    (replace 'check check)
    (replace 'install install)))

(define* (cargo-build #:key inputs (phases %standard-phases)
                      #:allow-other-keys #:rest args)
  "Build the given Cargo package, applying all of PHASES in order."
  (display "Source: ")
  (write (assoc-ref inputs "source"))
  (newline)
  (apply gnu:gnu-build #:inputs inputs #:phases phases args))

;;; cargo-build-system.scm ends here


;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2013, 2014, 2015, 2016 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2013 Andreas Enge <andreas@enge.fr>
;;; Copyright © 2013 Nikita Karetnikov <nikita@karetnikov.org>
;;; Copyright © 2016 David Craven <david@craven.ch>
;;;
;;; This file is part of GNU Guix.
;;;
;;; GNU Guix is free software; you can redistribute it and/or modify it
;;; under the terms of the GNU General Public License as published by
;;; the Free Software Foundation; either version 3 of the License, or (at
;;; your option) any later version.
;;;
;;; GNU Guix is distributed in the hope that it will be useful, but
;;; WITHOUT ANY WARRANTY; without even the implied warranty of
;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;;; GNU General Public License for more details.
;;;
;;; You should have received a copy of the GNU General Public License
;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.

(define-module (guix build-system cargo)
  #:use-module (guix search-paths)
  #:use-module (guix store)
  #:use-module (guix utils)
  #:use-module (guix derivations)
  #:use-module (guix packages)
  #:use-module (guix build-system)
  #:use-module (guix build-system gnu)
  #:use-module (ice-9 match)
  #:use-module (srfi srfi-26)
  #:export (cargo-build-system
            crate-url
            crate-url?
            crate-uri))

(define crate-url "https://crates.io/api/v1/crates/")
(define crate-url? (cut string-prefix? crate-url <>))

(define (crate-uri name version)
  "Return a URI string for the crate package hosted at crates.io corresponding
to NAME and VERSION."
  (string-append crate-url name "/" version "/download"))

(define (default-cargo)
  "Return the default Cargo package."
  ;; Lazily resolve the binding to avoid a circular dependency.
  (let ((rust (resolve-interface '(gnu packages rust))))
    (module-ref rust 'cargo)))

(define (default-rustc)
  "Return the default Rustc package."
  ;; Lazily resolve the binding to avoid a circular dependency.
  (let ((rust (resolve-interface '(gnu packages rust))))
    (module-ref rust 'rustc-bootstrap)))

(define (default-libgcrypt)
  "Return the default Libgcrypt package."
  ;; Lazily resolve the binding to avoid a circular dependency.
  (let ((gnupg (resolve-interface '(gnu packages gnupg))))
    (module-ref gnupg 'libgcrypt)))

(define %cargo-build-system-modules
  ;; Build-side modules imported by default.
  `((guix build cargo-build-system)
    (guix hash)
    (guix base32)
    (guix gcrypt)
    (guix config)
    ,@%gnu-build-system-modules))

(define* (cargo-build store name inputs
                      #:key
                      (tests? #t)
                      (test-target #f)
                      (configure-flags #f)
                      (phases '(@ (guix build cargo-build-system)
                                  %standard-phases))
                      (outputs '("out"))
                      (search-paths '())
                      (system (%current-system))
                      (guile #f)
                      (imported-modules %cargo-build-system-modules)
                      (modules '((guix build cargo-build-system)
                                 (guix build utils))))
  "Build SOURCE using CARGO, and with INPUTS."

  (define builder
    `(begin
       (use-modules ,@modules)
       (cargo-build #:name ,name
                    #:source ,(match (assoc-ref inputs "source")
                                (((? derivation? source))
                                 (derivation->output-path source))
                                ((source)
                                 source)
                                (source
                                 source))
                    #:system ,system
                    #:test-target ,test-target
                    #:tests? ,tests?
                    #:phases ,phases
                    #:outputs %outputs
                    #:search-paths ',(map search-path-specification->sexp
                                          search-paths)
                    #:inputs %build-inputs)))

  (define guile-for-build
    (match guile
      ((? package?)
       (package-derivation store guile system #:graft? #f))
      (#f                                         ; the default
       (let* ((distro (resolve-interface '(gnu packages commencement)))
              (guile  (module-ref distro 'guile-final)))
         (package-derivation store guile system #:graft? #f)))))

  (build-expression->derivation store name builder
                                #:inputs inputs
                                #:system system
                                #:modules imported-modules
                                #:outputs outputs
                                #:guile-for-build guile-for-build))

(define* (lower name
                #:key source inputs native-inputs outputs system target
                (cargo (default-cargo))
                (rustc (default-rustc))
                (libgcrypt (default-libgcrypt))
                #:allow-other-keys
                #:rest arguments)
  "Return a bag for NAME."

  (define private-keywords
    '(#:source #:target #:cargo #:rustc #:inputs #:native-inputs))

  (and (not target) ;; TODO: support cross-compilation
       (bag
         (name name)
         (system system)
         (target target)
         (host-inputs `(,@(if source
                              `(("source" ,source))
                              '())
                        ("libgcrypt" ,libgcrypt) ;   <========================================= HERE
                        ,@inputs
                        ;; Keep the standard inputs of 'gnu-build-system'
                        ,@(standard-packages)))
         (build-inputs `(("cargo" ,cargo)
                         ("rustc" ,rustc)
                         ("libgcrypt" ,libgcrypt) ;   <========================================= HERE
                         ,@native-inputs))
         (outputs outputs)
         (build (if target cargo-cross-build cargo-build))
         (arguments (strip-keyword-arguments private-keywords arguments)))))

(define cargo-build-system
  (build-system
    (name 'cargo)
    (description
     "Cargo build system, to build Rust crates")
    (lower lower)))

Re: Guix build side - including modules (guix hash), (guix gcrypt): How to provide shared library?

[David Craven]

I find it useful to look at the generated drv
(/gnu/store/asr76labrpqq0kcg64yni0sqsxj0d30y-rust-gtk-0.1.1.drv) and
build script (/gnu/store/hash-rust-gtk-0.1.1.builder) and see if those
make sense.

Sounds like a hard problem dough. Good luck! :)

I notice you haven't rebased yet, I added a couple of commits to the
cargo-build-system yesterday...

> Also, why is sha256 used via FFI via an external library? It's not exactly difficult to implement in Guile :P

I'm not sure we want to write our own crypto primitives... And it's
not quite as trivial, I don't know or care what a Newton-Raphson
algorithm is ;)
http://stackoverflow.com/questions/24093199/a-pure-scheme-implementation-r5rs-of-sha256

>  @dfeuer Not for the purposes of deriving the SHA-2 initialisation values. Most implementations use IEEE-754 doubles, which have only 53 bits of significand, not nearly enough for the 64-bit values used in the constants. I basically convert the doubles to rationals, then use Newton-Raphson to get the extra precision necessary. – Chris Jester-Young Jun 19 '14 at 6:10

> @dfeuer And now, on Mark Weaver's suggestion, I don't even use doubles at all, but instead use pure Newton-Raphson. It does make the startup slower, but that's a one-time cost. – Chris Jester-Young Aug 12 '14 at 15:26

Re: Guix build side - including modules (guix hash), (guix gcrypt): How to provide shared library?

[Ludovic Courtès]

David Craven <david@craven.ch> skribis:

>> Also, why is sha256 used via FFI via an external library? It's not exactly difficult to implement in Guile :P
>
> I'm not sure we want to write our own crypto primitives... And it's
> not quite as trivial, I don't know or care what a Newton-Raphson
> algorithm is ;)
> http://stackoverflow.com/questions/24093199/a-pure-scheme-implementation-r5rs-of-sha256

I suspect it would be slow with Guile 2.0, and (guix hash) is used in
performance-critical locations.

That said, if there’s a need for it on the build side where performance
does not matter much, it makes sense to add a pure-Scheme
implementation.  I think Weinhölt’s Industria library has one.

Alternately, we could make it easier to import (guix hash).  All we need
is to create a custom (guix config) module where ‘%libgcrypt’ is found
to the right thing, or something like that.

HTH!

Ludo’.