LLVM: "FileCheck" is missing

LLVM: "FileCheck" is missing

[Danny Milosavljevic]

Hi,

is anyone familiar with our LLVM packages? 

I'm trying to package rust and run into

  /gnu/store/pmiw7p7a6hwl6k2kblsk3zwhxh3gln21-llvm-3.6.2/bin/FileCheck is missing 

when configuring rust. Indeed, it is missing in the entire tree

  /gnu/store/pmiw7p7a6hwl6k2kblsk3zwhxh3gln21-llvm-3.6.2

.

It seems to be a tool for unit testing.

The rusts I'm trying to use are:

;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2012, 2013, 2014, 2015 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2014, 2015 Mark H Weaver <mhw@netris.org>
;;; Copyright © 2014, 2015, 2016 Ricardo Wurmus <rekado@elephly.net>
;;; Copyright © 2015 Andreas Enge <andreas@enge.fr>
;;; Copyright © 2015 Efraim Flashner <efraim@flashner.co.il>
;;;
;;; This file is part of GNU Guix.
;;;
;;; GNU Guix is free software; you can redistribute it and/or modify it
;;; under the terms of the GNU General Public License as published by
;;; the Free Software Foundation; either version 3 of the License, or (at
;;; your option) any later version.
;;;
;;; GNU Guix is distributed in the hope that it will be useful, but
;;; WITHOUT ANY WARRANTY; without even the implied warranty of
;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;;; GNU General Public License for more details.
;;;
;;; You should have received a copy of the GNU General Public License
;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.

(define-module (gnu packages rust)
  #:use-module ((guix licenses)
                #:select (asl2.0))
  #:use-module (gnu packages)
  #:use-module (gnu packages curl)
  #:use-module (gnu packages python)
  #:use-module (gnu packages llvm)
  #:use-module (guix packages)
  #:use-module (guix download)
  #:use-module (guix build-system gnu)
  #:use-module (guix build-system trivial)
  #:use-module (guix utils)
)

(define-public rust
  (package
    (name "rust")
    (version "1.9.0")
    (home-page "http://www.rust-lang.org/")
    (synopsis "Rust Programming Language")
    (description "Safe, Concurrent Programming Language")
    (license (list asl2.0)) ; FIXME and which MIT
    (build-system gnu-build-system) ; FIXME rust-build-system
    (inputs `(
; TODO llvm python jemalloc nacl musl
("llvm" ,llvm)

))
    (native-inputs `(
("curl" ,curl) ; FIXME remove
("python-2" ,python-2)
("llvm" ,llvm) ; 3.6 ??
; TODO ?
))
    (source
      (origin
         (method url-fetch)
         (uri "https://static.rust-lang.org/dist/2016-03-25/rustc-nightly-src.tar.gz")
         (sha256 (base32 "01sxhssllwqckmmzck6b17kaqqx622mwslwhjdrrkwjwa5qy7v54"))
         ;(uri (string-append "https://github.com/rust-lang/rust/archive/" version ".tar.gz"))
         ;(sha256 (base32 "1m2d1dc243s7ym8fq2yag3fr5jvki0q9c39llfwgcpq1gc8jvcn8"))
))
    (arguments
      `(#:configure-flags `( ;"-DCFG_BUILD=x86_64-unknown-linux-gnu"
                              ;"-disable-codegen-tests" ; because FileCheck is not built in llvm. https://github.com/alexcrichton/rust/commit/295a70f4b92d961c2e74cc750e47083e118ee10a
                             ) ; FIXME don't hard-code
        #:phases
          (modify-phases %standard-phases
            (replace 'configure
                     (lambda* (#:key inputs outputs configure-flags #:allow-other-keys)
                      (let ((out (assoc-ref outputs "out")))
                        (zero? (apply system*
                                      `("./configure"
                                        ,(string-append "--prefix=" out)
                                        ,(string-append "--llvm-root=" (assoc-ref inputs "llvm"))
                                        ; FIXME (string-append "--extra-ldflags=-Wl,-rpath-=" out "/lib")
                                        ,@configure-flags)))))))))

; FIXME remove src/llvm
;configure: CFG_LLVM_ROOT        :=  
;configure: CFG_PYTHON           :=  
;configure: CFG_JEMALLOC_ROOT    :=  
;configure: CFG_NACL_CROSS_PATH  :=  
;configure: CFG_MUSL_ROOT        := /usr/local 

))

Re: LLVM: "FileCheck" is missing

[Nils Gillmann]

Danny Milosavljevic <dannym@scratchpost.org> writes:

Hi Danny,

in my long journey to panopticon (https://panopticon.re) I am at
rust now.
Through what I tested and searched while I packaged rust, the
next release candidate of rust will have an option for us where
we can disable the llvm tests because they are optional for rust.

I don't have the links at hand, but there were open tickets on
this at rust.

I need to send out another message, I'll append my breaking
(new) rust.scm file to this thread later. Maybe we could
co-author this? I would be happy for some kind of workload
balance.

> Hi,
>
> is anyone familiar with our LLVM packages? 
>
> I'm trying to package rust and run into
>
>   /gnu/store/pmiw7p7a6hwl6k2kblsk3zwhxh3gln21-llvm-3.6.2/bin/FileCheck is missing 
>
> when configuring rust. Indeed, it is missing in the entire tree
>
>   /gnu/store/pmiw7p7a6hwl6k2kblsk3zwhxh3gln21-llvm-3.6.2
>
> .
>
> It seems to be a tool for unit testing.
>
> The rusts I'm trying to use are:
>
> ;;; GNU Guix --- Functional package management for GNU
> ;;; Copyright © 2012, 2013, 2014, 2015 Ludovic Courtès <ludo@gnu.org>
> ;;; Copyright © 2014, 2015 Mark H Weaver <mhw@netris.org>
> ;;; Copyright © 2014, 2015, 2016 Ricardo Wurmus <rekado@elephly.net>
> ;;; Copyright © 2015 Andreas Enge <andreas@enge.fr>
> ;;; Copyright © 2015 Efraim Flashner <efraim@flashner.co.il>
> ;;;
> ;;; This file is part of GNU Guix.
> ;;;
> ;;; GNU Guix is free software; you can redistribute it and/or modify it
> ;;; under the terms of the GNU General Public License as published by
> ;;; the Free Software Foundation; either version 3 of the License, or (at
> ;;; your option) any later version.
> ;;;
> ;;; GNU Guix is distributed in the hope that it will be useful, but
> ;;; WITHOUT ANY WARRANTY; without even the implied warranty of
> ;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> ;;; GNU General Public License for more details.
> ;;;
> ;;; You should have received a copy of the GNU General Public License
> ;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
>
> (define-module (gnu packages rust)
>   #:use-module ((guix licenses)
>                 #:select (asl2.0))
>   #:use-module (gnu packages)
>   #:use-module (gnu packages curl)
>   #:use-module (gnu packages python)
>   #:use-module (gnu packages llvm)
>   #:use-module (guix packages)
>   #:use-module (guix download)
>   #:use-module (guix build-system gnu)
>   #:use-module (guix build-system trivial)
>   #:use-module (guix utils)
> )
>
> (define-public rust
>   (package
>     (name "rust")
>     (version "1.9.0")
>     (home-page "http://www.rust-lang.org/")
>     (synopsis "Rust Programming Language")
>     (description "Safe, Concurrent Programming Language")
>     (license (list asl2.0)) ; FIXME and which MIT
>     (build-system gnu-build-system) ; FIXME rust-build-system
>     (inputs `(
> ; TODO llvm python jemalloc nacl musl
> ("llvm" ,llvm)
>
> ))
>     (native-inputs `(
> ("curl" ,curl) ; FIXME remove
> ("python-2" ,python-2)
> ("llvm" ,llvm) ; 3.6 ??
> ; TODO ?
> ))
>     (source
>       (origin
>          (method url-fetch)
>          (uri "https://static.rust-lang.org/dist/2016-03-25/rustc-nightly-src.tar.gz")
>          (sha256 (base32 "01sxhssllwqckmmzck6b17kaqqx622mwslwhjdrrkwjwa5qy7v54"))
>          ;(uri (string-append "https://github.com/rust-lang/rust/archive/" version ".tar.gz"))
>          ;(sha256 (base32 "1m2d1dc243s7ym8fq2yag3fr5jvki0q9c39llfwgcpq1gc8jvcn8"))
> ))
>     (arguments
>       `(#:configure-flags `( ;"-DCFG_BUILD=x86_64-unknown-linux-gnu"
>                               ;"-disable-codegen-tests" ; because FileCheck is not built in llvm. https://github.com/alexcrichton/rust/commit/295a70f4b92d961c2e74cc750e47083e118ee10a
>                              ) ; FIXME don't hard-code
>         #:phases
>           (modify-phases %standard-phases
>             (replace 'configure
>                      (lambda* (#:key inputs outputs configure-flags #:allow-other-keys)
>                       (let ((out (assoc-ref outputs "out")))
>                         (zero? (apply system*
>                                       `("./configure"
>                                         ,(string-append "--prefix=" out)
>                                         ,(string-append "--llvm-root=" (assoc-ref inputs "llvm"))
>                                         ; FIXME (string-append "--extra-ldflags=-Wl,-rpath-=" out "/lib")
>                                         ,@configure-flags)))))))))
>
> ; FIXME remove src/llvm
> ;configure: CFG_LLVM_ROOT        :=  
> ;configure: CFG_PYTHON           :=  
> ;configure: CFG_JEMALLOC_ROOT    :=  
> ;configure: CFG_NACL_CROSS_PATH  :=  
> ;configure: CFG_MUSL_ROOT        := /usr/local 
>
> ))
>
>

-- 
ng
personal contact: http://krosos.sdf.org
EDN: https://wiki.c3d2.de/EDN

rust work in progress conflicts (was: Re: LLVM: "FileCheck" is missing)

[Nils Gillmann]

Here is my work in progress rust.scm, it is on pause until the
next release of rust as mentioned before:


~/projects/guix_project/guix/gnu/packages $ cat rust.scm
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2016 Nils Gillmann <niasterisk@grrlz.net>
;;;
;;; This file is part of GNU Guix.
;;;
;;; GNU Guix is free software; you can redistribute it and/or modify it
;;; under the terms of the GNU General Public License as published by
;;; the Free Software Foundation; either version 3 of the License, or (at
;;; your option) any later version.
;;;
;;; GNU Guix is distributed in the hope that it will be useful, but
;;; WITHOUT ANY WARRANTY; without even the implied warranty of
;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;;; GNU General Public License for more details.
;;;
;;; You should have received a copy of the GNU General Public License
;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.

(define-module (gnu packages rust)
  #:use-module ((guix licenses) #:prefix license:)
  #:use-module (guix packages)
  #:use-module (guix download)
  #:use-module (guix git-download)
  #:use-module (guix utils)
  #:use-module (guix build-system gnu)
  ;; #:use-module (gnu packages gcc)
  #:use-module (gnu packages perl)
  #:use-module (gnu packages libffi)
  #:use-module (gnu packages valgrind)
  #:use-module (gnu packages version-control)
  #:use-module (gnu packages curl)
  #:use-module (gnu packages python)
  #:use-module (gnu packages llvm))

;;(list
;; (string-append
;; "https://github.com/rust-lang/rust/archive/"
;; version ".tar.gz")
;;gh "1m2d1dc243s7ym8fq2yag3fr5jvki0q9c39llfwgcpq1gc8jvcn8"))))

(define-public rustc
  (package
    (name "rustc")
    (version "1.7.0")
    (source (origin
              (method url-fetch)
              (uri (string-append
                    "https://static.rust-lang.org/dist/" name "-"
                    version "-src.tar.gz"))
              (file-name (string-append name "-" version ".tar.gz"))
              (sha256
               (base32
                "0fpiggrnvdmmnp6b95ik16v1h972wir9d1ydv5v8cwbvv1cn1ybd"))))
    (build-system gnu-build-system)
    (arguments
     `(#:phases
       (alist-replace
        'configure
        (lambda* (#:key outputs #:allow-other-keys)
          ;; This old `configure' script doesn't support
          ;; variables passed as arguments.
          (let ((out (assoc-ref outputs "out"))
                (llvm (assoc-ref %build-inputs "llvm")))
            (setenv "CONFIG_SHELL" (which "bash"))
            (zero?
             (system* "./configure"
                      (string-append "--prefix=" out)
                      (string-append "--llvm-root=" llvm)))))
        %standard-phases)))
       ;; #:configure-flags
       ;; (list
       ;;  (string-append "--llvm-root="
       ;;                 (assoc-ref %build-inputs "llvm")))))
    (inputs
     `(("python-2" ,python-2)
       ("curl" ,curl)
       ("git" ,git)
       ("valgrind" ,valgrind)
       ("libffi" ,libffi)
       ("perl" ,perl)
       ("llvm" ,llvm)))
    (home-page "https://www.rust-lang.org/")
    (synopsis
     "The Rust Programming Language")
    (description
     "LOREM IPSUM BLA")
    (license license:gpl3+)))
-- 
ng
personal contact: http://krosos.sdf.org
EDN: https://wiki.c3d2.de/EDN

Re: rust work in progress conflicts (was: Re: LLVM: "FileCheck" is missing)

[Jelle Licht]

I have taken the liberty to try my hand at finishing this, as I figured
it would be a good way for me to get more familiar with 'the Guix way'
of packaging things.

Wow, did I misjudge this rabbit hole though. It seems to be the case that
rust needs the (most recent) snapshotted binary stage-0 compiler as part
of the build process. This was not the case some years ago[1], but since
then, some 319 snapshots have been released.

Now there are two approaches which might make sense to me:

1) We package a recent stage-0 binary (thus adding yet another random
binary to the mix)

2) We bootstrap all the way from the original rust compiler, written in
ocaml. This would then presumably need to be repeated for each snapshot,
leading to about 319 iterative compiler build. On my kind-of-okay i7,
compiling a single rust iteration takes about 25 to 40 minutes.

I tentatively went with option 1, if only because I would like to see
results this decade still, and ran into several hurdles that became
quite manageable with help from the good people of #guix and
#rust-internals. One more issue yet remains: part of the rust
compilation process actually calls the 'cc linker'. This part does not
respect make flags, setenv calls or even rust's special configure flag
for setting cc.

Option 1 does not seem feasible at this point of time, but there is some
light at the end of the tunnel: rust is at some point going to follow a
convention that will allow bootstrapping compilers via 'master from
beta, beta from stable and stable from previous stable'[2].

I am currently thinking of a compromise; basically, at this moment go
for option 1, and once the policy previously described is properly
implemented by the rust team, start iteratively bootstrapping rust from
that point in time.


tldr: If we can get 'cc' in the build environment, we can have a 'dirty'
bootstrapped rust very soon. If we want to do it properly, it might take
a lot longer.  

WDYT?

[1]: https://news.ycombinator.com/item?id=8732669
[2]: https://botbot.me/mozilla/rust-internals/2016-04-29/?page=3, look
for eddyb

Nils Gillmann <niasterisk@grrlz.net> writes:

> Here is my work in progress rust.scm, it is on pause until the
> next release of rust as mentioned before:
>
>
> ~/projects/guix_project/guix/gnu/packages $ cat rust.scm
> ;;; GNU Guix --- Functional package management for GNU
> ;;; Copyright © 2016 Nils Gillmann <niasterisk@grrlz.net>
> ;;;
> ;;; This file is part of GNU Guix.
> ;;;
> ;;; GNU Guix is free software; you can redistribute it and/or modify it
> ;;; under the terms of the GNU General Public License as published by
> ;;; the Free Software Foundation; either version 3 of the License, or (at
> ;;; your option) any later version.
> ;;;
> ;;; GNU Guix is distributed in the hope that it will be useful, but
> ;;; WITHOUT ANY WARRANTY; without even the implied warranty of
> ;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> ;;; GNU General Public License for more details.
> ;;;
> ;;; You should have received a copy of the GNU General Public License
> ;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
>
> (define-module (gnu packages rust)
>   #:use-module ((guix licenses) #:prefix license:)
>   #:use-module (guix packages)
>   #:use-module (guix download)
>   #:use-module (guix git-download)
>   #:use-module (guix utils)
>   #:use-module (guix build-system gnu)
>   ;; #:use-module (gnu packages gcc)
>   #:use-module (gnu packages perl)
>   #:use-module (gnu packages libffi)
>   #:use-module (gnu packages valgrind)
>   #:use-module (gnu packages version-control)
>   #:use-module (gnu packages curl)
>   #:use-module (gnu packages python)
>   #:use-module (gnu packages llvm))
>
> ;;(list
> ;; (string-append
> ;; "https://github.com/rust-lang/rust/archive/"
> ;; version ".tar.gz")
> ;;gh "1m2d1dc243s7ym8fq2yag3fr5jvki0q9c39llfwgcpq1gc8jvcn8"))))
>
> (define-public rustc
>   (package
>     (name "rustc")
>     (version "1.7.0")
>     (source (origin
>               (method url-fetch)
>               (uri (string-append
>                     "https://static.rust-lang.org/dist/" name "-"
>                     version "-src.tar.gz"))
>               (file-name (string-append name "-" version ".tar.gz"))
>               (sha256
>                (base32
>                 "0fpiggrnvdmmnp6b95ik16v1h972wir9d1ydv5v8cwbvv1cn1ybd"))))
>     (build-system gnu-build-system)
>     (arguments
>      `(#:phases
>        (alist-replace
>         'configure
>         (lambda* (#:key outputs #:allow-other-keys)
>           ;; This old `configure' script doesn't support
>           ;; variables passed as arguments.
>           (let ((out (assoc-ref outputs "out"))
>                 (llvm (assoc-ref %build-inputs "llvm")))
>             (setenv "CONFIG_SHELL" (which "bash"))
>             (zero?
>              (system* "./configure"
>                       (string-append "--prefix=" out)
>                       (string-append "--llvm-root=" llvm)))))
>         %standard-phases)))
>        ;; #:configure-flags
>        ;; (list
>        ;;  (string-append "--llvm-root="
>        ;;                 (assoc-ref %build-inputs "llvm")))))
>     (inputs
>      `(("python-2" ,python-2)
>        ("curl" ,curl)
>        ("git" ,git)
>        ("valgrind" ,valgrind)
>        ("libffi" ,libffi)
>        ("perl" ,perl)
>        ("llvm" ,llvm)))
>     (home-page "https://www.rust-lang.org/")
>     (synopsis
>      "The Rust Programming Language")
>     (description
>      "LOREM IPSUM BLA")
>     (license license:gpl3+)))

Re: rust work in progress conflicts

[Ludovic Courtès]

Jelle Licht <jlicht@fsfe.org> skribis:

> Now there are two approaches which might make sense to me:
>
> 1) We package a recent stage-0 binary (thus adding yet another random
> binary to the mix)

This is something we want to avoid as much as possible.  Part of the
solution is to raise awareness about the security and freedom issues
that arise from the use of binary blobs to build compilers.

This has been discussed earlier:

  https://lists.gnu.org/archive/html/guix-devel/2016-03/msg00828.html

Based on this discussion…

> 2) We bootstrap all the way from the original rust compiler, written in
> ocaml. This would then presumably need to be repeated for each snapshot,
> leading to about 319 iterative compiler build. On my kind-of-okay i7,
> compiling a single rust iteration takes about 25 to 40 minutes.

… I strongly encourage you to try and take this route.

Hopefully, not every version of Rust needs to be compiled with exactly
the previous version of itself.

Anyway, your experience in trying this will be invaluable!

> I tentatively went with option 1, if only because I would like to see
> results this decade still, and ran into several hurdles that became
> quite manageable with help from the good people of #guix and
> #rust-internals. One more issue yet remains: part of the rust
> compilation process actually calls the 'cc linker'. This part does not
> respect make flags, setenv calls or even rust's special configure flag
> for setting cc.

At worst, you can always add a ‘cc’ script that does:

  #!$SHELL
  exec gcc "$@"

in $PATH, in a pre-build phase.

> Option 1 does not seem feasible at this point of time, but there is some
> light at the end of the tunnel: rust is at some point going to follow a
> convention that will allow bootstrapping compilers via 'master from
> beta, beta from stable and stable from previous stable'[2].
>
> I am currently thinking of a compromise; basically, at this moment go
> for option 1, and once the policy previously described is properly
> implemented by the rust team, start iteratively bootstrapping rust from
> that point in time.

Yeah it’s OK to do it in the order: write the package bootstrapped from
the binary blob, and then work on bootstrapping it from OCaml.

Thanks for looking into it!

Ludo’.

Re: rust work in progress conflicts

[Alex Griffin]

On Thu, May 5, 2016, at 08:35 AM, Ludovic Courtès wrote:
> 
> Yeah it’s OK to do it in the order: write the package bootstrapped from
> the binary blob, and then work on bootstrapping it from OCaml.

I do not think that bootstrapping Rust from OCaml is a practical goal.
When you guys have talked about bootstrapping gcc from an older version,
I don't think it has involved even a dozen gcc versions, let alone 319!
It's a huge amount of work, to the point where it may even be easier to
write a new Rust compiler from scratch. Even if it succeeded, it would
take over a full week to build the current Rust release, meanwhile OCaml
requires a blob anyway.

I'd love to be proven wrong, but in any case I don't think it's
reasonable to expect Jelle Licht to take on this project just because he
assumed the mantle of rustc packager.
-- 
Alex Griffin

Re: rust work in progress conflicts

[Andy Wingo]

On Thu 05 May 2016 16:46, Alex Griffin <a@ajgrf.com> writes:

> On Thu, May 5, 2016, at 08:35 AM, Ludovic Courtès wrote:
>> 
>> Yeah it’s OK to do it in the order: write the package bootstrapped from
>> the binary blob, and then work on bootstrapping it from OCaml.
>
> I do not think that bootstrapping Rust from OCaml is a practical goal.
> When you guys have talked about bootstrapping gcc from an older version,
> I don't think it has involved even a dozen gcc versions, let alone 319!
> It's a huge amount of work, to the point where it may even be easier to
> write a new Rust compiler from scratch. Even if it succeeded, it would
> take over a full week to build the current Rust release, meanwhile OCaml
> requires a blob anyway.
>
> I'd love to be proven wrong, but in any case I don't think it's
> reasonable to expect Jelle Licht to take on this project just because he
> assumed the mantle of rustc packager.

I agree with Alex FWIW.

Also FWIW, I think in the future a sane bootstrap is probably more
likely from a MIR interpreter (something like this but implemented from
another language: https://github.com/tsion/miri/tree/master/src).

But given that even OCaml includes a binary blob (!), I don't think that
we can practically require bootstrapped language implementations to
bootstrap all the way.

Andy

Re: rust work in progress conflicts

[Andy Wingo]

On Fri 06 May 2016 11:05, Andy Wingo <wingo@igalia.com> writes:

> But given that even OCaml includes a binary blob (!), I don't think that
> we can practically require bootstrapped language implementations to
> bootstrap all the way.

To clarify!  Obviously it would be better from a reproducibility POV if
we had fewer root binary blobs.  I don't mean to argue against that.
Just that the tradeoff between "include OCaml and its world" and "always
bootstrap only from GCC" in the short term leans towards inclusion
rather than purity.  Just my POV though.

A

Re: rust work in progress conflicts

[Ludovic Courtès]

Alex Griffin <a@ajgrf.com> skribis:

> On Thu, May 5, 2016, at 08:35 AM, Ludovic Courtès wrote:
>> 
>> Yeah it’s OK to do it in the order: write the package bootstrapped from
>> the binary blob, and then work on bootstrapping it from OCaml.
>
> I do not think that bootstrapping Rust from OCaml is a practical goal.
> When you guys have talked about bootstrapping gcc from an older version,
> I don't think it has involved even a dozen gcc versions, let alone 319!

If it really has to be 319 versions of Rust, then I agree.
I thought/hoped that it would in fact be just a few of them.

Andy Wingo <wingo@igalia.com> skribis:

> Also FWIW, I think in the future a sane bootstrap is probably more
> likely from a MIR interpreter (something like this but implemented from
> another language: https://github.com/tsion/miri/tree/master/src).

Yeah, though MIR it not source (not the “preferred form”), AIUI.

Thanks,
Ludo’.

Re: rust work in progress conflicts

[ng0]

Jelle Licht <jlicht@fsfe.org> writes:

> I have taken the liberty to try my hand at finishing this, as I figured
> it would be a good way for me to get more familiar with 'the Guix way'
> of packaging things.

Thanks!
Also, could you please use this email address for further CC
things (I tend to avoid them completely if possible), as the
@grrlz.net is no longer subscribed to the mailinglist afaik.

> Wow, did I misjudge this rabbit hole though. It seems to be the case that
> rust needs the (most recent) snapshotted binary stage-0 compiler as part
> of the build process. This was not the case some years ago[1], but since
> then, some 319 snapshots have been released.
>
> Now there are two approaches which might make sense to me:
>
> 1) We package a recent stage-0 binary (thus adding yet another random
> binary to the mix)
>
> 2) We bootstrap all the way from the original rust compiler, written in
> ocaml. This would then presumably need to be repeated for each snapshot,
> leading to about 319 iterative compiler build. On my kind-of-okay i7,
> compiling a single rust iteration takes about 25 to 40 minutes.

This sounds expensive. Isn't there a chance to speed this up,
some other developer with a small Cluster of CPUs for computing
at hand?

> I tentatively went with option 1, if only because I would like to see
> results this decade still, and ran into several hurdles that became
> quite manageable with help from the good people of #guix and
> #rust-internals. One more issue yet remains: part of the rust
> compilation process actually calls the 'cc linker'. This part does not
> respect make flags, setenv calls or even rust's special configure flag
> for setting cc.
>
> Option 1 does not seem feasible at this point of time, but there is some
> light at the end of the tunnel: rust is at some point going to follow a
> convention that will allow bootstrapping compilers via 'master from
> beta, beta from stable and stable from previous stable'[2].
>
> I am currently thinking of a compromise; basically, at this moment go
> for option 1, and once the policy previously described is properly
> implemented by the rust team, start iteratively bootstrapping rust from
> that point in time.
>
>
> tldr: If we can get 'cc' in the build environment, we can have a 'dirty'
> bootstrapped rust very soon. If we want to do it properly, it might take
> a lot longer.  
>
> WDYT?
>
> [1]: https://news.ycombinator.com/item?id=8732669
> [2]: https://botbot.me/mozilla/rust-internals/2016-04-29/?page=3, look
> for eddyb

It's good that there seems to be light at the end, though I did
not expect rust to be that challenging when I started with it to
just package panopticon[0] through cargo import which needs to be
written once rust is done.


[0]: https://panopticon.re

>>---snip-snap----

-- 
♥Ⓐ ng0

Re: rust work in progress conflicts

[ng0]

Hi,

ng0 <ng0@libertad.pw> writes:

> Jelle Licht <jlicht@fsfe.org> writes:
>
>> I have taken the liberty to try my hand at finishing this, as I figured
>> it would be a good way for me to get more familiar with 'the Guix way'
>> of packaging things.
>
> Thanks!
> Also, could you please use this email address for further CC
> things (I tend to avoid them completely if possible), as the
> @grrlz.net is no longer subscribed to the mailinglist afaik.
>
>> Wow, did I misjudge this rabbit hole though. It seems to be the case that
>> rust needs the (most recent) snapshotted binary stage-0 compiler as part
>> of the build process. This was not the case some years ago[1], but since
>> then, some 319 snapshots have been released.
>>
>> Now there are two approaches which might make sense to me:
>>
>> 1) We package a recent stage-0 binary (thus adding yet another random
>> binary to the mix)
>>
>> 2) We bootstrap all the way from the original rust compiler, written in
>> ocaml. This would then presumably need to be repeated for each snapshot,
>> leading to about 319 iterative compiler build. On my kind-of-okay i7,
>> compiling a single rust iteration takes about 25 to 40 minutes.
>
> This sounds expensive. Isn't there a chance to speed this up,
> some other developer with a small Cluster of CPUs for computing
> at hand?
>
>> I tentatively went with option 1, if only because I would like to see
>> results this decade still, and ran into several hurdles that became
>> quite manageable with help from the good people of #guix and
>> #rust-internals. One more issue yet remains: part of the rust
>> compilation process actually calls the 'cc linker'. This part does not
>> respect make flags, setenv calls or even rust's special configure flag
>> for setting cc.
>>
>> Option 1 does not seem feasible at this point of time, but there is some
>> light at the end of the tunnel: rust is at some point going to follow a
>> convention that will allow bootstrapping compilers via 'master from
>> beta, beta from stable and stable from previous stable'[2].
>>
>> I am currently thinking of a compromise; basically, at this moment go
>> for option 1, and once the policy previously described is properly
>> implemented by the rust team, start iteratively bootstrapping rust from
>> that point in time.
>>
>>
>> tldr: If we can get 'cc' in the build environment, we can have a 'dirty'
>> bootstrapped rust very soon. If we want to do it properly, it might take
>> a lot longer.  
>>
>> WDYT?
>>
>> [1]: https://news.ycombinator.com/item?id=8732669
>> [2]: https://botbot.me/mozilla/rust-internals/2016-04-29/?page=3, look
>> for eddyb
>
> It's good that there seems to be light at the end, though I did
> not expect rust to be that challenging when I started with it to
> just package panopticon[0] through cargo import which needs to be
> written once rust is done.
>
>
> [0]: https://panopticon.re
>
>>>---snip-snap----
>
> -- 
> ♥Ⓐ ng0
>

So I picked up rust.scm again and forgot about this thread, only a
search for rust brought it up again.
As this will be a long task obviously, however we finish it, can we file
a bug on it so it is obvious that work is being done on it and there'll
be no dual work on this?

Recently released version 1.10.0 of rust merged the
"--disable-codegen-tests" I needed back when i worked on it.
-- 
♥Ⓐ  ng0
Current Keys: https://we.make.ritual.n0.is/ng0.txt
For non-prism friendly talk find me on http://www.psyced.org

Re: rust work in progress conflicts

[Eric Le Bihan]

[-- Attachment #1: Type: text/plain, Size: 1373 bytes --]

Hi ng0, all!

Le Thu, 28 Jul 2016 08:28:18 +0000,
ng0 <ng0@we.make.ritual.n0.is> a écrit :

> ng0 <ng0@libertad.pw> writes:
[...]
>
> So I picked up rust.scm again and forgot about this thread, only a
> search for rust brought it up again.
> As this will be a long task obviously, however we finish it, can we
> file a bug on it so it is obvious that work is being done on it and
> there'll be no dual work on this?
> 
> Recently released version 1.10.0 of rust merged the
> "--disable-codegen-tests" I needed back when i worked on it.

It happens that I tried to package Rust, as an introduction to Guix.
Here is my version, inspired by the Haskell package, where Rust 1.10.0
is built, bootstrapped by a binary version of rustc 1.9.0. It uses the
"cc" wrapper trick previously presented.

Some questions, though:

1. I can compile a sample program in a guix environment created using
`guix environment gcc glibc binutils rust`, but the program
generated fails to run because libgcc_s.so.1 can not be found. How can
it be added to the environment?

2. Having a Rust compiler is cool, but having Cargo, the Rust package
manager, would be even better. Cargo is also bootstrapped, and it is
also built using zillions of crates (Rust packages) downloaded from the
Internet. How could this case be handled in Guix?

Best regards,

-- 
ELB

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-Add-rust.patch --]
[-- Type: text/x-patch, Size: 6884 bytes --]

From fb1fbc92cd68331b3dea94c238274f8a01b98afa Mon Sep 17 00:00:00 2001
From: Eric Le Bihan <eric.le.bihan.dev@free.fr>
Date: Thu, 28 Jul 2016 20:09:01 +0200
Subject: [PATCH 1/1] gnu: Add rust

* gnu/packages/rust.scm(rust): New variable.

Signed-off-by: Eric Le Bihan <eric.le.bihan.dev@free.fr>
---
 gnu/packages/rust.scm | 141 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 141 insertions(+)
 create mode 100644 gnu/packages/rust.scm

diff --git a/gnu/packages/rust.scm b/gnu/packages/rust.scm
new file mode 100644
index 0000000..6c6ec0c
--- /dev/null
+++ b/gnu/packages/rust.scm
@@ -0,0 +1,141 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2016 Eric Le Bihan <eric.le.bihan.dev@free.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu packages rust)
+  #:use-module (ice-9 regex)
+  #:use-module ((guix licenses) #:select (asl2.0 x11-style))
+  #:use-module (guix packages)
+  #:use-module (guix download)
+  #:use-module (guix build-system gnu)
+  #:use-module (gnu packages base)
+  #:use-module (gnu packages bootstrap)
+  #:use-module (gnu packages curl)
+  #:use-module (gnu packages elf)
+  #:use-module (gnu packages gcc)
+  #:use-module (gnu packages jemalloc)
+  #:use-module (gnu packages perl)
+  #:use-module (gnu packages python))
+
+(define rust-bootstrap-x86_64-1.9.0
+  (origin
+   (method url-fetch)
+   (uri
+    "https://static.rust-lang.org/dist/2016-05-24/rustc-1.9.0-x86_64-unknown-linux-gnu.tar.gz")
+   (sha256
+    (base32
+     "1i44rlvvn3pr81sli6bdbkzd78ar1ibybxx6mzpw6rkw4c84sw6h"))))
+
+(define rust-bootstrap-i686-1.9.0
+  (origin
+   (method url-fetch)
+   (uri
+    "https://static.rust-lang.org/dist/2016-05-24/rustc-1.9.0-i686-unknown-linux-gnu.tar.gz")
+   (sha256
+    (base32
+     "0fdf5xvh3g4hdza0y80w1r9vnfczjqnbzbvs7k878yc26p4dwl99"))))
+
+(define-public rust
+  (package
+   (name "rust")
+   (version "1.10.0")
+   (source (origin
+            (method url-fetch)
+            (uri (string-append "https://static.rust-lang.org/dist/rustc-"
+                                version "-src.tar.gz"))
+            (sha256
+             (base32
+              "0sb82xb8y2pzs8l1hk91z228bambwx3dmi2kj8isin7nyjn5l0d4"))))
+   (build-system gnu-build-system)
+   (native-inputs
+    `(("curl" ,curl)
+      ("gcc" ,gcc)
+      ("gcc-lib" ,gcc "lib")
+      ("jemalloc" ,jemalloc)
+      ("patchelf" ,patchelf)
+      ("perl" ,perl)
+      ("python" ,python-2)
+      ("rust-bootstrap"
+       ,(if (string-match "x86_64" (or (%current-target-system) (%current-system)))
+            rust-bootstrap-x86_64-1.9.0
+            rust-bootstrap-i686-1.9.0))
+      ("which" ,which)))
+   (arguments
+    `(#:phases
+      (modify-phases %standard-phases
+        (add-after 'unpack 'unpack-bootstrap
+                   (lambda* (#:key inputs #:allow-other-keys)
+                     (with-directory-excursion (getcwd)
+                       (zero? (system*
+                               "tar"
+                               "--strip-components=1"
+                               "-xzf"
+                               (assoc-ref inputs "rust-bootstrap"))))))
+        (replace 'configure
+                 (lambda* (#:key inputs outputs #:allow-other-keys)
+                   (let ((out (assoc-ref outputs "out"))
+                         (binutils (assoc-ref inputs "binutils"))
+                         (gcc (assoc-ref inputs "gcc"))
+                         (gcc-lib (assoc-ref inputs "gcc-lib"))
+                         (jemalloc (assoc-ref inputs "jemalloc"))
+                         (ld-so (string-append
+                                 (assoc-ref inputs "libc")
+                                 ,(glibc-dynamic-linker)))
+                         (python (assoc-ref inputs "python")))
+            (setenv "SHELL" (which "sh"))
+            (setenv "CONFIG_SHELL" (which "sh"))
+            ;; Tell where to find libgcc_s.so
+            (setenv "LD_LIBRARY_PATH" (string-append gcc-lib "/lib"))
+            ;; Remove reference to "/lib64/ld-linux-x86-64.so.2" from binary
+            (zero? (system*
+                    "patchelf"
+                    "--set-interpreter" ld-so
+                    (string-append (getcwd) "/rustc/bin/rustc")))
+            (zero? (system*
+                    "./configure"
+                    (string-append "--prefix=" out)
+                    (string-append "--default-linker=" gcc "/bin/gcc")
+                    (string-append "--default-ar=" binutils "/bin/ar")
+                    (string-append "--jemalloc-root=" jemalloc "/lib")
+                    (string-append "--enable-rpath")
+                    (string-append "--enable-local-rust")
+                    (string-append "--local-rust-root=" (getcwd) "/rustc")
+                    (string-append "--python=" python "/bin/python2")
+                    "--disable-manage-submodules")))))
+        (add-before 'build 'pre-build
+                    (lambda _
+                      (let* ((bindir (string-append (getcwd) "/bin"))
+                             (cc (string-append bindir "/cc")))
+                        (mkdir bindir)
+                        (call-with-output-file cc
+                          (lambda (port)
+                            (format port
+                                    "#!~a\n\nexec gcc \"$@\"\n" (which "sh"))))
+                        (chmod cc #o755))))
+      (replace 'build
+               (lambda* (#:key outputs #:allow-other-keys)
+                 (setenv "PATH"
+                         (string-append (getcwd) "/bin:" (getenv "PATH")))
+                 (mkdir (assoc-ref outputs "out"))
+                 (zero? (system* "make")))))
+      #:tests? #f))
+   (synopsis "Compiler for the Rust progamming language")
+   (description
+    "Rust is a systems programming language that runs blazingly fast, prevents
+ segfaults, and guarantees thread safety.")
+   (home-page "https://www.rust-lang.org")
+   (license (list asl2.0 (x11-style "file://LICENSE-MIT")))))
-- 
2.4.11

Re: rust work in progress conflicts

[Andreas Enge]

Hello,

On Thu, Jul 28, 2016 at 08:31:44PM +0200, Eric Le Bihan wrote:
> 1. I can compile a sample program in a guix environment created using
> `guix environment gcc glibc binutils rust`, but the program
> generated fails to run because libgcc_s.so.1 can not be found. How can
> it be added to the environment?

I would suggest to replace "gcc glibc binutils" by "gcc-toolchain", which
installs a specially modified linker.

Andreas

Re: rust work in progress conflicts

[Vincent Legoll]

> On Thu, Jul 28, 2016 at 08:31:44PM +0200, Eric Le Bihan wrote:
>> 1. I can compile a sample program in a guix environment created using
>> `guix environment gcc glibc binutils rust`, but the program
>> generated fails to run because libgcc_s.so.1 can not be found. How can
>> it be added to the environment?
>
> I would suggest to replace "gcc glibc binutils" by "gcc-toolchain", which
> installs a specially modified linker.

Looks like I'm not alone doing the gcc mistake. Maybe this should be explained
in the doc.

WDYT

-- 
Vincent Legoll

Re: rust work in progress conflicts

[ng0]

Vincent Legoll <vincent.legoll@gmail.com> writes:

>> On Thu, Jul 28, 2016 at 08:31:44PM +0200, Eric Le Bihan wrote:
>>> 1. I can compile a sample program in a guix environment created using
>>> `guix environment gcc glibc binutils rust`, but the program
>>> generated fails to run because libgcc_s.so.1 can not be found. How can
>>> it be added to the environment?
>>
>> I would suggest to replace "gcc glibc binutils" by "gcc-toolchain", which
>> installs a specially modified linker.
>
> Looks like I'm not alone doing the gcc mistake. Maybe this should be explained
> in the doc.
>
> WDYT
>
> -- 
> Vincent Legoll
>

Yes, definitely from what I read and watched in the time involved here
and my own experiences.
Can you try and come up with an addition?

Thanks.
-- 
♥Ⓐ  ng0
Current Keys: https://we.make.ritual.n0.is/ng0.txt
For non-prism friendly talk find me on http://www.psyced.org

Re: rust work in progress conflicts

[Eric Le Bihan]

Le Fri, 29 Jul 2016 11:03:41 +0200,
Andreas Enge <andreas@enge.fr> a écrit :

> On Thu, Jul 28, 2016 at 08:31:44PM +0200, Eric Le Bihan wrote:
> > 1. I can compile a sample program in a guix environment created
> > using `guix environment gcc glibc binutils rust`, but the program
> > generated fails to run because libgcc_s.so.1 can not be found. How
> > can it be added to the environment?  
> 
> I would suggest to replace "gcc glibc binutils" by "gcc-toolchain",
> which installs a specially modified linker.

This solves the problem. Thank you.

-- 
ELB

Rust

[Ludovic Courtès]

Hello,

Eric Le Bihan <eric.le.bihan.dev@free.fr> skribis:

> It happens that I tried to package Rust, as an introduction to Guix.
> Here is my version, inspired by the Haskell package, where Rust 1.10.0
> is built, bootstrapped by a binary version of rustc 1.9.0. It uses the
> "cc" wrapper trick previously presented.

As you might have seen from previous discussions, we try hard to reduce
the number of binary blobs needed to bootstrap packages, such that we
have a full source-to-binary path that everyone can audit.
Unfortunately, this usually hard to achieve for self-hosted compilers.

Do you know what’s Rust’s bootstrapping story is?  Can we reasonably
expect to bootstrap it from source, using a series of previous Rust
versions, or using an alternative implementation?

> Some questions, though:
>
> 1. I can compile a sample program in a guix environment created using
> `guix environment gcc glibc binutils rust`, but the program
> generated fails to run because libgcc_s.so.1 can not be found. How can
> it be added to the environment?

As Andreas notes, ‘gcc-toolchain’, which includes ‘ld-wrapper’, should
fix this.

Does Rust use GCC, or just ld?

> 2. Having a Rust compiler is cool, but having Cargo, the Rust package
> manager, would be even better. Cargo is also bootstrapped, and it is
> also built using zillions of crates (Rust packages) downloaded from the
> Internet. How could this case be handled in Guix?

Assuming Cargo itself is just a regular Rust program, it should be
possible to make a Guix package of Cargo.  Then, Guix users can install
it and use it the normal way; we won’t be able to use Cargo in package
recipes though, because our package build environments purposefully
lacks network access.

Besides, I would encourage you or anyone interested to write a crate
importer, like we do for most other language packages:

  https://www.gnu.org/software/guix/manual/html_node/Invoking-guix-import.html

Having crates available as normal Guix packages is the best option for
Guix users: uniform interface, the ability to use ‘guix environment’ and
all the tools, transactional upgrade and rollback, etc.

> From fb1fbc92cd68331b3dea94c238274f8a01b98afa Mon Sep 17 00:00:00 2001
> From: Eric Le Bihan <eric.le.bihan.dev@free.fr>
> Date: Thu, 28 Jul 2016 20:09:01 +0200
> Subject: [PATCH 1/1] gnu: Add rust
>
> * gnu/packages/rust.scm(rust): New variable.
>
> Signed-off-by: Eric Le Bihan <eric.le.bihan.dev@free.fr>

Apart from the bootstrapping thing discussed above, this looks good to
me (and a great first package!).

> +            ;; Tell where to find libgcc_s.so
> +            (setenv "LD_LIBRARY_PATH" (string-append gcc-lib "/lib"))

“LIBRARY_PATH” may be enough.

> +            ;; Remove reference to "/lib64/ld-linux-x86-64.so.2" from binary
> +            (zero? (system*
> +                    "patchelf"
> +                    "--set-interpreter" ld-so
> +                    (string-append (getcwd) "/rustc/bin/rustc")))

“rustc/bin/rustc” is enough here.

> +        (add-before 'build 'pre-build
> +                    (lambda _
> +                      (let* ((bindir (string-append (getcwd) "/bin"))
> +                             (cc (string-append bindir "/cc")))
> +                        (mkdir bindir)
> +                        (call-with-output-file cc
> +                          (lambda (port)
> +                            (format port
> +                                    "#!~a\n\nexec gcc \"$@\"\n" (which "sh"))))
> +                        (chmod cc #o755))))

Can we avoid this trick using a configure flag
(--with-compiler=/path/to/gcc) or a configure or environment variable
(CC=gcc)?  If not, that’s fine.

> +      (replace 'build
> +               (lambda* (#:key outputs #:allow-other-keys)
> +                 (setenv "PATH"
> +                         (string-append (getcwd) "/bin:" (getenv "PATH")))
> +                 (mkdir (assoc-ref outputs "out"))
> +                 (zero? (system* "make")))))

Rather do:

  (add-before 'build 'change-PATH
    (lambda _
      (setenv …)
      #t))

so we can reuse the normal ‘build’ phase, which passes -jX to ‘make’.

> +      #:tests? #f))

We normally run test suites, unless we have a good reason not to do so.
:-)  Any ideas why “make check” fails?

Thanks!

Ludo’.

Re: Rust

[Alex Griffin]

On Fri, Jul 29, 2016, at 10:16 AM, Ludovic Courtès wrote:
> Do you know what’s Rust’s bootstrapping story is?  Can we reasonably
> expect to bootstrap it from source, using a series of previous Rust
> versions, or using an alternative implementation?

Yes, Rust 1.10 builds with the previous stable release (1.9) for the
first time. So we will only need one binary to bootstrap Rust. Although
this will quickly require a long chain of builds because Rust releases
every 6 weeks and 1.11 is only guaranteed to build with 1.10, etc.

So after only two years we may need to compile like 17 different
releases to get the current version.
-- 
Alex Griffin

Re: Rust

[Jelle Licht]

[-- Attachment #1: Type: text/plain, Size: 2427 bytes --]

I looked into this once;
I found a Rust compiler written in OCaml on the web.
There should be a path from that compiler to the current version of Rust.
The problem lies in the fact that the entire "1.9 build 1.10 builds 1.11"
spiel only become the official policy of the Rust project recently.

disclaimer: the following is how I interpreted my communications with
the rust community. My apologies to anyone involved for any inaccuracies
and/or misrepresentations.

This means that you could be looking at individual commits, as before this,
I believe nightlies (snapshots) were used to compile the Rust compiler.

So, to ad to Alex' point, we would also need to expend (one-time) effort to
find this 'bootstrap-path' from the OCaml-based compiler to a more recent
rustc.

In my admittedly little amounts of experience, the people working on rust
were not really seeing the problem with the current state of affairs, so
good luck getting support with some arcane issues one might encounter
packaging these ancient versions of rustc.

To quote kibwen from [1]:
"We can determine exactly how many builds you'd need by looking at the
master snapshot file:
https://github.com/rust-lang/rust/blob/master/src/snapshots....
<https://github.com/rust-lang/rust/blob/master/src/snapshots.txt>

According to this, there have been 290 snapshots in total. And keep in mind
that you would also need to rebuild LLVM quite a few times as well during
this process, as Rust has continually upgraded its custom LLVM fork over
the years."

Not sure if all this is worth the effort...

- Jelle


[1]: https://news.ycombinator.com/item?id=8732669


2016-07-29 17:34 GMT+02:00 Alex Griffin <a@ajgrf.com>:

> On Fri, Jul 29, 2016, at 10:16 AM, Ludovic Courtès wrote:
> > Do you know what’s Rust’s bootstrapping story is?  Can we reasonably
> > expect to bootstrap it from source, using a series of previous Rust
> > versions, or using an alternative implementation?
>
> Yes, Rust 1.10 builds with the previous stable release (1.9) for the
> first time. So we will only need one binary to bootstrap Rust. Although
> this will quickly require a long chain of builds because Rust releases
> every 6 weeks and 1.11 is only guaranteed to build with 1.10, etc.
>
> So after only two years we may need to compile like 17 different
> releases to get the current version.
> --
> Alex Griffin
>
>

[-- Attachment #2: Type: text/html, Size: 3286 bytes --]

Re: Rust

[Ludovic Courtès]

Alex Griffin <a@ajgrf.com> skribis:

> On Fri, Jul 29, 2016, at 10:16 AM, Ludovic Courtès wrote:
>> Do you know what’s Rust’s bootstrapping story is?  Can we reasonably
>> expect to bootstrap it from source, using a series of previous Rust
>> versions, or using an alternative implementation?
>
> Yes, Rust 1.10 builds with the previous stable release (1.9) for the
> first time. So we will only need one binary to bootstrap Rust. Although
> this will quickly require a long chain of builds because Rust releases
> every 6 weeks and 1.11 is only guaranteed to build with 1.10, etc.
>
> So after only two years we may need to compile like 17 different
> releases to get the current version.

Bah.  If that were the only problem, we could regularly cut the
dependency graph by storing a pre-built binary built using Guix,
assuming Rust builds are reproducible.  (Note that each node in the
dependency graph contributes to the ‘guix build -d’ time by less than a
hundred milliseconds, so the chain can be quite long before it’s a
practical problem.)

Jelle Licht <jlicht@fsfe.org> skribis:

> So, to ad to Alex' point, we would also need to expend (one-time) effort to
> find this 'bootstrap-path' from the OCaml-based compiler to a more recent
> rustc.

I suspect this one-time effort would be quite huge.

> To quote kibwen from [1]:
> "We can determine exactly how many builds you'd need by looking at the
> master snapshot file:
> https://github.com/rust-lang/rust/blob/master/src/snapshots....
> <https://github.com/rust-lang/rust/blob/master/src/snapshots.txt>
>
> According to this, there have been 290 snapshots in total. And keep in mind
> that you would also need to rebuild LLVM quite a few times as well during
> this process, as Rust has continually upgraded its custom LLVM fork over
> the years."
>
> Not sure if all this is worth the effort...

It’s discouraging, indeed.

We need to find a reasonable way forward.  What about starting from the
version that Eric submitted (bootstrapped from the opaque binary), and
from there on do the “build with previous version” thing?

When the chain becomes too long, we’ll host new bootstrap binaries
produced with Guix and cut the initial part of the chain.

Thoughts?

Obviously not ideal, but at least we’d have a clearer track record of
our binaries: we’d document the Guix commit needed to reproduce a
specific binary, and anyone could verify it.

Ludo’.

Re: Rust

[Pjotr Prins]

On Sat, Jul 30, 2016 at 03:34:08PM +0200, Ludovic Courtès wrote:
> When the chain becomes too long, we’ll host new bootstrap binaries
> produced with Guix and cut the initial part of the chain.
> 
> Thoughts?

This sounds like a very good approach for compilers written in their
own language. Same will be true for the D compilers. The D compiler
was recently ported to D and the authors say they will aim to keep
support for bootstrapping from C - but they added it may grow
complicated over time. I think the Elixir compiler will also be ported
to Elixer in time... there have been some noises.

The bootstrapping from source is a fine idea(l), but not always
practical for package authors.

Interesting point is if we can share one such cross-building binary
compiler for all target platforms, or are we going to store binaries
for each of them?

Pj.

Re: Rust

[Eric Le Bihan]

Le Fri, 29 Jul 2016 17:16:29 +0200,
ludo@gnu.org (Ludovic Courtès) a écrit :

> Eric Le Bihan <eric.le.bihan.dev@free.fr> skribis:
> 
> > It happens that I tried to package Rust, as an introduction to Guix.
> > Here is my version, inspired by the Haskell package, where Rust
> > 1.10.0 is built, bootstrapped by a binary version of rustc 1.9.0.
> > It uses the "cc" wrapper trick previously presented.  
> 
> As you might have seen from previous discussions, we try hard to
> reduce the number of binary blobs needed to bootstrap packages, such
> that we have a full source-to-binary path that everyone can audit.
> Unfortunately, this usually hard to achieve for self-hosted compilers.
> 
> Do you know what’s Rust’s bootstrapping story is?  Can we reasonably
> expect to bootstrap it from source, using a series of previous Rust
> versions, or using an alternative implementation?

As explained by others, starting with version 1.10.0, Rust version N
will be guaranteed to bootstrap with version N-1.

> > Some questions, though:
> >
> > 1. I can compile a sample program in a guix environment created
> > using `guix environment gcc glibc binutils rust`, but the program
> > generated fails to run because libgcc_s.so.1 can not be found. How
> > can it be added to the environment?  
> 
> As Andreas notes, ‘gcc-toolchain’, which includes ‘ld-wrapper’, should
> fix this.

Yes. It solved my problem.
 
> Does Rust use GCC, or just ld?

It only uses the linker, i.e. ld on GNU/Linux.

> > 2. Having a Rust compiler is cool, but having Cargo, the Rust
> > package manager, would be even better. Cargo is also bootstrapped,
> > and it is also built using zillions of crates (Rust packages)
> > downloaded from the Internet. How could this case be handled in
> > Guix?  
> 
> Assuming Cargo itself is just a regular Rust program, it should be
> possible to make a Guix package of Cargo.  Then, Guix users can
> install it and use it the normal way; we won’t be able to use Cargo
> in package recipes though, because our package build environments
> purposefully lacks network access.
> 
> Besides, I would encourage you or anyone interested to write a crate
> importer, like we do for most other language packages:
> 
>   https://www.gnu.org/software/guix/manual/html_node/Invoking-guix-import.html
> 
> Having crates available as normal Guix packages is the best option for
> Guix users: uniform interface, the ability to use ‘guix environment’
> and all the tools, transactional upgrade and rollback, etc.

IIUC, to provide a Guix package for Cargo, the following should be done:

1. write a crate importer.
2. list all the crates needed by Cargo to build itself.
3. package each crate with the importer.
4. add a Cargo package which depends on the newly-imported crates and
uses a binary version of Cargo to bootstrap itself (though this is not
the best option in terms of auditing/reproducibility).

Unlike Rust, Cargo still uses an "anonymous" binary version of itself
for bootstrapping. I hope this may change soon.

> > From fb1fbc92cd68331b3dea94c238274f8a01b98afa Mon Sep 17 00:00:00
> > 2001 From: Eric Le Bihan <eric.le.bihan.dev@free.fr>
> > Date: Thu, 28 Jul 2016 20:09:01 +0200
> > Subject: [PATCH 1/1] gnu: Add rust
> >
> > * gnu/packages/rust.scm(rust): New variable.
> >
> > Signed-off-by: Eric Le Bihan <eric.le.bihan.dev@free.fr>  
> 
> Apart from the bootstrapping thing discussed above, this looks good to
> me (and a great first package!).
> 
> > +            ;; Tell where to find libgcc_s.so
> > +            (setenv "LD_LIBRARY_PATH" (string-append gcc-lib
> > "/lib"))  
> 
> “LIBRARY_PATH” may be enough.

OK.
 
> > +            ;; Remove reference to "/lib64/ld-linux-x86-64.so.2"
> > from binary
> > +            (zero? (system*
> > +                    "patchelf"
> > +                    "--set-interpreter" ld-so
> > +                    (string-append (getcwd) "/rustc/bin/rustc")))  
> 
> “rustc/bin/rustc” is enough here.

OK.

> 
> > +        (add-before 'build 'pre-build
> > +                    (lambda _
> > +                      (let* ((bindir (string-append (getcwd)
> > "/bin"))
> > +                             (cc (string-append bindir "/cc")))
> > +                        (mkdir bindir)
> > +                        (call-with-output-file cc
> > +                          (lambda (port)
> > +                            (format port
> > +                                    "#!~a\n\nexec gcc
> > \"$@\"\n" (which "sh"))))
> > +                        (chmod cc #o755))))  
> 
> Can we avoid this trick using a configure flag
> (--with-compiler=/path/to/gcc) or a configure or environment variable
> (CC=gcc)?  If not, that’s fine.

To build the Rust standard library, only the linker is needed. The
default value is "cc". I thought this could be overridden using the
"--default-linker=" of the ./configure script, but it looks like it is
not properly handled. Hence the need for the wrapper. 

I haven't checked too deeply to know if it is a bug or a feature. On
the Rust issue tracker, an entry [1] relates to this problem, but only
in terms of documentation. I'll bump the topic upstream.

> > +      (replace 'build
> > +               (lambda* (#:key outputs #:allow-other-keys)
> > +                 (setenv "PATH"
> > +                         (string-append (getcwd) "/bin:" (getenv
> > "PATH")))
> > +                 (mkdir (assoc-ref outputs "out"))
> > +                 (zero? (system* "make")))))  
> 
> Rather do:
> 
>   (add-before 'build 'change-PATH
>     (lambda _
>       (setenv …)
>       #t))
> 
> so we can reuse the normal ‘build’ phase, which passes -jX to ‘make’.

OK.

> 
> > +      #:tests? #f))  
> 
> We normally run test suites, unless we have a good reason not to do
> so. :-)  Any ideas why “make check” fails?

Out of laziness, I skipped the tests. I'll give it a look.

[1] https://github.com/rust-lang/rust/issues/32208

Best regards,

-- 
ELB

Re: Rust

[Ludovic Courtès]

Eric Le Bihan <eric.le.bihan.dev@free.fr> skribis:

> Le Fri, 29 Jul 2016 17:16:29 +0200,
> ludo@gnu.org (Ludovic Courtès) a écrit :

[...]

>> > Some questions, though:
>> >
>> > 1. I can compile a sample program in a guix environment created
>> > using `guix environment gcc glibc binutils rust`, but the program
>> > generated fails to run because libgcc_s.so.1 can not be found. How
>> > can it be added to the environment?  
>> 
>> As Andreas notes, ‘gcc-toolchain’, which includes ‘ld-wrapper’, should
>> fix this.
>
> Yes. It solved my problem.
>  
>> Does Rust use GCC, or just ld?
>
> It only uses the linker, i.e. ld on GNU/Linux.

Then it should be enough to add ‘ld-wrapper’ to the inputs (no need for
‘gcc-toolchain’ or ‘gcc’.)  Just make sure Rust captures the absolute
file name of ‘ld’.

> IIUC, to provide a Guix package for Cargo, the following should be done:
>
> 1. write a crate importer.
> 2. list all the crates needed by Cargo to build itself.
> 3. package each crate with the importer.
> 4. add a Cargo package which depends on the newly-imported crates and
> uses a binary version of Cargo to bootstrap itself (though this is not
> the best option in terms of auditing/reproducibility).
>
> Unlike Rust, Cargo still uses an "anonymous" binary version of itself
> for bootstrapping. I hope this may change soon.

Something along these lines, yes.  We may also need a
‘rust-build-system’, like we did for other languages:

  https://www.gnu.org/software/guix/manual/html_node/Build-Systems.html

If that build system works without invoking Cargo, then we probably
don’t need a Cargo binary to build Cargo.

Anyway, we’re not there yet.  :-)

>> > +        (add-before 'build 'pre-build
>> > +                    (lambda _
>> > +                      (let* ((bindir (string-append (getcwd)
>> > "/bin"))
>> > +                             (cc (string-append bindir "/cc")))
>> > +                        (mkdir bindir)
>> > +                        (call-with-output-file cc
>> > +                          (lambda (port)
>> > +                            (format port
>> > +                                    "#!~a\n\nexec gcc
>> > \"$@\"\n" (which "sh"))))
>> > +                        (chmod cc #o755))))  
>> 
>> Can we avoid this trick using a configure flag
>> (--with-compiler=/path/to/gcc) or a configure or environment variable
>> (CC=gcc)?  If not, that’s fine.
>
> To build the Rust standard library, only the linker is needed. The
> default value is "cc". I thought this could be overridden using the
> "--default-linker=" of the ./configure script, but it looks like it is
> not properly handled. Hence the need for the wrapper. 

OK, makes sense.  Please leave this explanation as a comment in the
code.

>> > +      #:tests? #f))  
>> 
>> We normally run test suites, unless we have a good reason not to do
>> so. :-)  Any ideas why “make check” fails?
>
> Out of laziness, I skipped the tests. I'll give it a look.

Let us know how it goes, even if the outcome is “too painful, giving
up”.  :-)

Thanks,
Ludo’.

Re: Rust

[David Craven]

> Can we reasonably expect to bootstrap it from source,
> using a series of previous Rust versions, or using an
> alternative implementation?

Currently as Jelle said no. But there is [0] that may be a
viable option in the future.

> Having crates available as normal Guix packages is the
> best option for Guix users: uniform interface, the ability
> to use ‘guix environment’ and all the tools, transactional
> upgrade and rollback, etc.

I think with a cargo plugin that interacts with the guix daemon,
we could achive the same advantages. If we want to package
servo (mozillas experimental rust webbrowser) we could add
a guix package for servo, specify it uses the cargo build
system and that would call cargo build-guix.
For other rust projects (that don't need packaging) it could add
a project profile and add the symlinks in $PROJECT_DIR/target/.

I don't think we should dismiss this possibility straight away...
Rust contributors have been very helpful with issues that arose on
nixos. I think it may be even possible that at some point in the
future cargo build could detect if there is a guix/nix daemon
available and use that as the storage backend instead of
$HOME/.cargo/registry.

[0] https://github.com/thepowersgang/mrustc

Re: Rust

[Ludovic Courtès]

David Craven <david@craven.ch> skribis:

>> Can we reasonably expect to bootstrap it from source,
>> using a series of previous Rust versions, or using an
>> alternative implementation?
>
> Currently as Jelle said no. But there is [0] that may be a
> viable option in the future.
>
> [0] https://github.com/thepowersgang/mrustc

That would be great.

>> Having crates available as normal Guix packages is the
>> best option for Guix users: uniform interface, the ability
>> to use ‘guix environment’ and all the tools, transactional
>> upgrade and rollback, etc.
>
> I think with a cargo plugin that interacts with the guix daemon,

The Guix daemon is very low-level, is has no idea what a package is, so
it’d be the wrong place for such things IMO.

What we did for other languages should work here: (1) provide an
importer, and (2) provide a build system (we have ‘guix import cpan’ and
‘perl-build-system’, and so on.)

Ludo’.

Re: Rust

[David Craven]

> IIUC, to provide a Guix package for Cargo, the following should be done:

> 1. write a crate importer.
> 2. list all the crates needed by Cargo to build itself.
> 3. package each crate with the importer.
> 4. add a Cargo package which depends on the newly-imported crates and
> uses a binary version of Cargo to bootstrap itself (though this is not
> the best option in terms of auditing/reproducibility).

Have you made any progress on this? =)

FYI I also think that that's the best course of action. I remember
dealing with non gnu-build-system packages in nixos was a bit of a
pain, so I was biased against the importer thing. I think guix does a
better job at it...

To build the crates needed to build cargo a good idea is probably to
have rust-build-system take a #:cargo and a #:rustc package. Then you
can pass the binary versions for bootstrapping cargo.

Re: Rust

[Eric Le Bihan]

Hi!

Le Sun, 4 Sep 2016 16:50:48 +0200,
David Craven <david@craven.ch> a écrit :

> > IIUC, to provide a Guix package for Cargo, the following should be
> > done:  
> 
> > 1. write a crate importer.
> > 2. list all the crates needed by Cargo to build itself.
> > 3. package each crate with the importer.
> > 4. add a Cargo package which depends on the newly-imported crates
> > and uses a binary version of Cargo to bootstrap itself (though this
> > is not the best option in terms of auditing/reproducibility).  
> 
> Have you made any progress on this? =)

Unfortunately no. Bandwith is a little bit low ATM :-(

I sent an updated version of the Rust package, taking into account the
suggestions from the original version.

> FYI I also think that that's the best course of action. I remember
> dealing with non gnu-build-system packages in nixos was a bit of a
> pain, so I was biased against the importer thing. I think guix does a
> better job at it...
> 
> To build the crates needed to build cargo a good idea is probably to
> have rust-build-system take a #:cargo and a #:rustc package. Then you
> can pass the binary versions for bootstrapping cargo.

Thanks for the suggestion.

Best regards, 

-- 
ELB