[PATCH 0/3]; ruby-build-system improvements.

[PATCH 0/3]; ruby-build-system improvements.

[Ben Woodcroft]

Hi all,

Attached is a patch to make the building gems with native extensions
reproducible, and a another to introduce a phase to remove `git ls-files` in
Rakefiles as suggested by Dave:
http://lists.gnu.org/archive/html/guix-devel/2016-06/msg00262.html

With these patches and some other small fixes not mentioned most/all of the
Ruby packages build reproducibly.

Thanks as usual for any review.
ben

[PATCH 1/3] guix: ruby-build-system: Build compiled gems reproducibly.

[Ben Woodcroft]

* guix/build/ruby-build-system.scm (log-file-deletion): New procedure.
(install): Remove files containing non-reproducible elements.  Print when each
file is deleted.
---
 guix/build/ruby-build-system.scm | 38 ++++++++++++++++++++++++++++++++------
 1 file changed, 32 insertions(+), 6 deletions(-)

diff --git a/guix/build/ruby-build-system.scm b/guix/build/ruby-build-system.scm
index 79ac380..95793f7 100644
--- a/guix/build/ruby-build-system.scm
+++ b/guix/build/ruby-build-system.scm
@@ -120,18 +120,44 @@ GEM-FLAGS are passed to the 'gem' invokation, if present."
                            1))
          (out (assoc-ref outputs "out"))
          (gem-home (string-append out "/lib/ruby/gems/" ruby-version ".0"))
-         (gem-name (first-matching-file "\\.gem$")))
+         (gem-file (first-matching-file "\\.gem$"))
+         (gem-file-basename (basename gem-file))
+         (gem-name (substring gem-file-basename
+                              0
+                              (- (string-length gem-file-basename) 4)))
+         (gem-directory (string-append gem-home "/gems/" gem-name)))
     (setenv "GEM_HOME" gem-home)
     (mkdir-p gem-home)
-    (and (apply system* "gem" "install" gem-name
+    (and (apply system* "gem" "install" gem-file
                 "--local" "--ignore-dependencies"
                 ;; Executables should go into /bin, not /lib/ruby/gems.
                 "--bindir" (string-append out "/bin")
                 gem-flags)
-         ;; Remove the cached gem file as this is unnecessary and contains
-         ;; timestamped files rendering builds not reproducible.
-         (begin (delete-file (string-append gem-home "/cache/" gem-name))
-                #t))))
+         (begin
+           ;; Remove the cached gem file as this is unnecessary and contains
+           ;; timestamped files rendering builds not reproducible.
+           (let ((cached-gem (string-append gem-home "/cache/" gem-file)))
+             (log-file-deletion cached-gem)
+             (delete-file cached-gem))
+           ;; For gems with native extensions, several Makefile-related files
+           ;; are created that contain timestamps or other elements making
+           ;; them not reproducible.  They are unnecessary so we remove them.
+           (if (file-exists? (string-append gem-directory "/ext"))
+               (begin
+                 (for-each (lambda (file)
+                             (log-file-deletion file)
+                             (delete-file file))
+                           (append
+                            (find-files (string-append gem-home "/doc")
+                                        "page-Makefile.ri")
+                            (find-files (string-append gem-home "/extensions")
+                                        "gem_make.out")
+                            (find-files (string-append gem-directory "/ext")
+                                        "Makefile")))))
+           #t))))
+
+(define (log-file-deletion file)
+  (display (string-append "deleting '" file "' for reproducibility\n")))
 
 (define %standard-phases
   (modify-phases gnu:%standard-phases
-- 
2.9.1

Re: [PATCH 1/3] guix: ruby-build-system: Build compiled gems reproducibly.

[Ricardo Wurmus]

Ben Woodcroft <donttrustben@gmail.com> writes:

> * guix/build/ruby-build-system.scm (log-file-deletion): New procedure.
> (install): Remove files containing non-reproducible elements.  Print when each
> file is deleted.

Thank you, this looks good to me.  Good job on making even more Ruby
gems build reproducibly!

~~ Ricardo

Re: [PATCH 1/3] guix: ruby-build-system: Build compiled gems reproducibly.

[Ben Woodcroft]

On 18/08/16 20:06, Ricardo Wurmus wrote:
> Ben Woodcroft <donttrustben@gmail.com> writes:
>
>> * guix/build/ruby-build-system.scm (log-file-deletion): New procedure.
>> (install): Remove files containing non-reproducible elements.  Print when each
>> file is deleted.
> Thank you, this looks good to me.  Good job on making even more Ruby
> gems build reproducibly!

Great, thanks Ricardo. I just pushed this series.

ben

[PATCH 2/3] guix: ruby-build-system: Add replace-git-ls-files.

[Ben Woodcroft]

* guix/build/ruby-build-system.scm (replace-git-ls-files): New variable.
(%standard-phases): Add it.
---
 guix/build/ruby-build-system.scm | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)

diff --git a/guix/build/ruby-build-system.scm b/guix/build/ruby-build-system.scm
index 95793f7..c2d2766 100644
--- a/guix/build/ruby-build-system.scm
+++ b/guix/build/ruby-build-system.scm
@@ -1,7 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2015 David Thompson <davet@gnu.org>
 ;;; Copyright © 2015 Pjotr Prins <pjotr.public01@thebird.nl>
-;;; Copyright © 2015 Ben Woodcroft <donttrustben@gmail.com>
+;;; Copyright © 2015, 2016 Ben Woodcroft <donttrustben@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -69,6 +69,16 @@ directory."
 (define (first-gemspec)
   (first-matching-file "\\.gemspec$"))
 
+(define* (replace-git-ls-files #:key source #:allow-other-keys)
+  "Many gemspec files downloaded from outside rubygems.org use `git ls-files`
+to list of the files to be included in the built gem.  However, since this
+operation is not deterministic, we replace it with `find`."
+  (when (not (gem-archive? source))
+    (let ((gemspec (first-gemspec)))
+      (substitute* gemspec
+        (("`git ls-files`") "`find . -type f |sort`"))))
+  #t)
+
 (define* (extract-gemspec #:key source #:allow-other-keys)
   "Remove the original gemspec, if present, and replace it with a new one.
 This avoids issues with upstream gemspecs requiring tools such as git to
@@ -162,11 +172,12 @@ GEM-FLAGS are passed to the 'gem' invokation, if present."
 (define %standard-phases
   (modify-phases gnu:%standard-phases
     (delete 'configure)
+    (replace 'unpack unpack)
     (add-before 'build 'extract-gemspec extract-gemspec)
+    (add-after 'extract-gemspec 'replace-git-ls-files replace-git-ls-files)
     (replace 'build build)
-    (replace 'unpack unpack)
-    (replace 'install install)
-    (replace 'check check)))
+    (replace 'check check)
+    (replace 'install install)))
 
 (define* (ruby-build #:key inputs (phases %standard-phases)
                      #:allow-other-keys #:rest args)
-- 
2.9.1

[PATCH 3/3] gnu: ruby-concurrent: Adjust for 'replace-git-ls-files'.

[Ben Woodcroft]

* gnu/packages/ruby.scm (ruby-concurrent)[arguments]: Adjust for new build
phase 'replace-git-ls-files'.
---
 gnu/packages/ruby.scm | 15 ++++++---------
 1 file changed, 6 insertions(+), 9 deletions(-)

diff --git a/gnu/packages/ruby.scm b/gnu/packages/ruby.scm
index 0ff7d56..88bb306 100644
--- a/gnu/packages/ruby.scm
+++ b/gnu/packages/ruby.scm
@@ -4095,21 +4095,18 @@ call.")
      `(#:test-target "spec"
        #:phases
        (modify-phases %standard-phases
-         (add-before 'build 'remove-git-lsfiles-and-extra-gemspecs
-           (lambda _
-             (for-each (lambda (file)
-                         (substitute* file
-                           (("git ls-files") "find * |sort")))
-                       (list "concurrent-ruby.gemspec"
-                             "support/file_map.rb"))
-             #t))
-         (add-before 'build 'remove-extra-gemspecs
+         (add-before 'replace-git-ls-files 'remove-extra-gemspecs
            (lambda _
              ;; Delete extra gemspec files so 'first-gemspec' chooses the
              ;; correct one.
              (delete-file "concurrent-ruby-edge.gemspec")
              (delete-file "concurrent-ruby-ext.gemspec")
              #t))
+         (add-before 'build 'replace-git-ls-files2
+           (lambda _
+             (substitute* "support/file_map.rb"
+               (("git ls-files") "find * |sort"))
+             #t))
          (add-before 'check 'rake-compile
            ;; Fix the test error described at
            ;; https://github.com/ruby-concurrency/concurrent-ruby/pull/408
-- 
2.9.1