Re: [PATCH] Help Ruby packages be reproducible

[Ben Woodcroft <b.woodcroft@uq.edu.au>]

[-- Attachment #1: Type: text/plain, Size: 3380 bytes --]

On 31/12/15 03:26, Ludovic Courtès wrote:
> Ben Woodcroft <b.woodcroft@uq.edu.au> skribis:
>
>> On 29/12/15 15:46, Ben Woodcroft wrote:
>>> Unfortunately none of these builds are reproducible because rubygems
>>> in Guix generally aren't. For one, this is because .gem files are
>>> archives whose contents are timestamped.
>> I should clarify. What I meant was the cache .gem files
>>
>> /gnu/store/ib83mg5zsyr5x2w0m3i1f84gdvdbp5x9-ruby-ascii85-1.0.2/lib/ruby/gems/2.2.0/cache$
>> tar tvf Ascii85-1.0.2.gem |head
>> -r--r--r-- wheel/wheel     703 2015-12-27 22:44 metadata.gz
>> -r--r--r-- wheel/wheel    7436 2015-12-27 22:44 data.tar.gz
>> -r--r--r-- wheel/wheel     268 2015-12-27 22:44 checksums.yaml.gz
> We should arrange so that gems are created with a fixed timestamp and
> UID/GID, and a well-defined file ordering, as with:
>
>    --mtime=@0 --sort=name --owner=root:0 --group=root:0
>
> We also need to make sure gzip is always run with -n/--no-name.  That
> way, the gz files above will not include an additional timestamp.
>
>  From what I can see in
> <git://git.debian.org/git/reproducible/notes.git>, this is not addressed
> yet in other distros.
Ludo are you suggesting we should abandon the deletion approach?


On 30/12/15 18:26, Ricardo Wurmus wrote:
> Ben Woodcroft <b.woodcroft@uq.edu.au> writes:
>> The .gem file stored in GEM_HOME after install is both redundant and an
>> archive that stores timestamped files which makes builds non-deterministic. So
>> delete it after 'gem install'.
> Good idea!  I don’t know if the existence of the cached gem is checked
> for by any Ruby tools (bundler or the like).  Is there some
> documentation about this cache?
I wondered that too, but I built all of the ruby packages again without 
issue and many of them use bundler. It also doesn't seem like a good 
idea for bundler to use cached gems since I would guess that gems that 
are downloaded but fail to install are kept in the cache. I also wasn't 
able to see any mention of the cache in the rubygems API.
>> -    (zero? (apply system* "gem" "install" (first-matching-file "\\.gem$")
>> -                  "--local" "--ignore-dependencies"
>> -                  ;; Executables should go into /bin, not /lib/ruby/gems.
>> -                  "--bindir" (string-append out "/bin")
>> -                  gem-flags))))
>> +    (apply system* "gem" "install" gem-name
>> +           "--local" "--ignore-dependencies"
>> +           ;; Executables should go into /bin, not /lib/ruby/gems.
>> +           "--bindir" (string-append out "/bin")
>> +           gem-flags)
>> +    ;; Remove the cached gem file as this is unnecessary and contains
>> +    ;; timestamped files rendering builds not reproducible.
>> +    (delete-file (string-append gem-home "/cache/" gem-name))
>> +    #t))
> I’d prefer to keep ‘(zero? ...)’ and only delete the file when the
> ‘system*’ call above succeeded.  It would be nice if we could propagate
> any bad return value from ‘system*’ to the end of the procedure.
>
> Maybe something like this:
>
>    (and (zero? (apply system* ...))
>         (begin (delete-file ...) #t))
>
> It’s a bit clunky but the return value would still be #f if ‘system*’
> fails.  What do you think?
I think you are right as usual. Better in attached?

Thanks,
ben

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-build-ruby-Remove-cached-gem-after-install.patch --]
[-- Type: text/x-patch; name="0001-build-ruby-Remove-cached-gem-after-install.patch", Size: 2498 bytes --]

From c00a032644c02474212be97c185c0953967de4e5 Mon Sep 17 00:00:00 2001
From: Ben Woodcroft <donttrustben@gmail.com>
Date: Wed, 30 Dec 2015 10:27:33 +1000
Subject: [PATCH] build: ruby: Remove cached gem after install.

The .gem file stored in GEM_HOME after install is both redundant and an
archive that stores timestamped files which makes builds non-deterministic. So
delete it after 'gem install'.

* guix/build/ruby-build-system.scm (install): Remove cached gem after install.
---
 guix/build/ruby-build-system.scm | 19 ++++++++++++-------
 1 file changed, 12 insertions(+), 7 deletions(-)

diff --git a/guix/build/ruby-build-system.scm b/guix/build/ruby-build-system.scm
index 2685da1..6439bf6 100644
--- a/guix/build/ruby-build-system.scm
+++ b/guix/build/ruby-build-system.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2015 David Thompson <davet@gnu.org>
 ;;; Copyright © 2015 Pjotr Prins <pjotr.public01@thebird.nl>
+;;; Copyright © 2015 Ben Woodcroft <donttrustben@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -115,15 +116,19 @@ GEM-FLAGS are passed to the 'gem' invokation, if present."
                                          (assoc-ref inputs "ruby"))
                            1))
          (out (assoc-ref outputs "out"))
-         (gem-home (string-append out "/lib/ruby/gems/" ruby-version ".0")))
-
+         (gem-home (string-append out "/lib/ruby/gems/" ruby-version ".0"))
+         (gem-name (first-matching-file "\\.gem$")))
     (setenv "GEM_HOME" gem-home)
     (mkdir-p gem-home)
-    (zero? (apply system* "gem" "install" (first-matching-file "\\.gem$")
-                  "--local" "--ignore-dependencies"
-                  ;; Executables should go into /bin, not /lib/ruby/gems.
-                  "--bindir" (string-append out "/bin")
-                  gem-flags))))
+    (and (apply system* "gem" "install" gem-name
+                "--local" "--ignore-dependencies"
+                ;; Executables should go into /bin, not /lib/ruby/gems.
+                "--bindir" (string-append out "/bin")
+                gem-flags)
+         ;; Remove the cached gem file as this is unnecessary and contains
+         ;; timestamped files rendering builds not reproducible.
+         (begin (delete-file (string-append gem-home "/cache/" gem-name))
+                #t))))
 
 (define %standard-phases
   (modify-phases gnu:%standard-phases
-- 
2.6.3