From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ben Woodcroft Subject: Re: [PATCH] Help Ruby packages be reproducible Date: Thu, 31 Dec 2015 09:52:21 +1000 Message-ID: <56846E35.6010805@uq.edu.au> References: <56821E47.9010400@uq.edu.au> <87io3h4ta3.fsf@elephly.net> <56832AA3.8030109@uq.edu.au> <56832BA6.6030806@uq.edu.au> <877fjwbavn.fsf@elephly.net> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------010603000605090106030505" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:42837) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aEQXm-0004yw-3P for guix-devel@gnu.org; Wed, 30 Dec 2015 18:52:39 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aEQXk-0005SX-D9 for guix-devel@gnu.org; Wed, 30 Dec 2015 18:52:38 -0500 In-Reply-To: <877fjwbavn.fsf@elephly.net> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: Ricardo Wurmus Cc: "guix-devel@gnu.org" This is a multi-part message in MIME format. --------------010603000605090106030505 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by newmailhub.uq.edu.au id tBUNqNHd007019 On 31/12/15 03:26, Ludovic Court=C3=A8s wrote: > Ben Woodcroft skribis: > >> On 29/12/15 15:46, Ben Woodcroft wrote: >>> Unfortunately none of these builds are reproducible because rubygems >>> in Guix generally aren't. For one, this is because .gem files are >>> archives whose contents are timestamped. >> I should clarify. What I meant was the cache .gem files >> >> /gnu/store/ib83mg5zsyr5x2w0m3i1f84gdvdbp5x9-ruby-ascii85-1.0.2/lib/rub= y/gems/2.2.0/cache$ >> tar tvf Ascii85-1.0.2.gem |head >> -r--r--r-- wheel/wheel 703 2015-12-27 22:44 metadata.gz >> -r--r--r-- wheel/wheel 7436 2015-12-27 22:44 data.tar.gz >> -r--r--r-- wheel/wheel 268 2015-12-27 22:44 checksums.yaml.gz > We should arrange so that gems are created with a fixed timestamp and > UID/GID, and a well-defined file ordering, as with: > > --mtime=3D@0 --sort=3Dname --owner=3Droot:0 --group=3Droot:0 > > We also need to make sure gzip is always run with -n/--no-name. That > way, the gz files above will not include an additional timestamp. > > From what I can see in > , this is not addresse= d > yet in other distros. Ludo are you suggesting we should abandon the deletion approach? On 30/12/15 18:26, Ricardo Wurmus wrote: > Ben Woodcroft writes: >> The .gem file stored in GEM_HOME after install is both redundant and a= n >> archive that stores timestamped files which makes builds non-determini= stic. So >> delete it after 'gem install'. > Good idea! I don=E2=80=99t know if the existence of the cached gem is = checked > for by any Ruby tools (bundler or the like). Is there some > documentation about this cache? I wondered that too, but I built all of the ruby packages again without=20 issue and many of them use bundler. It also doesn't seem like a good=20 idea for bundler to use cached gems since I would guess that gems that=20 are downloaded but fail to install are kept in the cache. I also wasn't=20 able to see any mention of the cache in the rubygems API. >> - (zero? (apply system* "gem" "install" (first-matching-file "\\.ge= m$") >> - "--local" "--ignore-dependencies" >> - ;; Executables should go into /bin, not /lib/ruby/g= ems. >> - "--bindir" (string-append out "/bin") >> - gem-flags)))) >> + (apply system* "gem" "install" gem-name >> + "--local" "--ignore-dependencies" >> + ;; Executables should go into /bin, not /lib/ruby/gems. >> + "--bindir" (string-append out "/bin") >> + gem-flags) >> + ;; Remove the cached gem file as this is unnecessary and contains >> + ;; timestamped files rendering builds not reproducible. >> + (delete-file (string-append gem-home "/cache/" gem-name)) >> + #t)) > I=E2=80=99d prefer to keep =E2=80=98(zero? ...)=E2=80=99 and only delet= e the file when the > =E2=80=98system*=E2=80=99 call above succeeded. It would be nice if we= could propagate > any bad return value from =E2=80=98system*=E2=80=99 to the end of the p= rocedure. > > Maybe something like this: > > (and (zero? (apply system* ...)) > (begin (delete-file ...) #t)) > > It=E2=80=99s a bit clunky but the return value would still be #f if =E2= =80=98system*=E2=80=99 > fails. What do you think? I think you are right as usual. Better in attached? Thanks, ben --------------010603000605090106030505 Content-Type: text/x-patch; name="0001-build-ruby-Remove-cached-gem-after-install.patch" Content-Disposition: attachment; filename="0001-build-ruby-Remove-cached-gem-after-install.patch" Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by newmailhub.uq.edu.au id tBUNqNHd007019 >From c00a032644c02474212be97c185c0953967de4e5 Mon Sep 17 00:00:00 2001 From: Ben Woodcroft Date: Wed, 30 Dec 2015 10:27:33 +1000 Subject: [PATCH] build: ruby: Remove cached gem after install. The .gem file stored in GEM_HOME after install is both redundant and an archive that stores timestamped files which makes builds non-deterministi= c. So delete it after 'gem install'. * guix/build/ruby-build-system.scm (install): Remove cached gem after ins= tall. --- guix/build/ruby-build-system.scm | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/guix/build/ruby-build-system.scm b/guix/build/ruby-build-sys= tem.scm index 2685da1..6439bf6 100644 --- a/guix/build/ruby-build-system.scm +++ b/guix/build/ruby-build-system.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2015 David Thompson ;;; Copyright =C2=A9 2015 Pjotr Prins +;;; Copyright =C2=A9 2015 Ben Woodcroft ;;; ;;; This file is part of GNU Guix. ;;; @@ -115,15 +116,19 @@ GEM-FLAGS are passed to the 'gem' invokation, if pr= esent." (assoc-ref inputs "ruby")) 1)) (out (assoc-ref outputs "out")) - (gem-home (string-append out "/lib/ruby/gems/" ruby-version ".0= "))) - + (gem-home (string-append out "/lib/ruby/gems/" ruby-version ".0= ")) + (gem-name (first-matching-file "\\.gem$"))) (setenv "GEM_HOME" gem-home) (mkdir-p gem-home) - (zero? (apply system* "gem" "install" (first-matching-file "\\.gem$"= ) - "--local" "--ignore-dependencies" - ;; Executables should go into /bin, not /lib/ruby/gems. - "--bindir" (string-append out "/bin") - gem-flags)))) + (and (apply system* "gem" "install" gem-name + "--local" "--ignore-dependencies" + ;; Executables should go into /bin, not /lib/ruby/gems. + "--bindir" (string-append out "/bin") + gem-flags) + ;; Remove the cached gem file as this is unnecessary and contai= ns + ;; timestamped files rendering builds not reproducible. + (begin (delete-file (string-append gem-home "/cache/" gem-name)= ) + #t)))) =20 (define %standard-phases (modify-phases gnu:%standard-phases --=20 2.6.3 --------------010603000605090106030505--