From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ben Woodcroft Subject: Re: [PATCH] Help Ruby packages be reproducible Date: Sun, 3 Jan 2016 01:02:02 +1000 Message-ID: <5687E66A.80101@uq.edu.au> References: <56821E47.9010400@uq.edu.au> <87io3h4ta3.fsf@elephly.net> <56832AA3.8030109@uq.edu.au> <56832BA6.6030806@uq.edu.au> <877fjwbavn.fsf@elephly.net> <56846E35.6010805@uq.edu.au> <87wprv9bqb.fsf@elephly.net> <87lh89l4hw.fsf@gnu.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------020201090508090207020808" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:42621) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aFNhG-0003Ae-AT for guix-devel@gnu.org; Sat, 02 Jan 2016 10:02:24 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aFNhC-0007XP-7G for guix-devel@gnu.org; Sat, 02 Jan 2016 10:02:22 -0500 In-Reply-To: <87lh89l4hw.fsf@gnu.org> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: =?UTF-8?Q?Ludovic_Court=c3=a8s?= , Ricardo Wurmus Cc: "guix-devel@gnu.org" This is a multi-part message in MIME format. --------------020201090508090207020808 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by newmailhub.uq.edu.au id u02F25b6021454 On 02/01/16 01:10, Ludovic Court=C3=A8s wrote: > Ricardo Wurmus skribis: > >> Ben Woodcroft writes: >> >>> On 31/12/15 03:26, Ludovic Court=C3=A8s wrote: >>>> Ben Woodcroft skribis: >>>> >>>>> On 29/12/15 15:46, Ben Woodcroft wrote: >>>>>> Unfortunately none of these builds are reproducible because rubyge= ms >>>>>> in Guix generally aren't. For one, this is because .gem files are >>>>>> archives whose contents are timestamped. >>>>> I should clarify. What I meant was the cache .gem files >>>>> >>>>> /gnu/store/ib83mg5zsyr5x2w0m3i1f84gdvdbp5x9-ruby-ascii85-1.0.2/lib/= ruby/gems/2.2.0/cache$ >>>>> tar tvf Ascii85-1.0.2.gem |head >>>>> -r--r--r-- wheel/wheel 703 2015-12-27 22:44 metadata.gz >>>>> -r--r--r-- wheel/wheel 7436 2015-12-27 22:44 data.tar.gz >>>>> -r--r--r-- wheel/wheel 268 2015-12-27 22:44 checksums.yaml.gz >>>> We should arrange so that gems are created with a fixed timestamp an= d >>>> UID/GID, and a well-defined file ordering, as with: >>>> >>>> --mtime=3D@0 --sort=3Dname --owner=3Droot:0 --group=3Droot:0 >>>> >>>> We also need to make sure gzip is always run with -n/--no-name. Tha= t >>>> way, the gz files above will not include an additional timestamp. >>>> >>>> From what I can see in >>>> , this is not addre= ssed >>>> yet in other distros. >>> Ludo are you suggesting we should abandon the deletion approach? > Ah no, I hadn=E2=80=99t read the proposal when I replied. Sorry for th= e > confusion! > >>> I think you are right as usual. Better in attached? >> It looks good to me, thank you. > So I guess you (Ricardo?) can push it now. > > Ben, do you confirm that the ruby-* packages you tested are indeed > bit-reproducible after this change, using --rounds=3D2 or so? I do confirm this. Well, I used build then check so that dependencies=20 weren't checked, but same thing. I noticed that ruby-lumberjack, a dependency of ruby-guard, now fails to=20 build (and fails before this patch and outside guix). This is because it=20 fails to pass tests in early January.. not a source of non-determinism I=20 was looking for, but thanks for writing tests into that package - picked=20 up the easily fixed bug. https://github.com/bdurand/lumberjack/pull/26 Thanks for pushing the patch. ben --------------020201090508090207020808 Content-Type: text/x-log; name="lumberjack.log" Content-Disposition: attachment; filename="lumberjack.log" Content-Transfer-Encoding: quoted-printable $ gp build ruby-lumberjack ;;; note: source file /home/ben/git/guix/gnu/packages/ruby.scm ;;; newer than compiled /home/ben/git/guix/gnu/packages/ruby.go The following derivation will be built: /gnu/store/7707bxq0x97vif50w6yhpijpv7k7in91-ruby-lumberjack-1.0.9.drv @ build-started /gnu/store/7707bxq0x97vif50w6yhpijpv7k7in91-ruby-lumberja= ck-1.0.9.drv - x86_64-linux /var/log/guix/drvs/77//07bxq0x97vif50w6yhpijp= v7k7in91-ruby-lumberjack-1.0.9.drv.bz2 starting phase `set-paths' environment variable `PATH' set to `/gnu/store/pgks1l9cl696j34v9mb35lk8x6= lac3b0-ruby-2.2.4/bin:/gnu/store/xz8c9ihn3p1d8xqkhzf54bjvjqzvqkc0-tar-1.2= 8/bin:/gnu/store/8m7mgixi5539vqrr1vd1nrvh96vjl66j-gzip-1.6/bin:/gnu/store= /yq9vkx7f4zw17gs6b09bg4arcgmmjgxn-bzip2-1.0.6/bin:/gnu/store/hk31a1xlgf6n= 3yi1yzbdyd2bnjmxc60m-xz-5.0.4/bin:/gnu/store/wl84nj6x2mz7zf5r0ajx5gpi0p68= 9ags-file-5.22/bin:/gnu/store/p1h463zxk0f56jl7snw7s8wxvz1k542q-diffutils-= 3.3/bin:/gnu/store/0xm4q87qi07gcig4riflrf65vyk1k0mp-patch-2.7.5/bin:/gnu/= store/g55pwwrmgi58n1yr57m7vc0y69v1ba89-sed-4.2.2/bin:/gnu/store/7fk9v37a5= q4r5sygywq0q0qg7iqrj517-findutils-4.4.2/bin:/gnu/store/k8qgvgwn5anbfy8r70= h938kxgd46cyxx-gawk-4.1.3/bin:/gnu/store/021rg9bs7xivw0acp1yr0rxw6mf3h64d= -grep-2.21/bin:/gnu/store/mnwjrkbfzkb5ifhqf8hssf3cxfvg11l6-coreutils-8.24= /bin:/gnu/store/22k2zmc74dvnri91ma0lwxyyhfi1srk0-make-4.1/bin:/gnu/store/= 311nvir0pz1mhf0mgsmfrw00qfj7yq0j-bash-4.3.39/bin:/gnu/store/k6r37137lfpg3= l3igi50c7lj2za7kqly-ld-wrapper-0/bin:/gnu/store/y5psndwpbbkjrf856x757psb7= 08y62dn-binutils-2.25.1/bin:/gnu/store/hddjjpkfvwaf1j1q3qwpvby0rid3k8by-g= cc-4.9.3/bin:/gnu/store/qv7bk62c22ms9i11dhfl71hnivyc82k2-glibc-2.22/bin:/= gnu/store/qv7bk62c22ms9i11dhfl71hnivyc82k2-glibc-2.22/sbin:/gnu/store/nmz= y77nd7adk19h2b90dakkv9lms66f5-ruby-rspec-core-3.2.3/bin:/gnu/store/fdh00c= pj5x6a2n7wdndq8cn8msa4s731-ruby-diff-lcs-1.2.5/bin' environment variable `GEM_PATH' set to `/gnu/store/pgks1l9cl696j34v9mb35l= k8x6lac3b0-ruby-2.2.4/lib/ruby/gems/2.2.0:/gnu/store/2f50n60msd5yn2xggnyk= 0jrfa0iddjc6-ruby-rspec-3.2.0/lib/ruby/gems/2.2.0:/gnu/store/xiwyf9r66hg1= gw79i1nm7nb51vi1pd0w-ruby-rspec-expectations-3.2.1/lib/ruby/gems/2.2.0:/g= nu/store/f1wzx5zsyf3r4cvg91x9n06si7640dp5-ruby-rspec-mocks-3.2.1/lib/ruby= /gems/2.2.0:/gnu/store/nmzy77nd7adk19h2b90dakkv9lms66f5-ruby-rspec-core-3= =2E2.3/lib/ruby/gems/2.2.0:/gnu/store/fdh00cpj5x6a2n7wdndq8cn8msa4s731-ru= by-diff-lcs-1.2.5/lib/ruby/gems/2.2.0:/gnu/store/fp7rn3m41g9nilk6bz3q8yjg= 17j0wkj9-ruby-rspec-support-3.2.2/lib/ruby/gems/2.2.0' environment variable `CPATH' set to `/gnu/store/pgks1l9cl696j34v9mb35lk8x= 6lac3b0-ruby-2.2.4/include:/gnu/store/yq9vkx7f4zw17gs6b09bg4arcgmmjgxn-bz= ip2-1.0.6/include:/gnu/store/hk31a1xlgf6n3yi1yzbdyd2bnjmxc60m-xz-5.0.4/in= clude:/gnu/store/wl84nj6x2mz7zf5r0ajx5gpi0p689ags-file-5.22/include:/gnu/= store/k8qgvgwn5anbfy8r70h938kxgd46cyxx-gawk-4.1.3/include:/gnu/store/22k2= zmc74dvnri91ma0lwxyyhfi1srk0-make-4.1/include:/gnu/store/y5psndwpbbkjrf85= 6x757psb708y62dn-binutils-2.25.1/include:/gnu/store/hddjjpkfvwaf1j1q3qwpv= by0rid3k8by-gcc-4.9.3/include:/gnu/store/qv7bk62c22ms9i11dhfl71hnivyc82k2= -glibc-2.22/include:/gnu/store/lyn2331ilik14yy2jqhndshvxmv9r6w5-linux-lib= re-headers-3.14.37/include' environment variable `LIBRARY_PATH' set to `/gnu/store/pgks1l9cl696j34v9m= b35lk8x6lac3b0-ruby-2.2.4/lib:/gnu/store/2f50n60msd5yn2xggnyk0jrfa0iddjc6= -ruby-rspec-3.2.0/lib:/gnu/store/yq9vkx7f4zw17gs6b09bg4arcgmmjgxn-bzip2-1= =2E0.6/lib:/gnu/store/hk31a1xlgf6n3yi1yzbdyd2bnjmxc60m-xz-5.0.4/lib:/gnu/= store/wl84nj6x2mz7zf5r0ajx5gpi0p689ags-file-5.22/lib:/gnu/store/k8qgvgwn5= anbfy8r70h938kxgd46cyxx-gawk-4.1.3/lib:/gnu/store/y5psndwpbbkjrf856x757ps= b708y62dn-binutils-2.25.1/lib:/gnu/store/qv7bk62c22ms9i11dhfl71hnivyc82k2= -glibc-2.22/lib:/gnu/store/xmkpq5mx40lv67368cq3zx315j3lxnax-glibc-utf8-lo= cales-2.22/lib:/gnu/store/xiwyf9r66hg1gw79i1nm7nb51vi1pd0w-ruby-rspec-exp= ectations-3.2.1/lib:/gnu/store/f1wzx5zsyf3r4cvg91x9n06si7640dp5-ruby-rspe= c-mocks-3.2.1/lib:/gnu/store/nmzy77nd7adk19h2b90dakkv9lms66f5-ruby-rspec-= core-3.2.3/lib:/gnu/store/fdh00cpj5x6a2n7wdndq8cn8msa4s731-ruby-diff-lcs-= 1.2.5/lib:/gnu/store/fp7rn3m41g9nilk6bz3q8yjg17j0wkj9-ruby-rspec-support-= 3.2.2/lib' environment variable `GUIX_LOCPATH' set to `/gnu/store/xmkpq5mx40lv67368c= q3zx315j3lxnax-glibc-utf8-locales-2.22/lib/locale' phase `set-paths' succeeded after 0.0 seconds starting phase `install-locale' using 'en_US.utf8' locale for category "LC_ALL" phase `install-locale' succeeded after 0.0 seconds starting phase `unpack' Unpacked gem: '/tmp/guix-build-ruby-lumberjack-1.0.9.drv-0/0m1ikqigchci66= hzcwszfyxjb5yq0aky-lumberjack-1.0.9' phase `unpack' succeeded after 0.5 seconds starting phase `patch-usr-bin-file' phase `patch-usr-bin-file' succeeded after 0.0 seconds starting phase `patch-source-shebangs' phase `patch-source-shebangs' succeeded after 0.0 seconds starting phase `patch-generated-file-shebangs' phase `patch-generated-file-shebangs' succeeded after 0.0 seconds starting phase `build' Successfully built RubyGem Name: lumberjack Version: 1.0.9 File: lumberjack-1.0.9.gem phase `build' succeeded after 0.5 seconds starting phase `check' /gnu/store/pgks1l9cl696j34v9mb35lk8x6lac3b0-ruby-2.2.4/bin/ruby -I/gnu/st= ore/fp7rn3m41g9nilk6bz3q8yjg17j0wkj9-ruby-rspec-support-3.2.2/lib/ruby/ge= ms/2.2.0/gems/rspec-support-3.2.2/lib:/gnu/store/nmzy77nd7adk19h2b90dakkv= 9lms66f5-ruby-rspec-core-3.2.3/lib/ruby/gems/2.2.0/gems/rspec-core-3.2.3/= lib /gnu/store/nmzy77nd7adk19h2b90dakkv9lms66f5-ruby-rspec-core-3.2.3/lib= /ruby/gems/2.2.0/gems/rspec-core-3.2.3/exe/rspec --pattern spec/\*\*\{,/\= */\*\*\}/\*_spec.rb =2EF.....................................................................= =2E....................... Failures: 1) Lumberjack::Device::DateRollingLogFile should roll the file weekly Failure/Error: File.read("#{log_file}.#{today.strftime('week-of-%Y-%= m-%d')}").should =3D=3D "test week one#{Lumberjack::LINE_SEPARATOR}" Errno::ENOENT: No such file or directory @ rb_sysopen - /tmp/guix-build-ruby-lumb= erjack-1.0.9.drv-0/gem/spec/tmp/b200452538.log.week-of-2016-01-02 # ./spec/device/date_rolling_log_file_spec.rb:45:in `read' # ./spec/device/date_rolling_log_file_spec.rb:45:in `block (2 levels= ) in ' Deprecation Warnings: Using `should` from rspec-expectations' old `:should` syntax without expl= icitly enabling the syntax is deprecated. Use the new `:expect` syntax or= explicitly enable `:should` with `config.expect_with(:rspec) { |c| c.syn= tax =3D :should }` instead. Called from /tmp/guix-build-ruby-lumberjack-1= =2E0.9.drv-0/gem/spec/device/date_rolling_log_file_spec.rb:28:in `block (= 2 levels) in '. Using `stub` from rspec-mocks' old `:should` syntax without explicitly en= abling the syntax is deprecated. Use the new `:expect` syntax or explicit= ly enable `:should` instead. Called from /tmp/guix-build-ruby-lumberjack-= 1.0.9.drv-0/gem/spec/device/date_rolling_log_file_spec.rb:23:in `block (2= levels) in '. If you need more of the backtrace for any of these deprecations to identify where to make the necessary changes, you can configure `config.raise_errors_for_deprecations!`, and it will turn the deprecation warnings into errors, giving you the full backtrace. 2 deprecation warnings total Finished in 1.16 seconds (files took 0.21941 seconds to load) 95 examples, 1 failure Failed examples: rspec ./spec/device/date_rolling_log_file_spec.rb:32 # Lumberjack::Device= ::DateRollingLogFile should roll the file weekly /gnu/store/pgks1l9cl696j34v9mb35lk8x6lac3b0-ruby-2.2.4/bin/ruby -I/gnu/st= ore/fp7rn3m41g9nilk6bz3q8yjg17j0wkj9-ruby-rspec-support-3.2.2/lib/ruby/ge= ms/2.2.0/gems/rspec-support-3.2.2/lib:/gnu/store/nmzy77nd7adk19h2b90dakkv= 9lms66f5-ruby-rspec-core-3.2.3/lib/ruby/gems/2.2.0/gems/rspec-core-3.2.3/= lib /gnu/store/nmzy77nd7adk19h2b90dakkv9lms66f5-ruby-rspec-core-3.2.3/lib= /ruby/gems/2.2.0/gems/rspec-core-3.2.3/exe/rspec --pattern spec/\*\*\{,/\= */\*\*\}/\*_spec.rb failed phase `check' failed after 1.8 seconds builder for `/gnu/store/7707bxq0x97vif50w6yhpijpv7k7in91-ruby-lumberjack-= 1.0.9.drv' failed with exit code 1 @ build-failed /gnu/store/7707bxq0x97vif50w6yhpijpv7k7in91-ruby-lumberjac= k-1.0.9.drv - 1 builder for `/gnu/store/7707bxq0x97vif50w6yhpijpv7k7in91-= ruby-lumberjack-1.0.9.drv' failed with exit code 1 guix build: error: build failed: build of `/gnu/store/7707bxq0x97vif50w6y= hpijpv7k7in91-ruby-lumberjack-1.0.9.drv' failed --------------020201090508090207020808--