Re: [PATCH] Help Ruby packages be reproducible

[Ben Woodcroft <b.woodcroft@uq.edu.au>]

[-- Attachment #1: Type: text/plain, Size: 2372 bytes --]



On 02/01/16 01:10, Ludovic Courtès wrote:
> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> Ben Woodcroft <b.woodcroft@uq.edu.au> writes:
>>
>>> On 31/12/15 03:26, Ludovic Courtès wrote:
>>>> Ben Woodcroft <b.woodcroft@uq.edu.au> skribis:
>>>>
>>>>> On 29/12/15 15:46, Ben Woodcroft wrote:
>>>>>> Unfortunately none of these builds are reproducible because rubygems
>>>>>> in Guix generally aren't. For one, this is because .gem files are
>>>>>> archives whose contents are timestamped.
>>>>> I should clarify. What I meant was the cache .gem files
>>>>>
>>>>> /gnu/store/ib83mg5zsyr5x2w0m3i1f84gdvdbp5x9-ruby-ascii85-1.0.2/lib/ruby/gems/2.2.0/cache$
>>>>> tar tvf Ascii85-1.0.2.gem |head
>>>>> -r--r--r-- wheel/wheel     703 2015-12-27 22:44 metadata.gz
>>>>> -r--r--r-- wheel/wheel    7436 2015-12-27 22:44 data.tar.gz
>>>>> -r--r--r-- wheel/wheel     268 2015-12-27 22:44 checksums.yaml.gz
>>>> We should arrange so that gems are created with a fixed timestamp and
>>>> UID/GID, and a well-defined file ordering, as with:
>>>>
>>>>     --mtime=@0 --sort=name --owner=root:0 --group=root:0
>>>>
>>>> We also need to make sure gzip is always run with -n/--no-name.  That
>>>> way, the gz files above will not include an additional timestamp.
>>>>
>>>>   From what I can see in
>>>> <git://git.debian.org/git/reproducible/notes.git>, this is not addressed
>>>> yet in other distros.
>>> Ludo are you suggesting we should abandon the deletion approach?
> Ah no, I hadn’t read the proposal when I replied.  Sorry for the
> confusion!
>
>>> I think you are right as usual. Better in attached?
>> It looks good to me, thank you.
> So I guess you (Ricardo?) can push it now.
>
> Ben, do you confirm that the ruby-* packages you tested are indeed
> bit-reproducible after this change, using --rounds=2 or so?
I do confirm this. Well, I used build then check so that dependencies 
weren't checked, but same thing.

I noticed that ruby-lumberjack, a dependency of ruby-guard, now fails to 
build (and fails before this patch and outside guix). This is because it 
fails to pass tests in early January.. not a source of non-determinism I 
was looking for, but thanks for writing tests into that package - picked 
up the easily fixed bug.
https://github.com/bdurand/lumberjack/pull/26

Thanks for pushing the patch.
ben

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: lumberjack.log --]
[-- Type: text/x-log; name="lumberjack.log", Size: 8448 bytes --]

$ gp build ruby-lumberjack
;;; note: source file /home/ben/git/guix/gnu/packages/ruby.scm
;;;       newer than compiled /home/ben/git/guix/gnu/packages/ruby.go
The following derivation will be built:
   /gnu/store/7707bxq0x97vif50w6yhpijpv7k7in91-ruby-lumberjack-1.0.9.drv
@ build-started /gnu/store/7707bxq0x97vif50w6yhpijpv7k7in91-ruby-lumberjack-1.0.9.drv - x86_64-linux /var/log/guix/drvs/77//07bxq0x97vif50w6yhpijpv7k7in91-ruby-lumberjack-1.0.9.drv.bz2
starting phase `set-paths'
environment variable `PATH' set to `/gnu/store/pgks1l9cl696j34v9mb35lk8x6lac3b0-ruby-2.2.4/bin:/gnu/store/xz8c9ihn3p1d8xqkhzf54bjvjqzvqkc0-tar-1.28/bin:/gnu/store/8m7mgixi5539vqrr1vd1nrvh96vjl66j-gzip-1.6/bin:/gnu/store/yq9vkx7f4zw17gs6b09bg4arcgmmjgxn-bzip2-1.0.6/bin:/gnu/store/hk31a1xlgf6n3yi1yzbdyd2bnjmxc60m-xz-5.0.4/bin:/gnu/store/wl84nj6x2mz7zf5r0ajx5gpi0p689ags-file-5.22/bin:/gnu/store/p1h463zxk0f56jl7snw7s8wxvz1k542q-diffutils-3.3/bin:/gnu/store/0xm4q87qi07gcig4riflrf65vyk1k0mp-patch-2.7.5/bin:/gnu/store/g55pwwrmgi58n1yr57m7vc0y69v1ba89-sed-4.2.2/bin:/gnu/store/7fk9v37a5q4r5sygywq0q0qg7iqrj517-findutils-4.4.2/bin:/gnu/store/k8qgvgwn5anbfy8r70h938kxgd46cyxx-gawk-4.1.3/bin:/gnu/store/021rg9bs7xivw0acp1yr0rxw6mf3h64d-grep-2.21/bin:/gnu/store/mnwjrkbfzkb5ifhqf8hssf3cxfvg11l6-coreutils-8.24/bin:/gnu/store/22k2zmc74dvnri91ma0lwxyyhfi1srk0-make-4.1/bin:/gnu/store/311nvir0pz1mhf0mgsmfrw00qfj7yq0j-bash-4.3.39/bin:/gnu/store/k6r37137lfpg3l3igi50c7lj2za7kqly-ld-wrapper-0/bin:/gnu/store/y5psndwpbbkjrf856x757psb708y62dn-binutils-2.25.1/bin:/gnu/store/hddjjpkfvwaf1j1q3qwpvby0rid3k8by-gcc-4.9.3/bin:/gnu/store/qv7bk62c22ms9i11dhfl71hnivyc82k2-glibc-2.22/bin:/gnu/store/qv7bk62c22ms9i11dhfl71hnivyc82k2-glibc-2.22/sbin:/gnu/store/nmzy77nd7adk19h2b90dakkv9lms66f5-ruby-rspec-core-3.2.3/bin:/gnu/store/fdh00cpj5x6a2n7wdndq8cn8msa4s731-ruby-diff-lcs-1.2.5/bin'
environment variable `GEM_PATH' set to `/gnu/store/pgks1l9cl696j34v9mb35lk8x6lac3b0-ruby-2.2.4/lib/ruby/gems/2.2.0:/gnu/store/2f50n60msd5yn2xggnyk0jrfa0iddjc6-ruby-rspec-3.2.0/lib/ruby/gems/2.2.0:/gnu/store/xiwyf9r66hg1gw79i1nm7nb51vi1pd0w-ruby-rspec-expectations-3.2.1/lib/ruby/gems/2.2.0:/gnu/store/f1wzx5zsyf3r4cvg91x9n06si7640dp5-ruby-rspec-mocks-3.2.1/lib/ruby/gems/2.2.0:/gnu/store/nmzy77nd7adk19h2b90dakkv9lms66f5-ruby-rspec-core-3.2.3/lib/ruby/gems/2.2.0:/gnu/store/fdh00cpj5x6a2n7wdndq8cn8msa4s731-ruby-diff-lcs-1.2.5/lib/ruby/gems/2.2.0:/gnu/store/fp7rn3m41g9nilk6bz3q8yjg17j0wkj9-ruby-rspec-support-3.2.2/lib/ruby/gems/2.2.0'
environment variable `CPATH' set to `/gnu/store/pgks1l9cl696j34v9mb35lk8x6lac3b0-ruby-2.2.4/include:/gnu/store/yq9vkx7f4zw17gs6b09bg4arcgmmjgxn-bzip2-1.0.6/include:/gnu/store/hk31a1xlgf6n3yi1yzbdyd2bnjmxc60m-xz-5.0.4/include:/gnu/store/wl84nj6x2mz7zf5r0ajx5gpi0p689ags-file-5.22/include:/gnu/store/k8qgvgwn5anbfy8r70h938kxgd46cyxx-gawk-4.1.3/include:/gnu/store/22k2zmc74dvnri91ma0lwxyyhfi1srk0-make-4.1/include:/gnu/store/y5psndwpbbkjrf856x757psb708y62dn-binutils-2.25.1/include:/gnu/store/hddjjpkfvwaf1j1q3qwpvby0rid3k8by-gcc-4.9.3/include:/gnu/store/qv7bk62c22ms9i11dhfl71hnivyc82k2-glibc-2.22/include:/gnu/store/lyn2331ilik14yy2jqhndshvxmv9r6w5-linux-libre-headers-3.14.37/include'
environment variable `LIBRARY_PATH' set to `/gnu/store/pgks1l9cl696j34v9mb35lk8x6lac3b0-ruby-2.2.4/lib:/gnu/store/2f50n60msd5yn2xggnyk0jrfa0iddjc6-ruby-rspec-3.2.0/lib:/gnu/store/yq9vkx7f4zw17gs6b09bg4arcgmmjgxn-bzip2-1.0.6/lib:/gnu/store/hk31a1xlgf6n3yi1yzbdyd2bnjmxc60m-xz-5.0.4/lib:/gnu/store/wl84nj6x2mz7zf5r0ajx5gpi0p689ags-file-5.22/lib:/gnu/store/k8qgvgwn5anbfy8r70h938kxgd46cyxx-gawk-4.1.3/lib:/gnu/store/y5psndwpbbkjrf856x757psb708y62dn-binutils-2.25.1/lib:/gnu/store/qv7bk62c22ms9i11dhfl71hnivyc82k2-glibc-2.22/lib:/gnu/store/xmkpq5mx40lv67368cq3zx315j3lxnax-glibc-utf8-locales-2.22/lib:/gnu/store/xiwyf9r66hg1gw79i1nm7nb51vi1pd0w-ruby-rspec-expectations-3.2.1/lib:/gnu/store/f1wzx5zsyf3r4cvg91x9n06si7640dp5-ruby-rspec-mocks-3.2.1/lib:/gnu/store/nmzy77nd7adk19h2b90dakkv9lms66f5-ruby-rspec-core-3.2.3/lib:/gnu/store/fdh00cpj5x6a2n7wdndq8cn8msa4s731-ruby-diff-lcs-1.2.5/lib:/gnu/store/fp7rn3m41g9nilk6bz3q8yjg17j0wkj9-ruby-rspec-support-3.2.2/lib'
environment variable `GUIX_LOCPATH' set to `/gnu/store/xmkpq5mx40lv67368cq3zx315j3lxnax-glibc-utf8-locales-2.22/lib/locale'
phase `set-paths' succeeded after 0.0 seconds
starting phase `install-locale'
using 'en_US.utf8' locale for category "LC_ALL"
phase `install-locale' succeeded after 0.0 seconds
starting phase `unpack'
Unpacked gem: '/tmp/guix-build-ruby-lumberjack-1.0.9.drv-0/0m1ikqigchci66hzcwszfyxjb5yq0aky-lumberjack-1.0.9'
phase `unpack' succeeded after 0.5 seconds
starting phase `patch-usr-bin-file'
phase `patch-usr-bin-file' succeeded after 0.0 seconds
starting phase `patch-source-shebangs'
phase `patch-source-shebangs' succeeded after 0.0 seconds
starting phase `patch-generated-file-shebangs'
phase `patch-generated-file-shebangs' succeeded after 0.0 seconds
starting phase `build'
  Successfully built RubyGem
  Name: lumberjack
  Version: 1.0.9
  File: lumberjack-1.0.9.gem
phase `build' succeeded after 0.5 seconds
starting phase `check'
/gnu/store/pgks1l9cl696j34v9mb35lk8x6lac3b0-ruby-2.2.4/bin/ruby -I/gnu/store/fp7rn3m41g9nilk6bz3q8yjg17j0wkj9-ruby-rspec-support-3.2.2/lib/ruby/gems/2.2.0/gems/rspec-support-3.2.2/lib:/gnu/store/nmzy77nd7adk19h2b90dakkv9lms66f5-ruby-rspec-core-3.2.3/lib/ruby/gems/2.2.0/gems/rspec-core-3.2.3/lib /gnu/store/nmzy77nd7adk19h2b90dakkv9lms66f5-ruby-rspec-core-3.2.3/lib/ruby/gems/2.2.0/gems/rspec-core-3.2.3/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb
.F.............................................................................................

Failures:

  1) Lumberjack::Device::DateRollingLogFile should roll the file weekly
     Failure/Error: File.read("#{log_file}.#{today.strftime('week-of-%Y-%m-%d')}").should == "test week one#{Lumberjack::LINE_SEPARATOR}"
     Errno::ENOENT:
       No such file or directory @ rb_sysopen - /tmp/guix-build-ruby-lumberjack-1.0.9.drv-0/gem/spec/tmp/b200452538.log.week-of-2016-01-02
     # ./spec/device/date_rolling_log_file_spec.rb:45:in `read'
     # ./spec/device/date_rolling_log_file_spec.rb:45:in `block (2 levels) in <top (required)>'

Deprecation Warnings:

Using `should` from rspec-expectations' old `:should` syntax without explicitly enabling the syntax is deprecated. Use the new `:expect` syntax or explicitly enable `:should` with `config.expect_with(:rspec) { |c| c.syntax = :should }` instead. Called from /tmp/guix-build-ruby-lumberjack-1.0.9.drv-0/gem/spec/device/date_rolling_log_file_spec.rb:28:in `block (2 levels) in <top (required)>'.

Using `stub` from rspec-mocks' old `:should` syntax without explicitly enabling the syntax is deprecated. Use the new `:expect` syntax or explicitly enable `:should` instead. Called from /tmp/guix-build-ruby-lumberjack-1.0.9.drv-0/gem/spec/device/date_rolling_log_file_spec.rb:23:in `block (2 levels) in <top (required)>'.


If you need more of the backtrace for any of these deprecations to
identify where to make the necessary changes, you can configure
`config.raise_errors_for_deprecations!`, and it will turn the
deprecation warnings into errors, giving you the full backtrace.

2 deprecation warnings total

Finished in 1.16 seconds (files took 0.21941 seconds to load)
95 examples, 1 failure

Failed examples:

rspec ./spec/device/date_rolling_log_file_spec.rb:32 # Lumberjack::Device::DateRollingLogFile should roll the file weekly

/gnu/store/pgks1l9cl696j34v9mb35lk8x6lac3b0-ruby-2.2.4/bin/ruby -I/gnu/store/fp7rn3m41g9nilk6bz3q8yjg17j0wkj9-ruby-rspec-support-3.2.2/lib/ruby/gems/2.2.0/gems/rspec-support-3.2.2/lib:/gnu/store/nmzy77nd7adk19h2b90dakkv9lms66f5-ruby-rspec-core-3.2.3/lib/ruby/gems/2.2.0/gems/rspec-core-3.2.3/lib /gnu/store/nmzy77nd7adk19h2b90dakkv9lms66f5-ruby-rspec-core-3.2.3/lib/ruby/gems/2.2.0/gems/rspec-core-3.2.3/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb failed
phase `check' failed after 1.8 seconds
builder for `/gnu/store/7707bxq0x97vif50w6yhpijpv7k7in91-ruby-lumberjack-1.0.9.drv' failed with exit code 1
@ build-failed /gnu/store/7707bxq0x97vif50w6yhpijpv7k7in91-ruby-lumberjack-1.0.9.drv - 1 builder for `/gnu/store/7707bxq0x97vif50w6yhpijpv7k7in91-ruby-lumberjack-1.0.9.drv' failed with exit code 1
guix build: error: build failed: build of `/gnu/store/7707bxq0x97vif50w6yhpijpv7k7in91-ruby-lumberjack-1.0.9.drv' failed