[PATCH] gnu: Add ruby-nokogiri

[PATCH] gnu: Add ruby-nokogiri

[Pjotr Prins]

[-- Attachment #1: Type: text/plain, Size: 889 bytes --]

Sorry, the second mail got sent off too early.

Patch for the Ruby-nokogiri gem, an infamous hard to install piece of
software. Not so hard to add to GNU Guix, though I had to run 'rake
gem' 2x to create the gem (the first time it merely installs the build
directory). The tests are skipped because they currently expect the
gem to be already installed (not only built!).

I documented the process of making the patch here:

  https://github.com/pjotrp/guix-notes/blob/master/RUBYGEMS-Nokogiri.org

David Thompson gave me advice on how to introduce the new 'gem-flags'
key. Though the blame is mine if I got it wrong :)

It is a work in progress - I'll send a sanity patch upstream for
Nokogiri and get testing working. But I thought I'd share this with
you first for comments. Inclusion can be useful to others as the
package does work. It is a critical package for many Ruby gems.

Pj.


[-- Attachment #2: 0001-gnu-Add-ruby-nokogiri.patch --]
[-- Type: text/x-diff, Size: 2711 bytes --]

From e2fd935a659cacde5cb74bef19406f056a262f79 Mon Sep 17 00:00:00 2001
From: pjotrp <pjotr.public01@thebird.nl>
Date: Mon, 13 Jul 2015 14:56:40 +0200
Subject: [PATCH] gnu: Add ruby-nokogiri

* gnu/packages/ruby.scm (gnu-nokogiri): New variable
---
 gnu/packages/ruby.scm |   40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/gnu/packages/ruby.scm b/gnu/packages/ruby.scm
index bb7d865..85cb0d4 100644
--- a/gnu/packages/ruby.scm
+++ b/gnu/packages/ruby.scm
@@ -34,6 +34,7 @@
   #:use-module (guix git-download)
   #:use-module (guix utils)
   #:use-module (guix build-system gnu)
+  #:use-module (gnu packages xml)
   #:use-module (guix build-system ruby))
 
 (define-public ruby
@@ -519,6 +520,45 @@ using Net::HTTP, supporting reconnection and retry according to RFC 2616.")
     (home-page "https://github.com/drbrain/net-http-persistent")
     (license license:expat)))
 
+(define-public ruby-nokogiri
+  (package
+    (name "ruby-nokogiri")
+    (version "1.6.6.2")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append
+		    "https://github.com/sparklemotion/nokogiri/archive/v"
+                    version ".tar.gz"))
+              (file-name (string-append name "-" version ".tar.gz"))
+	      (patches (map search-patch
+                            (list "ruby-nokogiri-Rakefile.patch")))
+              (sha256
+               (base32
+                "1dpmmxr8azbyvhhmw9hpyk3dds577vsd6c312gh2s7kgjd98nd9j"))))
+    (build-system ruby-build-system)
+    (arguments
+     '(
+       #:tests? #f  ;; test fails because nokogiri can only test with a built extension (now part of install phase)
+       #:gem-flags (list "--use-system-libraries" (string-append "--with-xml2-include=" (assoc-ref %build-inputs "libxml2") "/include/libxml2" ))
+       #:phases (alist-replace
+                 'build
+                 (lambda _
+		   ;; calling rake gem 2x begets a gem
+		   (system* "rake" "gem")
+		   (zero? (system* "rake" "gem")))
+		  %standard-phases)))
+    (native-inputs
+     `(("ruby-hoe" ,ruby-hoe)
+       ("ruby-rake-compiler", ruby-rake-compiler)))
+    (inputs
+     `(("zlib" ,zlib)
+       ("libxml2" ,libxml2)
+       ("libxslt" ,libxslt)))
+    (synopsis "Nokogiri (鋸) is an HTML, XML, SAX, and Reader parser")
+    (description "Nokogiri parses and searches XML/HTML very quickly, and also has correctly implemented CSS3 selector support as well as XPath 1.0 support.")
+    (home-page "http://www.nokogiri.org/")
+    (license license:x11)))
+
 (define-public ruby-minitest
   (package
     (name "ruby-minitest")
-- 
1.7.10.4

Re: [PATCH] gnu: Add ruby-nokogiri

[Ricardo Wurmus]

Hi Pjotr,

following are a couple of comments about the patch.  (I did not try it
myself.)

> From e2fd935a659cacde5cb74bef19406f056a262f79 Mon Sep 17 00:00:00 2001
> From: pjotrp <pjotr.public01@thebird.nl>
> Date: Mon, 13 Jul 2015 14:56:40 +0200
> Subject: [PATCH] gnu: Add ruby-nokogiri

> * gnu/packages/ruby.scm (gnu-nokogiri): New variable

The name of the variable is ‘ruby-nokogiri’, not ‘gnu-nokogiri’.  There
also should be a period at the end.

> +(define-public ruby-nokogiri
> +  (package
> +    (name "ruby-nokogiri")
> +    (version "1.6.6.2")
> +    (source (origin
> +              (method url-fetch)
> +              (uri (string-append
> +		    "https://github.com/sparklemotion/nokogiri/archive/v"
> +                    version ".tar.gz"))
> +              (file-name (string-append name "-" version ".tar.gz"))
> +	      (patches (map search-patch
> +                            (list "ruby-nokogiri-Rakefile.patch")))

‘(patches ...)’ should be aligned with the previous line.  ‘(list’
should be aligned with ‘search-patch’.  Since you only have one patch,
though, I think using ‘(map search-patch’ is odd.

> +    (arguments
> +     '(
> +       #:tests? #f  ;; test fails because nokogiri can only test with a built extension (now part of install phase)

‘'(’ should not be on a line of its own.  The margin comment should only
have one semicolon and the text should be broken up to avoid long lines.

> +       #:gem-flags (list "--use-system-libraries" (string-append "--with-xml2-include=" (assoc-ref %build-inputs "libxml2") "/include/libxml2" ))

This is a very long line.  It can be broken up easily.

> +       #:phases (alist-replace
> +                 'build
> +                 (lambda _
> +		   ;; calling rake gem 2x begets a gem
> +		   (system* "rake" "gem")
> +		   (zero? (system* "rake" "gem")))
> +		  %standard-phases)))

The alignment here is off.  Also consider using the ‘modify-phases’
syntax instead.  Why exactly is “rake gem” called twice?  What is
different the second time around?

> +    (native-inputs
> +     `(("ruby-hoe" ,ruby-hoe)
> +       ("ruby-rake-compiler", ruby-rake-compiler)))
> +    (inputs
> +     `(("zlib" ,zlib)
> +       ("libxml2" ,libxml2)
> +       ("libxslt" ,libxslt)))
> +    (synopsis "Nokogiri (鋸) is an HTML, XML, SAX, and Reader parser")
> +    (description "Nokogiri parses and searches XML/HTML very quickly, and also has correctly implemented CSS3 selector support as well as XPath 1.0 support.")

Please break apart the long description into multiple lines.  In Emacs
you can do this easily with ‘fill-paragraph’ (bound to ‘M-q’ by
default).

> +    (home-page "http://www.nokogiri.org/")
> +    (license license:x11)))
> +

~~ Ricardo

Re: [PATCH] gnu: Add ruby-nokogiri

[Pjotr Prins]

Hi Ricardo,

Many thanks for your help and bearing with me. Here the improved
version of the code, would this be OK? I'll repost the patches when
this looks good.

Pj.


diff --git a/gnu/packages/patches/ruby-nokogiri-Rakefile.patch b/gnu/packages/patches/ruby-nokogiri-Rakefile.patch
new file mode 100644
index 0000000..91923eb
--- /dev/null
+++ b/gnu/packages/patches/ruby-nokogiri-Rakefile.patch
@@ -0,0 +1,10 @@
+--- nokogiri-1.6.6.2/Rakefile	2015-01-23 19:22:19.000000000 +0100
++++ /tmp/Rakefile	2015-07-06 10:15:16.153659819 +0200
+@@ -134,7 +134,6 @@
+       # for more details, see:
+       # - https://github.com/sparklemotion/nokogiri/issues/1102
+       # - https://github.com/luislavena/mini_portile/issues/32
+-      ["mini_portile",    "~> 0.6.0"],
+     ]
+   end
+ 
diff --git a/gnu/packages/ruby.scm b/gnu/packages/ruby.scm
index bb7d865..987b8c1 100644
--- a/gnu/packages/ruby.scm
+++ b/gnu/packages/ruby.scm
@@ -34,6 +34,7 @@
   #:use-module (guix git-download)
   #:use-module (guix utils)
   #:use-module (guix build-system gnu)
+  #:use-module (gnu packages xml)
   #:use-module (guix build-system ruby))
 
 (define-public ruby
@@ -519,6 +520,52 @@ using Net::HTTP, supporting reconnection and retry according to RFC 2616.")
     (home-page "https://github.com/drbrain/net-http-persistent")
     (license license:expat)))
 
+(define-public ruby-nokogiri
+  (package
+    (name "ruby-nokogiri")
+    (version "1.6.6.2")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append
+                    "https://github.com/sparklemotion/nokogiri/archive/v"
+                    version ".tar.gz"))
+              (file-name (string-append name "-" version ".tar.gz"))
+              (patches (list (search-patch "ruby-nokogiri-Rakefile.patch")))
+              (sha256
+               (base32
+                "1dpmmxr8azbyvhhmw9hpyk3dds577vsd6c312gh2s7kgjd98nd9j"))))
+    (build-system ruby-build-system)
+    (arguments
+     '(#:tests? #f  ;; test fails because nokogiri can only test with
+                    ;; an installed extension (now part of install
+                    ;; phase
+       #:gem-flags (list "--use-system-libraries"
+                     (string-append "--with-xml2-include="
+                       (assoc-ref %build-inputs "libxml2")
+                       "/include/libxml2" ))
+       #:phases
+         (modify-phases %standard-phases
+           (replace 'build
+                    (lambda _
+                      ;; calling rake gem 2x begets a gem. The first time
+                      ;; only the build-dir is created
+                      (zero? (begin
+                               (system* "rake" "gem")
+                               (system* "rake" "gem"))))))))
+    (native-inputs
+     `(("ruby-hoe" ,ruby-hoe)
+       ("ruby-rake-compiler", ruby-rake-compiler)))
+    (inputs
+     `(("zlib" ,zlib)
+       ("libxml2" ,libxml2)
+       ("libxslt" ,libxslt)))
+    (synopsis "Nokogiri (鋸) is an HTML, XML, SAX, and Reader parser")
+    (description "Nokogiri parses and searches XML/HTML very quickly,
+and also has correctly implemented CSS3 selector support as well as
+XPath 1.0 support.")
+    (home-page "http://www.nokogiri.org/")
+    (license license:x11)))
+
 (define-public ruby-minitest
   (package
     (name "ruby-minitest")

Re: [PATCH] gnu: Add ruby-nokogiri

[Ricardo Wurmus]

> +    (arguments
> +     '(#:tests? #f  ;; test fails because nokogiri can only test with
> +                    ;; an installed extension (now part of install
> +                    ;; phase

You should use a single semicolon for each line here or convert the
comment to not share a line with ‘#:tests?’.

> +       #:gem-flags (list "--use-system-libraries"
> +                     (string-append "--with-xml2-include="
> +                       (assoc-ref %build-inputs "libxml2")
> +                       "/include/libxml2" ))

The alignment here is wrong.  ‘(string-append’ should be aligned with
‘"--use-system-libraries"’ and ‘(assoc-ref’ should be aligned with
‘"--with-xml2-include="’, if I’m not mistaken.

> +       #:phases
> +         (modify-phases %standard-phases
> +           (replace 'build
> +                    (lambda _
> +                      ;; calling rake gem 2x begets a gem. The first time
> +                      ;; only the build-dir is created

If this is so, could we not just create the build directory before
running rake?

> +                      (zero? (begin
> +                               (system* "rake" "gem")
> +                               (system* "rake" "gem"))))))))
> +    (native-inputs
> +     `(("ruby-hoe" ,ruby-hoe)
> +       ("ruby-rake-compiler", ruby-rake-compiler)))
> +    (inputs
> +     `(("zlib" ,zlib)
> +       ("libxml2" ,libxml2)
> +       ("libxslt" ,libxslt)))
> +    (synopsis "Nokogiri (鋸) is an HTML, XML, SAX, and Reader parser")
> +    (description "Nokogiri parses and searches XML/HTML very quickly,
> +and also has correctly implemented CSS3 selector support as well as
> +XPath 1.0 support.")
> +    (home-page "http://www.nokogiri.org/")
> +    (license license:x11)))
> +

~~ Ricardo

Re: [PATCH] gnu: Add ruby-nokogiri

[Pjotr Prins]

There you go :) Share the love.

http://www.nokogiri.org/tutorials/installing_nokogiri.html#gnu_guix

On Tue, Jul 14, 2015 at 11:06:13AM +0200, Pjotr Prins wrote:
> Hi Ricardo,
> 
> Many thanks for your help and bearing with me. Here the improved
> version of the code, would this be OK? I'll repost the patches when
> this looks good.
> 
> Pj.

Re: [PATCH] gnu: Add ruby-nokogiri

[Ben Woodcroft]

On 18/02/16 08:37, Pjotr Prins wrote:
> There you go :) Share the love.
>
> http://www.nokogiri.org/tutorials/installing_nokogiri.html#gnu_guix
Excellent. However, I'm not sure that is entirely correct:

 >   guix package -i ruby-nokogiri
 >will install Nokogiri with tool and libraries and all its dependencies 
(including a recent Ruby, libxml2 and libxslt).

It doesn't install Ruby unless you make it

guix package -i ruby-nokogiri ruby


right?

ben

Foreign packages (formerly Re: [PATCH] gnu: Add ruby-nokogiri)

[Pjotr Prins]

On Thu, Feb 18, 2016 at 09:05:26AM +1000, Ben Woodcroft wrote:
> On 18/02/16 08:37, Pjotr Prins wrote:
> >There you go :) Share the love.
> >
> >http://www.nokogiri.org/tutorials/installing_nokogiri.html#gnu_guix
> Excellent. However, I'm not sure that is entirely correct:
> 
> >   guix package -i ruby-nokogiri
> >will install Nokogiri with tool and libraries and all its
> dependencies (including a recent Ruby, libxml2 and libxslt).
> 
> It doesn't install Ruby unless you make it
> 
> guix package -i ruby-nokogiri ruby

It does install Ruby. But it does not propagate it automatically. And
maybe it is up for garbage collection - I am not sure of that right
now.

We have the same issue with R packages, I think.

Another issue: for me the main problem with foreign modules in Guix is
that they are not completely isolated in the profile. No one caught me
out on that yet 

   ~/.guix-profile/lib/ruby/2.2.0/

(in my talk I showed the path). But we symlink against major version
numbers.

So any Ruby interpreter 2.2.x version will share the same gems. It is not
necessarily a problem because the Ruby world is built around this
assumption. But when I look at developer support and reproducibility I
don't like it much. You can have software running with different Ruby
interpreters under the hood - and you won't know it.

I realise this is different from what you are saying Ben, but both
these problems exist in my mind.

I would prefer to isolate the against the full hash in the profile - or at least
Ruby version - that way there can be no mixing. E.g.

  ~/.guix-profile/lib/ruby/pgks1l9cl696j34v9mb35lk8x6lac3b0-ruby-2.2.4/

It does not look as nice in the profile - but who cares.

Pj.

Re: Foreign packages (formerly Re: [PATCH] gnu: Add ruby-nokogiri)

[Ricardo Wurmus]

Pjotr Prins <pjotr.public12@thebird.nl> writes:

> Another issue: for me the main problem with foreign modules in Guix is
> that they are not completely isolated in the profile. No one caught me
> out on that yet 
>
>    ~/.guix-profile/lib/ruby/2.2.0/
>
> (in my talk I showed the path). But we symlink against major version
> numbers.
>
> So any Ruby interpreter 2.2.x version will share the same gems. It is not
> necessarily a problem because the Ruby world is built around this
> assumption. But when I look at developer support and reproducibility I
> don't like it much. You can have software running with different Ruby
> interpreters under the hood - and you won't know it.

Do they really reference different variants of Ruby in the background?
The ruby-build-system adds the “ruby” package only to the build inputs.
I don’t have different “ruby-*” packages in my store right now (after
“guix gc”), so I cannot verify this myself.  I would be interested to
know if references to the “ruby” package are actually retained when
building, say, “ruby-nokogiri”.

> I realise this is different from what you are saying Ben, but both
> these problems exist in my mind.
>
> I would prefer to isolate the against the full hash in the profile - or at least
> Ruby version - that way there can be no mixing. E.g.
>
>   ~/.guix-profile/lib/ruby/pgks1l9cl696j34v9mb35lk8x6lac3b0-ruby-2.2.4/
>
> It does not look as nice in the profile - but who cares.

I haven’t thought enough about this to have an opinion about this.
Generally, though, I dislike to use environment variables to point the
interpreter to a prefix where all libraries can be found.  I think that
it’s up to the libraries themselves to find their dependencies (like
it’s done with RUNPATH).

Can we avoid all of this by replacing “require "library"” in Ruby source
files with “require "/gnu/store/.../path/to/library"”?  Would we still
need to have a single in-profile prefix for Ruby libs?

~~ Ricardo

Re: Foreign packages (formerly Re: [PATCH] gnu: Add ruby-nokogiri)

[Ben Woodcroft]

On 21/02/16 21:16, Ricardo Wurmus wrote:
> Pjotr Prins <pjotr.public12@thebird.nl> writes:
>
>> Another issue: for me the main problem with foreign modules in Guix is
>> that they are not completely isolated in the profile. No one caught me
>> out on that yet
>>
>>     ~/.guix-profile/lib/ruby/2.2.0/
>>
>> (in my talk I showed the path). But we symlink against major version
>> numbers.
>>
>> So any Ruby interpreter 2.2.x version will share the same gems. It is not
>> necessarily a problem because the Ruby world is built around this
>> assumption. But when I look at developer support and reproducibility I
>> don't like it much. You can have software running with different Ruby
>> interpreters under the hood - and you won't know it.
> Do they really reference different variants of Ruby in the background?
> The ruby-build-system adds the “ruby” package only to the build inputs.
> I don’t have different “ruby-*” packages in my store right now (after
> “guix gc”), so I cannot verify this myself.  I would be interested to
> know if references to the “ruby” package are actually retained when
> building, say, “ruby-nokogiri”.

And either way, I'd argue that just being present in the store doesn't 
qualify as "installed" for its usual definition.

>> I realise this is different from what you are saying Ben, but both
>> these problems exist in my mind.
>>
>> I would prefer to isolate the against the full hash in the profile - or at least
>> Ruby version - that way there can be no mixing. E.g.
>>
>>    ~/.guix-profile/lib/ruby/pgks1l9cl696j34v9mb35lk8x6lac3b0-ruby-2.2.4/
>>
>> It does not look as nice in the profile - but who cares.

I'm not sure I follow Pjotr. There is only one GEM_PATH environment 
variable used by all rubies, no?

> I haven’t thought enough about this to have an opinion about this.
> Generally, though, I dislike to use environment variables to point the
> interpreter to a prefix where all libraries can be found.  I think that
> it’s up to the libraries themselves to find their dependencies (like
> it’s done with RUNPATH).

Indeed, please do make progress in the parallel thread about Python...

Maybe Ruby is easier though. Simply replace the foo in require 'foo' 
with the first input where 
/gnu/store/.../lib/ruby/gems/2.2.0/gems/.../lib/foo.rb is a file, I 
think. Presumably the reality will be more complex, but does this sound 
bad Pjotr?

Thanks,
ben

>
> Can we avoid all of this by replacing “require "library"” in Ruby source
> files with “require "/gnu/store/.../path/to/library"”?  Would we still
> need to have a single in-profile prefix for Ruby libs?
>
> ~~ Ricardo

Re: Foreign packages (formerly Re: [PATCH] gnu: Add ruby-nokogiri)

[Pjotr Prins]

On Sun, Feb 21, 2016 at 12:16:58PM +0100, Ricardo Wurmus wrote:
> > So any Ruby interpreter 2.2.x version will share the same gems. It is not
> > necessarily a problem because the Ruby world is built around this
> > assumption. But when I look at developer support and reproducibility I
> > don't like it much. You can have software running with different Ruby
> > interpreters under the hood - and you won't know it.
> 
> Do they really reference different variants of Ruby in the background?

Potentially: when you have a gem already installed and you update Ruby
to a newer version, you can run the old gem against the newer Ruby
provided it looks in the profile path for modules.

> The ruby-build-system adds the “ruby” package only to the build inputs.
> I don’t have different “ruby-*” packages in my store right now (after
> “guix gc”), so I cannot verify this myself.  I would be interested to
> know if references to the “ruby” package are actually retained when
> building, say, “ruby-nokogiri”.

Ruby is an input in the build system, right? After that...

> I haven’t thought enough about this to have an opinion about this.
> Generally, though, I dislike to use environment variables to point the
> interpreter to a prefix where all libraries can be found.  I think that
> it’s up to the libraries themselves to find their dependencies (like
> it’s done with RUNPATH).

You are right. The same holds for Guile, python, perl, R and Ruby at
this point. 

I think it would be OK to have a profile for every Ruby with libs. The
main problem here is that ruby interpreters can get mixed in with
existing (installed) modules.

I think Dave has pointed this out in the past. 

Pj.

Re: Foreign packages (formerly Re: [PATCH] gnu: Add ruby-nokogiri)

[Pjotr Prins]

On Sun, Feb 21, 2016 at 09:50:59PM +1000, Ben Woodcroft wrote:
> Maybe Ruby is easier though. Simply replace the foo in require 'foo'
> with the first input where
> /gnu/store/.../lib/ruby/gems/2.2.0/gems/.../lib/foo.rb is a file, I
> think. Presumably the reality will be more complex, but does this
> sound bad Pjotr?

I can't oversee what it means to Guix. Traversing all the files would
take some logic...

But I don't think it is necessary. If the gem load path is correct
there should be no real issues.

Main problem is the shared profile between rubies and gems. See my
previous E-mail in this thread.

Pj.

Foreign packages and reproducibility (formerly Re: [PATCH] gnu: Add ruby-nokogiri)

[Pjotr Prins]

I am thinking about this again. It is relevant for all interpreters.

The real pain is that between Ruby interpreters the profile can be shared which
contains a path to the modules, e.g. say ruby-nokogiri. Currently you can run

  guix package -i ruby ruby-nokogiri
  ruby -e "require-2.2.1 'nokogiri'"

next update ruby to a new (minor) version 

  guix package -i ruby-2.2.2

and run again

  ruby -e "require 'nokogiri'"

will work.

which is wrong because nokogiri was built against the first ruby interpreter.
It is a problem from the view point of reproducible software execution.

To prevent this we should *not* allow modules (i.e. nokogiri) in the
profile.

But now a user doing a simple

  ruby -e "require 'nokogiri'"

will fail, because the interpreter does not know where nokogiri
is. This is wrong because users should be able to access installed
gems (that is the point of having gems!).

One suggestion is wrappers and runpaths. I.e. wrap the invocation of
'ruby' with a script that will pass in the path to all installed gems.

Q1: is there a limitation to the length of such a path (generally a
    bash variable)?

Q2: would the wrapper be updated every time a new gem gets installed?
    How can we be sure it is deterministic? Installation order may matter.

Q3: can we query guix on-the-fly for paths to all leaf nodes that
    depend on a ruby? That way a wrapper could be installed
    once. Would that be too slow? How can we be sure it is
    deterministic?

I do not know how nix solves it exactly, but I do know they use a
wrapper script.

Finally, I do think that most Ruby gems should propagate ruby
itself. Only rarely are gems executables. Mostly they are libraries,
which implies you need to run ruby to run them. Right? It does not
make sense to install a Ruby library without the ruby interpreter.

Pj.

Re: Foreign packages and reproducibility (formerly Re: [PATCH] gnu: Add ruby-nokogiri)

[Pjotr Prins]

Another option would be have a symlinked profile for every ruby interpreter with
the full hash

/var/guix/profile/per-ruby/ziy7a6zib846426kprc7fgimggh8bz97-ruby-2.1.3/

which contains symlinks to the libraries/gems installed against that ruby.

NOW we only need to tell the interpreter where to find the libraries using this
path. This could actually be patched in trivially at installation time.

(I think this is rather exciting, though again I have the feeling
someone has thought of this before).

Pj.


On Sun, Feb 21, 2016 at 06:22:31PM +0100, Pjotr Prins wrote:
> I am thinking about this again. It is relevant for all interpreters.
> 
> The real pain is that between Ruby interpreters the profile can be shared which
> contains a path to the modules, e.g. say ruby-nokogiri. Currently you can run
> 
>   guix package -i ruby ruby-nokogiri
>   ruby -e "require-2.2.1 'nokogiri'"
> 
> next update ruby to a new (minor) version 
> 
>   guix package -i ruby-2.2.2
> 
> and run again
> 
>   ruby -e "require 'nokogiri'"
> 
> will work.
> 
> which is wrong because nokogiri was built against the first ruby interpreter.
> It is a problem from the view point of reproducible software execution.
> 
> To prevent this we should *not* allow modules (i.e. nokogiri) in the
> profile.
> 
> But now a user doing a simple
> 
>   ruby -e "require 'nokogiri'"
> 
> will fail, because the interpreter does not know where nokogiri
> is. This is wrong because users should be able to access installed
> gems (that is the point of having gems!).
> 
> One suggestion is wrappers and runpaths. I.e. wrap the invocation of
> 'ruby' with a script that will pass in the path to all installed gems.
> 
> Q1: is there a limitation to the length of such a path (generally a
>     bash variable)?
> 
> Q2: would the wrapper be updated every time a new gem gets installed?
>     How can we be sure it is deterministic? Installation order may matter.
> 
> Q3: can we query guix on-the-fly for paths to all leaf nodes that
>     depend on a ruby? That way a wrapper could be installed
>     once. Would that be too slow? How can we be sure it is
>     deterministic?
> 
> I do not know how nix solves it exactly, but I do know they use a
> wrapper script.
> 
> Finally, I do think that most Ruby gems should propagate ruby
> itself. Only rarely are gems executables. Mostly they are libraries,
> which implies you need to run ruby to run them. Right? It does not
> make sense to install a Ruby library without the ruby interpreter.
> 
> Pj.
> 

-- 

Re: Foreign packages and reproducibility (formerly Re: [PATCH] gnu: Add ruby-nokogiri)

[Pjotr Prins]

On Sun, Feb 21, 2016 at 06:31:27PM +0100, Pjotr Prins wrote:
> Another option would be have a symlinked profile for every ruby interpreter with
> the full hash
> 
> /var/guix/profile/per-ruby/ziy7a6zib846426kprc7fgimggh8bz97-ruby-2.1.3/
> 
> which contains symlinks to the libraries/gems installed against that ruby.

Hmmm. One problem is that this will be a global profile. We should do

  /var/guix/profile/per-user/per-ruby/ziy7a6zib846426kprc7fgimggh8bz97-ruby-2.1.3/

instead. Now the Ruby interpreter has to be aware about users too :/

Pj.

Re: Foreign packages and reproducibility (formerly Re: [PATCH] gnu: Add ruby-nokogiri)

[Pjotr Prins]

OK, back on the topic of conflicts between interpreters and modules:

I realise the current system is fine as is!

Sometimes it is good to go for a cycle run in the cold ;)

GNU Guix does the right thing. It builds isolated packages, including
interpreters and modules. Next it puts them in a profile.

Now the key insight is that it is *not* Guix' responsibility when
people mess up their profiles. It is easily possible, even with emacs,
to mess up the profile.

Also, Ruby allows for minor version mixing of modules. It is a Ruby
problem. They should fix that.

We could have the interpreter test modules for being true dependencies
and we could test the interpreter for belonging with a module. But the
thing is, it is not necessary.

Users can easily use different profiles for different Rubies. They
just need to be aware. I can create a profile named
~/.guix-pgks1l9cl696j34v9mb35lk8x6lac3b0-ruby-2.2.4 if I want to. It
is good enough. That profile will contain the modules I install with
that Ruby.

So, after much thought, my final suggestion is to leave it like it is
:). The symlink in the profile is cool. We don't need a runpath or
wrapper script.

Pj.

Re: Foreign packages and reproducibility (formerly Re: [PATCH] gnu: Add ruby-nokogiri)

[Ricardo Wurmus]

Pjotr Prins <pjotr.public12@thebird.nl> writes:

> OK, back on the topic of conflicts between interpreters and modules:
>
> I realise the current system is fine as is!
>
> Sometimes it is good to go for a cycle run in the cold ;)
>
> GNU Guix does the right thing. It builds isolated packages, including
> interpreters and modules. Next it puts them in a profile.
>
> Now the key insight is that it is *not* Guix' responsibility when
> people mess up their profiles. It is easily possible, even with emacs,
> to mess up the profile.
>
> Also, Ruby allows for minor version mixing of modules. It is a Ruby
> problem. They should fix that.
>
> We could have the interpreter test modules for being true dependencies
> and we could test the interpreter for belonging with a module. But the
> thing is, it is not necessary.
>
> Users can easily use different profiles for different Rubies. They
> just need to be aware. I can create a profile named
> ~/.guix-pgks1l9cl696j34v9mb35lk8x6lac3b0-ruby-2.2.4 if I want to. It
> is good enough. That profile will contain the modules I install with
> that Ruby.

That’s not a bad idea!  But it may cause problems in this situation:

    * install that ruby
    * install ruby packages built with that ruby
    * wait until the ruby recipe is updated (or changes hashes because
      of a change to the inputs)
    * install another ruby package (built with the new ruby variant)

There is no way to fix the ruby variant when installing new ruby
packages.  You’d have to freeze the whole closure of the ruby package
expression to keep it stable.  When we commit to master we don’t
separate updates affecting ruby from those that simply add new packages
for ruby gems.

As a result you wouldn’t be able to add new gems to your profile without
mixing Ruby versions.

I guess we might eventually use the dependency rewriting framework to
replace any “ruby” in the dependency graph with a particular store item
(e.g. “pgks1l9cl696j34v9mb35lk8x6lac3b0-ruby-2.2.4”), so that you can
even install new ruby gem packages with the very same version of ruby
that you already have in your profile.  (Currently we can only override
explicit inputs, not inputs that are added implicitly by the build
system.)

For convenience we might also want to have per-profile settings that
allow us to specify default overrides, so that no matter what ruby gem
you add to the profile it will always use a fixed variant of Ruby to
build it.

The same applies to R.  It would be good if we could ensure that all
additional packages in a profile reference the same interpreter version,
because at runtime we only have one interpreter available.

~~ Ricardo

Re: Foreign packages and reproducibility (formerly Re: [PATCH] gnu: Add ruby-nokogiri)

[Pjotr Prins]

On Wed, Mar 02, 2016 at 11:33:26AM +0100, Ricardo Wurmus wrote:
> For convenience we might also want to have per-profile settings that
> allow us to specify default overrides, so that no matter what ruby gem
> you add to the profile it will always use a fixed variant of Ruby to
> build it.
> 
> The same applies to R.  It would be good if we could ensure that all
> additional packages in a profile reference the same interpreter version,
> because at runtime we only have one interpreter available.

Sounds good to me.

Pj.
--