Giving up on RubyGems

Giving up on RubyGems

[David Thompson]

Hello Guix hackers,

As some of you know, I've been working on Ruby support for Guix for
about a year now.  In that time, I helped write and rewrite a Ruby gem
build system, wrote an importer for <https://rubygems.org>, and packaged
many Ruby gems.

At various points, I've had my doubts about the gem archives hosted on
the RubyGems website: Are they source code?  Are they binaries?  After a
good deal of debate, we came to the conclusion that they are source
code.  This seems to be the case when you inspect any given gem.  The
Ruby source code is there, and so is the C source code needed for native
extensions when there is a native extension to be built.  Furthermore,
the RubyGems website says that, among other things, gems should contain
"code (including tests and supporting utilities)." [0]

However, it has become clear that the RubyGems maintainers do not
actually feel this way.  From their perspective, gems are binaries, not
source code.  I discovered this once I noticed that several popular Ruby
gems such as Arel do not, and refuse to [1], ship the test suite in
their releases.  This is because they view gems as binaries that need to
be as slim as possible, containing only necessary runtime files, to cut
down on bandwidth usage and storage space.  Unfortunately, they have no
notion of a source package that corresponds to a given binary.

In practice, I've found that all the gems I've packaged come with source
code and no binaries, they might just be missing the test suite.  So, I
asked the RubyGems maintainers to consider the use-cases for including
test suites, which spawned a large thread on their GitHub page
yesterday. [2]  The end result is this depressing quote:

    Yes, gems are effectively binary packages delivered to
    end-users. Some gems contain ruby source code, some contain
    pre-compiled binaries, some contain both. The internals of a
    particular gem aren't relevant from the perspective of RubyGems
    itself.
    
    As has been pointed out here, RubyGems does not provide packages
    containing gem source code. To be honest, RubyGems as a system does
    not care about gem source code—it accepts .gem files from gem
    authors, and distributes those files on request. Any gem author who
    wishes to provide a link to the source code used to produce a gem is
    welcome to use the gemspec metadata fields to do so.

I've grown very tired of trying to convince people that independent user
verification of binary releases is an important thing to prioritize, but
they think that users do not want the source code.  I've tried to make
my arguments as clear as I could, yet they've been misunderstood by some
and rejected entirely by others, and now it is time to give up.  I don't
know what the best way forward for Ruby support in Guix is.  Things like
the RubyGems importer seem useless now.  Just downloading release
tarballs from GitHub doesn't work without major hacks because almost
every gem (thanks to a terrible script in Bundler that generates
boilerplate for new gems) relies on running 'git ls-files', which of
course requires a Git commit database, in order to build at all.  This
won't do because the '.git' directory is non-deterministic when running
'git clone', as many of us know.  The entire stack, from the build
system to the package management system are broken and are effectively
beyond repair because no one else believes that there are problems.
This effort has drained too much of my enthusiasm, and now I need a
break.

Sorry if this comes across too ranty and complainy, I suppose it is
both.  I hope your hacking is happier than mine.

-- 
David Thompson
GPG Key: 0FF1D807

[0] http://guides.rubygems.org/what-is-a-gem/
[1] https://github.com/rails/arel/issues/384
[2] https://github.com/rubygems/rubygems/issues/1364

Re: Giving up on RubyGems

[Taylan Ulrich Bayırlı/Kammer]

David Thompson <dthompson2@worcester.edu> writes:

> Hello Guix hackers,
>
> As some of you know, I've been working on Ruby support for Guix for
> about a year now.  In that time, I helped write and rewrite a Ruby gem
> build system, wrote an importer for <https://rubygems.org>, and packaged
> many Ruby gems.
>
> At various points, I've had my doubts about the gem archives hosted on
> the RubyGems website: Are they source code?  Are they binaries?  After a
> good deal of debate, we came to the conclusion that they are source
> code.  This seems to be the case when you inspect any given gem.  The
> Ruby source code is there, and so is the C source code needed for native
> extensions when there is a native extension to be built.  Furthermore,
> the RubyGems website says that, among other things, gems should contain
> "code (including tests and supporting utilities)." [0]
>
> However, it has become clear that the RubyGems maintainers do not
> actually feel this way.  From their perspective, gems are binaries, not
> source code.  I discovered this once I noticed that several popular Ruby
> gems such as Arel do not, and refuse to [1], ship the test suite in
> their releases.  This is because they view gems as binaries that need to
> be as slim as possible, containing only necessary runtime files, to cut
> down on bandwidth usage and storage space.  Unfortunately, they have no
> notion of a source package that corresponds to a given binary.
>
> In practice, I've found that all the gems I've packaged come with source
> code and no binaries, they might just be missing the test suite.  So, I
> asked the RubyGems maintainers to consider the use-cases for including
> test suites, which spawned a large thread on their GitHub page
> yesterday. [2]  The end result is this depressing quote:
>
>     Yes, gems are effectively binary packages delivered to
>     end-users. Some gems contain ruby source code, some contain
>     pre-compiled binaries, some contain both. The internals of a
>     particular gem aren't relevant from the perspective of RubyGems
>     itself.
>     
>     As has been pointed out here, RubyGems does not provide packages
>     containing gem source code. To be honest, RubyGems as a system does
>     not care about gem source code—it accepts .gem files from gem
>     authors, and distributes those files on request. Any gem author who
>     wishes to provide a link to the source code used to produce a gem is
>     welcome to use the gemspec metadata fields to do so.
>
> I've grown very tired of trying to convince people that independent user
> verification of binary releases is an important thing to prioritize, but
> they think that users do not want the source code.  I've tried to make
> my arguments as clear as I could, yet they've been misunderstood by some
> and rejected entirely by others, and now it is time to give up.  I don't
> know what the best way forward for Ruby support in Guix is.  Things like
> the RubyGems importer seem useless now.  Just downloading release
> tarballs from GitHub doesn't work without major hacks because almost
> every gem (thanks to a terrible script in Bundler that generates
> boilerplate for new gems) relies on running 'git ls-files', which of
> course requires a Git commit database, in order to build at all.  This
> won't do because the '.git' directory is non-deterministic when running
> 'git clone', as many of us know.  The entire stack, from the build
> system to the package management system are broken and are effectively
> beyond repair because no one else believes that there are problems.
> This effort has drained too much of my enthusiasm, and now I need a
> break.
>
> Sorry if this comes across too ranty and complainy, I suppose it is
> both.  I hope your hacking is happier than mine.

Hi David. :-)

No matter the results, thanks a lot for all the work.

After just having had a quarrel on a certain development list that shall
not be named, I fully sympathize with your frustration with people who
continuously misunderstand your ideas and refuse to accept problems you
point out for them.  It can truly drive one mad...

Take a well deserved rest!

Taylan

Re: Giving up on RubyGems

[Ludovic Courtès]

Hi!

I think letting the dust settle down cannot hurt–and you have awesome
projects in the pipeline, like container support!  ;-)

From an outsider’s perspective, it may be that the outcome is that we’ll
have to act on a case-by-case basis: using gems when they actually
contain source, and starting from Git checkouts elsewhere.

For the latter, it might be that ‘git ls-files’ is not too ubiquitous,
or that we can find workarounds.  It certainly means more manual work.

Anyway, thanks for all the work and for the clear summary.  We’ll see
how things evolve and what other Rubyists would like to do.

Ludo’.

Re: Giving up on RubyGems

[John Darrington]

[-- Attachment #1: Type: text/plain, Size: 4792 bytes --]

You got a lot further than I did.

I gave up when ruby refused to run if $HOME did not match the entry in /etc/passwd

The Ruby maintainers insisted that my environment was "broken" if HOME was ever
set to anything else.

:(

On Tue, Oct 20, 2015 at 08:51:22AM -0400, David Thompson wrote:
     Hello Guix hackers,
     
     As some of you know, I've been working on Ruby support for Guix for
     about a year now.  In that time, I helped write and rewrite a Ruby gem
     build system, wrote an importer for <https://rubygems.org>, and packaged
     many Ruby gems.
     
     At various points, I've had my doubts about the gem archives hosted on
     the RubyGems website: Are they source code?  Are they binaries?  After a
     good deal of debate, we came to the conclusion that they are source
     code.  This seems to be the case when you inspect any given gem.  The
     Ruby source code is there, and so is the C source code needed for native
     extensions when there is a native extension to be built.  Furthermore,
     the RubyGems website says that, among other things, gems should contain
     "code (including tests and supporting utilities)." [0]
     
     However, it has become clear that the RubyGems maintainers do not
     actually feel this way.  From their perspective, gems are binaries, not
     source code.  I discovered this once I noticed that several popular Ruby
     gems such as Arel do not, and refuse to [1], ship the test suite in
     their releases.  This is because they view gems as binaries that need to
     be as slim as possible, containing only necessary runtime files, to cut
     down on bandwidth usage and storage space.  Unfortunately, they have no
     notion of a source package that corresponds to a given binary.
     
     In practice, I've found that all the gems I've packaged come with source
     code and no binaries, they might just be missing the test suite.  So, I
     asked the RubyGems maintainers to consider the use-cases for including
     test suites, which spawned a large thread on their GitHub page
     yesterday. [2]  The end result is this depressing quote:
     
         Yes, gems are effectively binary packages delivered to
         end-users. Some gems contain ruby source code, some contain
         pre-compiled binaries, some contain both. The internals of a
         particular gem aren't relevant from the perspective of RubyGems
         itself.
         
         As has been pointed out here, RubyGems does not provide packages
         containing gem source code. To be honest, RubyGems as a system does
         not care about gem source code???it accepts .gem files from gem
         authors, and distributes those files on request. Any gem author who
         wishes to provide a link to the source code used to produce a gem is
         welcome to use the gemspec metadata fields to do so.
     
     I've grown very tired of trying to convince people that independent user
     verification of binary releases is an important thing to prioritize, but
     they think that users do not want the source code.  I've tried to make
     my arguments as clear as I could, yet they've been misunderstood by some
     and rejected entirely by others, and now it is time to give up.  I don't
     know what the best way forward for Ruby support in Guix is.  Things like
     the RubyGems importer seem useless now.  Just downloading release
     tarballs from GitHub doesn't work without major hacks because almost
     every gem (thanks to a terrible script in Bundler that generates
     boilerplate for new gems) relies on running 'git ls-files', which of
     course requires a Git commit database, in order to build at all.  This
     won't do because the '.git' directory is non-deterministic when running
     'git clone', as many of us know.  The entire stack, from the build
     system to the package management system are broken and are effectively
     beyond repair because no one else believes that there are problems.
     This effort has drained too much of my enthusiasm, and now I need a
     break.
     
     Sorry if this comes across too ranty and complainy, I suppose it is
     both.  I hope your hacking is happier than mine.
     
     -- 
     David Thompson
     GPG Key: 0FF1D807
     
     [0] http://guides.rubygems.org/what-is-a-gem/
     [1] https://github.com/rails/arel/issues/384
     [2] https://github.com/rubygems/rubygems/issues/1364
     
     

-- 
Avoid eavesdropping.  Send strong encryted email.
PGP Public key ID: 1024D/2DE827B3 
fingerprint = 8797 A26D 0854 2EAB 0285  A290 8A67 719C 2DE8 27B3
See http://sks-keyservers.net or any PGP keyserver for public key.


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Giving up on RubyGems

[Leo Famulari]

On Tue, Oct 20, 2015 at 08:51:22AM -0400, David Thompson wrote:
> I've grown very tired of trying to convince people that independent user
> verification of binary releases is an important thing to prioritize, but
> they think that users do not want the source code.  I've tried to make
> my arguments as clear as I could, yet they've been misunderstood by some
> and rejected entirely by others, and now it is time to give up.

As someone with zero Ruby experience, I find this very confusing. Are
you saying that they don't think it's necessary to be able to build the
Gem "binaries" independently? If that were true, how can RubyGems be a
trusted intermediary between software authors and users? How can users
verify that the RubyGems people are not distributing malicious
"binaries". Am I totally misunderstanding the role of RubyGems in the
Ruby world?

> This effort has drained too much of my enthusiasm, and now I need a
> break.

:( We all have experiences like this once in a while. Everyone
appreciates the work you do on Guix. Hopefully you can come back with
some fresh energy when you are ready!

Re: Giving up on RubyGems

[Pjotr Prins]

Hi Dave,

You know I am not so much interested in fixing upstream concepts which
appear to be mixed up (indeed). I think you are heroic for trying to
discuss this with the Rubygem system authors. Kudos for trying.

Still: GNU Guix Rubygem support is a major achievement.

Fact is that:

1. We have successfully packaged rubygems for guix in a simple and
   elegant way. I use them daily.

2. Enough tests are in there to make sure things work - in fact I was
   mighty surprised that Nokogiri works on ARM+MIPS, despite the fact
   that we haven't gotten the Nokogiri test system to work - and that
   it works was tested by virtue of my bio-blastxmlparser gem which
   *has* working tests.

3. Rubygems does not dictate how people package their gems. In fact, 
   if need be, we can also repackage into rubygems and deploy those.
   We may even influence the upstream authors. 

I agree the Rubygem situation is not ideal and the maybe Rubygems
people are misguided in their architectural choices. But I think they
will probably converge towards our ideas in time. When a choice, such
as no tests, or using git fuzzily, starts to bite, they will want to
revert on that. I also love your argumentation that they should
provide the source code next to the 'binary' distribution. Totally
valid and in line with most licenses in use, including the Ruby
license. Maybe the FSF should threaten a case.

With Linux distributions and languages you can just see people
reinventing the wheel (npm, pip, docker, firefox anyone?). It is
amazing how much energy goes into all the combined packaging systems
when they could simply be using Guix ;)

Still, it is a form of evolution. Each his own (imperfect) niche.

For us, I think we have to be purely pragmatic. I love the current
Ruby build system in Guix and will use it for those gems that allow
inclusion. It has to be source (anyway) to build against the Guix
ruby-build-system.

And when it fails we find some other way. It is simply software, i.e.,
we can always fix it.

So, yes, I think it is wise to give up on the rubygem authors for now.

But please don't stop loving - the people who use - Ruby :)

Pj.



On Tue, Oct 20, 2015 at 08:51:22AM -0400, David Thompson wrote:
> Hello Guix hackers,
> 
> As some of you know, I've been working on Ruby support for Guix for
> about a year now.  In that time, I helped write and rewrite a Ruby gem
> build system, wrote an importer for <https://rubygems.org>, and packaged
> many Ruby gems.
> 
> At various points, I've had my doubts about the gem archives hosted on
> the RubyGems website: Are they source code?  Are they binaries?  After a
> good deal of debate, we came to the conclusion that they are source
> code.  This seems to be the case when you inspect any given gem.  The
> Ruby source code is there, and so is the C source code needed for native
> extensions when there is a native extension to be built.  Furthermore,
> the RubyGems website says that, among other things, gems should contain
> "code (including tests and supporting utilities)." [0]
> 
> However, it has become clear that the RubyGems maintainers do not
> actually feel this way.  From their perspective, gems are binaries, not
> source code.  I discovered this once I noticed that several popular Ruby
> gems such as Arel do not, and refuse to [1], ship the test suite in
> their releases.  This is because they view gems as binaries that need to
> be as slim as possible, containing only necessary runtime files, to cut
> down on bandwidth usage and storage space.  Unfortunately, they have no
> notion of a source package that corresponds to a given binary.
> 
> In practice, I've found that all the gems I've packaged come with source
> code and no binaries, they might just be missing the test suite.  So, I
> asked the RubyGems maintainers to consider the use-cases for including
> test suites, which spawned a large thread on their GitHub page
> yesterday. [2]  The end result is this depressing quote:
> 
>     Yes, gems are effectively binary packages delivered to
>     end-users. Some gems contain ruby source code, some contain
>     pre-compiled binaries, some contain both. The internals of a
>     particular gem aren't relevant from the perspective of RubyGems
>     itself.
>     
>     As has been pointed out here, RubyGems does not provide packages
>     containing gem source code. To be honest, RubyGems as a system does
>     not care about gem source code—it accepts .gem files from gem
>     authors, and distributes those files on request. Any gem author who
>     wishes to provide a link to the source code used to produce a gem is
>     welcome to use the gemspec metadata fields to do so.
> 
> I've grown very tired of trying to convince people that independent user
> verification of binary releases is an important thing to prioritize, but
> they think that users do not want the source code.  I've tried to make
> my arguments as clear as I could, yet they've been misunderstood by some
> and rejected entirely by others, and now it is time to give up.  I don't
> know what the best way forward for Ruby support in Guix is.  Things like
> the RubyGems importer seem useless now.  Just downloading release
> tarballs from GitHub doesn't work without major hacks because almost
> every gem (thanks to a terrible script in Bundler that generates
> boilerplate for new gems) relies on running 'git ls-files', which of
> course requires a Git commit database, in order to build at all.  This
> won't do because the '.git' directory is non-deterministic when running
> 'git clone', as many of us know.  The entire stack, from the build
> system to the package management system are broken and are effectively
> beyond repair because no one else believes that there are problems.
> This effort has drained too much of my enthusiasm, and now I need a
> break.
> 
> Sorry if this comes across too ranty and complainy, I suppose it is
> both.  I hope your hacking is happier than mine.
> 
> -- 
> David Thompson
> GPG Key: 0FF1D807
> 
> [0] http://guides.rubygems.org/what-is-a-gem/
> [1] https://github.com/rails/arel/issues/384
> [2] https://github.com/rubygems/rubygems/issues/1364
> 
> 

-- 

Re: Giving up on RubyGems

[Thompson, David]

Hi Pjotr,

This cheered me up!

On Wed, Oct 21, 2015 at 5:46 AM, Pjotr Prins <pjotr.public12@thebird.nl> wrote:
> Hi Dave,
>
> You know I am not so much interested in fixing upstream concepts which
> appear to be mixed up (indeed). I think you are heroic for trying to
> discuss this with the Rubygem system authors. Kudos for trying.

Thanks!

> Still: GNU Guix Rubygem support is a major achievement.
>
> Fact is that:
>
> 1. We have successfully packaged rubygems for guix in a simple and
>    elegant way. I use them daily.
>
> 2. Enough tests are in there to make sure things work - in fact I was
>    mighty surprised that Nokogiri works on ARM+MIPS, despite the fact
>    that we haven't gotten the Nokogiri test system to work - and that
>    it works was tested by virtue of my bio-blastxmlparser gem which
>    *has* working tests.
>
> 3. Rubygems does not dictate how people package their gems. In fact,
>    if need be, we can also repackage into rubygems and deploy those.
>    We may even influence the upstream authors.

All good points.  You're right.  I shouldn't think that we've
accomplished nothing, because we have done a lot.

> I agree the Rubygem situation is not ideal and the maybe Rubygems
> people are misguided in their architectural choices. But I think they
> will probably converge towards our ideas in time. When a choice, such
> as no tests, or using git fuzzily, starts to bite, they will want to
> revert on that. I also love your argumentation that they should
> provide the source code next to the 'binary' distribution. Totally
> valid and in line with most licenses in use, including the Ruby
> license. Maybe the FSF should threaten a case.

Perhaps I could find a gem that is GPL licensed whose archive doesn't
include the corresponding source code, but it seems like finding a
needle in a haystack.  It probably doesn't exist.

> With Linux distributions and languages you can just see people
> reinventing the wheel (npm, pip, docker, firefox anyone?). It is
> amazing how much energy goes into all the combined packaging systems
> when they could simply be using Guix ;)
>
> Still, it is a form of evolution. Each his own (imperfect) niche.
>
> For us, I think we have to be purely pragmatic. I love the current
> Ruby build system in Guix and will use it for those gems that allow
> inclusion. It has to be source (anyway) to build against the Guix
> ruby-build-system.
>
> And when it fails we find some other way. It is simply software, i.e.,
> we can always fix it.

Yes, agreed.  We will evaluate on a case-by-case basis whether or not
the gem archive on rubygems.org meets our standards.  I will write a
patch sometime to add the 'gitify' phase back to the Ruby build system
as an optional phase for cases where we need to build from a tarball
or git checkout.

> So, yes, I think it is wise to give up on the rubygem authors for now.
>
> But please don't stop loving - the people who use - Ruby :)

I won't.  Thanks for cheering me up, Pjotr. :)

- Dave