[GSoC] Integrating npm packages into the Guix ecosystem

[GSoC] Integrating npm packages into the Guix ecosystem

[Jelle Licht]

[-- Attachment #1.1: Type: text/plain, Size: 352 bytes --]

Hello Guix,

Seeing as this time next year I'll be finishing up my studies, this was a
now-or-never moment for me. I would love to spend this summer hacking on
both guix, and reading up on npm.


Attached you will find the .org and corresponding pdf export of my draft
proposal for GSoC 2016!

If anyone has some feedback, I'd love to hear it.

~Jelle

[-- Attachment #1.2: Type: text/html, Size: 487 bytes --]

[-- Attachment #2: proposal_jlicht.org --]
[-- Type: application/octet-stream, Size: 4872 bytes --]

#+TITLE: Integrating npm into the Guix ecosystem
#+AUTHOR: Jelle Licht,  jlicht@fsfe.org
#+DATE: 21-03-2016
#+OPTIONS: toc:nil

* Overview

This project will allow Guix hackers to more easily package software that is
distributed through the Node Package Manager (npm), as well as allowing Node
developers on Guix to make use of the reproducible builds guarantee of Guix.

After completing this project, it should be possible to easily make use of the
less-problematic packages in the npm registry on the Guix Software Distribution.

* Project structure

Depending on findings in the early stages of the project, I foresee the
following distinct parts:

1. Extend Guix so it can 'simulate' the dependency graph generation of both the
   old and new npm [fn:6].
2. Extend guix with an algorithm that matches npm's package.json flexible
   version specification to a specific version.
3. Add a ~guix import~ backend for the npm registry
4. Package npm modules in guix
5. Interface

Right now, dependency resolution in npm is as stateful as can be, with even the
installation order [fn:6] making a difference for where npm expects to find a
certain dependency. As two different dependency resolution mechanism are in use,
of which especially the newer one is problematic because of its habit of
propagating dependencies upwards in the folder structure, both have to be
supported by a guix module.

npm uses SemanticVersioning range patterns [fn:7] to declare dependencies
between packages. A problem with this approach is that the same package
declaration can lead to an entirely different dependency graph, which defeats
the purpose of having a system with a focus on reproducible builds. If npm
packages are to be used, these version numbers need to be locked down to a
specific version, corresponding to the version that npm would install if left to
its devices. A potential problem is that the entire dependency graph has to be
known ahead of time in order to pinpoint a 'correct' version of the dependency.
A consistent ordering for 'installing' dependencies also has to be decided upon.

After the dependency resolution has been worked out, it should be possible to
create a guix import backend to leverage the code that has been produced up till
now to allow the packaging of npm modules. The last part of the project
essentially serves as a starting point for packaging up useful npm packages.

A stretch goal for the summer would be to create a guix build system for a
subset of npm packages, such as gulp.

* Planning

As I am currently a novice with regards to the internals of guix and the guile
programming language, up to the start of the actual project I will mostly be
reading up and hacking on guix. As such, the planning becomes:

April 22 -  May 22:
  - Getting to know Guix(SD)
  - Package programs using the guix import module
  - An informal specification of the npm dependency resolution mechanism
  - Getting the know the guix community and what everyone is working on
  - Getting familiar with the contributing work flows
  
May 23 - Jun 5:
  - Formal specification to allow guix to simulate npm dependency resolution (1)

Jun 6 - Jun 19:
  - npm Version pinning should be working (2)
  - start working on the guix import backend

Jun 20 - Jul 3: 
  - Guix import backend should be finished by now (3)
  - Start testing npm packages 

Jul 4 - Jul 10:
  - Holidays! 

Jul 11 - Aug 7:
  - Solve any problems and corner cases with building and installing npm
    packages (4)

Aug 8 - Aug 23:
  - If life goes a planned, Get all contributions ready to be merged back in the
    main Guix codebase.


* About me

My name is Jelle Licht, and I am currently studying Data Science at the
University of Technology Delft. I finished my BSc in Computer Science in 2015,
at the University of Technology Delft as well. Somewhere in the second semester
of my studies, my roommate was using Vi in front of me, and since then I've been
falling into the rabbit hole that is GNU/Linux, free software and the 'open
source' community. Somewhere along the line I became a huge fan of Clojure, and
for a short while was a member of the very much unofficial Lisp Community Delft.

I have been a small-time contributor to some free software projects on github,
as well as having a day job to make ends meet using mostly Node. My online
handles include 'wordempire', as I am quite fond of reading, and 'jlicht'. For
the past month, I have been lurking on-and-off again in the #guix irc channel,
as well as reading up on some of the motivations behind reproducible research
and reproducible builds.

Besides one week for visiting family and general holidays, this project would be
my full time focus during the summer.


* Footnotes

[fn:7] http://developer.telerik.com/featured/mystical-magical-semver-ranges-used-npm-bower/

[fn:6] https://docs.npmjs.com/how-npm-works/npm3

[-- Attachment #3: proposal_jlicht.pdf --]
[-- Type: application/pdf, Size: 31490 bytes --]

Re: [GSoC] Integrating npm packages into the Guix ecosystem

[Christopher Allan Webber]

Jelle Licht writes:

> Hello Guix,
>
> Seeing as this time next year I'll be finishing up my studies, this was a
> now-or-never moment for me. I would love to spend this summer hacking on
> both guix, and reading up on npm.
>
>
> Attached you will find the .org and corresponding pdf export of my draft
> proposal for GSoC 2016!
>
> If anyone has some feedback, I'd love to hear it.
>
> ~Jelle

This sounds great!  I appreciate the detail and the exploration of the
problem in the document... and the use of org-mode ;)

You might want to look at
  http://sandervanderburg.blogspot.com/2014/10/deploying-npm-packages-with-nix-package.html
and
  http://sandervanderburg.blogspot.com/2016/02/managing-npm-flat-module-installations.html
also if you haven't already, but this looks like a quite solid proposal
to me.  If it succeeds, it could have a large amount of impact.

 - Chris

Re: [GSoC] Integrating npm packages into the Guix ecosystem

[Ludovic Courtès]

Hello!

Christopher Allan Webber <cwebber@dustycloud.org> skribis:

> This sounds great!  I appreciate the detail and the exploration of the
> problem in the document... and the use of org-mode ;)
>
> You might want to look at
>   http://sandervanderburg.blogspot.com/2014/10/deploying-npm-packages-with-nix-package.html
> and
>   http://sandervanderburg.blogspot.com/2016/02/managing-npm-flat-module-installations.html
> also if you haven't already, but this looks like a quite solid proposal
> to me.  If it succeeds, it could have a large amount of impact.

That was also my impression, but you know better than me.  :-)

I assume you would mentor it, Christopher, possibly with David?

Thanks!

Ludo’.

Re: [GSoC] Integrating npm packages into the Guix ecosystem

[Christopher Allan Webber]

Ludovic Courtès writes:

> Hello!
>
> Christopher Allan Webber <cwebber@dustycloud.org> skribis:
>
>> This sounds great!  I appreciate the detail and the exploration of the
>> problem in the document... and the use of org-mode ;)
>>
>> You might want to look at
>>   http://sandervanderburg.blogspot.com/2014/10/deploying-npm-packages-with-nix-package.html
>> and
>>   http://sandervanderburg.blogspot.com/2016/02/managing-npm-flat-module-installations.html
>> also if you haven't already, but this looks like a quite solid proposal
>> to me.  If it succeeds, it could have a large amount of impact.
>
> That was also my impression, but you know better than me.  :-)
>
> I assume you would mentor it, Christopher, possibly with David?
>
> Thanks!
>
> Ludo’.

Hm, I could try... I don't *know* enough about npm to be confident I'd
be the best mentor.  But I do know enough about Guix probably.

I'd prefer to co-mentor it with Dave if that were an option, and Dave
would like to not handle it all on his own.

Re: [GSoC] Integrating npm packages into the Guix ecosystem

[Thompson, David]

On Wed, Mar 23, 2016 at 11:56 AM, Christopher Allan Webber
<cwebber@dustycloud.org> wrote:

> I'd prefer to co-mentor it with Dave if that were an option, and Dave
> would like to not handle it all on his own.

Co-mentoring sounds good to me!

- Dave

Re: [GSoC] Integrating npm packages into the Guix ecosystem

[Ludovic Courtès]

"Thompson, David" <dthompson2@worcester.edu> skribis:

> On Wed, Mar 23, 2016 at 11:56 AM, Christopher Allan Webber
> <cwebber@dustycloud.org> wrote:
>
>> I'd prefer to co-mentor it with Dave if that were an option, and Dave
>> would like to not handle it all on his own.
>
> Co-mentoring sounds good to me!

Cool, thanks to both of you!  :-)

Ludo’.

Re: [GSoC] Integrating npm packages into the Guix ecosystem

[David Thompson]

Jelle Licht <jlicht@fsfe.org> writes:

> Hello Guix,
>
> Seeing as this time next year I'll be finishing up my studies, this was a
> now-or-never moment for me. I would love to spend this summer hacking on
> both guix, and reading up on npm.

I think this project is a great idea!  Feedback inline below.

> #+AUTHOR: Jelle Licht,  jlicht@fsfe.org
> #+DATE: 21-03-2016
> #+OPTIONS: toc:nil
> 
> * Overview
> 
> This project will allow Guix hackers to more easily package software that is
> distributed through the Node Package Manager (npm), as well as allowing Node
> developers on Guix to make use of the reproducible builds guarantee of Guix.
> 
> After completing this project, it should be possible to easily make use of the
> less-problematic packages in the npm registry on the Guix Software Distribution.

I assume that by “less-problematic” you are referring to the packages
that actually upload their corresponding source code and not some
minified blob.  Maybe also the ones with the least insane dependency
graph.  Is this correct?

> * Project structure
> 
> Depending on findings in the early stages of the project, I foresee the
> following distinct parts:
> 
> 1. Extend Guix so it can 'simulate' the dependency graph generation of both the
>    old and new npm [fn:6].

I don’t quite get this part.  Why does it matter how old and new
versions of NPM store dependencies?  IIRC, there’s a NODE_PATH
environment variable that we could set to specify where to find each
library.

Even if we have to make a similar directory structure as NPM for some
reason, shouldn’t we just do what NPM 3 does and forget about older
versions?

> 2. Extend guix with an algorithm that matches npm's package.json flexible
>    version specification to a specific version.

This sounds like it could be part of ’guix import npm’, but not in the
Guix package recipes themselves.  To date, we do not have a single
importer that recursively imports packages, so this would be somewhat
new territory.

> 3. Add a ~guix import~ backend for the npm registry

Yup. :)

> 4. Package npm modules in guix

Could you perhaps come up with an application or library to target for
packaging that will exercise all of the code you will be adding to Guix?
As you probably know, many non-trivial Node modules typically have
*massive* dependency graphs (hundreds of nodes), so I think we need to
be strategic in what we pick to package.

I should also note that NPM, like other language package management
systems, obscures the full dependency graph of a module by
differentiating between “dependencies” and “development dependencies”.
In Guix, we build from source, so we need those “development” (misnomer:
they are really build-time dependencies, but apparently only a developer
would want to build from source) dependencies, too.

Ideally, we would like to run test suites for node modules as well.
Unfortunately, npm packages are viewed more like binaries, so they strip
out things like tests.  It’s an open question if the tarballs available
on npmjs.com are worth anything at all, or if we should instead build
from the upstream source code releases.  Unfortunately again, those
upstream releases are usually just auto-generated tarballs from git tags
on GitHub, which may prove problematic because no one actually uses
them.

Long story short, we need to be diligent about ensuring that we are
truly building from source. :)

> 5. Interface
> 
> Right now, dependency resolution in npm is as stateful as can be, with even the
> installation order [fn:6] making a difference for where npm expects to find a
> certain dependency. As two different dependency resolution mechanism are in use,
> of which especially the newer one is problematic because of its habit of
> propagating dependencies upwards in the folder structure, both have to be
> supported by a guix module.

As I mentioned earlier, I don’t think that we need to support this at
all.  We should just arrange that NODE_PATH be configured correctly.
We’re *replacing* NPM, after all, so we should use a more sane
architecture if we can.  Does that sound doable or are there technical
reasons why it wouldn’t work?
 
> npm uses SemanticVersioning range patterns [fn:7] to declare dependencies
> between packages. A problem with this approach is that the same package
> declaration can lead to an entirely different dependency graph, which defeats
> the purpose of having a system with a focus on reproducible builds. If npm
> packages are to be used, these version numbers need to be locked down to a
> specific version, corresponding to the version that npm would install if left to
> its devices. A potential problem is that the entire dependency graph has to be
> known ahead of time in order to pinpoint a 'correct' version of the dependency.
> A consistent ordering for 'installing' dependencies also has to be decided upon.

This is indeed a problem that needs solving *if* we consider recursive
package imports a requirement for this project.  Recursive imports
haven’t been done before, so maybe this could be a “nice to have” part
of the project rather than a “must have”.  This functionality would be
part of ’guix import npm’.

> After the dependency resolution has been worked out, it should be possible to
> create a guix import backend to leverage the code that has been produced up till
> now to allow the packaging of npm modules. The last part of the project
> essentially serves as a starting point for packaging up useful npm packages.
> 
> A stretch goal for the summer would be to create a guix build system for a
> subset of npm packages, such as gulp.

I would promote this from a stretch goal to a “must have”.  One of the
first steps of this project should be to create (guix build-system node)
and (guix build node-build-system) modules that will do whatever the
standard build+test+install process is for node modules.  From what I’ve
seen, a lot of node modules need gulp or grunt in order to build, so
boostrapping one or both of those tools will probably be necessary in
order to package anything substantial.
 
> * Planning
> 
> As I am currently a novice with regards to the internals of guix and the guile
> programming language, up to the start of the actual project I will mostly be
> reading up and hacking on guix. As such, the planning becomes:
> 
> April 22 -  May 22:
>   - Getting to know Guix(SD)
>   - Package programs using the guix import module
>   - An informal specification of the npm dependency resolution mechanism
>   - Getting the know the guix community and what everyone is working on
>   - Getting familiar with the contributing work flows
> 
> May 23 - Jun 5:
>   - Formal specification to allow guix to simulate npm dependency resolution (1)
> 
> Jun 6 - Jun 19:
>   - npm Version pinning should be working (2)
>   - start working on the guix import backend
> 
> Jun 20 - Jul 3: 
>   - Guix import backend should be finished by now (3)
>   - Start testing npm packages 
> 
> Jul 4 - Jul 10:
>   - Holidays! 
> 
> Jul 11 - Aug 7:
>   - Solve any problems and corner cases with building and installing npm
>     packages (4)
> 
> Aug 8 - Aug 23:
>   - If life goes a planned, Get all contributions ready to be merged back in the
>     main Guix codebase.

Please add to the timeline when you would expect to have
node-build-system implemented.  I may have some code laying around that
could accelerate this a bit.

> * About me
> 
> My name is Jelle Licht, and I am currently studying Data Science at the
> University of Technology Delft. I finished my BSc in Computer Science in 2015,
> at the University of Technology Delft as well. Somewhere in the second semester
> of my studies, my roommate was using Vi in front of me, and since then I've been
> falling into the rabbit hole that is GNU/Linux, free software and the 'open
> source' community. Somewhere along the line I became a huge fan of Clojure, and
> for a short while was a member of the very much unofficial Lisp Community Delft.
> 
> I have been a small-time contributor to some free software projects on github,
> as well as having a day job to make ends meet using mostly Node. My online
> handles include 'wordempire', as I am quite fond of reading, and 'jlicht'. For
> the past month, I have been lurking on-and-off again in the #guix irc channel,
> as well as reading up on some of the motivations behind reproducible research
> and reproducible builds.
> 
> Besides one week for visiting family and general holidays, this project would be
> my full time focus during the summer.

Thank you for your proposal!  Making JavaScript software packagable in
Guix would be a *huge* advance in our quest for reproducible web
applications.

- Dave

Re: [GSoC] Integrating npm packages into the Guix ecosystem

[Jelle Licht]

[-- Attachment #1: Type: text/plain, Size: 3417 bytes --]

Thanks for the feedback.

You are correct in your interpretation of my description. Some packages only
publish a blob, and while this should eventually also be handled
gracefully, a
first working version of an importer would surely be easier to realize if we
focus on the more well-behaved packages at first.

Using NODE_PATH could work, but in my mind it would just make managing all
these
paths more complex, compared to just symlinking the correct packages.
Symlinking
might lead to some problems with '..' not evaluating to the right place.
Some
research for an approach that works for us might still be required, but it
seems
that while the location of where npm places the packages has changed since
npm
3, how node loads modules at run-time seems to be more or less the same. Any
scheme we come up with that allows node to run the code would suffice,
indeed,
as we wouldn't use or depend on npm's package management functionalities.
The
latest folder structure as generated by npm seems as good as any starting
point
though.

Seeing as testing would be an important part to ensure that our builds are
functioning correctly, I would propose that a first goal would be the mocha
package. As far as I can see, it has only 15 unique dependencies, none of
which
seem too big to grok.

After having a functioning test runner, we could choose a package that
makes use
of the test runner. Which packages would be most suitable, I do not know
right
now, but I am open to any suggestions/wish-lists.

Another issue that you highlighted is of course the fact that, to us, npm is
more of a binary distribution platform, than an actual repository of code
artifacts. I am well aware of this fact, but in theory there is no reason a
checkout via git should not suffice. Whether it will be possible to properly
pinpoint the checkout for a specific version of each project, and if these
sources are in a functional state, could worst-case scenario be solved with
some
elbow-grease when packaging them up.

If recursive imports are not deemed a must-have, maybe we can bump this
down to
the stretch goals? It would make sense to chart new territory after having
some
experience with Guix under my belt as well ;-)

A potential problem with Gulp, and I am assuming the situation with Grunt
is not
much better, is that it (indirectly) depends on an enormous list of
dependencies. I have been keeping half an eye on this mailing-list,
especially
with regards to what to do with some of the 'bootstrap-binaries' for
Haskell and
other languages. If possible, I would like to not add more fuel to this
fire,
but packaging up all the dependencies of the de facto build tools might
prove to
be too monumental a task for now. In which case, a temporary solution using
a
bootstrapped Gulp/Grunt/both might work. If possible, some advice from
people
involved with similar situations would be nice.

The node-build-system seems doable, depending on what we need to cover.
Node-gyp
might take a bit of time to properly support, but I will look into this
tomorrow. Most other build steps for npm modules include running some
external
program, either binary or a globally installed node package, and shuffling
some
files around. When we get the popular build systems to work, the hardest
part
(w.r.t. the build system) will have been taken care of.

I will send a revised planning tomorrow.

~Jelle

[fn:1] https://docs.npmjs.com/misc/scripts

[-- Attachment #2: Type: text/html, Size: 3729 bytes --]