[GSoC] Integrating npm into the Guix ecosystem

[GSoC] Integrating npm into the Guix ecosystem

[Jelle Licht]

Greetings Guix hackers,

It has been some time since my last mail to this list, so I wanted to
share what I have been up to. For the people who might want to watch
along after today, I will be posting the changes that should not break
everything immediately to [1]. (Apologies for it being web-only, my
sysadminfu is still lacking. If you know how to easily and most of all
securely expose a git repo for read-only anonymous checkouts, please do
tell).

These first two weeks, I have mostly been keeping busy with writing some
of the features that will quickly allow me to identify problems with the
import and build procedures as I envision them. I was in luck; there
were many of these problems as I found out ;-).

One of the things that kind of worried me for a while was the fact that
npm was broken after my first attempts at writing _anything_. At this
moment, still being somehow intimidated by the guix code base, I had
quite some trouble finding out what was happening; it turns out
somewhere along the way, the way Node was built since version 6 (using
guix, at least) has changed, making the `npm' in the PATH of your guix
profile a _copy_ instead of a symlink to the actual executable
`npm-cli.js' script.

A patch to fix this is currently still brewing, so expect to see it
soon. In order to continue a bit (and implement something that actually
works), up till now I have worked with the 5.10 release of node.

After toiling a bit to find a nice way of getting node to find
dependencies, I started out with the NODE_PATH variable. Although some
alarming messages I read some time ago indicated this method of allowing
node to load modules is discouraged, but not deprecated.

At first I wanted to make use of a variant of the npm command to
actually take care of installing node modules to the store outputs; some
of the advantages include:

- The 'correct' files are ignored/always included
- The 'correct' symlinks are generated to executable scripts, man-pages
  and other documentation.
- Automatically up to date with new conventions in npm-land

While that all seemed (and still seems) quite nice, it happens to be the
case that npm is quite particular about dependencies that it will
accept. Patching the actual dependencies and versions in the
`package.json' file was another option, but after some consideration and
advice from my mentors I decided this would not work nicely in the long
term. Instead, I want to implement the relevant set of behaviors in
guix, of course allowing for our particular kind of dependency
management.

As such, the current installation phase mostly involves copying files to
their expected locations. I added support (as an exercise in build
system arguments) for 'global' installs[3], which as far as my current
bare-bones implementation is concerned means properly symlinking
executable scripts. In the (near-)future, I am looking into properly
wrapping these scripts to make sure the amount of propagated inputs is
kept to a minimum. As a sanity test, I packaged [4] in my own little
scratchpad. If you want to follow along, it only takes 4 `guix import
npm' invocations ;).

The current importer functions as expected for the "90%" of packages
that I tried. A problem that I ran into that I could not recognize as
easily in other importers is the fact that the npm community only
distributes the artifacts that you need to run the npm modules, but not
to build them. In most trivial cases, there are literally no
differences, but especially for more complicated packages involving
transpilation steps, this poses a problem.

As such, the importer can not actually 'know' of the location of the
source. Right now it uses some limited heuristics to probe GitHub
repositories for a tarball release, and if these are not found or the
sources are hosted at non-GH sites, it tries to check out a tag
according to the npm packaging conventions (SemVer).

The most important thing that needs to happen right now would be to
extend the range of packages that are buildable by the build system. A
combination of working towards having working 'large' packages and test
frameworks should help me quickly identify problems. This will be my
main focus for the next week.

I am aware that the current importer is quite brittle, so this needs to
change as well. As I run into problems (or someone brings them to my
attention), I will improve the quality of this subsystem as well.

If you have some advice, questions or problems, please do not hesitate
to reply. The same goes if you have some npm packages you would love to
see packaged.

Thanks a bunch for reading this WoT :)
- Jelle

[1]https://gitweb.jlicht.org/?p=guix.git;a=shortlog;h=refs/heads/gsoc-npm
[2]https://github.com/nodejs/node/issues/1627
[3]https://docs.npmjs.com/getting-started/installing-npm-packages-globally
[4]https://www.npmjs.com/package/package-json-validator

Re: [GSoC] Integrating npm into the Guix ecosystem

[John Darrington]

[-- Attachment #1: Type: text/plain, Size: 445 bytes --]

Hello Jelle,

I suggest that you avoid the word "ecosystem" when describing the work of
people who have designed and contributed to Guix.  

See https://www.gnu.org/philosophy/words-to-avoid#Ecosystem

J'


-- 
Avoid eavesdropping.  Send strong encryted email.
PGP Public key ID: 1024D/2DE827B3 
fingerprint = 8797 A26D 0854 2EAB 0285  A290 8A67 719C 2DE8 27B3
See http://sks-keyservers.net or any PGP keyserver for public key.


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [GSoC] Integrating npm into the Guix ecosystem

[ng0]

[-- Attachment #1: Type: text/plain, Size: 1376 bytes --]

On 2016-06-07(02:56:34AM+0200), Jelle Licht wrote:
>
> Greetings Guix hackers,
>
> It has been some time since my last mail to this list, so I wanted to
> share what I have been up to. For the people who might want to watch
> along after today, I will be posting the changes that should not break
> everything immediately to [1]. (Apologies for it being web-only, my
> sysadminfu is still lacking. If you know how to easily and most of all
> securely expose a git repo for read-only anonymous checkouts, please do
> tell).

I am preparing a guide on our (youbroketheinternet-overlay) way we work
with git for a long time now, it would be relatively easy to do this with
a domain name and not just tor.
The tor way involves setting up tor, torsocks, creating a dedicated git
user, set up a hidden service on port 9418 (git default), and for the git
repositories which should be exported touch a file "git-daemon-export-ok"
in the bare repository on the machine running this.

I will post this on the Gentoo wiki and probably finish it this month,
it should be easy to apply for any system though.
Adding dynamic dns name with OpenNIC or commercial/free providers to it
should be doable too, as I used it with a domain name for some time too.

> -- snip --

--
♥Ⓐ ng0
For non-prism friendly talk find me on
psyced.org / loupsycedyglgamf.onion

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: [GSoC] Integrating npm into the Guix ecosystem

[Ludovic Courtès]

Hello!

Jelle Licht <jlicht@fsfe.org> skribis:

> It has been some time since my last mail to this list, so I wanted to
> share what I have been up to. For the people who might want to watch
> along after today, I will be posting the changes that should not break
> everything immediately to [1].

Glad to see there’s already a build system and an importer!

I think it’s indeed a good idea to have them in master early.  That will
hopefully allow others to give it a try and provide feedback.

As a pre-review ;-), could be make sure you add a bunch of tests for the
importer (see tests/{cran,cpan,elpa,gem,hackage}.scm for examples), and
mention ‘node-build-system’ under “Build Systems” in doc/guix.texi?

> The current importer functions as expected for the "90%" of packages
> that I tried. A problem that I ran into that I could not recognize as
> easily in other importers is the fact that the npm community only
> distributes the artifacts that you need to run the npm modules, but not
> to build them. In most trivial cases, there are literally no
> differences, but especially for more complicated packages involving
> transpilation steps, this poses a problem.
>
> As such, the importer can not actually 'know' of the location of the
> source. Right now it uses some limited heuristics to probe GitHub
> repositories for a tarball release, and if these are not found or the
> sources are hosted at non-GH sites, it tries to check out a tag
> according to the npm packaging conventions (SemVer).

Interesting.

> The most important thing that needs to happen right now would be to
> extend the range of packages that are buildable by the build system. A
> combination of working towards having working 'large' packages and test
> frameworks should help me quickly identify problems. This will be my
> main focus for the next week.

Sounds like you’re on the right track.  :-)

I’m not qualified to comment on the specific npm issues.  I think you
may be interested in the recursive importer that Ricardo recently
submitted for CRAN, because recursive imports probably makes a lot of
sense for a report as big as npm (as in ‘guix import npm -r jquery’).

Thanks for the update!

Ludo’.