GSoC NPM

GSoC NPM

[Jelle Licht]

[-- Attachment #1: Type: text/plain, Size: 7030 bytes --]

Hello Guix,

In the last hours of GSoC, the time has come to report on my progress,
challenges and ideas regarding my project. To reiterate, the goals of this
project:

 - The ability to parse npm version data
 - An npm backend for ~guix import~
 - Npm modules in guix
 - An actual build system for npm packages

When the project started, there was some code written by David Thompson
that was
exactly what I needed to start on a node build system. For the importer of
things, I started to look at the gem importer; it seemed simple enough to
grok,
while still offering the basic functionality I needed to get a running
start.

To start of with something that did not work out as well as I had hoped,
getting
a popular build system (e.g. Gulp, Grunt, Broccoli and others) packaged. As
mentioned in my earlier mails, the list of transitive dependencies of any of
these suffer from at least the following:
- It is a list with more than 4000 packages on it
- It is a list with at some point the package itself on it
As a compromise I wanted to get a testing framework packaged instead,
because everyone likes testing. While looking at the dependencies of a
testing
framework, I noticed that I had a need for CoffeeScript. Having a passing
familiarity with the CoffeeScript dialect, I researched how one could
achieve
this. At the moment of this writing, I have packaged CoffeeScript v1.0.0,
to be
found in my git repo. What I did not account for nor foresaw was that
bootstrapping CoffeeScript took some effort. My earlier, optimistic
estimates
were based on a flaw that lacked the isolation which is needed for a proper
reproducible build. Anyway, if any of you want to play around with
CoffeeScript
v1.0.0 or any of the 50+(!) preceding versions, be my guest.

I also took both Ludovic', as well as Catonano's detailed feedback on the
initial draft of the recursive importer into account when rewriting it. It
should now only visit each node in the dependency graph once, and be a
whole lot
more efficient as well. It is still based on the multi-valued return values
that drove Ricardo's initial work on the CRAN recursive importer.

The amount of npm packages and the complexity of how they depend on one
another
is enormous. As discussed on the guix-devel ML[0], it would be useful to
gain
some insights into which packages would be worthwile to get into guix. As
Catonano noted, the problem is not on Guix' side; we have the (elementary)
building blocks with which to do the graph processing. The issue here is on
how
to implement something akin to `fold-packages' for npm packages in order to
traverse the dependency graph. After rewriting the recursive importer to be
more
sane, I scrawled some notes on my notepad that basically boil down to the
following:
1. We should only look up each npm package once, if possible
2. We should have a list of all npm package names.
3. We should be able to specify the maximum traversal depth

For (1.), a simplified version of the recursive npm importer can be used.
For
(2.), once one has installed node (with npm) and executed some `npm search'
commands, there should be a file in
`$HOME/.npm/registry.npmjs.org/-/all/.cache.json' that contains, among other
things, a listing of all package names. npm can be configured to updated
this
cache quite often, (or almost never). It does weigh in at a hefty 160MB.
What is
left is wiring all this together, which I did not have my priority these
months.

Regarding `guix refresh', one has to re-import an npm package in order to
get an
up-to-date package-representation usable by guix. Originally I had thought
that
this would be of similar difficulty to the other importers. Because we only
use
the npm registry [1] to retrieve metadata and the location of the actual
source
archive, we have no way of knowing whether a particular guix package
originated
from the npm registry.

An easy-yet-inelegant solution would be to include the package name as used
within the npm registry as metadata via an argument to the
node-build-system.
Think an `#:npm-name' key in the `arguments' field of the guix package
definition.


The importer should be able to handle most of the valid (and invalid) source
uri's you can find in the wild, especially github-related urls and
shorthands.
See [3] for a list of packages that might need some changes to either their
package.json/npmregistry metadata, or obviate a change to the importer
logic.

The current version of the importer only looks at the latest version of
packages. It should be easy to fix this by handling the `@version' suffix
like
the hackage importer does. This could be useful to break some of the
dependency
cycles that exist between npm packages. For this to work, a scheme different
from the current NODE_PATH will have to be considered. The first module
with a
certain name found in NODE_PATH will be loaded at runtime, so in the current
implementation it is not possible to have multiple versions of a package
with
the same name loaded at one moment.

Ricardo's idea of a recursive importer is pretty nice, imho. It should be
doable
to implement some more of them in a similar fashion what has been done for
cran
and npm.


While I hope nobody (including myself) has to package so many variants of
the
same package again, it would be nice to somehow download _only_ the
revision you
are interested in. AFAIK, there is no proper way for git to do this for the
general 'give me this commit' case. Something that I eventually did in
order to
alleviate the ~3 minute checkout times for each iteration of CS, was the
following hack[2]. It basically puts a recent-enough copy of the CS git
repo in
my store, and then made a shallow copy from that when using git-fetch. This
took
my build times down to less than 10 seconds per iteration.

If you are interested in my work, have a look at:
https://github.com/wordempire/guix/commits/gsoc-final
, or just
`git clone https://github.com/wordempire/guix.git`
`git checkout gsoc-final`.

I will be trickling in a patch series onto the ML the next few days.

I guess that is enough text from me again. I would still like to express my
gratitude to my mentors David Thompson and Christopher Allan Webber, as
well as
the rest of #guix and guix-devel (and some folks at GHM as well) for dealing
with my ramblings, questions and helping me keep this project fun. Special
thanks to Catonano as well for having a close look at my code as well.

With just some tweaks to the importer, we should be able to at least
package a huge subset of all the packages that require zero to few
dependencies, once we are able to identify them.


I probably forgot quite some important and unimportant details, so if you
have
any questions, tips or just want to blame me for getting more messy
JavaScript into guix-land, send me a mail ;-).

- Jelle Licht

[0] https://lists.gnu.org/archive/html/guix-devel/2016-07/msg01726.html
[1] https://www.npmjs.com/
[2] http://paste.lisp.org/display/323999 <- beware, here be dragons etc
[3] http://paste.lisp.org/display/324007

[-- Attachment #2: Type: text/html, Size: 7900 bytes --]

Re: GSoC NPM

[Ricardo Wurmus]

Hi

> I also took both Ludovic', as well as Catonano's detailed feedback on the
> initial draft of the recursive importer into account when rewriting it. It
> should now only visit each node in the dependency graph once, and be a
> whole lot
> more efficient as well. It is still based on the multi-valued return values
> that drove Ricardo's initial work on the CRAN recursive importer.

[...]

> After rewriting the recursive importer to be
> more
> sane, I scrawled some notes on my notepad that basically boil down to the
> following:
> 1. We should only look up each npm package once, if possible
> 2. We should have a list of all npm package names.
> 3. We should be able to specify the maximum traversal depth

I’m not sure I understand.  The CRAN recursive importer visits
packages only once because it keeps track of previously imported
packages (in addition to those that are already in Guix).

> An easy-yet-inelegant solution would be to include the package name as used
> within the npm registry as metadata via an argument to the
> node-build-system.

That’s not so inelegant; or at least we have precedent in Guix.  For
CRAN and Bioconductor packages we often add something like this:

    (properties `((upstream-name . "ACSNMineR")))

This is already used by the updater.

~~ Ricardo

Re: GSoC NPM

[Jelle Licht]

Hi

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> writes:

> Hi
>
>> I also took both Ludovic', as well as Catonano's detailed feedback on the
>> initial draft of the recursive importer into account when rewriting it. It
>> should now only visit each node in the dependency graph once, and be a
>> whole lot
>> more efficient as well. It is still based on the multi-valued return values
>> that drove Ricardo's initial work on the CRAN recursive importer.
>
> [...]
>
>> After rewriting the recursive importer to be
>> more
>> sane, I scrawled some notes on my notepad that basically boil down to the
>> following:
>> 1. We should only look up each npm package once, if possible
>> 2. We should have a list of all npm package names.
>> 3. We should be able to specify the maximum traversal depth
>
> I’m not sure I understand.  The CRAN recursive importer visits
> packages only once because it keeps track of previously imported
> packages (in addition to those that are already in Guix).


With the recursive fold approach, I had the issue that
sometimes packages that were imported in a 'leaf fold' had to be
imported again in a different 'leaf fold'. This might very well also be
a mistake I made in my original adaptation of the CRAN importer ;-).

If the general consensus on the ML is that using higher order functions
+ recursion is more elegant than a big, bold loop, I can still
reimplement it as such.

>
>> An easy-yet-inelegant solution would be to include the package name as used
>> within the npm registry as metadata via an argument to the
>> node-build-system.
>
> That’s not so inelegant; or at least we have precedent in Guix.  For
> CRAN and Bioconductor packages we often add something like this:
>
>     (properties `((upstream-name . "ACSNMineR")))
>
> This is already used by the updater.

Well, that is most likely what I will be implementing then. An
advantage of this is that it would also make my life easier when making
runtime npm module loading more robust :-).

>
> ~~ Ricardo

Thanks for your feedback

- Jelle

Re: GSoC NPM

[Catonano]

[-- Attachment #1: Type: text/plain, Size: 1848 bytes --]

Hi Jelle,

I tried to import jquery with the last version of your code

This is the resulting expression (they are a lot, this is just one)

(define-public node-jquery
  (package
    (name "node-jquery")
    (version "3.1.0")
    (source
      (origin
        (method url-fetch)
        (uri "
https://github.com/jquery/jquery/archive/3.1.0/jquery-3.1.0.tar.gz")
        (sha256
          (base32
            "05jcaa0xg041hpq3xdswf45jbkm6shvinjm3scawwjqqlrc39xca"))))
    (build-system node-build-system)
    (propagated-inputs `())
    (native-inputs `())
    (synopsis
      "JavaScript library for DOM operations")
    (description
      "JavaScript library for DOM operations")
    (home-page "https://jquery.com")
    (license expat)))


As you can see, there are no native-inputs

I understand that dev-dependencies (as on line 467) is used by
make-npm-sexp to populate native-inputs, so in the terminal I inspected
dev-dependencies  expecting to find it empty and I was wrong, it was
correctly populated

Like this

("node-native-promise-only"
 "node-insight"
 "node-strip-json-comments"
 "node-cross-spawn"
 "node-q"
 "node-grunt-git-authors"
 "node-grunt-contrib-watch"
 "node-eslint-config-jquery"
 "node-husky"
 "node-load-grunt-tasks"
 "node-qunitjs"
 "node-sinon"
 "node-commitplease"
 "node-grunt-eslint"
 "node-sizzle"
 "node-grunt-babel"
 "node-jsdom"
 "node-babel-preset-es2015"
 "node-grunt-npmcopy"
 "node-grunt-contrib-uglify"
 "node-gzip-js"
 "node-qunit-assert-step"
 "node-requirejs"
 "node-grunt"
 "node-promises-aplus-tests"
 "node-grunt-compare-size"
 "node-grunt-jsonlint"
 "node-testswarm"
 "node-core-js"
 "node-grunt-cli"
 "node-grunt-newer")

So now I can't explain why native-inputs is empty.

Maybe I should run this a few times to ensure that the result is always the
same

But this could be worth your attention.

[-- Attachment #2: Type: text/html, Size: 2850 bytes --]

Re: GSoC NPM

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1703 bytes --]

Hi, Jelle!

Jelle Licht <jlicht@fsfe.org> skribis:

> To start of with something that did not work out as well as I had hoped,
> getting
> a popular build system (e.g. Gulp, Grunt, Broccoli and others) packaged. As
> mentioned in my earlier mails, the list of transitive dependencies of any of
> these suffer from at least the following:
> - It is a list with more than 4000 packages on it
> - It is a list with at some point the package itself on it
> As a compromise I wanted to get a testing framework packaged instead,
> because everyone likes testing. While looking at the dependencies of a
> testing
> framework, I noticed that I had a need for CoffeeScript. Having a passing
> familiarity with the CoffeeScript dialect, I researched how one could
> achieve
> this. At the moment of this writing, I have packaged CoffeeScript v1.0.0,
> to be
> found in my git repo. What I did not account for nor foresaw was that
> bootstrapping CoffeeScript took some effort. My earlier, optimistic
> estimates
> were based on a flaw that lacked the isolation which is needed for a proper
> reproducible build. Anyway, if any of you want to play around with
> CoffeeScript
> v1.0.0 or any of the 50+(!) preceding versions, be my guest.

I haven’t looked at the code but, as discussed at the GHM, I think that
importing CoffeeScript and especially finding the right path in this
maze of dependencies and versions to bootstrap it is an achievement.

It provides a concrete illustration of how hard it is to build from
source even relatively recent JS code.

Anyway, make sure to ping Chris and Dave so we can get this code in
‘master’ soon!  :-)

Thank you!

Ludo’.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: GSoC NPM

[Christopher Allan Webber]

Ludovic Courtès writes:

> I haven’t looked at the code but, as discussed at the GHM, I think that
> importing CoffeeScript and especially finding the right path in this
> maze of dependencies and versions to bootstrap it is an achievement.
>
> It provides a concrete illustration of how hard it is to build from
> source even relatively recent JS code.
>
> Anyway, make sure to ping Chris and Dave so we can get this code in
> ‘master’ soon!  :-)
>
> Thank you!
>
> Ludo’.

Yes, agreed, I think this is a great achievement... I'd really like to
get it into Guix's master branch as soon as we can.

Unfortunately, I won't be able to help until next month at soonest.  So
if there is anyone else able to help...
 - Chris

Re: GSoC NPM

[Jan Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 868 bytes --]

Jelle Licht writes:

Hi Jelle!

> - The ability to parse npm version data
> - An npm backend for ~guix import~
> - Npm modules in guix
> - An actual build system for npm packages

That's amazing.  I played with it today and noticed that it always
downloads devDependencies.  Why is that...I disabled that because
I think I don't need those?

Also, I found that you prefer going through the repository/github
instead of using the dist tarball.  Why is that?  Some packages do not
have a repository field, such as `http'.  I changed that to prefer using
the dist tarball and use repository as fallback.  You probably want to
change that order?

I made some other small changes, see attached patch, to be able to
download all packages that I need, notably: cjson, http and xmldom.

Thanks again for your amazing work, hoping to have this in master soon.

Greetings,
Jan


[-- Attachment #2: 0001-npm-importer-updates-fixes-downloading-of-e.g.-cjson.patch --]
[-- Type: text/x-patch, Size: 11830 bytes --]

From 151f5d338199f94651d499070240ff5f1e75058c Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke@gnu.org>
Date: Fri, 2 Sep 2016 16:16:35 +0200
Subject: [PATCH] npm importer: updates; fixes downloading of e.g.: cjson,
 http, xmldom.

* gnu/nmp.scm: New file.
* gnu/local.mk (GNU_SYSTEM_MODULES): Add it.
* scripts/npm-import: New file.
* guix/import/npm.scm (gh-fuzzy-tag-match): Add two fallbacks: missing /TAGS
and VERSION mismatch.
(source-uri): Prefer using (dist . tarball) over (repository . url).
(spdx-string->license): Add LGPL.
(package-origin): Handle registry.npmjs.org url.
* (npm->guix-package): Discard devDependencies.
---
 gnu/local.mk         |   1 +
 gnu/packages/npm.scm |  12 +++++
 guix/import/npm.scm  | 125 ++++++++++++++++++++++++++++++++++++---------------
 scripts/npm-import   |  31 +++++++++++++
 4 files changed, 132 insertions(+), 37 deletions(-)
 create mode 100644 gnu/packages/npm.scm
 create mode 100755 scripts/npm-import

diff --git a/gnu/local.mk b/gnu/local.mk
index b9d2a11..4fa94c7 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -255,6 +255,7 @@ GNU_SYSTEM_MODULES =				\
   %D%/packages/nettle.scm			\
   %D%/packages/networking.scm			\
   %D%/packages/ninja.scm			\
+  %D%/packages/npm.scm				\
   %D%/packages/node.scm				\
   %D%/packages/noweb.scm			\
   %D%/packages/ntp.scm				\
diff --git a/gnu/packages/npm.scm b/gnu/packages/npm.scm
new file mode 100644
index 0000000..0a483d2
--- /dev/null
+++ b/gnu/packages/npm.scm
@@ -0,0 +1,12 @@
+(define-module (gnu packages npm)
+  #:use-module (guix licenses)
+  #:use-module (guix packages)
+  #:use-module (guix download)
+  #:use-module (guix build-system node))
+
+;; FIXME
+(define npm-license-unknown public-domain)
+
+#!
+scripts/npm-import async-q q cjson http fs-extra  xmldom >> gnu/packages/npm.scm
+!#
diff --git a/guix/import/npm.scm b/guix/import/npm.scm
index b6c9120..1e7f2c4 100644
--- a/guix/import/npm.scm
+++ b/guix/import/npm.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2015 David Thompson <davet@gnu.org>
 ;;; Copyright © 2016 Jelle Licht <jlicht@fsfe.org>
+;;; Copyright © 2016 Jan Nieuwenhuizen <janneke@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -187,10 +188,10 @@ GITHUB-REPO"
                       "https://api.github.com/repos/"
                       (github-user-slash-repository github-repo)
                       "/tags"))
-         (json (json-fetch*
-                (if token
-                    (string-append api-url "?access_token=" token)
-                    api-url))))
+         (api-url (if token
+                      (string-append api-url "?access_token=" token)
+                      api-url))
+         (json (json-fetch* api-url)))
     (if (eq? json #f)
         (if token
             (error "Error downloading release information through the GitHub
@@ -208,8 +209,25 @@ api-url))
                     (member name fuzzy-tags)))
                 json)))
           (match proper-release
-            (()                       ;empty release list
-             #f)
+            (()                       ;fuzzy version mismatch
+             (if (pair? json)
+                 (begin
+                   ;;XXX: Just pick first release
+                   ;; e.g.: xmldom 0.1.16 vs 0.1.22
+                   (hash-ref (car json) "name"))
+                 ;;XXX: No tags: Just pick latest commit from master
+                 ;; e.g.: cjson
+                 ;; TODO: iso master, snarf default_branch from /
+                 (let* ((branches-url (string-replace-substring api-url "/tags" "/branches"))
+                        (branches (json-fetch* branches-url))
+                        (first-or-master
+                         (or
+                          (find (lambda (x) (equal? (hash-ref x "name") "master"))
+                                branches)
+                          (car branches)))
+                        (commit (hash-ref first-or-master "commit"))
+                        (sha (hash-ref commit "sha")))
+                   sha)))
             ((release . rest)         ;one or more releases
              ;;XXX: Just pick the first release
              (let ((tag (hash-ref release "name")))
@@ -265,8 +283,19 @@ GITHUB-URL."
 
 (define (source-uri npm-meta version)
   "Return the repository url for version VERSION of NPM-META"
-  (let* ((v    (assoc-ref* npm-meta "versions" version)))
-    (normalise-url (assoc-ref* v "repository" "url"))))
+  (let* ((v    (assoc-ref* npm-meta "versions" version))
+         (repo (assoc-ref v "repository"))
+         (dist (assoc-ref v "dist")))
+    (or
+     ;; e.g.: http
+     ;; FIXME: this will prefer registry.npmjs.org
+     (and dist
+          (assoc-ref dist "tarball"))
+
+     ;; FIXME: this will prefer github.org
+     (and repo
+          (and=> (assoc-ref repo "url") normalise-url))
+     )))
 
 (define (guix-hash-url path)
   "Return the hash of PATH in nix-base32 format. PATH can be either a file or
@@ -319,6 +348,7 @@ package."
     ("IJG" 'ijg)
     ("Imlib2" 'imlib2)
     ("IPA" 'ipa)
+    ("LGPL" 'lgpl2.0)
     ("LGPL-2.0" 'lgpl2.0)
     ("LGPL-2.0+" 'lgpl2.0+)
     ("LGPL-2.1" 'lgpl2.1)
@@ -359,32 +389,46 @@ command."
 located at REPO-URL. Tries to locate a released tarball before falling back to
 a git checkout."
   (let ((uri (string->uri repo-url)))
-    (if (equal? (uri-host uri) "github.com")
-        (call-with-temporary-output-file
-         (lambda (temp port)
-           (let* ((gh-version (gh-fuzzy-tag-match repo-url version))
-                  (tb (github-release-url repo-url gh-version))
-                  (result (url-fetch tb temp))
-                  (hash (bytevector->nix-base32-string (port-sha256 port))))
-             (close-port port)
-             `(origin
-                (method url-fetch)
-                (uri ,tb)
-                (sha256
-                 (base32
-                  ,hash))))))
-        (call-with-temporary-directory
-         (lambda (temp-dir)
-           (let ((fuzzy-version (generic-fuzzy-tag-match repo-url version)))
-             (and (node-git-fetch repo-url fuzzy-version temp-dir)
-                  `(origin
-                     (method git-fetch)
-                     (uri (git-reference
-                           (url ,repo-url)
-                           (commit ,fuzzy-version)))
-                     (sha256
-                      (base32
-                       ,(guix-hash-url temp-dir)))))))))))
+    (cond
+     ((equal? (uri-host uri) "registry.npmjs.org")
+      (call-with-temporary-output-file
+       (lambda (temp port)
+         (let* ((result (url-fetch repo-url temp))
+                (hash (bytevector->nix-base32-string (port-sha256 port))))
+           (close-port port)
+           `(origin
+              (method url-fetch)
+              (uri ,repo-url)
+              (sha256
+               (base32
+                ,hash)))))))
+     ((equal? (uri-host uri) "github.com")
+      (call-with-temporary-output-file
+       (lambda (temp port)
+         (let* ((gh-version (gh-fuzzy-tag-match repo-url version))
+                (tb (github-release-url repo-url gh-version))
+                (result (url-fetch tb temp))
+                (hash (bytevector->nix-base32-string (port-sha256 port))))
+           (close-port port)
+           `(origin
+              (method url-fetch)
+              (uri ,tb)
+              (sha256
+               (base32
+                ,hash)))))))
+     (else
+      (call-with-temporary-directory
+       (lambda (temp-dir)
+         (let ((fuzzy-version (generic-fuzzy-tag-match repo-url version)))
+           (and (node-git-fetch repo-url fuzzy-version temp-dir)
+                `(origin
+                   (method git-fetch)
+                   (uri (git-reference
+                         (url ,repo-url)
+                         (commit ,fuzzy-version)))
+                   (sha256
+                    (base32
+                     ,(guix-hash-url temp-dir))))))))))))
 
 (define (make-npm-sexp name version home-page description
                        dependencies dev-dependencies license source-url)
@@ -444,11 +488,16 @@ npm list of dependencies DEPENDENCIES."
       (spdx-string->license (assoc-ref license-entry "type")))
      ((string? license-legacy)
       (spdx-string->license license-legacy))
+     ((and (pair? license-legacy) (string? (car license-legacy)))
+      (if (= (length license-legacy) 1)
+          (spdx-string->license (car license-legacy))
+          (map spdx-string->license license-legacy)))
      ((and license-legacy (positive? (length license-legacy)))
       `(list ,@(map
                 (lambda (l) (spdx-string->license (assoc-ref l "type")))
                 license-legacy)))
      (else
+      (format (current-error-port) "extract-license: no license found: ~a\n" package-json)
       #f))))
 
 (define (npm->guix-package package-name)
@@ -460,7 +509,9 @@ npm list of dependencies DEPENDENCIES."
                (version (latest-source-release package))
                (curr (assoc-ref* package "versions" version))
                (raw-dependencies (assoc-ref curr "dependencies"))
-               (raw-dev-dependencies (assoc-ref curr "devDependencies"))
+               ;; TODO: do not recurse into devDependencies
+               (raw-dev-dependencies #f;;(assoc-ref curr "devDependencies")
+                )
                (dependencies (extract-guix-dependencies raw-dependencies))
                (dev-dependencies (extract-guix-dependencies
                                   raw-dev-dependencies))
@@ -469,8 +520,8 @@ npm list of dependencies DEPENDENCIES."
                  (extract-npm-dependencies raw-dependencies)
                  (extract-npm-dependencies raw-dev-dependencies)))
                (description (assoc-ref package "description"))
-               (home-page (assoc-ref package "homepage"))
-               (license (extract-license curr))
+               (home-page (or (assoc-ref package "homepage") "http://npmjs.com"))
+               (license (or (extract-license curr) 'npm-license-unknown))
                (source-url (source-uri package version)))
           (values 
            (make-npm-sexp name version home-page description
diff --git a/scripts/npm-import b/scripts/npm-import
new file mode 100755
index 0000000..3f45aa0
--- /dev/null
+++ b/scripts/npm-import
@@ -0,0 +1,31 @@
+#! /bin/sh
+# -*- scheme -*-
+unset LANG LC_ALL
+unset GUILE_AUTO_COMPILE GUILE_LOAD_COMPILED_PATH
+exec ${GUILE-guile} --no-auto-compile -L $PWD -C $PWD -e '(@@ (npm-import) main)' -s "$0" ${1+"$@"}
+!#
+
+(define-module (npm-import)
+  #:use-module (ice-9 pretty-print)
+  #:use-module (srfi srfi-26)
+  #:use-module (guix import npm)
+  #:use-module (gnu packages npm))
+
+(define (package->define entry)
+  `(define-public ,(string->symbol (string-append "node-" (car entry)))
+     ,(cadr entry)))
+
+(define (name->node.scm package-name)
+  (format (current-error-port) "package: ~a\n" package-name)
+  (let* ((packages-alist (recursive-import package-name))
+         (defines (map package->define packages-alist))
+         (file (open-file (string-append "node-" package-name ".scm") "w")))
+    (map (cut pretty-print <> file) defines)
+    (close file)))
+
+(define (main args)
+  (let ((files (cdr (command-line))))
+   (when (null? files)
+     (format (current-error-port) "Usage: npm-import NPM-PACKAGE-NAME\n")
+     (exit 1))
+   (for-each name->node.scm files)))
-- 
2.9.3


[-- Attachment #3: Type: text/plain, Size: 154 bytes --]


-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar®  http://AvatarAcademy.nl  

Re: GSoC NPM

[Thompson, David]

On Fri, Sep 2, 2016 at 10:24 AM, Jan Nieuwenhuizen <janneke@gnu.org> wrote:

> Also, I found that you prefer going through the repository/github
> instead of using the dist tarball.  Why is that?

The tarballs distributed by NPM are considered binaries, not source.

- Dave

Re: GSoC NPM

[Jan Nieuwenhuizen]

Thompson, David writes:

> On Fri, Sep 2, 2016 at 10:24 AM, Jan Nieuwenhuizen <janneke@gnu.org> wrote:
>
>> Also, I found that you prefer going through the repository/github
>> instead of using the dist tarball.  Why is that?
>
> The tarballs distributed by NPM are considered binaries, not source.

Ah I see.  In some cases there are indeed some differences.  So we'll
probably want to reverse the default: if a repository is present it is
most probably be better to get that.  I only found significant
differences with the fibers package, though.  Others just differ in
having an additional .gitignore and .npmignore file.

I found in some cases that repositories do not have release tags.  In
such cases, it may be "better" to use the dist tarball, as you have a
better chance of getting the exact released version commit?

Greetings,
Jan

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar®  http://AvatarAcademy.nl  

Re: GSoC NPM

[Jelle Licht]

[-- Attachment #1: Type: text/plain, Size: 2023 bytes --]

Hi Jan,

Thanks for your interest and work. I am currently quite occupied with
getting ready
for my next year of studies, so I will only shortly address your points;

The short of it is that the dist tarball does not always contain the actual
source code.
Examples of this include generated code, minified code etc.

The devDependencies are, in these cases, the things we need to be able to
actually
build the package. Examples of this include gulp, grunt, and several
testing frameworks.

For simple packages, the difference between a npm tarball and a GH
tarball/repo are
non-existent. I made the choice to skip the npm tarball because I'd rather
err on the
side of caution, and not let people download and run these non-source
packages by accident ;-).

I will have more time to see this through next week.

- Jelle


2016-09-02 16:24 GMT+02:00 Jan Nieuwenhuizen <janneke@gnu.org>:

> Jelle Licht writes:
>
> Hi Jelle!
>
> > - The ability to parse npm version data
> > - An npm backend for ~guix import~
> > - Npm modules in guix
> > - An actual build system for npm packages
>
> That's amazing.  I played with it today and noticed that it always
> downloads devDependencies.  Why is that...I disabled that because
> I think I don't need those?
>
> Also, I found that you prefer going through the repository/github
> instead of using the dist tarball.  Why is that?  Some packages do not
> have a repository field, such as `http'.  I changed that to prefer using
> the dist tarball and use repository as fallback.  You probably want to
> change that order?
>
> I made some other small changes, see attached patch, to be able to
> download all packages that I need, notably: cjson, http and xmldom.
>
> Thanks again for your amazing work, hoping to have this in master soon.
>
> Greetings,
> Jan
>
>
>
> --
> Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
> Freelance IT http://JoyofSource.com | Avatar®  http://AvatarAcademy.nl
>
>

[-- Attachment #2: Type: text/html, Size: 2838 bytes --]

Re: GSoC NPM

[Jan Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 2707 bytes --]

Jelle Licht writes:

Hi Jelle!

Here's a new patch replacing the previous one, summary

   * add --binary option to importer, sets (arguments (#:binary? #t))
   * use `npm build' for non-binary packages as fallback (WAS: skip)
   * use `npm install -g' for non-binary packages; fixes e.g. loadash
   * fallback for packages dist tarball/binary-only: e.g.: http
   * handle packages without any tags in git, e.g.: cjson
   * handle packages with version mismatch, e.g.: xmldom

With these small additions to your work I'm able to automagically fetch
the full list of 318 (binary) packages that I need for my client's
project (which has 40 toplevel dependencies such as bunyan, express,
jison, jquery, nodemailer, pg, q, socket.io, underscore, xmldom).

> The short of it is that the dist tarball does not always contain the
> actual source code.  Examples of this include generated code, minified
> code etc.

Yes, I see that now.  David remarked that the dist tarball should be
considered to be a binary package.

> The devDependencies are, in these cases, the things we need to be able
> to actually build the package. Examples of this include gulp, grunt,
> and several testing frameworks.

Yes...and here is where it starts getting interesting.

I made several attempts to build packages from source, but except for
packages that imho should not be allowed to exist such as `array-equal',
that seemed next to impossible.  Maybe I was unlucky, or maybe I am
missing something?

As a first attempt, I tried to recursively import `q', a fairly basic
package from my possibly ignorant perspective: can you write anything
non-trivial in node without using q?.  When that resulted in over 6004
dependencies (using build systems grunt, gulp and node-gyp, listing 582
errors), I was pretty sure there was a problem with your importer.
Using the --binary option, q has no dependencies.  None.  Single
package.  Hmm.

The `babel' package, a prerequisite for the the `gulp' build system
which is needed to build the `har-validator' library needed to run the
`node-gyp' build system, has a list of over 6000 dependencies.

Build systems building build systems...

> For simple packages, the difference between a npm tarball and a GH
> tarball/repo are non-existent. I made the choice to skip the npm
> tarball because I'd rather err on the side of caution, and not let
> people download and run these non-source packages by accident ;-).

Yes, that makes sense.  I found that the `http' package has this binary
form only so I added it as a fallback for now.

> I will have more time to see this through next week.

That's great, thanks.

Greetings,
Jan

See https://gitlab.com/janneke/guix.git -- branch npm-binary


[-- Attachment #2: 0001-npm-importer-support-binary-and-fixes-for-e.g.-cjson.patch --]
[-- Type: text/x-patch, Size: 22588 bytes --]

From c60e72504a8ba4bb6a90c07bef7844d461a12467 Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke@gnu.org>
Date: Fri, 2 Sep 2016 16:16:35 +0200
Subject: [PATCH] npm importer: support --binary and fixes for e.g.: cjson,
 http, xmldom.

* gnu/nmp.scm: New file.
* gnu/local.mk (GNU_SYSTEM_MODULES): Add it.
* guix/scripts/import/npm.scm: Add --binary option.
* guix/import/npm.scm (gh-fuzzy-tag-match): Add two fallbacks: missing /TAGS
and VERSION mismatch.
(strip-.git-if-needed project): New function.
(github-user-slash-repository, github-repository): Use it.
(source-uri): Fallback to use `binary' (dist . tarball).  Add optional binary?
parameter to prefer binary fallback.
(spdx-string->license): Add LGPL, fix LGPL-3.0.
(make-npm-sexp): Add optional binary? parameter to set #:binary? argument.
(npm->guix-package): Add optional binary? parameter to set #:binary? argument
to ignore devDependencies.
(recursive-import): Add optional binary? parameter.
* guix/build-system/node.scm (node-build): Add binary? and make-flags keys.
* guix/build/node-build-system (build): Also check for `Gulpfile.js', fallback
to generic `npm build'.  Skip build if #:binary?.
(binary-install): Rename from install.
(npm-install): New function.
(install): Have #:binary? switch between binary-install, and npm-install.
(package-origin): Handle registry.npmjs.org url.
(npm->guix-package)[npm-binary?]: Discard devDependencies.
---
 gnu/local.mk                     |   1 +
 gnu/packages/npm.scm             |  34 +++++++++
 guix/build-system/node.scm       |   4 +
 guix/build/node-build-system.scm |  30 ++++++--
 guix/import/npm.scm              | 161 +++++++++++++++++++++++++++------------
 guix/scripts/import/npm.scm      |  13 +++-
 6 files changed, 186 insertions(+), 57 deletions(-)
 create mode 100644 gnu/packages/npm.scm

diff --git a/gnu/local.mk b/gnu/local.mk
index b9d2a11..4fa94c7 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -255,6 +255,7 @@ GNU_SYSTEM_MODULES =				\
   %D%/packages/nettle.scm			\
   %D%/packages/networking.scm			\
   %D%/packages/ninja.scm			\
+  %D%/packages/npm.scm				\
   %D%/packages/node.scm				\
   %D%/packages/noweb.scm			\
   %D%/packages/ntp.scm				\
diff --git a/gnu/packages/npm.scm b/gnu/packages/npm.scm
new file mode 100644
index 0000000..43b7774
--- /dev/null
+++ b/gnu/packages/npm.scm
@@ -0,0 +1,34 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2016 Jan Nieuwenhuizen <janneke@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu packages npm)
+  #:use-module (guix licenses)
+  #:use-module (guix packages)
+  #:use-module (guix download)
+  #:use-module (guix build-system node)
+  #:use-module (gnu packages base)
+  #:use-module (gnu packages commencement)
+  #:use-module (gnu packages gcc)
+  #:use-module (gnu packages perl)
+  #:use-module (gnu packages python))
+
+(define npm-license-unknown public-domain)
+
+#!
+for i in array-equal async-q q cjson http fs-extra xmldom; do ./pre-inst-env guix import import --recursive --binary $i >> gnu/packages/npm.scm; make; done
+!#
diff --git a/guix/build-system/node.scm b/guix/build-system/node.scm
index a7b71e6..99e0ef0 100644
--- a/guix/build-system/node.scm
+++ b/guix/build-system/node.scm
@@ -75,10 +75,12 @@ registry."
 
 (define* (node-build store name inputs
                      #:key
+                     (binary? #f)
                      (npm-flags ''())
                      (global? #f)
                      (test-target "test")
                      (tests? #f)
+                     (make-flags ''())
                      (phases '(@ (guix build node-build-system)
                                  %standard-phases))
                      (outputs '("out"))
@@ -103,6 +105,8 @@ registry."
                                 source))
                    #:system ,system
                    #:npm-flags ,npm-flags
+                   #:make-flags ,make-flags                   
+                   #:binary? ,binary?
                    #:global? ,global?
                    #:test-target ,test-target
                    #:tests? ,tests?
diff --git a/guix/build/node-build-system.scm b/guix/build/node-build-system.scm
index 35767d6..1077201 100644
--- a/guix/build/node-build-system.scm
+++ b/guix/build/node-build-system.scm
@@ -50,17 +50,23 @@
             (find-files "." "(min\\.js|min\\.js\\.map|min\\.map)$"))
   #t)
 
-(define* (build #:key outputs inputs #:allow-other-keys)
+(define* (build #:key outputs binary? (make-flags '()) (npm-flags '())
+                #:allow-other-keys)
   "Build a new node module using the appropriate build system."
   ;; XXX: Develop a more robust heuristic, allow override
-  (cond ((file-exists? "gulpfile.js")
+  (cond (binary? #t)
+        ((or (file-exists? "gulpfile.js")
+             (file-exists? "Gulpfile.js"))
          (zero? (system* "gulp")))
         ((file-exists? "gruntfile.js")
          (zero? (system* "grunt")))
+        ((file-exists? "binding.gyp")
+         (and (zero? (system* "node-gyp.js" "configure"))
+              (zero? (system* "node-gyp.js" "build"))))
         ((file-exists? "Makefile")
-         (zero? (system* "make")))
+         (zero? (apply system* "make" `(,@make-flags))))
         (else
-         #t)))
+         (zero? (apply system* "npm" "build" `(,@npm-flags))))))
 
 (define* (check #:key tests? #:allow-other-keys)
   "Run 'npm test' if TESTS?"
@@ -69,7 +75,7 @@
       (zero? (system* "npm" "test"))
       #t))
 
-(define* (install #:key outputs inputs global? #:allow-other-keys)
+(define* (binary-install #:key outputs inputs global? #:allow-other-keys)
   "Install the node module to the output store item. MODULENAME defines how
 under which name the module will be installed, GLOBAL? determines whether this
 is an npm global install."
@@ -86,6 +92,20 @@ is an npm global install."
       (symlink (string-append tgt-dir "/node_modules/" modulename "/bin") bin-dir))
     #t))
 
+(define* (npm-install #:key outputs inputs (npm-flags '()) #:allow-other-keys)
+  "Install the node module to the output store item. MODULENAME defines how
+under which name the module will be installed, GLOBAL? determines whether this
+is an npm global install."
+  (let* ((out (assoc-ref outputs "out"))
+         (home (string-append "/tmp/home")))
+    (setenv "HOME" home)
+    (zero? (apply system* "npm" "install" "-g" "--prefix" out `(,@npm-flags)))))
+
+(define* (install #:key outputs inputs binary? global? (npm-flags '())
+                  #:allow-other-keys)
+  (if binary?
+      (binary-install #:outputs outputs #:inputs inputs #:global? global?)
+      (npm-install #:outputs outputs #:global? global? #:npm-flags #:npm-flags)))
 
 (define %standard-phases
   (modify-phases gnu:%standard-phases
diff --git a/guix/import/npm.scm b/guix/import/npm.scm
index b6c9120..5d6bd9e 100644
--- a/guix/import/npm.scm
+++ b/guix/import/npm.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2015 David Thompson <davet@gnu.org>
 ;;; Copyright © 2016 Jelle Licht <jlicht@fsfe.org>
+;;; Copyright © 2016 Jan Nieuwenhuizen <janneke@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -47,6 +48,7 @@
   #:use-module (guix packages)
   #:use-module (gnu packages)
   #:use-module (guix build-system node)
+  #:use-module (guix build node-build-system)
   #:export (npm->guix-package
             recursive-import))
 
@@ -187,10 +189,10 @@ GITHUB-REPO"
                       "https://api.github.com/repos/"
                       (github-user-slash-repository github-repo)
                       "/tags"))
-         (json (json-fetch*
-                (if token
-                    (string-append api-url "?access_token=" token)
-                    api-url))))
+         (api-url (if token
+                      (string-append api-url "?access_token=" token)
+                      api-url))
+         (json (json-fetch* api-url)))
     (if (eq? json #f)
         (if token
             (error "Error downloading release information through the GitHub
@@ -208,28 +210,50 @@ api-url))
                     (member name fuzzy-tags)))
                 json)))
           (match proper-release
-            (()                       ;empty release list
-             #f)
+            (()                       ;fuzzy version mismatch
+             (if (pair? json)
+                 (begin
+                   ;;XXX: Just pick first release
+                   ;; e.g.: xmldom 0.1.16 vs 0.1.22
+                   (hash-ref (car json) "name"))
+                 ;;XXX: No tags: Just pick latest commit from master
+                 ;; e.g.: cjson
+                 ;; TODO: iso master, snarf default_branch from /
+                 (let* ((branches-url (string-replace-substring api-url "/tags" "/branches"))
+                        (branches (json-fetch* branches-url))
+                        (first-or-master
+                         (or
+                          (find (lambda (x) (equal? (hash-ref x "name") "master"))
+                                branches)
+                          (car branches)))
+                        (commit (hash-ref first-or-master "commit"))
+                        (sha (hash-ref commit "sha")))
+                   sha)))
             ((release . rest)         ;one or more releases
              ;;XXX: Just pick the first release
              (let ((tag (hash-ref release "name")))
                tag)))))))
 
+(define (strip-.git-if-needed project)
+  ;; for babel, e.g. project does not end in `.git'
+  (if (string-suffix? ".git" project)
+      (string-drop-right project 4)
+      project))
 
 (define (github-user-slash-repository github-url)
   "Return a string e.g. arq5x/bedtools2 of the owner and the name of the
 repository separated by a forward slash, from a string URL of the form
 'https://github.com/arq5x/bedtools2.git'"
   (match (string-split (uri-path (string->uri github-url)) #\/)
     ((_ owner project . rest)
-     (string-append owner "/" (string-drop-right project 4)))))
+     (string-append owner "/" (strip-.git-if-needed project)))))
 
 (define (github-repository github-url)
   "Return a string e.g. bedtools2 of the name of the repository, from a string
 URL of the form 'https://github.com/arq5x/bedtools2.git'"
   (match (string-split (uri-path (string->uri github-url)) #\/)
     ((_ owner project . rest)
-     (string-drop-right project 4))))
+     (strip-.git-if-needed project))))
 
 (define (github-release-url github-url version)
   "Return the url for the tagged release VERSION on the github repo found at
@@ -263,10 +288,19 @@ GITHUB-URL."
   "Return true if PACKAGE is a node package."
   (string-prefix? "node-" (package-name package)))
 
-(define (source-uri npm-meta version)
+(define* (source-uri npm-meta version #:optional binary?)
   "Return the repository url for version VERSION of NPM-META"
-  (let* ((v    (assoc-ref* npm-meta "versions" version)))
-    (normalise-url (assoc-ref* v "repository" "url"))))
+  (let* ((v    (assoc-ref* npm-meta "versions" version))
+         (repo (assoc-ref v "repository"))
+         (dist (assoc-ref v "dist")))
+    (or
+     (and binary? dist
+          (assoc-ref dist "tarball"))
+     (and repo
+          (and=> (assoc-ref repo "url") normalise-url))
+     ;; fallback for `binary'-only packages, e.g.: http
+     (and dist
+          (assoc-ref dist "tarball")))))
 
 (define (guix-hash-url path)
   "Return the hash of PATH in nix-base32 format. PATH can be either a file or
@@ -319,11 +353,12 @@ package."
     ("IJG" 'ijg)
     ("Imlib2" 'imlib2)
     ("IPA" 'ipa)
+    ("LGPL" 'lgpl2.0)
     ("LGPL-2.0" 'lgpl2.0)
     ("LGPL-2.0+" 'lgpl2.0+)
     ("LGPL-2.1" 'lgpl2.1)
     ("LGPL-2.1+" 'lgpl2.1+)
-    ("LGPL-3.0" 'lgpl3.0)
+    ("LGPL-3.0" 'lgpl3)
     ("MPL-1.0" 'mpl1.0)
     ("MPL-1.1" 'mpl1.1)
     ("MPL-2.0" 'mpl2.0)
@@ -359,35 +394,50 @@ command."
 located at REPO-URL. Tries to locate a released tarball before falling back to
 a git checkout."
   (let ((uri (string->uri repo-url)))
-    (if (equal? (uri-host uri) "github.com")
-        (call-with-temporary-output-file
-         (lambda (temp port)
-           (let* ((gh-version (gh-fuzzy-tag-match repo-url version))
-                  (tb (github-release-url repo-url gh-version))
-                  (result (url-fetch tb temp))
-                  (hash (bytevector->nix-base32-string (port-sha256 port))))
-             (close-port port)
-             `(origin
-                (method url-fetch)
-                (uri ,tb)
-                (sha256
-                 (base32
-                  ,hash))))))
-        (call-with-temporary-directory
-         (lambda (temp-dir)
-           (let ((fuzzy-version (generic-fuzzy-tag-match repo-url version)))
-             (and (node-git-fetch repo-url fuzzy-version temp-dir)
-                  `(origin
-                     (method git-fetch)
-                     (uri (git-reference
-                           (url ,repo-url)
-                           (commit ,fuzzy-version)))
-                     (sha256
-                      (base32
-                       ,(guix-hash-url temp-dir)))))))))))
+    (cond
+     ((equal? (uri-host uri) "registry.npmjs.org")
+      (call-with-temporary-output-file
+       (lambda (temp port)
+         (let* ((result (url-fetch repo-url temp))
+                (hash (bytevector->nix-base32-string (port-sha256 port))))
+           (close-port port)
+           `(origin
+              (method url-fetch)
+              (uri ,repo-url)
+              (sha256
+               (base32
+                ,hash)))))))
+     ((equal? (uri-host uri) "github.com")
+      (call-with-temporary-output-file
+       (lambda (temp port)
+         (let* ((gh-version (gh-fuzzy-tag-match repo-url version))
+                (tb (github-release-url repo-url gh-version))
+                (result (url-fetch tb temp))
+                (hash (bytevector->nix-base32-string (port-sha256 port))))
+           (close-port port)
+           `(origin
+              (method url-fetch)
+              (uri ,tb)
+              (sha256
+               (base32
+                ,hash)))))))
+     (else
+      (call-with-temporary-directory
+       (lambda (temp-dir)
+         (let ((fuzzy-version (generic-fuzzy-tag-match repo-url version)))
+           (and (node-git-fetch repo-url fuzzy-version temp-dir)
+                `(origin
+                   (method git-fetch)
+                   (uri (git-reference
+                         (url ,repo-url)
+                         (commit ,fuzzy-version)))
+                   (sha256
+                    (base32
+                     ,(guix-hash-url temp-dir))))))))))))
 
 (define (make-npm-sexp name version home-page description
-                       dependencies dev-dependencies license source-url)
+                       dependencies dev-dependencies license source-url
+                       binary?)
   "Return the `package' s-expression for a Node package with the given NAME,
 VERSION, HOME-PAGE, DESCRIPTION, DEPENDENCIES, DEV-DEPENDENCIES, LICENSES and
 SOURCE-URL."
@@ -415,6 +465,9 @@ SOURCE-URL."
                            (,'unquote
                             ,(string->symbol name))))
                        dev-dependencies)))))
+       ,@(if (not binary?)
+             '()
+             '((arguments `(#:binary? #t))))
        (synopsis ,description) ; no synopsis field in package.json files
        (description ,description)
        (home-page ,home-page)
@@ -444,23 +497,32 @@ npm list of dependencies DEPENDENCIES."
       (spdx-string->license (assoc-ref license-entry "type")))
      ((string? license-legacy)
       (spdx-string->license license-legacy))
+     ((and (pair? license-legacy) (string? (car license-legacy)))
+      (if (= (length license-legacy) 1)
+          (spdx-string->license (car license-legacy))
+          (map spdx-string->license license-legacy)))
      ((and license-legacy (positive? (length license-legacy)))
       `(list ,@(map
                 (lambda (l) (spdx-string->license (assoc-ref l "type")))
                 license-legacy)))
      (else
+      (format (current-error-port) "extract-license: no license found: ~a\n" package-json)
       #f))))
 
-(define (npm->guix-package package-name)
+(define* (npm->guix-package package-name #:optional binary?)
   "Fetch the metadata for PACKAGE-NAME from registry.npmjs.com and return the
- `package' s-expression corresponding to that package, or  on failure."
+`package' s-expression corresponding to that package, or on failure.  If
+BINARY?, use the `binary' dist tarball as source url and ignore any
+devDependencies."
   (let ((package (npm-fetch package-name)))
     (if package
         (let* ((name (assoc-ref package "name"))
                (version (latest-source-release package))
                (curr (assoc-ref* package "versions" version))
                (raw-dependencies (assoc-ref curr "dependencies"))
-               (raw-dev-dependencies (assoc-ref curr "devDependencies"))
+               (raw-dev-dependencies (if binary?
+                                         #f
+                                         (assoc-ref curr "devDependencies")))
                (dependencies (extract-guix-dependencies raw-dependencies))
                (dev-dependencies (extract-guix-dependencies
                                   raw-dev-dependencies))
@@ -469,19 +531,20 @@ npm list of dependencies DEPENDENCIES."
                  (extract-npm-dependencies raw-dependencies)
                  (extract-npm-dependencies raw-dev-dependencies)))
                (description (assoc-ref package "description"))
-               (home-page (assoc-ref package "homepage"))
-               (license (extract-license curr))
-               (source-url (source-uri package version)))
+               (home-page (or (assoc-ref package "homepage") "http://npmjs.com"))
+               (license (or (extract-license curr) 'npm-license-unknown))
+               (source-url (source-uri package version binary?)))
           (values 
            (make-npm-sexp name version home-page description
-                          dependencies dev-dependencies license source-url)
+                          dependencies dev-dependencies license source-url
+                          binary?)
            npm-dependencies))
         (error "Could not download metadata:" package-name))))
 
-(define* (recursive-import package-name)
+(define* (recursive-import package-name #:optional binary?)
   "Recursively fetch the metadata for PACKAGE-NAME and its dependencies from
 registry.npmjs.com and return a list of 'package-name, package s-expression'
-tuples."
+tuples.  If BINARY?, use the `binary' tarball from the dist field."
   (define (seen? item seen)
     (or (vhash-assoc item seen)
         (not (null? (find-packages-by-name (node-package-name item))))))
@@ -501,7 +564,7 @@ tuples."
              (receive (package dependencies)
                  (catch #t
                    (lambda ()
-                     (npm->guix-package package-name))
+                     (npm->guix-package package-name binary?))
                    (lambda (key . parameters)
                      (format (current-error-port)
                              "Uncaught throw to '~a: ~a\n" key parameters)
diff --git a/guix/scripts/import/npm.scm b/guix/scripts/import/npm.scm
index 79abcf0..8e39381 100644
--- a/guix/scripts/import/npm.scm
+++ b/guix/scripts/import/npm.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2015 David Thompson <davet@gnu.org>
+;;; Copyright © 2016 Jan Nieuwenhuizen <janneke@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -40,6 +41,8 @@
   (display (_ "Usage: guix import npm PACKAGE-NAME
    Import and convert the npm package for PACKAGE-NAME.\n"))
   (display (_ "
+     -b, --binary           use binary dist tarball for source url"))
+  (display (_ "
      -h, --help             display this help and exit"))
   (display (_ "
      -V, --version          display version information and exit"))
@@ -48,7 +51,10 @@
 
 (define %options
   ;; Specification of the command-line options.
-  (cons* (option '(#\h "help") #f #f
+  (cons* (option '(#\b "binary") #f #f
+                 (lambda (opt name arg result)
+                   (alist-cons 'binary? #t result)))
+         (option '(#\h "help") #f #f
                  (lambda args
                    (show-help)
                    (exit 0)))
@@ -73,6 +79,7 @@
                   (alist-cons 'argument arg result))
                 %default-options))
   (let* ((opts (parse-options))
+         (binary? (assoc-ref opts 'binary?))
          (args (filter-map (match-lambda
                              (('argument . value)
                               value)
@@ -88,9 +95,9 @@
                    `(define-public ,(string->symbol name)
                       ,pkg))
                   (_ #f))
-                (recursive-import package-name))
+                (recursive-import package-name binary?))
            ;; Single import
-           (let ((sexp (npm->guix-package package-name)))
+           (let ((sexp (npm->guix-package package-name binary?)))
              (unless sexp
                (leave (_ "failed to download meta-data for package '~a'~%")
                       package-name))
-- 
2.9.3


[-- Attachment #3: Type: text/plain, Size: 154 bytes --]


-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar®  http://AvatarAcademy.nl  

Re: GSoC NPM

[Thompson, David]

On Sun, Sep 4, 2016 at 10:11 AM, Jan Nieuwenhuizen <janneke@gnu.org> wrote:

>    * add --binary option to importer, sets (arguments (#:binary? #t))

This violates a core principle of Guix: reproducible builds.  I don't
support patches that encourage using pre-built binaries.

> As a first attempt, I tried to recursively import `q', a fairly basic
> package from my possibly ignorant perspective: can you write anything
> non-trivial in node without using q?.  When that resulted in over 6004
> dependencies (using build systems grunt, gulp and node-gyp, listing 582
> errors), I was pretty sure there was a problem with your importer.
> Using the --binary option, q has no dependencies.  None.  Single
> package.  Hmm.

That's because the thing on npm has already been built for you.  q
*does* have dependencies if you want to build it from source, which is
what we should all be striving for.

- Dave

NPM and trusted binaries

[Pjotr Prins]

On Tue, Sep 06, 2016 at 11:48:04AM -0400, Thompson, David wrote:
> On Sun, Sep 4, 2016 at 10:11 AM, Jan Nieuwenhuizen <janneke@gnu.org> wrote:
> 
> >    * add --binary option to importer, sets (arguments (#:binary? #t))
> 
> This violates a core principle of Guix: reproducible builds.  I don't
> support patches that encourage using pre-built binaries.

In principle I agree. We want to be able to read the code.

Still, I think Guix would benefit from a somewhat more relaxed stance
in this. Especially where it comes to cross-platform binary
deployments we could be accelerate things now and then - and maybe
work on source deployment later. I am thinking of Erlang Beam and the
JVM mostly. If binaries are *trusted* we could do that. Point of note,
we distribute *trusted* binaries already. Who builds those?

I am becoming increasingly of the opinion that Guix can be a 'small'
core of rock solid software and we should provide mechanisms to wave
out in other maybe less controlled directions. Whether it is in source
or in binary form.

Pj.

Re: NPM and trusted binaries

[Ludovic Courtès]

Howdy,

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> On Tue, Sep 06, 2016 at 11:48:04AM -0400, Thompson, David wrote:
>> On Sun, Sep 4, 2016 at 10:11 AM, Jan Nieuwenhuizen <janneke@gnu.org> wrote:
>> 
>> >    * add --binary option to importer, sets (arguments (#:binary? #t))
>> 
>> This violates a core principle of Guix: reproducible builds.  I don't
>> support patches that encourage using pre-built binaries.
>
> In principle I agree. We want to be able to read the code.
>
> Still, I think Guix would benefit from a somewhat more relaxed stance
> in this.

It’s part of Guix’s mission to build from source whenever that is
possible, which is the case here, AIUI.

We know from previous discussions that some compilers can no longer be
built from source; we already have exceptions for these.

Ludo’.

Re: NPM and trusted binaries

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

>> Still, I think Guix would benefit from a somewhat more relaxed stance
>> in this.
>
> It’s part of Guix’s mission to build from source whenever that is
> possible, which is the case here, AIUI.

Yes, I thought so too...I shared my findings to get more viewpoints on
this assertion and also on the validity of the source/binary metaphor
for npm packages.

I spent the weekend to attempt building `q' using the repository+version
= source package metaphor.  Because given the facts that

  * an npm package must specify a name and a version
  * an npm package may optionally list a source repository
  * an npm package may optionally lists devDependencies
  * using repository + the devDepencies you can build the installable
    npm package

I also thought it would be nice to build as many packages as possible
from their repsository urls.  I encountered several small problems with
the importer (or with inconsistencies/breakages in npm packages) that I
fixed.

Quoting my earlier mail

   ... that resulted in over 6004 dependencies (using build systems
   grunt, gulp and node-gyp, listing 582 errors)

Finally, I became discouraged and Sunday night I added the --binary flag
to the importer, decided to taint the resulting package description
doing

  (arguments `(#:binary #t))

to enable breaking this enormous dependency chain.  How many more
package dependencies will result from the 582 packages that have import
problems?

Oh, please note that the `binary' or installable version of this `q'
package consists of two javascript files: q.js and query.js which are
identical to the ones in the `source' package.  The git repository
additionally has tests and documentation (and history).

Another example is the `http' package: it does not list a repository url
and the installable package is plain readable javascript.  Does the
source/binary metaphor apply here?

The `fibers' package comes with precompiled binaries for popular
platforms.  It also includes all C sources to build these binaries.  It
could be possible that some npm package has only minimalized javascript
(i.e.: can really be considered binary) but I haven't seen such a
package yet.

WDYT, do we have enough information to decide if building from `source'
the right metaphor?  Is it pracically feasible and does feasibilty have
any weight?  What's the next step I could take to help to bring `q' and
`http' (and the other 316 packages I need) into Guix?

Greetings,
Jan

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar®  http://AvatarAcademy.nl  

Re: NPM and trusted binaries

[Pjotr Prins]

On Wed, Sep 07, 2016 at 07:51:46PM +0200, Jan Nieuwenhuizen wrote:
> Ludovic Courtès writes:
> 
> >> Still, I think Guix would benefit from a somewhat more relaxed stance
> >> in this.
> >
> > It’s part of Guix’s mission to build from source whenever that is
> > possible, which is the case here, AIUI.

Mission is fine and I agree with that (in principle).

> WDYT, do we have enough information to decide if building from `source'
> the right metaphor?  Is it pracically feasible and does feasibilty have
> any weight?  What's the next step I could take to help to bring `q' and
> `http' (and the other 316 packages I need) into Guix?

I think we are clear we do not want binaries in the main project
unless there is no way to do it from source.

Personally I think we should be easier on ourselves which implies that
we get multiple flavours of Guix. 

Another reason to make 'guix channels' work.

Pj.

Re: NPM and trusted binaries

[Jelle Licht]

Just a quick note from me;

AFAIK, the http module is a built-in of node, so you can probably save
yourselves the efforts of packaging it ;-).

Furthermore, lots of development dependencies are not strictly
necessary; e.g. a minifier/uglifier is not required for most
functionality of a package, and ditto for linters and to a certain
extent test frameworks, at least for our initial set of node packages.
This initial set of packages can then (hopefully) be used to package the
rest of npm properly, including tests etc.

The biggest issue here is that an importer can not decide for you which
devDependency is actually is needed to properly build a source archive,
and which just provides convenience functions. The importer should
become more useful when we have a solid set of npm packages in guix.
Before that, the importer will probably be useful to a lesser degree for
any packages besides the most trivial.

Regarding feasibility and its weight, I would say that a simple
transformation such as concatenating files should not be an issue,
whereas more involved transformations such as tree shaking,
uglification, or tranpilation do involve a transformation that take away
much of our freedoms to modify the software, at least in practice.

- Jelle

Pjotr Prins <pjotr.public12@thebird.nl> writes:

> On Wed, Sep 07, 2016 at 07:51:46PM +0200, Jan Nieuwenhuizen wrote:
>> Ludovic Courtès writes:
>> 
>> >> Still, I think Guix would benefit from a somewhat more relaxed stance
>> >> in this.
>> >
>> > It’s part of Guix’s mission to build from source whenever that is
>> > possible, which is the case here, AIUI.
>
> Mission is fine and I agree with that (in principle).
>
>> WDYT, do we have enough information to decide if building from `source'
>> the right metaphor?  Is it pracically feasible and does feasibilty have
>> any weight?  What's the next step I could take to help to bring `q' and
>> `http' (and the other 316 packages I need) into Guix?
>
> I think we are clear we do not want binaries in the main project
> unless there is no way to do it from source.
>
> Personally I think we should be easier on ourselves which implies that
> we get multiple flavours of Guix. 
>
> Another reason to make 'guix channels' work.
>
> Pj.

Re: NPM and trusted binaries

[Mike Gerwitz]

[-- Attachment #1: Type: text/plain, Size: 908 bytes --]

On Tue, Sep 06, 2016 at 18:50:48 +0200, Pjotr Prins wrote:
> On Tue, Sep 06, 2016 at 11:48:04AM -0400, Thompson, David wrote:
>> This violates a core principle of Guix: reproducible builds.  I don't
>> support patches that encourage using pre-built binaries.
>
> In principle I agree. We want to be able to read the code.
>
> Still, I think Guix would benefit from a somewhat more relaxed stance
> in this.

If a user is able to build from source, shouldn't Guix be able to?  And
if neither can, how can we guarantee that the provided binary is even
free and actually corresponds to the given source?

From a software freedom perspective, the source code _is_ the
program.  If that is unworkable, then so is the software itself.

-- 
Mike Gerwitz
Free Software Hacker+Activist | GNU Maintainer & Volunteer
GPG: 2217 5B02 E626 BC98 D7C0  C2E5 F22B B815 8EE3 0EAB
https://mikegerwitz.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: NPM and trusted binaries

[Jan Nieuwenhuizen]

Mike Gerwitz writes:

> If a user is able to build from source

That's a question that I like to explore.

If a user builds an npm package from its source repository, I assume
that they install the devDependencies needed for that using npm?

The transitive closure of installing all devDependencies for the `q'
package by building them all from their source repositories, means
building > 6000 packages.

> , shouldn't Guix be able to?

> And if neither can, how can we guarantee that the provided binary is
> even free and actually corresponds to the given source?

I would also like to explore if the source/binary package metaphor is
a valid one for npm.

For the packages that I considered, I used the `diff' command to assert
that the installable npm package includes javascript and C files and are
identical to the ones in the repository.

Greetings,
Jan

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar®  http://AvatarAcademy.nl  

Re: NPM and trusted binaries

[Mike Gerwitz]

[-- Attachment #1: Type: text/plain, Size: 2018 bytes --]

On Thu, Sep 08, 2016 at 10:45:57 +0200, Jan Nieuwenhuizen wrote:
> If a user builds an npm package from its source repository, I assume
> that they install the devDependencies needed for that using npm?

Unfortunately that depends on the project.  Some projects use
devDependencies only for things like linters, test runners, assertion
systems, etc; others might need them for building.

> The transitive closure of installing all devDependencies for the `q'
> package by building them all from their source repositories, means
> building > 6000 packages.

Many of those packages are shared between others.  Given a sufficiently
large pool of npm packages, there'll be a great deal of intersections in
the graph.

I haven't been following this closely enough to speak intelligently
about the conversion, though.

> I would also like to explore if the source/binary package metaphor is
> a valid one for npm.

Sure it is.

> For the packages that I considered, I used the `diff' command to assert
> that the installable npm package includes javascript and C files and are
> identical to the ones in the repository.

In some cases, this will be true.  Possibly in a majority.  Think of npm
as publishing the results of `make dist` (literally, that's what I
do).  That could do anything, it could do pretty much nothing.  If a
Perl/Python/PHP/Ruby/Scheme/<insert interpreted language here> script is
in the distribution tarball unmodified from the source, what
considerations do we give it when packaging for, say, Debian?

But we'd have to know that on a case-by-case basis.  If we want a
general solution to this problem, we wouldn't want to add a bunch of
exceptions.

If it's literally publishing the source code repository (which many
are), then there is no distinction.  But we'd have to know that to be
true.

-- 
Mike Gerwitz
Free Software Hacker+Activist | GNU Maintainer & Volunteer
GPG: 2217 5B02 E626 BC98 D7C0  C2E5 F22B B815 8EE3 0EAB
https://mikegerwitz.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: NPM and trusted binaries

[Jan Nieuwenhuizen]

Mike Gerwitz writes:

> On Thu, Sep 08, 2016 at 10:45:57 +0200, Jan Nieuwenhuizen wrote:
>> If a user builds an npm package from its source repository, I assume
>> that they install the devDependencies needed for that using npm?
>
> Unfortunately that depends on the project.  Some projects use
> devDependencies only for things like linters, test runners, assertion
> systems, etc; others might need them for building.

The question I'm trying to answer is: how does `a user' who builds a
package from the repository install the needed dependencies.

I very much doubt that users install the essential dependencies all by
building those from the source repository.  How would they do that?

My working hypothesis is that it's impossible to do so for any
moderately interesting npm package.  And I would very much like someone
to show me (with working code) that instead it is possible.

>> The transitive closure of installing all devDependencies for the `q'
>> package by building them all from their source repositories, means
>> building > 6000 packages.
>
> Many of those packages are shared between others.

Not so.  The total sum of interrelated dependencies to build `q' is over
41,000.  The number of imported packages for `q' using Jelle's importer
with some small fixes by me is over 6,000 unique dependencies and over
500 that can currently not be resolved by the importer and error out.

Please show me that building `q' this way is possible and what the
benefits are (in terms of software freedom) of spending our energy by
upholding the source/binary metaphor (even if for a majority of packages
there may not be a difference).

Greetings,
Jan

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar®  http://AvatarAcademy.nl  

Re: NPM and trusted binaries

[Mike Gerwitz]

[-- Attachment #1: Type: text/plain, Size: 5040 bytes --]

On Thu, Sep 08, 2016 at 21:54:36 +0200, Jan Nieuwenhuizen wrote:
> The question I'm trying to answer is: how does `a user' who builds a
> package from the repository install the needed dependencies.

Sorry, I misinterpreted.

`npm install <pkg>' will by default install all devDependencies; the
`--production' flag suppresses that behavior.

Many packages define a command to be run when `npm test` is invoked,
which would in turn need the devDependencies to run the test suite.

> I very much doubt that users install the essential dependencies all by
> building those from the source repository.  How would they do that?

No, they don't.  I'm not sure if it's even possible with how npm works,
though I haven't done that sort of research.

But that'd be Guix' responsibility---just because npm doesn't offer a
way to do that doesn't mean that Guix can't, provided that there is an
automated way to track down each of the packages and determine how they
are built.  Some might use Make, some Grunt, nothing, etc.

> My working hypothesis is that it's impossible to do so for any
> moderately interesting npm package.  And I would very much like someone
> to show me (with working code) that instead it is possible.

I'm hoping such code is precisely what this project produces. :)

> what the benefits are (in terms of software freedom) of spending our
> energy by upholding the source/binary metaphor (even if for a majority
> of packages there may not be a difference).

As I mentioned, I don't see a difference between this situation and
packaging other software that has no distinction between source code and
"binary" distribution.  It's just a hell of a lot more complex and the
package manager used to manage these packages (npm) doesn't care about
these issues.

Corresponding source code must include everything needed to build the
software, and must be the preferred form of modifying that
software.  This assumption cannot be made with the state of the packages
in the npm repository.  Some of the files might not even be in the
source repository (e.g. generated).

I have great faith in Guix and its mission; it would be a shame to see
that tainted by something like this.  Normally someone will look over a
package manually before adding it; but mass-adding thousands of packages
in an automated manner is even _more_ of an argument for the importance
not trusting binary distributions.


With all that said, I have no idea how it'll be done.  Someone could
just add any old file to the published npm package and never commit it
to any source repository.  I've done that accidentally.  I don't know
how you'd handle situations like that.  I don't know how you'd handle a
situation where a build script doesn't even work on a machine other than
the author's.  I don't know how you confirm that the software produced
from a build will actually be the same as the software normally
installed when you invoke `npm install <pkg>`.

If a package doesn't build from source, contain all the necessary files,
etc, it's not practical to exercise your freedoms, and so including it
would be bad.  But if one dependency out of thousands has that problem,
then what?  If one of the dependencies happens to actually contain
non-free code, then what?

When I evaluate software offered to GNU, I have to consider each and
every dependency.  This usually isn't a difficult task---many libraries
are standard, have already been reviewed by a project like Debian, and
there's rarely more than a few dozen of them.  I then build it from
source.  If I have the packages on my system (Trisquel at present), I
know that they're free and can be built from source.  Otherwise, I must
compile the library myself, recursively as needed for all
dependencies (which is usually not an issue).  If any of those were a
problem at any point, then the whole of the project is a problem.  If
any of those dependencies are non-free, the whole of the project is
non-free unless it can be swapped out.  If one obscure library requires
several dark incantations and a few dead chickens, users can't
practically exercise their freedoms, and that would be a problem for the
package.

Now how the hell is this enforced with thousands of dependencies that
have not undergone any review, within a community that really couldn't
care less?  Even something as simple as the license: package.json has no
legal force; it's _metadata_.

I feel like this will have to be manually checked no matter how it is
done; any automated process would just be a tool to aid in a
transition and keeping a package up-to-date.  I don't really see any
other way.

So I think that I share in your concern with how such a thing would
possible be done.  My point is that if it can't, it shouldn't be at all
(where is where we differ).

-- 
Mike Gerwitz
Free Software Hacker+Activist | GNU Maintainer & Volunteer
GPG: 2217 5B02 E626 BC98 D7C0  C2E5 F22B B815 8EE3 0EAB
https://mikegerwitz.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: NPM and trusted binaries

[Ludovic Courtès]

Hi Mike,

Mike Gerwitz <mtg@gnu.org> skribis:

> Now how the hell is this enforced with thousands of dependencies that
> have not undergone any review, within a community that really couldn't
> care less?  Even something as simple as the license: package.json has no
> legal force; it's _metadata_.
>
> I feel like this will have to be manually checked no matter how it is
> done; any automated process would just be a tool to aid in a
> transition and keeping a package up-to-date.  I don't really see any
> other way.
>
> So I think that I share in your concern with how such a thing would
> possible be done.  My point is that if it can't, it shouldn't be at all
> (where is where we differ).

Yes, that’s a serious concern.  Maybe all we can reasonably hope to
achieve is to provide a core subset of the free NPM packages in Guix
proper, built from source.

People may still end up using automatically-generated, unchecked
packages for the rest.  Nevertheless, that would be an improvement over
the status quo.

(PyPI, Hackage, CPAN, and CRAN seem to be less problematic in this
regard, maybe because they are “culturally closer” to the free software
movement.)

Ludo’.

Re: NPM and trusted binaries

[Pjotr Prins]

On Fri, Sep 09, 2016 at 10:45:43AM +0200, Ludovic Courtès wrote:
 
> Yes, that’s a serious concern.  Maybe all we can reasonably hope to
> achieve is to provide a core subset of the free NPM packages in Guix
> proper, built from source.
> 
> People may still end up using automatically-generated, unchecked
> packages for the rest.  Nevertheless, that would be an improvement over
> the status quo.
> 
> (PyPI, Hackage, CPAN, and CRAN seem to be less problematic in this
> regard, maybe because they are “culturally closer” to the free software
> movement.)

Not quite true, though there are generally less dependencies to deal
with. I still install packages using those language systems -
especially with Ruby, R, D and Elixir. It does not matter. Once I want
robustness I make sure to package in Guix. npm is just the worst of
the lot because of the sheer size, stupidity and circular
dependencies.

We should really think a bit harder about the transitional phase.
Also, software development goes faster in general than that we can
package. 

My take is that GNU Guix proper should be lean, mean and robust. That
way we can maintain and rely on stuff. 

For the more experimental packages and other 'solutions' we ought to
depend on channels - or distributed package sources. These need not
take the purist view.

Pj.