Re: Improved NPM importer with blacklist (Was: Re: NPM importer)

[swedebugia <swedebugia@riseup.net>]

[-- Attachment #1: Type: text/plain, Size: 710 bytes --]

Hi

I programmed the whole day and learned a lot. :D

On 2018-11-30 17:13, swedebugia wrote:

snip

> TODO:
> * make npm-recursive-import work by not fetching blacklisted packages

See the attached patches :)

> 
> * implement keyword blacklisting based on the descriptions
> 
> * match not just the whole string of blacklisted packages:
>    e.g. match also "rollup-plugin" when "rollup" is in the blacklist.

Done. See patch

> * get the tarballs from npm-registry instead as they are never missing
>    (githubs sometimes are) and likely reproducible.

Done, but not tested yet.

Also added a npm-uri (similar to pypi-uri) see patch.

Happy hacking!

-- 
Cheers Swedebugia

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-guix-Add-a-npm-uri-helper-method.patch --]
[-- Type: text/x-patch; name="0001-guix-Add-a-npm-uri-helper-method.patch", Size: 1708 bytes --]

From 9ad29ede5b26a9052518adb422bc66398c38c9b4 Mon Sep 17 00:00:00 2001
From: swedebugia <swedebugia@riseup.net>
Date: Fri, 30 Nov 2018 20:52:42 +0100
Subject: [PATCH 1/2] guix: Add a "npm-uri" helper method.

* guix/build-system/node.scm (npm-uri): New method.
---
 guix/build-system/node.scm | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/guix/build-system/node.scm b/guix/build-system/node.scm
index b6d362904..ce0d9f84c 100644
--- a/guix/build-system/node.scm
+++ b/guix/build-system/node.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2016 Jelle Licht <jlicht@fsfe.org>
+;;; Copyright © 2018 swedebugia <swedebugia@riseup.net>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -30,7 +31,22 @@
   #:export (npm-meta-uri
             %node-build-system-modules
             node-build
-            node-build-system))
+            node-build-system
+            npm-uri))
+
+;; Commentary:
+;;
+;; Standard build procedure for Node packages using 'packages.json' and
+;; Makefiles.  This is implemented as an extension of 'gnu-build-system'.
+;;
+;; Code:
+
+(define* (npm-uri name version #:optional (extension ".tgz"))
+  "Return a URI string for the NPM package hosted on the NPM Registry
+corresponding to NAME and VERSION.  EXTENSION is the file name extension, such
+as '.tar.gz'. E.g. https://registry.npmjs.org/async/-/async-1.8.0.tgz"
+  (string-append "https://registry.npmjs.org/" name "/-/" name "-" version
+                 extension))
 
 (define (npm-meta-uri name)
   "Return a URI string for the metadata of node module NAME found in the npm
-- 
2.19.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-guix-Add-blacklisting-to-recursive-importer.patch --]
[-- Type: text/x-patch; name="0002-guix-Add-blacklisting-to-recursive-importer.patch", Size: 6473 bytes --]

From 54515dbb9e30e3cdf0ac1835c465f157302aefd1 Mon Sep 17 00:00:00 2001
From: swedebugia <swedebugia@riseup.net>
Date: Sat, 1 Dec 2018 00:15:08 +0100
Subject: [PATCH 2/2] guix: Add blacklisting to recursive importer

* guix/import/npm.scm (blacklisted?, npm->guix-package): Remove blacklisted
packages from the list passed to the recursive importer.
* guix/scripts/import/npm.scm (show-help, %options): Document and improve
"recursive" option.
* doc/guix.texi (Running guix import): Document the blacklist function.

Inspired by ac906cb7bb2ec77821ddec291db4857cc812599d by Ricardo
---
 doc/guix.texi               | 33 ++++++++++++++++++++++++++++++++-
 guix/import/npm.scm         | 16 ++++++++++++----
 guix/scripts/import/npm.scm | 10 ++++++++--
 3 files changed, 52 insertions(+), 7 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 8f694a20e..ccb3c45f1 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -7258,7 +7258,7 @@ guix import npm minimist
 @end example
 
 The importer implements an experimental recursive subsystem, allowing
-for a significant of npm packages to be imported.  The command below
+for a significant number of npm packages to be imported.  The command below
 imports metadata for the @code{optimist} npm package, as well as its
 dependencies.
 
@@ -7266,6 +7266,37 @@ dependencies.
 guix import npm -r optimist
 @end example
 
+The importer currently has does not warn when a cyclic dependency has been
+encountered. It will run until the buffer overruns which could take a long
+time. You should generally see output every couple of seconds, if not a cyclic
+dependency has probably been encountered. These are best avoided by first
+packaging an earlier version of one of the packages without the offending
+input to avoid the cycle.
+
+@quotation Note
+We have implemented an experimental blacklist functionality, enabled by
+default, to help you avoid importing a lot of garbage development
+dependencies. You can customize this blacklist by editing
+@file{guix/import/npm-blacklist.scm} only when running @command{guix import
+npm} from a git checkout, see @pxref{Running Guix Before It Is Installed}.
+
+These packages and every package that begins with these 103 names are
+currently blacklisted: (matcha benchmark babel @babel webpack rollup
+browserify async ember broccoli nsp uglifyjs2 uglify-js electron statsd vega
+grunt-release lineman lerna openlayers openpgp yarnpkg wekan etherpad-lite
+meteor keybase docco jsdoc markdown eslint lint markdownlint prettier standard
+jscs @ljharb/eslint-config editorconfig-tools jshint xo tick tsml
+tsd-check @commitlint safe grunt lerna vows husky nps rimraf colors cli
+source-map chai unexpected karma karma-qunit karma-sauce-launcher
+karma-browserify coveralls covert nyc qunit mocha yargs sinon code tape
+colortape airtap tap tap-spec lab mock nyc proxyquire coffe-script cross-swawn
+terst stream-spigot make-generator-function forking-tap bogota faucet gremlins
+jest redux-mock-store webdriverio ava enzyme testem protractor testcafe
+test262 storybook jasmine volkswagen nightwatch supertest istanbul sizzy
+zombie PhantomCSS)
+This expands to thousands of packages.
+@end quotation
+
 @item elpa
 @cindex elpa
 Import metadata from an Emacs Lisp Package Archive (ELPA) package
diff --git a/guix/import/npm.scm b/guix/import/npm.scm
index 2efe81fa9..1d46fd384 100644
--- a/guix/import/npm.scm
+++ b/guix/import/npm.scm
@@ -350,8 +350,12 @@ a git checkout."
                        ,(guix-hash-url temp-dir)))))))))))
 
 (define (blacklisted? name)
-  "Check if the pair name-version is blacklisted. RETURN #t if yes, else #f."
-  (member name blacklist))
+  "Check if the string name is blacklisted. RETURN #t if yes, else #f."
+  ;; Split the string to enable ut so blacklist scoped packages like
+  ;; @babel/core and packages like eslint-popup without having to type in
+  ;; every single combination.
+  (if (member (car (string-split name (char-set #\- #\/))) blacklist)
+      #t #f))
 
 (define (sanitize-npm-version version)
   "Return version without prefixed ^."
@@ -518,8 +522,12 @@ npm list of dependencies DEPENDENCIES."
                (npm-dependencies
                 (append
                  (extract-npm-dependencies dependencies)
-                 ;; TODO blacklist!
-                 (extract-npm-dependencies dev-dependencies)))
+                 ;; Remove the blacklisted devdeps to avoid big dependency
+                 ;; cycles and all the linters, cli-tools, benchmarking,
+                 ;; etc. JS-devs adds to their development environment.
+                 (remove
+                  blacklisted?
+                  (extract-npm-dependencies dev-dependencies))))
                (description (assoc-ref package "description"))
                (home-page (assoc-ref package "homepage"))
                (license (extract-license curr))
diff --git a/guix/scripts/import/npm.scm b/guix/scripts/import/npm.scm
index 1f4ed777c..c540c3574 100644
--- a/guix/scripts/import/npm.scm
+++ b/guix/scripts/import/npm.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2015 David Thompson <davet@gnu.org>
+;;; Copyright © 2018 swedebugia <swedebugia@riseup.net>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -25,6 +26,7 @@
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-11)
   #:use-module (srfi srfi-37)
+  #:use-module (srfi srfi-41)
   #:use-module (ice-9 match)
   #:use-module (ice-9 format)
   #:export (guix-import-npm))
@@ -41,7 +43,9 @@
    Import and convert the npm package for PACKAGE-NAME.\n"))
   (display (G_ "
      -h, --help             display this help and exit"))
-  (display (G_ "
+   (display (G_ "
+     -r, --recursive        import packages recursively"))
+   (display (G_ "
      -V, --version          display version information and exit"))
   (newline)
   (show-bug-report-information))
@@ -88,7 +92,9 @@
                    `(define-public ,(string->symbol name)
                       ,pkg))
                   (G_ #f))
-                (recursive-import package-name))
+                (reverse
+                 (stream->list
+                (npm-recursive-import package-name))))
            ;; Single import
            (let ((sexp (npm->guix-package package-name)))
              (unless sexp
-- 
2.19.1