From mboxrd@z Thu Jan 1 00:00:00 1970 From: swedebugia Subject: Re: Improved NPM importer with blacklist (Was: Re: NPM importer) Date: Sat, 1 Dec 2018 00:27:57 +0100 Message-ID: References: <70F182DB-C157-4763-A4C6-89985545661C@lepiller.eu> <12fdf913-eb03-b898-f9ff-8dd455935975@riseup.net> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------3304040D449BA540336B1B76" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:50755) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gSs6n-0006yr-5T for guix-devel@gnu.org; Fri, 30 Nov 2018 18:22:08 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gSs6h-0000SZ-4b for guix-devel@gnu.org; Fri, 30 Nov 2018 18:22:04 -0500 Received: from mx1.riseup.net ([198.252.153.129]:35256) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gSs6f-0000Rk-4w for guix-devel@gnu.org; Fri, 30 Nov 2018 18:21:59 -0500 Received: from piha.riseup.net (piha-pn.riseup.net [10.0.1.163]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "*.riseup.net", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.riseup.net (Postfix) with ESMTPS id 3EE961A01F7 for ; Fri, 30 Nov 2018 15:21:55 -0800 (PST) Received: from [127.0.0.1] (localhost [127.0.0.1]) by piha.riseup.net with ESMTPSA id 645BC68AD2 for ; Fri, 30 Nov 2018 15:21:54 -0800 (PST) In-Reply-To: <12fdf913-eb03-b898-f9ff-8dd455935975@riseup.net> Content-Language: en-US List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: guix-devel@gnu.org This is a multi-part message in MIME format. --------------3304040D449BA540336B1B76 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Hi I programmed the whole day and learned a lot. :D On 2018-11-30 17:13, swedebugia wrote: snip > TODO: > * make npm-recursive-import work by not fetching blacklisted packages See the attached patches :) >=20 > * implement keyword blacklisting based on the descriptions >=20 > * match not just the whole string of blacklisted packages: > =C2=A0 e.g. match also "rollup-plugin" when "rollup" is in the blackli= st. Done. See patch > * get the tarballs from npm-registry instead as they are never missing > =C2=A0 (githubs sometimes are) and likely reproducible. Done, but not tested yet. Also added a npm-uri (similar to pypi-uri) see patch. Happy hacking! --=20 Cheers Swedebugia --------------3304040D449BA540336B1B76 Content-Type: text/x-patch; name="0001-guix-Add-a-npm-uri-helper-method.patch" Content-Disposition: attachment; filename="0001-guix-Add-a-npm-uri-helper-method.patch" Content-Transfer-Encoding: quoted-printable >From 9ad29ede5b26a9052518adb422bc66398c38c9b4 Mon Sep 17 00:00:00 2001 From: swedebugia Date: Fri, 30 Nov 2018 20:52:42 +0100 Subject: [PATCH 1/2] guix: Add a "npm-uri" helper method. * guix/build-system/node.scm (npm-uri): New method. --- guix/build-system/node.scm | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/guix/build-system/node.scm b/guix/build-system/node.scm index b6d362904..ce0d9f84c 100644 --- a/guix/build-system/node.scm +++ b/guix/build-system/node.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2016 Jelle Licht +;;; Copyright =C2=A9 2018 swedebugia ;;; ;;; This file is part of GNU Guix. ;;; @@ -30,7 +31,22 @@ #:export (npm-meta-uri %node-build-system-modules node-build - node-build-system)) + node-build-system + npm-uri)) + +;; Commentary: +;; +;; Standard build procedure for Node packages using 'packages.json' and +;; Makefiles. This is implemented as an extension of 'gnu-build-system'= . +;; +;; Code: + +(define* (npm-uri name version #:optional (extension ".tgz")) + "Return a URI string for the NPM package hosted on the NPM Registry +corresponding to NAME and VERSION. EXTENSION is the file name extension= , such +as '.tar.gz'. E.g. https://registry.npmjs.org/async/-/async-1.8.0.tgz" + (string-append "https://registry.npmjs.org/" name "/-/" name "-" versi= on + extension)) =20 (define (npm-meta-uri name) "Return a URI string for the metadata of node module NAME found in the= npm --=20 2.19.1 --------------3304040D449BA540336B1B76 Content-Type: text/x-patch; name="0002-guix-Add-blacklisting-to-recursive-importer.patch" Content-Disposition: attachment; filename="0002-guix-Add-blacklisting-to-recursive-importer.patch" Content-Transfer-Encoding: quoted-printable >From 54515dbb9e30e3cdf0ac1835c465f157302aefd1 Mon Sep 17 00:00:00 2001 From: swedebugia Date: Sat, 1 Dec 2018 00:15:08 +0100 Subject: [PATCH 2/2] guix: Add blacklisting to recursive importer * guix/import/npm.scm (blacklisted?, npm->guix-package): Remove blacklist= ed packages from the list passed to the recursive importer. * guix/scripts/import/npm.scm (show-help, %options): Document and improve "recursive" option. * doc/guix.texi (Running guix import): Document the blacklist function. Inspired by ac906cb7bb2ec77821ddec291db4857cc812599d by Ricardo --- doc/guix.texi | 33 ++++++++++++++++++++++++++++++++- guix/import/npm.scm | 16 ++++++++++++---- guix/scripts/import/npm.scm | 10 ++++++++-- 3 files changed, 52 insertions(+), 7 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index 8f694a20e..ccb3c45f1 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -7258,7 +7258,7 @@ guix import npm minimist @end example =20 The importer implements an experimental recursive subsystem, allowing -for a significant of npm packages to be imported. The command below +for a significant number of npm packages to be imported. The command be= low imports metadata for the @code{optimist} npm package, as well as its dependencies. =20 @@ -7266,6 +7266,37 @@ dependencies. guix import npm -r optimist @end example =20 +The importer currently has does not warn when a cyclic dependency has be= en +encountered. It will run until the buffer overruns which could take a lo= ng +time. You should generally see output every couple of seconds, if not a = cyclic +dependency has probably been encountered. These are best avoided by firs= t +packaging an earlier version of one of the packages without the offendin= g +input to avoid the cycle. + +@quotation Note +We have implemented an experimental blacklist functionality, enabled by +default, to help you avoid importing a lot of garbage development +dependencies. You can customize this blacklist by editing +@file{guix/import/npm-blacklist.scm} only when running @command{guix imp= ort +npm} from a git checkout, see @pxref{Running Guix Before It Is Installed= }. + +These packages and every package that begins with these 103 names are +currently blacklisted: (matcha benchmark babel @babel webpack rollup +browserify async ember broccoli nsp uglifyjs2 uglify-js electron statsd = vega +grunt-release lineman lerna openlayers openpgp yarnpkg wekan etherpad-li= te +meteor keybase docco jsdoc markdown eslint lint markdownlint prettier st= andard +jscs @ljharb/eslint-config editorconfig-tools jshint xo tick tsml +tsd-check @commitlint safe grunt lerna vows husky nps rimraf colors cli +source-map chai unexpected karma karma-qunit karma-sauce-launcher +karma-browserify coveralls covert nyc qunit mocha yargs sinon code tape +colortape airtap tap tap-spec lab mock nyc proxyquire coffe-script cross= -swawn +terst stream-spigot make-generator-function forking-tap bogota faucet gr= emlins +jest redux-mock-store webdriverio ava enzyme testem protractor testcafe +test262 storybook jasmine volkswagen nightwatch supertest istanbul sizzy +zombie PhantomCSS) +This expands to thousands of packages. +@end quotation + @item elpa @cindex elpa Import metadata from an Emacs Lisp Package Archive (ELPA) package diff --git a/guix/import/npm.scm b/guix/import/npm.scm index 2efe81fa9..1d46fd384 100644 --- a/guix/import/npm.scm +++ b/guix/import/npm.scm @@ -350,8 +350,12 @@ a git checkout." ,(guix-hash-url temp-dir))))))))))) =20 (define (blacklisted? name) - "Check if the pair name-version is blacklisted. RETURN #t if yes, else= #f." - (member name blacklist)) + "Check if the string name is blacklisted. RETURN #t if yes, else #f." + ;; Split the string to enable ut so blacklist scoped packages like + ;; @babel/core and packages like eslint-popup without having to type i= n + ;; every single combination. + (if (member (car (string-split name (char-set #\- #\/))) blacklist) + #t #f)) =20 (define (sanitize-npm-version version) "Return version without prefixed ^." @@ -518,8 +522,12 @@ npm list of dependencies DEPENDENCIES." (npm-dependencies (append (extract-npm-dependencies dependencies) - ;; TODO blacklist! - (extract-npm-dependencies dev-dependencies))) + ;; Remove the blacklisted devdeps to avoid big dependen= cy + ;; cycles and all the linters, cli-tools, benchmarking, + ;; etc. JS-devs adds to their development environment. + (remove + blacklisted? + (extract-npm-dependencies dev-dependencies)))) (description (assoc-ref package "description")) (home-page (assoc-ref package "homepage")) (license (extract-license curr)) diff --git a/guix/scripts/import/npm.scm b/guix/scripts/import/npm.scm index 1f4ed777c..c540c3574 100644 --- a/guix/scripts/import/npm.scm +++ b/guix/scripts/import/npm.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2015 David Thompson +;;; Copyright =C2=A9 2018 swedebugia ;;; ;;; This file is part of GNU Guix. ;;; @@ -25,6 +26,7 @@ #:use-module (srfi srfi-1) #:use-module (srfi srfi-11) #:use-module (srfi srfi-37) + #:use-module (srfi srfi-41) #:use-module (ice-9 match) #:use-module (ice-9 format) #:export (guix-import-npm)) @@ -41,7 +43,9 @@ Import and convert the npm package for PACKAGE-NAME.\n")) (display (G_ " -h, --help display this help and exit")) - (display (G_ " + (display (G_ " + -r, --recursive import packages recursively")) + (display (G_ " -V, --version display version information and exit")) (newline) (show-bug-report-information)) @@ -88,7 +92,9 @@ `(define-public ,(string->symbol name) ,pkg)) (G_ #f)) - (recursive-import package-name)) + (reverse + (stream->list + (npm-recursive-import package-name)))) ;; Single import (let ((sexp (npm->guix-package package-name))) (unless sexp --=20 2.19.1 --------------3304040D449BA540336B1B76--