Improved NPM importer with blacklist (Was: Re: NPM importer)

[swedebugia <swedebugia@riseup.net>]

[-- Attachment #1: Type: text/plain, Size: 2505 bytes --]

Hi :)

On 2018-11-11 16:37, Julien Lepiller wrote:
> I improved a bit over jlicht's work here, but there still a few tgings we want co work on: https://framagit.org/tyreunom/guix/tree/npm
> 
> There is an importer and a build system as well as a few packages. One of tge issue is that the importer is not recursive, so it doesn't get the specified version, and the packages aren't tested because the tests depend on cylic dependencies (and sometimes very big circles).

There is a stub of a recursive importer, but it does not seem to work.

--

I improved on Julliens work and added a blacklister. After much sweat it 
now works and the blacklist is populated with a lot of unneeded 
development dependencies and complex packages.

I added the version to all imported npm-packages and to the inputs as it 
does not make sense to reference the packages without a version given 
all the cyclic dependencies.

Thanks for all the help!

TODO:
* make npm-recursive-import work by not fetching blacklisted packages

* implement keyword blacklisting based on the descriptions

* match not just the whole string of blacklisted packages:
   e.g. match also "rollup-plugin" when "rollup" is in the blacklist.

* get the tarballs from npm-registry instead as they are never missing
   (githubs sometimes are) and likely reproducible.

* Output a (define-public <guixname> (inherit <guixname>-<version>)) for
   all imported npm-packages.

* Make it possible to specify a specific version to import (and perhaps
   the latest of all minor versions of a package :D).
(For async that would be "0.1.22", "0.2.10", "0.3.0", etc all the way up 
to "2.6.1" which is the current beast. This would mean that we in total 
import about 477.000 packages times the number of minor releases (mean 
~10?) that equals 4,7 mio. npm-packages :p) Then we will definitely need 
to speed up guile. My guess is that we will have to import at least 1,5 
versions for every npm package to mitigate cyclic dependencies (this 
means 477.000*1,5 = 715.500 npm-package-versions).

* Make it easy to analyze a given npm-package to see when deps/devdeps 
were added. In the case async, propose we import 0.9.0 first which is 
the last version without lodash as devdep. From 1.0.0 more devdeps were 
added. (source: https://registry.npmjs.org/async)

Perhaps some kind of tree output for these complex packages with 
versions as branches and dependencies as subbranches would be nice?
--

See the files I changed attached.

-- 
Cheers
Swedebugia

[-- Attachment #2: npm.scm.bz2 --]
[-- Type: application/x-bzip, Size: 5882 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: npm-blacklist.scm --]
[-- Type: text/x-scheme; name="npm-blacklist.scm", Size: 6251 bytes --]

;;; Copyright © 2018 swedebugia <swedebugia@riseup.net>
;;;
;;; This file is part of GNU Guix.
;;;
;;; GNU Guix is free software; you can redistribute it and/or modify it
;;; under the terms of the GNU General Public License as published by
;;; the Free Software Foundation; either version 3 of the License, or (at
;;; your option) any later version.
;;;
;;; GNU Guix is distributed in the hope that it will be useful, but
;;; WITHOUT ANY WARRANTY; without even the implied warranty of
;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;;; GNU General Public License for more details.
;;;
;;; You should have received a copy of the GNU General Public License
;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.

(define-module (guix import npm-blacklist)
  #:export (blacklist
            keyword-blacklist))

;; Many of these are already in debian.

(define blacklist-benchmark
  '(
    "matcha"
    "benchmark"
    ))
  
(define blacklist-complex
  ;; Complex packages we should not begin with
  '(
    ;; From debian wiki /Javascript:
    ;; Packages used to transform code written for nodejs to run in
    ;; the browser:
    "babel"
    "@babel"
    "babel-core"
    "webpack"
    "rollup"
    "rollup-plugin-node-resolve"
    "browserify"			;This can be replaced by one
					;of the above.
    ;; jquery dependant
    "async"
    "ember" ;A JavaScript framework for creating ambitious web applications
    "broccoli"                          ;Browser compilation library
    "nsp"                               ;security related
    ;; Other complex packages not yet in debian:
    "electron"
    "statsd"
    "vega"
    "grunt-release"
    "lineman"
    "lineman-angular"
    "lerna"
    "openlayers"
    "openpgp"
    "yarnpkg"
    "wekan"
    "etherpad-lite"
    "meteor"
    ))

(define blacklist-deprecated
  '(
   ;; From debian wiki:
    "keybase"
    ))

(define blacklist-doc
  '(
    ;; From underscore:
    "docco"
    ;; From mocha:
    "jsdoc"
    "markdown-toc"))

(define blacklist-lint
  '(
    ;; From underscore:
    "eslint"
    ;; From mocha:
    "eslint-config-prettier"
    "eslint-config-semistandard"
    "eslint-config-standard"
    "eslint-plugin-import"
    "eslint-plugin-node"
    "eslint-plugin-prettier"
    "eslint-plugin-promise"
    "eslint-plugin-standard"
    "lint-staged"
    "markdownlint-cli"
    "prettier-eslint-cli"
    ;;From yargs
    "standard"				;linter & automatic code fixer
    "standard-version"	 ;Automate versioning and CHANGELOG generation
    ;;From tap
    "jscs"		 ;JavaScript Code Style checker
    ;; From nsp
    "eslint-config-nodesecurity"
    ;; From
    "@ljharb/eslint-config"
    ;; From eslint-config
    "editorconfig-tools"		;A tool to verify
    ;; From requirejs
    "jshint"
    ;; from path-is-absolute
    "xo"
    ;; from glob
    "tick"
    ;; video.js
    "tsml"
    ;; callsites
    "tsd-check"
    ;; promptly
    "@commitlint/config-conventional"
    "eslint-config-moxy"
    "@commitlint/cli"
    ;; async
    "jshint-stylish"
    ;; fs-extra
    "standard-markdown"
    ;; is
    "safe-publish-latest"
    ))

(define blacklist-grunt
  '(
    ;; From tap:
    "grunt"				;The JavaScript Task Runner - is in
					;debian
    "grunt-contrib-concat"
    "grunt-contrib-uglify"
    ;; From jquery
    "grunt-compare-size"
    
    ))

(define blacklist-management
  '(
    ;; From parser5:
    "lerna"
    ;; CI
    "vows"))

(define blacklist-other
  '(
    ;; From mocha:
    "husky"
    "nps"
    "rimraf"                            ;complex also
    ;;async
    "colors"                            ;cli
    "cli-table"
    
    ))

(define blacklist-test-assertion
  '(
    ;; mocha
   "chai"				;Chai is an assertion library.
   "unexpected"
    ))

(define blacklist-test-browser
  '(
    ;; mocha
    "karma"
    "karma-qunit"
    "karma-sauce-launcher"
    ;; async
    "karma-browserify"
    ))

(define blacklist-test-coverage
  '(
    ;; mocha
    "coveralls"				;is in debian
    "covert"				;code coverage command
    "nyc"
    ))

(define blacklist-test-general
  ;; sinon and tape are the most common
  '(
   ;; From underscore:
   "qunit-cli"
   "qunit"
   ;; From yargs
   "mocha"				;is in debian
   "yargs-test-extends"
   ;; From commander
   "sinon"				;Test spies, stubs and mocks
   ;; From nsp
   "code"				;assertion, fork of chai
   ;; From minimist
   "tape"				;tap-producing test harness
   ;; universalify
   "colortape"
   "airtap"
   "tap"				;Test-Anything-Protocol
					;library
   ;; klaw
   "tap-spec"
   "lab"				;Node.js test framework
   ;; From nsp
   "mock-fs"
   ;; From many :)
   "nyc"				;istanbul coverage, testing
   ;; from is-wsl
   "proxyquire"
   ;; mocha
   "coffe-script"
   "cross-swawn"
   ;; secure-random
   "terst"
   ;; through2
   "stream-spigot"
   ;; is
   "make-generator-function"
   ;; From github search "test" javascript, sort by stars
   "forking-tap"
   "bogota"
   "faucet"
   "gremlins"
   "jest"
   "redux-mock-store"
   "webdriverio"
   "ava"
   "enzyme"
   "testem"
   "protractor"
   "testcafe"
   "test262"
   "storybook"
   "jasmine"
   "volkswagen"
   "nightwatch"
   "supertest"
   "istanbul"
   "sizzy"
   "zombie"
   "PhantomCSS"
   ))

(define blacklist
  ;; Combine blacklists into one.
  (append
   blacklist-benchmark
   blacklist-complex
   blacklist-deprecated
   blacklist-doc
   blacklist-lint
   blacklist-grunt
   blacklist-management
   blacklist-other
   blacklist-test-assertion
   blacklist-test-browser
   blacklist-test-coverage
   blacklist-test-general
   ))

(define keyword-blacklist
  ;; Blacklist based on description or keywords
  '("assert"
    "assertion"
    "check"
    "test"
    "testing"
    "mock"
    "coverage"
    "lint"
    "linter"
    "security")) 

;; Debugging
;; (format (current-error-port) "Blacklisted: ~a \n" blacklist)
;; (force-output)
;; (format (current-error-port) "Total: ~a packages blacklisted \n" (length blacklist))