NPM importer

NPM importer

[swedebugia]

Hi

Has anybody worked on that?

-- 
Cheers
Swedebugia

Re: NPM importer

[Julien Lepiller]

I improved a bit over jlicht's work here, but there still a few tgings we want co work on: https://framagit.org/tyreunom/guix/tree/npm

There is an importer and a build system as well as a few packages. One of tge issue is that the importer is not recursive, so it doesn't get the specified version, and the packages aren't tested because the tests depend on cylic dependencies (and sometimes very big circles).

Le 11 novembre 2018 11:41:34 GMT+01:00, swedebugia <swedebugia@riseup.net> a écrit :
>Hi
>
>Has anybody worked on that?

Re: NPM importer

[swedebugia]

On 2018-11-11 16:37, Julien Lepiller wrote:
> I improved a bit over jlicht's work here, but there still a few tgings we want co work on: https://framagit.org/tyreunom/guix/tree/npm
> 
> There is an importer and a build system as well as a few packages. One of tge issue is that the importer is not recursive, so it doesn't get the specified version, and the packages aren't tested because the tests depend on cylic dependencies (and sometimes very big circles).

Ok.
Is there a reason to have this out-of-tree?
Could you push it to a wip-npm branch in guix?

Some npm packages has no or almost no dependencies.
E.g. https://github.com/substack/minimist/blob/master/package.json

These we could start with if the build-system works and incrementally 
improve the build-system and the complexity of added packages.

Thoughts?
-- 
Cheers
Swedebugia

Re: NPM importer

[Julien Lepiller]

Well, minimist is not a small package. I have a small script somewhere that browse the registry to find recursively the list of dependencies of a package. I tried to run it on minimist and I stopped it after a few thousand different packages were found. That's also why we can't make a recursive importer: it would almost never end.

I think my plan was to build these packages without testing them at all, focusing on packaging test frameworks that will be needed to test other packages.

See this script that builds a graph of dependencies, ignoring devDependencies (so none of the packages listed by this script is going to be tested): https://framagit.org/tyreunom/guix/snippets/2534

Not testing these packages is maybe less problematic than not testing other kinds of packages because we are packaging the same thing as developpers test, in a similar environment.

I'll push what I have to a wip-branch latter today. The reason why I develop out-of-tree is because I can always rebase my patches.

Le 20 novembre 2018 00:29:02 GMT+01:00, swedebugia <swedebugia@riseup.net> a écrit :
>On 2018-11-11 16:37, Julien Lepiller wrote:
>> I improved a bit over jlicht's work here, but there still a few
>tgings we want co work on: https://framagit.org/tyreunom/guix/tree/npm
>> 
>> There is an importer and a build system as well as a few packages.
>One of tge issue is that the importer is not recursive, so it doesn't
>get the specified version, and the packages aren't tested because the
>tests depend on cylic dependencies (and sometimes very big circles).
>
>Ok.
>Is there a reason to have this out-of-tree?
>Could you push it to a wip-npm branch in guix?
>
>Some npm packages has no or almost no dependencies.
>E.g. https://github.com/substack/minimist/blob/master/package.json
>
>These we could start with if the build-system works and incrementally 
>improve the build-system and the complexity of added packages.
>
>Thoughts?

Re: NPM importer

[swedebugia]

Hi :)

On 2018-11-20 08:50, Julien Lepiller wrote:
> Well, minimist is not a small package. I have a small script somewhere that browse the registry to find recursively the list of dependencies of a package. I tried to run it on minimist and I stopped it after a few thousand different packages were found. That's also why we can't make a recursive importer: it would almost never end.
> 
> I think my plan was to build these packages without testing them at all, focusing on packaging test frameworks that will be needed to test other packages.

ok.

> 
> See this script that builds a graph of dependencies, ignoring devDependencies (so none of the packages listed by this script is going to be tested): https://framagit.org/tyreunom/guix/snippets/2534

Unfortunately I did not understand how to use the script :/
I got this after wgetting the snippet:
scheme@(guile-user)> (load "npm-explorer.scm")
digraph dependencies {
overlap=false;
splines=true;
"mocha" -> "supports-color";
"supports-color" -> "has-flag";
"mocha" -> "glob";s
"glob" -> "inherits";
"glob" -> "fs.realpath";
"glob" -> "minimatch";
"minimatch" -> "brace-expansion";
"brace-expansion" -> "concat-map";
"brace-expansion" -> "balanced-match";
"glob" -> "path-is-absolute";
"glob" -> "once";ges
"once" -> "wrappy";s
^Conce [color=red];
"glob" -> "inflight";
"inflight" -> "once";
"inflight" -> "wrappy";
"mocha" -> "mkdirp";
"mkdirp" -> "minimist";
"mocha" -> "commander";
"mocha" -> "he";ages
"mocha" -> "diff";es
"mocha" -> "minimatch";
"mocha" -> "escape-string-regexp";
"mocha" -> "growl";s
"mocha" -> "debug";s
"debug" -> "ms";ages
"mocha" -> "browser-stdout";

}
$1 = #t

Can you explain how to use it?

> 
> Not testing these packages is maybe less problematic than not testing other kinds of packages because we are packaging the same thing as developpers test, in a similar environment.

It sounds like a good start. This mess is not a situation we created, we 
do our best to shoehorn these packages into a reproducible environment 
that the js-devs does not really care about or gave up on long ago.

Even if we succeed to build every major Node web app in a few years it 
is not certain that the js-devs or the users are going to use them. But 
we can hope and enjoy them ourselves :D

> 
> I'll push what I have to a wip-branch latter today. The reason why I develop out-of-tree is because I can always rebase my patches.

That would be nice. I did not see it yet in the list 
http://git.savannah.gnu.org/cgit/guix.git/refs/heads

BTW: Did you see that Jelle did two commits after your fork?
https://github.com/jellelicht/guix/commits/wip-node-build-system

-- 
Cheers
Swedebugia

Re: NPM importer

[swedebugia]

[-- Attachment #1: Type: text/plain, Size: 1247 bytes --]

Hi

On 2018-11-20 20:58, swedebugia wrote:
> On 2018-11-20 08:50, Julien Lepiller wrote:

snip

>> See this script that builds a graph of dependencies, ignoring 
>> devDependencies (so none of the packages listed by this script is 
>> going to be tested): https://framagit.org/tyreunom/guix/snippets/2534

What is devDependencies? Required to build the npm package?

I understood the script now I think. :)

I replaced "mocha" with name in the call in the bottom.

Then did:
$guile
scheme@(guile-user)> (define name "ssb-patchwork")
scheme@(guile-user)> (load "npm-explorer.scm")
...

As you said it runs for(ever) (in my case 5 minutes) listing 
dependencies of dependencies of dependencies of dependencies of 
dependencies of dependencies ...

1100 last lines of the output is attached. :p

One thing is for sure. There are a LOT of npm packages. "Over 477,000 
packages are available on the main npm registry." according to WP. :O

I wonder how many are free software? 90%? 50%?

I hope we can automate this some way.

npm.scm will be a VERY VERY long file. Maybe we should rethink about how 
to best store all these variables...

Just 1 npm application (ssb-patchwork) will add at least 1000 package 
definitions to it.

-- 
Cheers
Swedebugia

[-- Attachment #2: screen-npm-dep-ssb-patchwork.log.bz2 --]
[-- Type: application/x-bzip, Size: 6621 bytes --]

Re: NPM importer

[Julien Lepiller]

Le Tue, 20 Nov 2018 22:12:18 +0100,
swedebugia <swedebugia@riseup.net> a écrit :

> Hi
> 
> On 2018-11-20 20:58, swedebugia wrote:
> > On 2018-11-20 08:50, Julien Lepiller wrote:  
> 
> snip
> 
> >> See this script that builds a graph of dependencies, ignoring 
> >> devDependencies (so none of the packages listed by this script is 
> >> going to be tested):
> >> https://framagit.org/tyreunom/guix/snippets/2534  
> 
> What is devDependencies? Required to build the npm package?

the package.json file declares two kinds (I think I've seen a third
kind, but I'm not sure, nor am I a js developper) of dependencies:
dependencies and devDependencies that map to our inputs and
native-inputs. devDependencies are tools used during the build or
development, but they are not needed at runtime. Sometimes, they are
required at build time (for instance the coffeescript compiler is a
devDependencies of some packages, but we need it to build the package),
usually they are required at test time (for instance mocha, tap, or
tape are test frameworks), otherwise, they can be documentation tools,
and sometimes really not useful for us (we probably don't want to run
linters or release-related tools).

> 
> I understood the script now I think. :)
> 
> I replaced "mocha" with name in the call in the bottom.
> 
> Then did:
> $guile
> scheme@(guile-user)> (define name "ssb-patchwork")
> scheme@(guile-user)> (load "npm-explorer.scm")
> ...

The way I wrote it, you were supposed to change the "mocha" with the
package you're interested in, and run the script with guile like so:

guile npm-explorer.scm > mocha.dot

and then:

fdp -Tpng mocha.dot > mocha.png

(fdp is part of graphviz, but produces nicer graphs than dot in my
opinion)

> 
> As you said it runs for(ever) (in my case 5 minutes) listing 
> dependencies of dependencies of dependencies of dependencies of 
> dependencies of dependencies ...
> 
> 1100 last lines of the output is attached. :p
> 
> One thing is for sure. There are a LOT of npm packages. "Over 477,000 
> packages are available on the main npm registry." according to WP. :O

That's probably the number of packages on which depend any node
packages through its dependencies and devDependencies... I suspect you
can always find a path between any two packages in this graph... in
either direction.

> 
> I wonder how many are free software? 90%? 50%?
> 
> I hope we can automate this some way.

It looks very difficult, but if you have any ideas, please tell us!

> 
> npm.scm will be a VERY VERY long file. Maybe we should rethink about
> how to best store all these variables...

Very long files are frowned upon, because they have very long
compilation times and require a lot of memory. We'll have to find a way
to split the file before it grows too much. Maybe we will do something
similar than python packages? We have python.scm, python-web.scm...

> 
> Just 1 npm application (ssb-patchwork) will add at least 1000 package 
> definitions to it.
> 

Re: NPM importer

[swedebugia]

On 2018-11-20 23:35, Julien Lepiller wrote:
> Le Tue, 20 Nov 2018 22:12:18 +0100,
> swedebugia <swedebugia@riseup.net> a écrit :
> 
>> Hi
>>
>> On 2018-11-20 20:58, swedebugia wrote:
>>> On 2018-11-20 08:50, Julien Lepiller wrote:
>>
>> snip
>>
>>>> See this script that builds a graph of dependencies, ignoring
>>>> devDependencies (so none of the packages listed by this script is
>>>> going to be tested):
>>>> https://framagit.org/tyreunom/guix/snippets/2534
>>
>> What is devDependencies? Required to build the npm package?
> 
> the package.json file declares two kinds (I think I've seen a third
> kind, but I'm not sure, nor am I a js developper) of dependencies:
> dependencies and devDependencies that map to our inputs and
> native-inputs. devDependencies are tools used during the build or
> development, but they are not needed at runtime. Sometimes, they are
> required at build time (for instance the coffeescript compiler is a
> devDependencies of some packages, but we need it to build the package),
> usually they are required at test time (for instance mocha, tap, or
> tape are test frameworks), otherwise, they can be documentation tools,
> and sometimes really not useful for us (we probably don't want to run
> linters or release-related tools).

Ok, that sounds resonable.
Maybe we can blacklist all these linters and release tools during import 
and import everything else as dependencies?

snip

> The way I wrote it, you were supposed to change the "mocha" with the
> package you're interested in, and run the script with guile like so:
> 
> guile npm-explorer.scm > mocha.dot
> 
> and then:
> 
> fdp -Tpng mocha.dot > mocha.png
> 
> (fdp is part of graphviz, but produces nicer graphs than dot in my
> opinion)

Thanks!

snip

>>
>> npm.scm will be a VERY VERY long file. Maybe we should rethink about
>> how to best store all these variables...
> 
> Very long files are frowned upon, because they have very long
> compilation times and require a lot of memory. We'll have to find a way
> to split the file before it grows too much. Maybe we will do something
> similar than python packages? We have python.scm, python-web.scm...

yeah good idea. but with half a miljon packages and maybe max a 1000 
definitons in each we will still have ~500 npm files. By then I suggest 
we put them in a separate dir like gnu/packages/npm/web.scm etc.


-- 
Cheers
Swedebugia

Re: NPM importer

[Mike Gerwitz]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Tue, Nov 20, 2018 at 22:12:18 +0100, swedebugia wrote:
> I wonder how many are free software? 90%? 50%?
>
> I hope we can automate this some way.

The JavaScript community has poor licensing practices, and the culture
is somewhat hostile to the ideals of the free software movement (they
focus on permissive licensing to empower non-free software developers
using those libraries).

The package.json has a license field, but package.json is often
auto-generated and I think is MIT Expat by default.  It is metadata---I
can't imagine it carries any legal weight by itself.  Consequently, we'd
have to fall back on COPYING or LICENSE files (of various sorts) in the
projects.  Even then, a project may contain things under various licenses.

Further, since there tend to be many really small packages, if _any_ one
of those is missing proper license information, then anything that
depends on it will be non-free.  Since npm doesn't ensure that its
packages are actually free, the odds of there being some sort of
licensing issue---just by sheer number---are probably higher than we
would like them to be.  I'm not suggesting malice; it may be
accidental, or maybe someone knows nothing about licensing and simply
never attached a license to begin with (making it non-free by default).[0]

There's also the risk of any of these projects using incompatible
licenses.

Both GitLab and GitHub detect licenses on projects.  I forget the name
of the software they use to do that (and it may not be the same for both
of them), and it's probably not perfect, but something like that may
help with automation.


[0]: https://blog.github.com/2015-03-09-open-source-license-usage-on-github-com/
     (as of 2015)

- -- 
Mike Gerwitz
Free Software Hacker+Activist | GNU Maintainer & Volunteer
GPG: D6E9 B930 028A 6C38 F43B  2388 FEF6 3574 5E6F 6D05
https://mikegerwitz.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCgAGBQJb9Le7AAoJEIyRe39dxRuijsQP/19DjT/G4/806rbtECrZAOeB
DhsWG6fm4rFkv9VbdVgKFASF2qIYSmq8p3yiVHwaNNoV0HcUurK/Gz44880S9EPO
EnHoN7naBYEzv1JRkGFdiHh6eEeUsGPdt4RB1/bunjY4K3EOfeSVQWWeXlDjI4GK
KKsLqNto00ESiwkDW2FOvElnUwfkFa1i+mtc48PbNtoy0VPie9kid51NDQLw4fZG
4VTU+kExeBHriq1AsNREWOQB2Ctg09ydkrzTvzlg4kucnFp8S3ohvbLoSoPil0YH
09hihN/sSwtws2BkMjgB2cO/y46MZeRsvazDD6ZZNyz3PqVZYsPa+5F0eFp7MlpV
a56lsND20IEtmITUZsaIx+W4F71iLJPIVWMN0NZA/rMgEntWSgexSx3w9RFZNYWH
m3UAyACIV9M9WDlTN/OJ/Q98clJ8G3AcZ3M5hKMyvtupctMWxZmDhY3dEGGeavvv
AKIXJ0Qau0mi9HFvT4eusfyih01kCFGddMcUCkY7QY6yuDxg1hHglM75t6YHvfwP
4lZlCywwC6UMazpv0QO6KMOhq7pwEVsXvn9agZR9gd5d18H+6EMv0AogNrh2a48u
HFtRNSMx4woFxXUGd2P50/Zy8hJFyijMbLyksLkxfr7HcTfJ+hUGb1e94xuEdZzr
xR/YjftLIOSzNUV9y20y
=F0Yh
-----END PGP SIGNATURE-----

Re: NPM importer

[Brett Gilio]

Mike Gerwitz writes:

> The JavaScript community has poor licensing practices, and the culture
> is somewhat hostile to the ideals of the free software movement (they
> focus on permissive licensing to empower non-free software developers
> using those libraries).

To say the least. It will take a good deal of implementing a license
checker on the importer, as well as human verification to ensure that we
are maintaining a high ethical standard.

Brett Gilio

Re: NPM importer

[swedebugia]

[-- Attachment #1: Type: text/plain, Size: 2589 bytes --]

On 2018-11-21 23:01, Brett Gilio wrote:
> 
> Mike Gerwitz writes:
> 
>> The JavaScript community has poor licensing practices, and the culture
>> is somewhat hostile to the ideals of the free software movement (they
>> focus on permissive licensing to empower non-free software developers
>> using those libraries).
> 
> To say the least. It will take a good deal of implementing a license
> checker on the importer, as well as human verification to ensure that we
> are maintaining a high ethical standard.

We might want to use the same approach as licensee:

"The solution
Licensee automates the process of reading LICENSE files and compares 
their contents to known licenses using a several strategies (which we 
call "Matchers"). It attempts to determine a project's license in the 
following order:

If the license file has an explicit copyright notice, and nothing more 
(e.g., Copyright (c) 2015 Ben Balter), we'll assume the author intends 
to retain all rights, and thus the project isn't licensed.
If the license is an exact match to a known license. If we strip away 
whitespace and copyright notice, we might get lucky, and direct string 
comparison in Ruby is cheap.
If we still can't match the license, we use a fancy math thing called 
the Sørensen–Dice coefficient, which is really good at calculating the 
similarity between two strings. By calculating the percent changed from 
the known license to the license file, you can tell, e.g., that a given 
license is 95% similar to the MIT license, that 5% likely representing 
legally insignificant changes to the license text."
https://github.com/benbalter/licensee

We could perhaps also semi-automate the generation of emails to authors 
of the offending npm packages with unclear packages.
Say only 1% of 470.000 has unclear license, that equals 4700 emails to 
authors. :)

In a hypothetical scenario with import of 20 npm packages a day it will 
take us 477.000/20 = 23850 days = 65 years to import them all.

In a hypothetical scenario with import of 500 npm packages a day it will 
take us 477.000/500 = 954 days = 2,6 years to import them all.

This is based on the assumption that all are free software, but that is 
probably not the case.

BTW
ssb-patchwork had over 400 dependants in 10+ levels and the dotfile is 
attached. The rendered png is crazy looking. Reminds me of the holy 
spaghetti monster.

A graph of all npm packages and top packages is also available: 
https://exploring-data.com/info/npm-packages-dependencies/
-- 
Cheers
Swedebugia

[-- Attachment #2: ssb-patchwork.dot.bz2 --]
[-- Type: application/x-bzip, Size: 6041 bytes --]

Re: NPM importer

[swedebugia]

Hi

On 2018-11-22 00:22, swedebugia wrote:
snip

> A graph of all npm packages and top packages is also available: 
> https://exploring-data.com/info/npm-packages-dependencies/

While investigating the top libraries* packages with most depends in npm 
I found the following:

Lib		Dep	DevDep	RecdevDep	Dependants	license
underscore	0	12	2400+		18000+		mit
async		1	37	2696		26069		mit

rec=recursive

See also the issue I created here: 
https://github.com/caolan/async/issues/1600 asking which of asyncs 37 
devdeps are needed to build or build+test async.

And similar here: https://github.com/jashkenas/underscore/issues/2790
-- 
Cheers
Swedebugia

* (according to the link above)

Re: NPM importer

[Brett Gilio]

swedebugia writes:

> Hi
>
> On 2018-11-22 00:22, swedebugia wrote:
> snip
>
>> A graph of all npm packages and top packages is also available:
>> https://exploring-data.com/info/npm-packages-dependencies/
>
> While investigating the top libraries* packages with most depends in
> npm I found the following:
>
> Lib		Dep	DevDep	RecdevDep	Dependants	license
> underscore	0	12	2400+		18000+		mit
> async		1	37	2696		26069		mit
>
> rec=recursive
>
> See also the issue I created here:
> https://github.com/caolan/async/issues/1600 asking which of asyncs 37
> devdeps are needed to build or build+test async.
>
> And similar here: https://github.com/jashkenas/underscore/issues/2790


Holy cow.

Thank you for taking the initiative on this, swedebugia. I tend to agree
with your licensee approach, but I think this will require an undoubted
good deal of man power before we are able to sufficiently move forward
on npm imports with a decent velocity.

I know that the largest issue of this as usual will be the JavaScript
trap, but there will have to be some compromises on the part of people
wanting to develop for Node on Guix when their libraries are missing due
to non-free or ambiguous licensing. Not the least of which being the
issues surrounding Electron on the Chromium licenses, which seems to be
where a lot of energy is lately on the Node development side.

I also do not want Guix to detract too much for "more important issues",
not to say this isn't important. Just perhaps it isn't as much of a
priority simply because that community and ours are not so cohesive at
the current moment.

I guess one could argue that we should be wanting to bring them to us,
but I also know how disuasive "a lack of convenience" can be to those
who are not as freedom and ethicality conscious as the rest of us.

Brett Gilio

import libjs-*.deb from Debian? (was Re: NPM importer)

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 2075 bytes --]

Hi Guix!

sorry if I'm going to reinvent some wheels but I've not found previous
discussions nor documentation about what I'm writing

what about reusing work already done from the Debian JavaScript
Maintainers team [1]?

they set up some interesting policies we could adapt to Guix needs and
pusblish somewhere (where?)

I mean pages like:

- https://wiki.debian.org/Javascript/Policy
- https://wiki.debian.org/Javascript/Nodejs

they have an npm2deb importer
https://wiki.debian.org/Javascript/Nodejs/Npm2Deb
... we could import what they import :-)

I see in Apr 2016 a brief discussion on a .deb importer was done

  https://lists.gnu.org/archive/html/guix-devel/2016-04/msg01053.html

what is the updated situation today?

Brett Gilio <brettg@posteo.net> writes:

[...]

> Thank you for taking the initiative on this, swedebugia. I tend to agree
> with your licensee approach, but I think this will require an undoubted
> good deal of man power before we are able to sufficiently move forward
> on npm imports with a decent velocity.

could a .deb importer help us to manage this *very big* workload,
sharing this burden with Debian JavaScript maintainers?

I found this project as a starting point

  https://gitlab.digitalcourage.de/htgoebel/guix-import-debian

[...]

> I guess one could argue that we should be wanting to bring them to us,
> but I also know how disuasive "a lack of convenience" can be to those
> who are not as freedom and ethicality conscious as the rest of us.

should we ignore/deprecate npm packages with licensing issues and work
with the ethically conscious like Debian JavaScript Maintainers?

having a fully free software _and_ reproducible [3] nodejs/javascript
dev environment would benefit a **huge** amount of web
developers... they will be so happy to be guixified! :-)

[...]

ciao
Giovanni



[1] https://wiki.debian.org/Javascript

[2] https://packages.debian.org/stretch/libjs-jquery

[3] and bootstrappable?

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: import libjs-*.deb from Debian? (was Re: NPM importer)

[Ricardo Wurmus]

Hi Giovanni,

> what about reusing work already done from the Debian JavaScript
> Maintainers team [1]?
[…]
> should we ignore/deprecate npm packages with licensing issues and work
> with the ethically conscious like Debian JavaScript Maintainers?

I think this would be a good thing to try.  Work on npm can of course
still continue, but it could be useful to have a couple of easier
JavaScript library packages sooner.

Would you like to adapt one or more of the libjs* Debian packages for
Guix to see if this is a reasonable way forward?

--
Ricardo

Re: NPM importer

[Julien Lepiller]

Thanks! It made me wonder if we could better approximate the set of needed dependencies by looking at package tags. I'll try to improve my script in that spirit, and share it again here.

I think once we have a clearer view of what we want, we should focus on finding the best way forward: what packages do we need first, etc. I think build tools are obviously more important than test tools, which should have a greater priority than the rest of packages.

Le 22 novembre 2018 02:02:52 GMT+01:00, swedebugia <swedebugia@riseup.net> a écrit :
>Hi
>
>On 2018-11-22 00:22, swedebugia wrote:
>snip
>
>> A graph of all npm packages and top packages is also available: 
>> https://exploring-data.com/info/npm-packages-dependencies/
>
>While investigating the top libraries* packages with most depends in
>npm 
>I found the following:
>
>Lib		Dep	DevDep	RecdevDep	Dependants	license
>underscore	0	12	2400+		18000+		mit
>async		1	37	2696		26069		mit
>
>rec=recursive
>
>See also the issue I created here: 
>https://github.com/caolan/async/issues/1600 asking which of asyncs 37 
>devdeps are needed to build or build+test async.
>
>And similar here: https://github.com/jashkenas/underscore/issues/2790

Re: NPM importer

[swedebugia]

On 2018-11-22 09:36, Julien Lepiller wrote:
> Thanks! It made me wonder if we could better approximate the set of needed dependencies by looking at package tags. I'll try to improve my script in that spirit, and share it again here.

Sounds good.

I would like to work on implementing a way to blacklist dependencies 
during guix import npm.

I look forward to see your snippet.

I suggest we work in tree on master going forward and I think we should 
submit both the node-build-system and import script to master as they 
seem to work fine.

Both apply cleanly.

Getting this in before 1.0 is possible as I see it.
We can improve the importer with blacklisting and recursive capacities 
in a wip-branch.

-- 
Cheers
Swedebugia

Re: NPM importer

[swedebugia]

On 2018-11-22 02:02, swedebugia wrote:
> Hi
> 
> On 2018-11-22 00:22, swedebugia wrote:
> snip
> 
>> A graph of all npm packages and top packages is also available: 
>> https://exploring-data.com/info/npm-packages-dependencies/

This graph is in-degree and does not consider versions. It is useful 
only to see the relative popularity of libraries.
It does not reveal the out-degree which is what we are interested in: 
recursive dependencies (given x what does it take to build x? build and 
test x?)
Also it leaves out any information about licensing. It would be nice 
with a graph of which npm packages has which licenses with licenses as 
vertices/nodes and packages as edges.

snip

> 
> See also the issue I created here: 
snip

> And similar here: https://github.com/jashkenas/underscore/issues/2790
Got an answer which is good news for building and bad news for testing:

   "devDependencies": {
     "coveralls": "^2.11.2", // NOT NEEDED: Posts coverage reports to 
converalls.io
     "docco": "*", // NOT NEEDED: Builds docs
     "eslint": "1.10.x", // Needed for linting (not sure if you think of 
linting as part of testing)
     "gzip-size-cli": "^1.0.0", // NOT NEEDED
     "karma": "^0.13.13", // Needed for testing
     "karma-qunit": "~2.0.1", // Needed for testing
     "karma-sauce-launcher": "^1.2.0", // Needed for testing
     "nyc": "^2.1.3", // Needed for testing
     "pretty-bytes-cli": "^1.0.0", // NOT NEEDED
     "qunit-cli": "~0.2.0", // Needed for testing
     "qunit": "^2.6.0", // Needed for testing
     "uglify-js": "3.3.21" // Needing for building
   },

In summary direct devdeps: 1 needed for building, 6 needed for testing.

uglify-js is a minifier used in a lot of projects and the only one 
needed for this specific project.

For packages like this we could probably patch the package to skip the 
minifying step and let the user choose which minifier (if any) to use.

In this particular case the minifier compacts the code from 57KB -> 17.6KB.

The most important thing I think is to be able to start packaging this 
mess so cutting an arm and a leg in the beginning (= skip minification 
and testing is warranted in my view).

In this case the only thing left to do to package underscore is to:
1) name it: e.g. node-underscore (like debian)
2) decide whether to delete the minified upstream or not in the guix 
package.

Hooray!

-- 
Cheers
Swedebugia

Re: NPM importer

[Ricardo Wurmus]

swedebugia <swedebugia@riseup.net> writes:

> uglify-js is a minifier used in a lot of projects and the only one
> needed for this specific project.
>
> For packages like this we could probably patch the package to skip the
> minifying step and let the user choose which minifier (if any) to use.
>
> In this particular case the minifier compacts the code from 57KB -> 17.6KB.

We already have a JS uglifier package (written in Common Lisp), so this
should be doable.

-- 
Ricardo

Packaging async and underscore (Was: Re: NPM importer)

[swedebugia]

[-- Attachment #1: Type: text/plain, Size: 811 bytes --]

On 2018-11-22 02:02, swedebugia wrote:
> Hi
> 
> On 2018-11-22 00:22, swedebugia wrote:
> snip
> 
snip

> While investigating the top libraries* packages with most depends in npm 
> I found the following:
> 
> Lib        Dep    DevDep    RecdevDep    Dependants    license
> underscore    0    12    2400+        18000+        mit
> async        1    37    2696        26069        mit

I succeded in packaging the latest underscore. I discovered a bug in the 
build system and fixed it to be able to build it. (see the diff attached)

Debian packaged a 4 year old async 0.8.0 that has 0 dep and only 
uglify-js as devdep (which we already have packaged). From 0.9.2 lodash 
was made a devdep.

-- 
Cheers
Swedebugia

[-- Attachment #2: diff node-build-system --]
[-- Type: text/plain, Size: 4007 bytes --]

modified   guix/build/node-build-system.scm                                                                                                                   
@@ -1,6 +1,7 @@                                                                                                                                               
 ;;; GNU Guix --- Functional package management for GNU                                                                                                       
 ;;; Copyright © 2015 David Thompson <davet@gnu.org>                                                                                                          
 ;;; Copyright © 2016 Jelle Licht <jlicht@fsfe.org>                                                                                                           
+;;; Copyright © 2018 swedebugia <swedebugia@riseup.net>                                                                                                      
 ;;;                                                                                                                                                          
 ;;; This file is part of GNU Guix.                                                                                                                           
 ;;;                                                                                                                                                          
@@ -108,9 +109,13 @@ is an npm global install."                                                                                                               
                          (#f #f))))                                                                                                                          
     (mkdir-p tgt-dir)                                                                                                                                        
     (copy-recursively "." (string-append tgt-dir "/node_modules/" modulename))                                                                               
-    ;; Remove references to dependencies                                                                                                                     
-    (delete-file-recursively                                                                                                                                 
-      (string-append tgt-dir "/node_modules/" modulename "/node_modules"))                                                                                   
+    ;; Remove references to bundled dependencies                                                                                                             
+    (let ((tgt-node-modules (string-append tgt-dir                                                                                                           
+                                           "/node_modules/" modulename "/node_modules")))                                                                    
+      ;; Check if tgt-node-modules exist and remove them if #t                                                                                               
+      (when (file-is-directory? tgt-node-modules)                                                                                                            
+        (delete-file-recursively                                                                                                                             
+         tgt-node-modules)))                                                                                                                                 
     (when global?                                                                                                                                            
       (cond                                                                                                                                                  
         ((string? bin-conf)

Re: Packaging async and underscore

[Julien Lepiller]

Le 2018-11-30 15:17, swedebugia a écrit :
> On 2018-11-22 02:02, swedebugia wrote:
>> Hi
>> 
>> On 2018-11-22 00:22, swedebugia wrote:
>> snip
>> 
> snip
> 
>> While investigating the top libraries* packages with most depends in 
>> npm I found the following:
>> 
>> Lib        Dep    DevDep    RecdevDep    Dependants    license
>> underscore    0    12    2400+        18000+        mit
>> async        1    37    2696        26069        mit
> 
> I succeded in packaging the latest underscore. I discovered a bug in
> the build system and fixed it to be able to build it. (see the diff
> attached)
> 
> Debian packaged a 4 year old async 0.8.0 that has 0 dep and only
> uglify-js as devdep (which we already have packaged). From 0.9.2
> lodash was made a devdep.

Thanks, I'll merge that with my version of the patches, and send them 
both for review. I think they are already very good, unless you think we 
need more time to make sure the build system is perfect.

Could you explain why this is necessary? In what case does this 
directory doesn't exist? I also wonder if it can be the case that 
node_modules is a file or a symlink that isn't catched by your check...

Re: Packaging async and underscore

[swedebugia]

[-- Attachment #1: Type: text/plain, Size: 840 bytes --]

Hi :)

On 2018-11-30 17:08, Julien Lepiller wrote:
> Le 2018-11-30 15:17, swedebugia a écrit :
snip

> 
> Thanks, I'll merge that with my version of the patches, and send them 
> both for review. I think they are already very good, unless you think we 
> need more time to make sure the build system is perfect.

Sounds good to me :) We can improve it further in-tree.

> Could you explain why this is necessary? In what case does this 
> directory doesn't exist? I also wonder if it can be the case that 
> node_modules is a file or a symlink that isn't catched by your check...

The following patch triggers the error. I build it again today and saw 
that my fix seemed not to work anymore for some reason I do not 
understand :-/

See this error log: https://paste.debian.net/1053871/.

-- 
Cheers
Swedebugia

[-- Attachment #2: 0001-gnu-Add-node-underscore.patch --]
[-- Type: text/x-patch, Size: 2959 bytes --]

From cdfbaa7c882a810756ec34ad3a4f885960c8a970 Mon Sep 17 00:00:00 2001
From: swedebugia <swedebugia@riseup.net>
Date: Fri, 30 Nov 2018 17:30:50 +0100
Subject: [PATCH] gnu: Add node-underscore

* gnu/packages/node.scm(node-underscore): New variable
---
 gnu/packages/node.scm | 47 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)

diff --git a/gnu/packages/node.scm b/gnu/packages/node.scm
index 70bbb1779..580071fb1 100644
--- a/gnu/packages/node.scm
+++ b/gnu/packages/node.scm
@@ -22,6 +22,10 @@
 ;;; You should have received a copy of the GNU General Public License
 ;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
 
+;; The node-build-system allows two ways to install node:
+;; Locally and globally. Default is "local".
+;; See https://nodejs.org/en/blog/npm/npm-1-0-global-vs-local-installation/
+
 (define-module (gnu packages node)
   #:use-module ((guix licenses) #:prefix license:)
   #:use-module (guix packages)
@@ -38,6 +42,7 @@
   #:use-module (gnu packages icu4c)
   #:use-module (gnu packages libevent)
   #:use-module (gnu packages linux)
+  #:use-module (gnu packages lisp)
   #:use-module (gnu packages perl)
   #:use-module (gnu packages pkg-config)
   #:use-module (gnu packages python)
@@ -547,3 +552,45 @@ commonjs, browser, AMD, electron, etc.")
   (home-page
     "https://github.com/kumavis/browser-stdout")
   (license license:expat)))
+
+(define-public node-underscore
+  (package
+   (name "node-underscore")
+   (version "1.9.1")
+   (source
+    (origin
+      (method git-fetch)
+      (uri (git-reference
+             (url "https://github.com/jashkenas/underscore")
+             (commit "ae037f7c41323807ae6f1533c45512e6d31a1574")))
+      (file-name (string-append name "-" version))
+      (sha256
+       (base32
+        "1f75wrln5kv5ihkbb9zwhyjqd9imwil801abhv36w09dkkabpjy5"))))
+   (build-system node-build-system)
+   (native-inputs `(("uglify-js" ,uglify-js)))
+   (arguments
+    `(#:tests? #f 
+      ;; FIXME: These test dependencies are missing:
+      ;; "karma": "^0.13.13",
+      ;; "karma-qunit": "~2.0.1",
+      ;; "karma-sauce-launcher": "^1.2.0",
+      ;; "nyc": "^2.1.3",
+      ;; "qunit-cli": "~0.2.0",
+      ;; "qunit": "^2.6.0",
+      #:phases                          ;No configure script
+      (modify-phases %standard-phases
+        (delete 'configure)
+        (add-before 'build 'patch-files
+          (lambda* (#:key inputs #:allow-other-keys)
+            ;; Fix wrong path to uglify-js
+            (substitute* "package.json"
+              (("uglifyjs") "uglify-js"))))))) 
+   (synopsis
+    "JavaScript's functional programming helper library.")
+   (description
+    "Underscore.js is a utility-belt library for JavaScript that provides
+support for the usual functional suspects (each, map, reduce, filter...)
+providing over 100 functions.")
+   (home-page "http://underscorejs.org")
+   (license license:expat)))
-- 
2.19.1

Re: NPM importer

[swedebugia]

Hi

On 2018-11-20 08:50, Julien Lepiller wrote:
snip

> I'll push what I have to a wip-branch latter today. The reason why I develop out-of-tree is because I can always rebase my patches.

I don't see it in guix yet. Could you push it?

I corrected a bug in node-build-system and added node-underscore.

See mangled changes below (mangled by hard-wrap). I will create commit 
and create 2 patches when I have somewhere to share it:

modified   gnu/packages/node.scm 
 

@@ -22,6 +22,10 @@ 
 

  ;;; You should have received a copy of the GNU General Public License 
 

  ;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>. 
 

 
 

+;; The node-build-system allows two ways to install node: 
 

+;; Locally and globally. Default is "local". 
 

+;; See 
https://nodejs.org/en/blog/npm/npm-1-0-global-vs-local-installation/ 
 

+ 
 

  (define-module (gnu packages node) 
 

    #:use-module ((guix licenses) #:prefix license:) 
 

    #:use-module (guix packages) 
 

@@ -38,6 +42,7 @@ 
 

    #:use-module (gnu packages icu4c) 
 

    #:use-module (gnu packages libevent) 
 

    #:use-module (gnu packages linux) 
 

+  #:use-module (gnu packages lisp) 
 

    #:use-module (gnu packages perl) 
 

    #:use-module (gnu packages pkg-config) 
 

    #:use-module (gnu packages python) 
 

@@ -547,3 +552,45 @@ commonjs, browser, AMD, electron, etc.") 
 

    (home-page 
 

      "https://github.com/kumavis/browser-stdout") 
 

    (license license:expat))) 
 

+ 
 

+(define-public node-underscore 
 

+  (package 
 

+   (name "node-underscore") 
 

+   (version "1.9.1") 
 

+   (source 
 

+    (origin 
 

+      (method git-fetch) 
 

+      (uri (git-reference 
 

+             (url "https://github.com/jashkenas/underscore") 
 

+             (commit "ae037f7c41323807ae6f1533c45512e6d31a1574"))) 
 

+      (file-name (string-append name "-" version)) 
 

+      (sha256 
 

+       (base32 
 

+        "1f75wrln5kv5ihkbb9zwhyjqd9imwil801abhv36w09dkkabpjy5"))))
+   (build-system node-build-system) 
 

+   (native-inputs `(("uglify-js" ,uglify-js))) 
 

+   (arguments 
 

+    `(#:tests? #f 
 

+      ;; FIXME: These test dependencies are missing: 
 

+      ;; "karma": "^0.13.13", 
 

+      ;; "karma-qunit": "~2.0.1", 
 

+      ;; "karma-sauce-launcher": "^1.2.0", 
 

+      ;; "nyc": "^2.1.3", 
 

+      ;; "qunit-cli": "~0.2.0", 
 

+      ;; "qunit": "^2.6.0", 
 

+      #:phases                          ;No configure script 
 

+      (modify-phases %standard-phases 
 

+        (delete 'configure) 
 

+        (add-before 'build 'patch-files 
 

+          (lambda* (#:key inputs #:allow-other-keys) 
 

+            ;; Fix wrong path to uglify-js 
 

+            (substitute* "package.json" 
 

+              (("uglifyjs") "uglify-js"))))))) 
 

+   (synopsis 
 

+    "JavaScript's functional programming helper library.") 
 

+   (description 
 

+    "Underscore.js is a utility-belt library for JavaScript that 
provides 

+support for the usual functional suspects (each, map, reduce, 
filter...) 

+providing over 100 functions.") 
 

+   (home-page "http://underscorejs.org") 
 

+   (license license:expat))) 
 

modified   guix/build/node-build-system.scm 
 

@@ -1,6 +1,7 @@ 
 

  ;;; GNU Guix --- Functional package management for GNU 
 

  ;;; Copyright © 2015 David Thompson <davet@gnu.org> 
 

  ;;; Copyright © 2016 Jelle Licht <jlicht@fsfe.org> 
 

+;;; Copyright © 2018 swedebugia <swedebugia@riseup.net> 
 

  ;;; 
 

  ;;; This file is part of GNU Guix. 
 

  ;;; 
 

@@ -109,8 +110,12 @@ is an npm global install." 
 

      (mkdir-p tgt-dir) 
 

      (copy-recursively "." (string-append tgt-dir "/node_modules/" 
modulename)) 

      ;; Remove references to dependencies 
 

-    (delete-file-recursively 
 

-      (string-append tgt-dir "/node_modules/" modulename 
"/node_modules")) 

+    ;; FIXME: clarify why this is done. 
 

+    (let ((tgt-node-modules (string-append tgt-dir "/node_modules/" 
modulename "/node_modules"))) 

+      ;; Check if tgt-node-modules exist and remove them if #t 
 

+      (when (file-is-directory? tgt-node-modules) 
 

+        (delete-file-recursively 
 

+         tgt-node-modules))) 
 

      (when global? 
 

        (cond 
 

          ((string? bin-conf)

-- 
Cheers Swedebugia

Improved NPM importer with blacklist (Was: Re: NPM importer)

[swedebugia]

[-- Attachment #1: Type: text/plain, Size: 2505 bytes --]

Hi :)

On 2018-11-11 16:37, Julien Lepiller wrote:
> I improved a bit over jlicht's work here, but there still a few tgings we want co work on: https://framagit.org/tyreunom/guix/tree/npm
> 
> There is an importer and a build system as well as a few packages. One of tge issue is that the importer is not recursive, so it doesn't get the specified version, and the packages aren't tested because the tests depend on cylic dependencies (and sometimes very big circles).

There is a stub of a recursive importer, but it does not seem to work.

--

I improved on Julliens work and added a blacklister. After much sweat it 
now works and the blacklist is populated with a lot of unneeded 
development dependencies and complex packages.

I added the version to all imported npm-packages and to the inputs as it 
does not make sense to reference the packages without a version given 
all the cyclic dependencies.

Thanks for all the help!

TODO:
* make npm-recursive-import work by not fetching blacklisted packages

* implement keyword blacklisting based on the descriptions

* match not just the whole string of blacklisted packages:
   e.g. match also "rollup-plugin" when "rollup" is in the blacklist.

* get the tarballs from npm-registry instead as they are never missing
   (githubs sometimes are) and likely reproducible.

* Output a (define-public <guixname> (inherit <guixname>-<version>)) for
   all imported npm-packages.

* Make it possible to specify a specific version to import (and perhaps
   the latest of all minor versions of a package :D).
(For async that would be "0.1.22", "0.2.10", "0.3.0", etc all the way up 
to "2.6.1" which is the current beast. This would mean that we in total 
import about 477.000 packages times the number of minor releases (mean 
~10?) that equals 4,7 mio. npm-packages :p) Then we will definitely need 
to speed up guile. My guess is that we will have to import at least 1,5 
versions for every npm package to mitigate cyclic dependencies (this 
means 477.000*1,5 = 715.500 npm-package-versions).

* Make it easy to analyze a given npm-package to see when deps/devdeps 
were added. In the case async, propose we import 0.9.0 first which is 
the last version without lodash as devdep. From 1.0.0 more devdeps were 
added. (source: https://registry.npmjs.org/async)

Perhaps some kind of tree output for these complex packages with 
versions as branches and dependencies as subbranches would be nice?
--

See the files I changed attached.

-- 
Cheers
Swedebugia

[-- Attachment #2: npm.scm.bz2 --]
[-- Type: application/x-bzip, Size: 5882 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: npm-blacklist.scm --]
[-- Type: text/x-scheme; name="npm-blacklist.scm", Size: 6251 bytes --]

;;; Copyright © 2018 swedebugia <swedebugia@riseup.net>
;;;
;;; This file is part of GNU Guix.
;;;
;;; GNU Guix is free software; you can redistribute it and/or modify it
;;; under the terms of the GNU General Public License as published by
;;; the Free Software Foundation; either version 3 of the License, or (at
;;; your option) any later version.
;;;
;;; GNU Guix is distributed in the hope that it will be useful, but
;;; WITHOUT ANY WARRANTY; without even the implied warranty of
;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;;; GNU General Public License for more details.
;;;
;;; You should have received a copy of the GNU General Public License
;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.

(define-module (guix import npm-blacklist)
  #:export (blacklist
            keyword-blacklist))

;; Many of these are already in debian.

(define blacklist-benchmark
  '(
    "matcha"
    "benchmark"
    ))
  
(define blacklist-complex
  ;; Complex packages we should not begin with
  '(
    ;; From debian wiki /Javascript:
    ;; Packages used to transform code written for nodejs to run in
    ;; the browser:
    "babel"
    "@babel"
    "babel-core"
    "webpack"
    "rollup"
    "rollup-plugin-node-resolve"
    "browserify"			;This can be replaced by one
					;of the above.
    ;; jquery dependant
    "async"
    "ember" ;A JavaScript framework for creating ambitious web applications
    "broccoli"                          ;Browser compilation library
    "nsp"                               ;security related
    ;; Other complex packages not yet in debian:
    "electron"
    "statsd"
    "vega"
    "grunt-release"
    "lineman"
    "lineman-angular"
    "lerna"
    "openlayers"
    "openpgp"
    "yarnpkg"
    "wekan"
    "etherpad-lite"
    "meteor"
    ))

(define blacklist-deprecated
  '(
   ;; From debian wiki:
    "keybase"
    ))

(define blacklist-doc
  '(
    ;; From underscore:
    "docco"
    ;; From mocha:
    "jsdoc"
    "markdown-toc"))

(define blacklist-lint
  '(
    ;; From underscore:
    "eslint"
    ;; From mocha:
    "eslint-config-prettier"
    "eslint-config-semistandard"
    "eslint-config-standard"
    "eslint-plugin-import"
    "eslint-plugin-node"
    "eslint-plugin-prettier"
    "eslint-plugin-promise"
    "eslint-plugin-standard"
    "lint-staged"
    "markdownlint-cli"
    "prettier-eslint-cli"
    ;;From yargs
    "standard"				;linter & automatic code fixer
    "standard-version"	 ;Automate versioning and CHANGELOG generation
    ;;From tap
    "jscs"		 ;JavaScript Code Style checker
    ;; From nsp
    "eslint-config-nodesecurity"
    ;; From
    "@ljharb/eslint-config"
    ;; From eslint-config
    "editorconfig-tools"		;A tool to verify
    ;; From requirejs
    "jshint"
    ;; from path-is-absolute
    "xo"
    ;; from glob
    "tick"
    ;; video.js
    "tsml"
    ;; callsites
    "tsd-check"
    ;; promptly
    "@commitlint/config-conventional"
    "eslint-config-moxy"
    "@commitlint/cli"
    ;; async
    "jshint-stylish"
    ;; fs-extra
    "standard-markdown"
    ;; is
    "safe-publish-latest"
    ))

(define blacklist-grunt
  '(
    ;; From tap:
    "grunt"				;The JavaScript Task Runner - is in
					;debian
    "grunt-contrib-concat"
    "grunt-contrib-uglify"
    ;; From jquery
    "grunt-compare-size"
    
    ))

(define blacklist-management
  '(
    ;; From parser5:
    "lerna"
    ;; CI
    "vows"))

(define blacklist-other
  '(
    ;; From mocha:
    "husky"
    "nps"
    "rimraf"                            ;complex also
    ;;async
    "colors"                            ;cli
    "cli-table"
    
    ))

(define blacklist-test-assertion
  '(
    ;; mocha
   "chai"				;Chai is an assertion library.
   "unexpected"
    ))

(define blacklist-test-browser
  '(
    ;; mocha
    "karma"
    "karma-qunit"
    "karma-sauce-launcher"
    ;; async
    "karma-browserify"
    ))

(define blacklist-test-coverage
  '(
    ;; mocha
    "coveralls"				;is in debian
    "covert"				;code coverage command
    "nyc"
    ))

(define blacklist-test-general
  ;; sinon and tape are the most common
  '(
   ;; From underscore:
   "qunit-cli"
   "qunit"
   ;; From yargs
   "mocha"				;is in debian
   "yargs-test-extends"
   ;; From commander
   "sinon"				;Test spies, stubs and mocks
   ;; From nsp
   "code"				;assertion, fork of chai
   ;; From minimist
   "tape"				;tap-producing test harness
   ;; universalify
   "colortape"
   "airtap"
   "tap"				;Test-Anything-Protocol
					;library
   ;; klaw
   "tap-spec"
   "lab"				;Node.js test framework
   ;; From nsp
   "mock-fs"
   ;; From many :)
   "nyc"				;istanbul coverage, testing
   ;; from is-wsl
   "proxyquire"
   ;; mocha
   "coffe-script"
   "cross-swawn"
   ;; secure-random
   "terst"
   ;; through2
   "stream-spigot"
   ;; is
   "make-generator-function"
   ;; From github search "test" javascript, sort by stars
   "forking-tap"
   "bogota"
   "faucet"
   "gremlins"
   "jest"
   "redux-mock-store"
   "webdriverio"
   "ava"
   "enzyme"
   "testem"
   "protractor"
   "testcafe"
   "test262"
   "storybook"
   "jasmine"
   "volkswagen"
   "nightwatch"
   "supertest"
   "istanbul"
   "sizzy"
   "zombie"
   "PhantomCSS"
   ))

(define blacklist
  ;; Combine blacklists into one.
  (append
   blacklist-benchmark
   blacklist-complex
   blacklist-deprecated
   blacklist-doc
   blacklist-lint
   blacklist-grunt
   blacklist-management
   blacklist-other
   blacklist-test-assertion
   blacklist-test-browser
   blacklist-test-coverage
   blacklist-test-general
   ))

(define keyword-blacklist
  ;; Blacklist based on description or keywords
  '("assert"
    "assertion"
    "check"
    "test"
    "testing"
    "mock"
    "coverage"
    "lint"
    "linter"
    "security")) 

;; Debugging
;; (format (current-error-port) "Blacklisted: ~a \n" blacklist)
;; (force-output)
;; (format (current-error-port) "Total: ~a packages blacklisted \n" (length blacklist))

Re: Improved NPM importer with blacklist

[Julien Lepiller]

Le 2018-11-30 17:13, swedebugia a écrit :
> Hi :)
> 
> On 2018-11-11 16:37, Julien Lepiller wrote:
>> I improved a bit over jlicht's work here, but there still a few tgings 
>> we want co work on: https://framagit.org/tyreunom/guix/tree/npm
>> 
>> There is an importer and a build system as well as a few packages. One 
>> of tge issue is that the importer is not recursive, so it doesn't get 
>> the specified version, and the packages aren't tested because the 
>> tests depend on cylic dependencies (and sometimes very big circles).
> 
> There is a stub of a recursive importer, but it does not seem to work.
> 
> --
> 
> I improved on Julliens work and added a blacklister. After much sweat
> it now works and the blacklist is populated with a lot of unneeded
> development dependencies and complex packages.
> 
> I added the version to all imported npm-packages and to the inputs as
> it does not make sense to reference the packages without a version
> given all the cyclic dependencies.
> 
> Thanks for all the help!

Hi,

I never used the recursive importer, so I didn't know it wasn't very 
good.

I wonder if we really need to import every version of the packages. That 
doesn't seem very practical. There are a few cyclic dependencies issues 
in Java packages too, and they are dealt with in a case-by-case basis. 
Most often, we made a degraded version of one of the packages, the 
second can use to build itself, then we rebuild the first with the 
second package.

Sometimes, we also have to adapt some of our packages for the newer 
versions of the dependencies we have. If we didn't, we'd have a lot of 
versions of every package, and most of them would be outdated, probably 
buggy or contain security holes. I'd prefer using the latest versions of 
dependencies, and contribute patches back to upstream, so they can use 
the latest and greatest too :)

That's obviously a lot more work, but that's also probably a saner way 
of doing things.

> 
> TODO:
> * make npm-recursive-import work by not fetching blacklisted packages

Let's be careful though: we don't want to fetch blacklisted packages 
when they are devDependencies, but we still want them if they are 
runtime dependencies.

> 
> * implement keyword blacklisting based on the descriptions

We can probably use tags instead of the description : '("test" "testing" 
"check" "doc" "coverage" "unit") seem like a good approximation of what 
we want to blacklist.

> 
> * match not just the whole string of blacklisted packages:
>   e.g. match also "rollup-plugin" when "rollup" is in the blacklist.
> 
> * get the tarballs from npm-registry instead as they are never missing
>   (githubs sometimes are) and likely reproducible.

Are they actual source tarballs, or are they somewhat different than the 
source used to build the "binary" npm package? With maven (for java) for 
instance, some sources are hosted, but they aren't supposed to be used 
to build the package, they're only here for the debugger.

> 
> * Output a (define-public <guixname> (inherit <guixname>-<version>)) 
> for
>   all imported npm-packages.

I don't think that's a good idea: if we have multiple versions of a 
package, we'll have multiple <guixname> packages...

> 
> * Make it possible to specify a specific version to import (and perhaps
>   the latest of all minor versions of a package :D).
> (For async that would be "0.1.22", "0.2.10", "0.3.0", etc all the way
> up to "2.6.1" which is the current beast. This would mean that we in
> total import about 477.000 packages times the number of minor releases
> (mean ~10?) that equals 4,7 mio. npm-packages :p) Then we will
> definitely need to speed up guile. My guess is that we will have to
> import at least 1,5 versions for every npm package to mitigate cyclic
> dependencies (this means 477.000*1,5 = 715.500 npm-package-versions).

Again, I'm more in favor of patching them, rather than importing more 
versions. Do we still have as many cyclic deps with the blacklist?

> 
> * Make it easy to analyze a given npm-package to see when deps/devdeps
> were added. In the case async, propose we import 0.9.0 first which is
> the last version without lodash as devdep. From 1.0.0 more devdeps
> were added. (source: https://registry.npmjs.org/async)
> 
> Perhaps some kind of tree output for these complex packages with
> versions as branches and dependencies as subbranches would be nice?
> --
> 
> See the files I changed attached.

Thanks for your work!

Re: Improved NPM importer with blacklist

[swedebugia]

[-- Attachment #1: Type: text/plain, Size: 4946 bytes --]

On 2018-11-30 17:24, Julien Lepiller wrote:
> Le 2018-11-30 17:13, swedebugia a écrit :

snip

> Hi,
> 
> I never used the recursive importer, so I didn't know it wasn't very good.
> 
> I wonder if we really need to import every version of the packages. That 
> doesn't seem very practical. There are a few cyclic dependencies issues 
> in Java packages too, and they are dealt with in a case-by-case basis. 
> Most often, we made a degraded version of one of the packages, the 
> second can use to build itself, then we rebuild the first with the 
> second package.

Sounds good.

> Sometimes, we also have to adapt some of our packages for the newer 
> versions of the dependencies we have. If we didn't, we'd have a lot of 
> versions of every package, and most of them would be outdated, probably 
> buggy or contain security holes. I'd prefer using the latest versions of 
> dependencies, and contribute patches back to upstream, so they can use 
> the latest and greatest too :)
> 
> That's obviously a lot more work, but that's also probably a saner way 
> of doing things.

Agreed, this seems better. With a good tree browser we can probably 
avoid importing more than 2-5 versions of the worst packages.

I collected a few cyclic devdeps. See attached. (these definitions is of 
little value as the versions of deps and devdeps are discarded)

>> TODO:
>> * make npm-recursive-import work by not fetching blacklisted packages
> 
> Let's be careful though: we don't want to fetch blacklisted packages 
> when they are devDependencies, but we still want them if they are 
> runtime dependencies.

Totally agree. This is exactly why I only implemented blacklisting of 
native-inputs.

>> * implement keyword blacklisting based on the descriptions
> 
> We can probably use tags instead of the description : '("test" "testing" 
> "check" "doc" "coverage" "unit") seem like a good approximation of what 
> we want to blacklist.

Fewer that half the npm packages have tags to my knowledge. We can do 
both though :D

> 
>>
>> * match not just the whole string of blacklisted packages:
>>   e.g. match also "rollup-plugin" when "rollup" is in the blacklist.
>>
>> * get the tarballs from npm-registry instead as they are never missing
>>   (githubs sometimes are) and likely reproducible.
> 
> Are they actual source tarballs, or are they somewhat different than the 
> source used to build the "binary" npm package? With maven (for java) for 
> instance, some sources are hosted, but they aren't supposed to be used 
> to build the package, they're only here for the debugger.

Fortunately it seems it is the full source. :D
See
https://registry.npmjs.org/underscore/-/underscore-1.9.1.tgz
https://registry.npmjs.org/nodeunit/-/nodeunit-0.5.5.tgz
https://registry.npmjs.org/async/-/async-0.9.0.tgz

> 
>>
>> * Output a (define-public <guixname> (inherit <guixname>-<version>)) for
>>   all imported npm-packages.
> 
> I don't think that's a good idea: if we have multiple versions of a 
> package, we'll have multiple <guixname> packages...

Ok, got it. I thought the define-publics would collide, but I guess not.

> 
>>
>> * Make it possible to specify a specific version to import (and perhaps
>>   the latest of all minor versions of a package :D).
>> (For async that would be "0.1.22", "0.2.10", "0.3.0", etc all the way
>> up to "2.6.1" which is the current beast. This would mean that we in
>> total import about 477.000 packages times the number of minor releases
>> (mean ~10?) that equals 4,7 mio. npm-packages :p) Then we will
>> definitely need to speed up guile. My guess is that we will have to
>> import at least 1,5 versions for every npm package to mitigate cyclic
>> dependencies (this means 477.000*1,5 = 715.500 npm-package-versions).
> 
> Again, I'm more in favor of patching them, rather than importing more 
> versions. Do we still have as many cyclic deps with the blacklist?

No, the blacklist makes a BIG difference (but only to the cycdevdeps. 
The deps still introduce just as many cycles. These can be avoided by 
carefully choosing a version just before the cycdep was added :) (or by 
patching but I know nothing about JS so I leave that to others)

>> * Make it easy to analyze a given npm-package to see when deps/devdeps
>> were added. In the case async, propose we import 0.9.0 first which is
>> the last version without lodash as devdep. From 1.0.0 more devdeps
>> were added. (source: https://registry.npmjs.org/async)
>>
>> Perhaps some kind of tree output for these complex packages with
>> versions as branches and dependencies as subbranches would be nice?

I will try parsing the registry to output something intelligently to 
help the user choose which version to import.

> Thanks for your work!

Thanks for sharing so we can improve this together :)

-- 
Cheers
Swedebugia

[-- Attachment #2: node-cyclic.scm --]
[-- Type: text/x-scheme, Size: 5635 bytes --]

(define-public node-rimraf
  (package
   (name "node-rimraf")
   (version "2.6.2")
   (source
    (origin
      (method url-fetch)
      (uri "https://github.com/isaacs/rimraf/archive/v2.6.2/rimraf-v2.6.2.tar.gz")
      (sha256
       (base32
        "0bmssxz3s30nhq5f8ldssf6s8ga5w0aarn71wjsmvqb1j15b2r6d"))))
   (build-system node-build-system)
   (inputs `(("node-glob" ,node-glob)))
   (native-inputs
    `(
      ;; tests("node-tap" ,node-tap)
      ("node-mkdirp" ,node-mkdirp)))
   (synopsis
    "A deep deletion module for node (like `rm -rf`)")
   (description
    "A deep deletion module for node (like `rm -rf`)")
   (home-page
    "https://github.com/isaacs/rimraf#readme")
   (license license:isc)))


(define-public node-glob
  (package
   (name "node-glob")
   (version "7.1.3")
   (source
    (origin
      (method url-fetch)
      (uri "https://github.com/isaacs/node-glob/archive/v7.1.3/node-glob-v7.1.3.tar.gz")
      (sha256
       (base32
        "0qcymwljbm947gvfn7g7871dnwv5s0jq0r8c8ih9xgrfcynfw3hx"))))
   (build-system node-build-system)
   (inputs
    `(("node-inflight" ,node-inflight)
      ("node-once" ,node-once)
      ("node-path-is-absolute" ,node-path-is-absolute)
      ("node-minimatch" ,node-minimatch)
      ("node-fs.realpath" ,node-fs.realpath)
      ("node-inherits" ,node-inherits)))
   (native-inputs
    `(
      ;; benchm ("node-tick" ,node-tick) 
      ;; tests ("node-tap" ,node-tap)
      ("node-rimraf" ,node-rimraf)
      ("node-mkdirp" ,node-mkdirp)))
   (synopsis "a little globber")
   (description "a little globber")
   (home-page
    "https://github.com/isaacs/node-glob#readme")
   (license license:isc)))

(define-public node-jasmine-core
  (package
   (name "node-jasmine-core")
   (version "3.3.0")
   (source
    (origin
      (method url-fetch)
      (uri "https://github.com/jasmine/jasmine/archive/v3.3.0/jasmine-v3.3.0.tar.gz")
      (sha256
       (base32
        "1rg4p487hf8mlxcj99wywzwp7jp3s4d114n4j12r3mkh8qyi8nck"))))
   (build-system node-build-system)
   (inputs `())
   (native-inputs
    `(
      ;; ("node-grunt" ,node-grunt)
      ;; ("node-grunt-contrib-compass"
      ;;  ,node-grunt-contrib-compass)
      ("node-jsdom" ,node-jsdom)
      ("node-shelljs" ,node-shelljs)
      ("node-jasmine" ,node-jasmine)
      ;; ("node-load-grunt-tasks" ,node-load-grunt-tasks)
      ;; ("node-grunt-contrib-compress"
      ;;  ,node-grunt-contrib-compress)
      ;; ("node-grunt-contrib-concat"
      ;;  ,node-grunt-contrib-concat)
      ;; ("node-grunt-cli" ,node-grunt-cli)
      ("node-temp" ,node-temp)
      ("node-glob" ,node-glob)
      ;; 
      ;; ("node-grunt-contrib-jshint"
      ;;  ,node-grunt-contrib-jshint)
      ))
   (synopsis
    "Official packaging of Jasmine's core files for use by Node.js projects.")
   (description
    "Official packaging of Jasmine's core files for use by Node.js projects.")
   (home-page "https://jasmine.github.io")
   (license license:expat)))


(define-public node-jasmine
  (package
   (name "node-jasmine")
   (version "3.3.0")
   (source
    (origin
      (method url-fetch)
      (uri "https://github.com/jasmine/jasmine-npm/archive/v3.3.0/jasmine-npm-v3.3.0.tar.gz")
      (sha256
       (base32
        "1b6mgxmxv71bpr4fg75azfyh1v0m469prb7srg990fkf7i5bszw9"))))
   (build-system node-build-system)
   (inputs
    `(("node-jasmine-core" ,node-jasmine-core)
      ("node-glob" ,node-glob)))
   (native-inputs
    `(("node-grunt" ,node-grunt)
      ("node-shelljs" ,node-shelljs)
      ("node-grunt-cli" ,node-grunt-cli)
      ("node-grunt-contrib-jshint"
       ,node-grunt-contrib-jshint)))
   (synopsis "Command line jasmine")
   (description "Command line jasmine")
   (home-page "http://jasmine.github.io/")
   (license license:expat)))

(define-public node-domhandler
  (package
   (name "node-domhandler")
   (version "2.4.2")
   (source
    (origin
      (method url-fetch)
      (uri "https://github.com/fb55/DomHandler/archive/v2.4.2/DomHandler-v2.4.2.tar.gz")
      (sha256
       (base32
        "16hi0vapmavw9g9s321b4c9nvwfg06cclj7pjnvjzk0imnzxjngp"))))
   (build-system node-build-system)
   (inputs
    `(("node-domelementtype" ,node-domelementtype)))
   (native-inputs
    `(("node-htmlparser2" ,node-htmlparser2)
      ("node-jshint" ,node-jshint)
      ("node-mocha" ,node-mocha)))
   (synopsis
    "handler for htmlparser2 that turns pages into a dom")
   (description
    "handler for htmlparser2 that turns pages into a dom")
   (home-page
    "https://github.com/fb55/DomHandler#readme")
   (license #f)))

(define-public node-htmlparser2
  (package
   (name "node-htmlparser2")
   (version "3.10.0")
   (source
    (origin
      (method url-fetch)
      (uri "https://github.com/fb55/htmlparser2/archive/v3.10.0/htmlparser2-v3.10.0.tar.gz")
      (sha256
       (base32
        "1qvsv4aixmgnh4h7q726wapg7qnk7srw4z9nmy71jc5r2krimnvn"))))
   (build-system node-build-system)
   (inputs
    `(("node-readable-stream" ,node-readable-stream)
      ("node-domhandler" ,node-domhandler)
      ("node-domelementtype" ,node-domelementtype)
      ("node-inherits" ,node-inherits)
      ("node-domutils" ,node-domutils)
      ("node-entities" ,node-entities)))
   (native-inputs
    `(("node-eslint" ,node-eslint)
      ("node-coveralls" ,node-coveralls)
      ("node-istanbul" ,node-istanbul)
      ("node-mocha-lcov-reporter"
       ,node-mocha-lcov-reporter)
      ("node-mocha" ,node-mocha)))
   (synopsis "Fast & forgiving HTML/XML/RSS parser")
   (description
    "Fast & forgiving HTML/XML/RSS parser")
   (home-page
    "https://github.com/fb55/htmlparser2#readme")
   (license license:expat)))

Re: Improved NPM importer with blacklist (Was: Re: NPM importer)

[swedebugia]

[-- Attachment #1: Type: text/plain, Size: 710 bytes --]

Hi

I programmed the whole day and learned a lot. :D

On 2018-11-30 17:13, swedebugia wrote:

snip

> TODO:
> * make npm-recursive-import work by not fetching blacklisted packages

See the attached patches :)

> 
> * implement keyword blacklisting based on the descriptions
> 
> * match not just the whole string of blacklisted packages:
>    e.g. match also "rollup-plugin" when "rollup" is in the blacklist.

Done. See patch

> * get the tarballs from npm-registry instead as they are never missing
>    (githubs sometimes are) and likely reproducible.

Done, but not tested yet.

Also added a npm-uri (similar to pypi-uri) see patch.

Happy hacking!

-- 
Cheers Swedebugia

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-guix-Add-a-npm-uri-helper-method.patch --]
[-- Type: text/x-patch; name="0001-guix-Add-a-npm-uri-helper-method.patch", Size: 1708 bytes --]

From 9ad29ede5b26a9052518adb422bc66398c38c9b4 Mon Sep 17 00:00:00 2001
From: swedebugia <swedebugia@riseup.net>
Date: Fri, 30 Nov 2018 20:52:42 +0100
Subject: [PATCH 1/2] guix: Add a "npm-uri" helper method.

* guix/build-system/node.scm (npm-uri): New method.
---
 guix/build-system/node.scm | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/guix/build-system/node.scm b/guix/build-system/node.scm
index b6d362904..ce0d9f84c 100644
--- a/guix/build-system/node.scm
+++ b/guix/build-system/node.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2016 Jelle Licht <jlicht@fsfe.org>
+;;; Copyright © 2018 swedebugia <swedebugia@riseup.net>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -30,7 +31,22 @@
   #:export (npm-meta-uri
             %node-build-system-modules
             node-build
-            node-build-system))
+            node-build-system
+            npm-uri))
+
+;; Commentary:
+;;
+;; Standard build procedure for Node packages using 'packages.json' and
+;; Makefiles.  This is implemented as an extension of 'gnu-build-system'.
+;;
+;; Code:
+
+(define* (npm-uri name version #:optional (extension ".tgz"))
+  "Return a URI string for the NPM package hosted on the NPM Registry
+corresponding to NAME and VERSION.  EXTENSION is the file name extension, such
+as '.tar.gz'. E.g. https://registry.npmjs.org/async/-/async-1.8.0.tgz"
+  (string-append "https://registry.npmjs.org/" name "/-/" name "-" version
+                 extension))
 
 (define (npm-meta-uri name)
   "Return a URI string for the metadata of node module NAME found in the npm
-- 
2.19.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-guix-Add-blacklisting-to-recursive-importer.patch --]
[-- Type: text/x-patch; name="0002-guix-Add-blacklisting-to-recursive-importer.patch", Size: 6473 bytes --]

From 54515dbb9e30e3cdf0ac1835c465f157302aefd1 Mon Sep 17 00:00:00 2001
From: swedebugia <swedebugia@riseup.net>
Date: Sat, 1 Dec 2018 00:15:08 +0100
Subject: [PATCH 2/2] guix: Add blacklisting to recursive importer

* guix/import/npm.scm (blacklisted?, npm->guix-package): Remove blacklisted
packages from the list passed to the recursive importer.
* guix/scripts/import/npm.scm (show-help, %options): Document and improve
"recursive" option.
* doc/guix.texi (Running guix import): Document the blacklist function.

Inspired by ac906cb7bb2ec77821ddec291db4857cc812599d by Ricardo
---
 doc/guix.texi               | 33 ++++++++++++++++++++++++++++++++-
 guix/import/npm.scm         | 16 ++++++++++++----
 guix/scripts/import/npm.scm | 10 ++++++++--
 3 files changed, 52 insertions(+), 7 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 8f694a20e..ccb3c45f1 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -7258,7 +7258,7 @@ guix import npm minimist
 @end example
 
 The importer implements an experimental recursive subsystem, allowing
-for a significant of npm packages to be imported.  The command below
+for a significant number of npm packages to be imported.  The command below
 imports metadata for the @code{optimist} npm package, as well as its
 dependencies.
 
@@ -7266,6 +7266,37 @@ dependencies.
 guix import npm -r optimist
 @end example
 
+The importer currently has does not warn when a cyclic dependency has been
+encountered. It will run until the buffer overruns which could take a long
+time. You should generally see output every couple of seconds, if not a cyclic
+dependency has probably been encountered. These are best avoided by first
+packaging an earlier version of one of the packages without the offending
+input to avoid the cycle.
+
+@quotation Note
+We have implemented an experimental blacklist functionality, enabled by
+default, to help you avoid importing a lot of garbage development
+dependencies. You can customize this blacklist by editing
+@file{guix/import/npm-blacklist.scm} only when running @command{guix import
+npm} from a git checkout, see @pxref{Running Guix Before It Is Installed}.
+
+These packages and every package that begins with these 103 names are
+currently blacklisted: (matcha benchmark babel @babel webpack rollup
+browserify async ember broccoli nsp uglifyjs2 uglify-js electron statsd vega
+grunt-release lineman lerna openlayers openpgp yarnpkg wekan etherpad-lite
+meteor keybase docco jsdoc markdown eslint lint markdownlint prettier standard
+jscs @ljharb/eslint-config editorconfig-tools jshint xo tick tsml
+tsd-check @commitlint safe grunt lerna vows husky nps rimraf colors cli
+source-map chai unexpected karma karma-qunit karma-sauce-launcher
+karma-browserify coveralls covert nyc qunit mocha yargs sinon code tape
+colortape airtap tap tap-spec lab mock nyc proxyquire coffe-script cross-swawn
+terst stream-spigot make-generator-function forking-tap bogota faucet gremlins
+jest redux-mock-store webdriverio ava enzyme testem protractor testcafe
+test262 storybook jasmine volkswagen nightwatch supertest istanbul sizzy
+zombie PhantomCSS)
+This expands to thousands of packages.
+@end quotation
+
 @item elpa
 @cindex elpa
 Import metadata from an Emacs Lisp Package Archive (ELPA) package
diff --git a/guix/import/npm.scm b/guix/import/npm.scm
index 2efe81fa9..1d46fd384 100644
--- a/guix/import/npm.scm
+++ b/guix/import/npm.scm
@@ -350,8 +350,12 @@ a git checkout."
                        ,(guix-hash-url temp-dir)))))))))))
 
 (define (blacklisted? name)
-  "Check if the pair name-version is blacklisted. RETURN #t if yes, else #f."
-  (member name blacklist))
+  "Check if the string name is blacklisted. RETURN #t if yes, else #f."
+  ;; Split the string to enable ut so blacklist scoped packages like
+  ;; @babel/core and packages like eslint-popup without having to type in
+  ;; every single combination.
+  (if (member (car (string-split name (char-set #\- #\/))) blacklist)
+      #t #f))
 
 (define (sanitize-npm-version version)
   "Return version without prefixed ^."
@@ -518,8 +522,12 @@ npm list of dependencies DEPENDENCIES."
                (npm-dependencies
                 (append
                  (extract-npm-dependencies dependencies)
-                 ;; TODO blacklist!
-                 (extract-npm-dependencies dev-dependencies)))
+                 ;; Remove the blacklisted devdeps to avoid big dependency
+                 ;; cycles and all the linters, cli-tools, benchmarking,
+                 ;; etc. JS-devs adds to their development environment.
+                 (remove
+                  blacklisted?
+                  (extract-npm-dependencies dev-dependencies))))
                (description (assoc-ref package "description"))
                (home-page (assoc-ref package "homepage"))
                (license (extract-license curr))
diff --git a/guix/scripts/import/npm.scm b/guix/scripts/import/npm.scm
index 1f4ed777c..c540c3574 100644
--- a/guix/scripts/import/npm.scm
+++ b/guix/scripts/import/npm.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2015 David Thompson <davet@gnu.org>
+;;; Copyright © 2018 swedebugia <swedebugia@riseup.net>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -25,6 +26,7 @@
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-11)
   #:use-module (srfi srfi-37)
+  #:use-module (srfi srfi-41)
   #:use-module (ice-9 match)
   #:use-module (ice-9 format)
   #:export (guix-import-npm))
@@ -41,7 +43,9 @@
    Import and convert the npm package for PACKAGE-NAME.\n"))
   (display (G_ "
      -h, --help             display this help and exit"))
-  (display (G_ "
+   (display (G_ "
+     -r, --recursive        import packages recursively"))
+   (display (G_ "
      -V, --version          display version information and exit"))
   (newline)
   (show-bug-report-information))
@@ -88,7 +92,9 @@
                    `(define-public ,(string->symbol name)
                       ,pkg))
                   (G_ #f))
-                (recursive-import package-name))
+                (reverse
+                 (stream->list
+                (npm-recursive-import package-name))))
            ;; Single import
            (let ((sexp (npm->guix-package package-name)))
              (unless sexp
-- 
2.19.1

Re: NPM importer

[Ludovic Courtès]

Hello,

swedebugia <swedebugia@riseup.net> skribis:

> Has anybody worked on that?

Yes, this was the topic of a GSoC project by Jelle Licht (Cc’d).  But
don’t hold your breath: as Chris Webber explained, the npm situation is
very hard to address sanely:

  http://dustycloud.org/blog/javascript-packaging-dystopia/

Ludo’.

Re: NPM importer

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 809 bytes --]


ludo@gnu.org (Ludovic Courtès) writes:

[...]

> Yes, this was the topic of a GSoC project by Jelle Licht (Cc’d).  But
> don’t hold your breath: as Chris Webber explained, the npm situation is
> very hard to address sanely:
>
>   http://dustycloud.org/blog/javascript-packaging-dystopia/

(semi OT: today Debian ships a recent jquery 3.2.1)

I'm not an expert in js (nor guix) packaging so I'm not able to judge
this:

 https://spin.atomicobject.com/2016/12/16/reproducible-builds-npm-yarn/

is yarn a viable solution to the NPM packaging problems?

can we achieve reproducible builds ala guix with a yarn importer and
some amount of yarn packages downloading/automation and offline
mirroring?

Ciao
Giovanni

P.S.: why 

--
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: NPM importer

[Julien Lepiller]

Le 2018-11-21 17:37, Giovanni Biscuolo a écrit :
> ludo@gnu.org (Ludovic Courtès) writes:
> 
> [...]
> 
>> Yes, this was the topic of a GSoC project by Jelle Licht (Cc’d).  But
>> don’t hold your breath: as Chris Webber explained, the npm situation 
>> is
>> very hard to address sanely:
>> 
>>   http://dustycloud.org/blog/javascript-packaging-dystopia/
> 
> (semi OT: today Debian ships a recent jquery 3.2.1)
> 
> I'm not an expert in js (nor guix) packaging so I'm not able to judge
> this:
> 
>  https://spin.atomicobject.com/2016/12/16/reproducible-builds-npm-yarn/
> 
> is yarn a viable solution to the NPM packaging problems?
> 
> can we achieve reproducible builds ala guix with a yarn importer and
> some amount of yarn packages downloading/automation and offline
> mirroring?

How different is it to build an npm package and a yarn package? Could 
you elaborate a bit on your idea?

We can already build packages with our wip node-build-system, as long as 
we have build- and run-time dependencies available. The real hard parts 
are: sometimes build-tools depend on what they build, there is just too 
many dependencies and some packages don't declare a license properly.

For instance, grunt is a build tool for node packages; it has 179 
dependencies at runtime (including recursive dependencies). All of them 
need to be built before grunt can be run. What's the chance that none of 
them require grunt? I haven't taken the time to look at these 
dependencies, so maybe I'm pessimistic with no good reason.

Another instance is application-config-path that declares its license 
only in the Makefile, in the form of "License: MIT". Do we consider this 
free software?

Now if yarn has some build recipes and has taken the time to make this 
whole mess more manageable, I'm all for a yarn importer. Otherwise, it's 
just another source of package information, which is fine, but npm seems 
to do the job already.

> 
> Ciao
> Giovanni
> 
> P.S.: why
> 
> --
> Giovanni Biscuolo
> 
> Xelera IT Infrastructures

Re: NPM importer

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 1177 bytes --]

Hi Guix,

sorry: reading at today messages from swdebugia and your comments below
I realize that mine was just noise from a packaging-ignorant :-S

Julien Lepiller <julien@lepiller.eu> writes:

[...]

>>  https://spin.atomicobject.com/2016/12/16/reproducible-builds-npm-yarn/
>> 
>> is yarn a viable solution to the NPM packaging problems?

[...]

> How different is it to build an npm package and a yarn package?

no difference

> Could you elaborate a bit on your idea?

I'm not able: I was mislead by the title of this article I cited

  https://spin.atomicobject.com/2016/12/16/reproducible-builds-npm-yarn/

"reproducuble builds" and "npm/yarn" seems like an oxymoron to me, now :-)

> We can already build packages with our wip node-build-system, as long as 
> we have build- and run-time dependencies available. The real hard parts 
> are: sometimes build-tools depend on what they build, there is just too 
> many dependencies and some packages don't declare a license properly.

ACK, now I finally understand the problems

[...]

thany you for your patience

ciao
Giovanni

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]