Re: NPM importer

[Julien Lepiller <julien@lepiller.eu>]

Le Tue, 20 Nov 2018 22:12:18 +0100,
swedebugia <swedebugia@riseup.net> a écrit :

> Hi
> 
> On 2018-11-20 20:58, swedebugia wrote:
> > On 2018-11-20 08:50, Julien Lepiller wrote:  
> 
> snip
> 
> >> See this script that builds a graph of dependencies, ignoring 
> >> devDependencies (so none of the packages listed by this script is 
> >> going to be tested):
> >> https://framagit.org/tyreunom/guix/snippets/2534  
> 
> What is devDependencies? Required to build the npm package?

the package.json file declares two kinds (I think I've seen a third
kind, but I'm not sure, nor am I a js developper) of dependencies:
dependencies and devDependencies that map to our inputs and
native-inputs. devDependencies are tools used during the build or
development, but they are not needed at runtime. Sometimes, they are
required at build time (for instance the coffeescript compiler is a
devDependencies of some packages, but we need it to build the package),
usually they are required at test time (for instance mocha, tap, or
tape are test frameworks), otherwise, they can be documentation tools,
and sometimes really not useful for us (we probably don't want to run
linters or release-related tools).

> 
> I understood the script now I think. :)
> 
> I replaced "mocha" with name in the call in the bottom.
> 
> Then did:
> $guile
> scheme@(guile-user)> (define name "ssb-patchwork")
> scheme@(guile-user)> (load "npm-explorer.scm")
> ...

The way I wrote it, you were supposed to change the "mocha" with the
package you're interested in, and run the script with guile like so:

guile npm-explorer.scm > mocha.dot

and then:

fdp -Tpng mocha.dot > mocha.png

(fdp is part of graphviz, but produces nicer graphs than dot in my
opinion)

> 
> As you said it runs for(ever) (in my case 5 minutes) listing 
> dependencies of dependencies of dependencies of dependencies of 
> dependencies of dependencies ...
> 
> 1100 last lines of the output is attached. :p
> 
> One thing is for sure. There are a LOT of npm packages. "Over 477,000 
> packages are available on the main npm registry." according to WP. :O

That's probably the number of packages on which depend any node
packages through its dependencies and devDependencies... I suspect you
can always find a path between any two packages in this graph... in
either direction.

> 
> I wonder how many are free software? 90%? 50%?
> 
> I hope we can automate this some way.

It looks very difficult, but if you have any ideas, please tell us!

> 
> npm.scm will be a VERY VERY long file. Maybe we should rethink about
> how to best store all these variables...

Very long files are frowned upon, because they have very long
compilation times and require a lot of memory. We'll have to find a way
to split the file before it grows too much. Maybe we will do something
similar than python packages? We have python.scm, python-web.scm...

> 
> Just 1 npm application (ssb-patchwork) will add at least 1000 package 
> definitions to it.
>