[GSoC update] Npm & guix

[GSoC update] Npm & guix

[Jelle Licht]

[-- Attachment #1: Type: text/plain, Size: 3745 bytes --]

Hello Guix!

After hopefully enough contemplation and a lot of elbow grease, I would
like to
give you an overview of what I have been up to these past weeks.

To start off with something that might make some people less than happy;
jQuery
and its dependencies will most likely not be packaged this summer. By me, at
least.

On Ludo's advice, I snarfed Ricardo's recursive importer and bolted it on
my npm
importer. After leaving the importer running for a quite some hours (and
making
it more robust in the face of inconsistent npm information), it turns out
that
jQuery has a direct or indirect dependcy on about everything. We are talking
pretty much all of the build systems, all of the testing frameworks and all
of
the test runners. Literally thousands of packages, and multiple
(conflicting)
versions of most.

While this is a sad realization indeed, it just makes it easier to focus on
other important packages of npm. Running the recursive importer on a
handful of
packages leads me to the following possibly redundant observations:

* Quite some packages have tests (Yay!)
* Running tests requires test frameworks and test runners

This makes it IMHO a worthwhile goal bootstrap to a working test framework,
with
of course at first tests disabled for the dependencies of this test
framework.
Test frameworks all have an (indirect) dependency on the Coffeescript
compiler,
of which the first version was written in Ruby. Using this initial (alpha)
compiler, and the awesome git-bisect command, I was able to subsequently
compile
and use the more modern (but still old) Coffeescript-in-coffeescript
compilers.

I am currently hovering between version 0.6 and 0.7, which can properly
recompile itself and slightly more contemporary version. Getting to version
1.6 from June 2013 should be doable using this exact same approach. This
will
allow us to package a 2014 version of the Mocha testing framework.

For the people more knowledgeable in these matters, how would you deal with
deprecated functionality in languages such as python, ruby etc? Because npm
packages are so interdependent, I simply need to start somewhere, by
packaging
things back when they did not have as many dependencies. Currently, I have a
file containing implementations of old Node (pre 1.0) functionality on Node
6.0.
I was thinking of releasing this 'hack' as an npm package and then package
it in
Guix.

The alternative would be to package each version of Node that was used to
build
these ancient packages. For bootstrapping Coffeescript, this already forces
us to
have 3~4 versions of Node, although it is conceptually a lot cleaner.

So my current view of our options:
* Backport ancient node features to a contemporary node version
* Package a significant variety of node versions
Please let me know if anyone has some thoughts, critiques or silver bullets
for this
problem.

A goal of this project is still to have a working jQuery by the end of this
summer, just
not via the procedures defined by npm. My current plan is to partially
reimplement the build procedures used for jquery, albeit in a much simpler
manner: just a single, some might say bloated, javascript file created from
the
sources, instead of the a-la-carte build afforded by Grunt.

In this approach, I also see something that might make packaging npm
packages more Guix-friendly: as long as we have _functional_ equivalents,
we should be able to build a
huge subset of packages. This can be either a blessed version of an npm
package, or a
simplistic yet correct reimplementation, depending on the complexity of the
specific
package. This would only work for build-time dependencies, or native-inputs.

Thanks for reading this WoT and looking forward to hearing your feedback.

- Jelle

[-- Attachment #2: Type: text/html, Size: 4044 bytes --]

Re: [GSoC update] Npm & guix

[Andy Wingo]

On Sun 24 Jul 2016 03:06, Jelle Licht <jlicht@fsfe.org> writes:

> On Ludo's advice, I snarfed Ricardo's recursive importer and bolted it
> on my npm importer. After leaving the importer running for a quite
> some hours (and making it more robust in the face of inconsistent npm
> information), it turns out that jQuery has a direct or indirect
> dependcy on about everything. We are talking pretty much all of the
> build systems, all of the testing frameworks and all of the test
> runners. Literally thousands of packages, and multiple (conflicting)
> versions of most.

Wow, how frustrating!

> This makes it IMHO a worthwhile goal bootstrap to a working test
> framework, with of course at first tests disabled for the dependencies
> of this test framework.  Test frameworks all have an (indirect)
> dependency on the Coffeescript compiler, of which the first version
> was written in Ruby. Using this initial (alpha) compiler, and the
> awesome git-bisect command, I was able to subsequently compile and use
> the more modern (but still old) Coffeescript-in-coffeescript
> compilers.

For what it's worth, to me as a bystander this looks like great progress
:)

Andy

Re: [GSoC update] Npm & guix

[Jan Nieuwenhuizen]

Jelle Licht writes:

> On Ludo's advice, I snarfed Ricardo's recursive importer and bolted it on my npm
> importer. After leaving the importer running for a quite some hours (and making
> it more robust in the face of inconsistent npm information), it turns out that
> jQuery has a direct or indirect dependcy on about everything. We are talking
> pretty much all of the build systems, all of the testing frameworks and all of
> the test runners. Literally thousands of packages, and multiple (conflicting)
> versions of most.

Have you asked the jQuery developers for their view on the number of
dependencies you found and some of the conflicts you see?

Thanks for your great effort!
Greetings, Jan

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar®  http://AvatarAcademy.nl  

Re: [GSoC update] Npm & guix

[Ludovic Courtès]

Hello!

Jelle Licht <jlicht@fsfe.org> skribis:

> On Ludo's advice, I snarfed Ricardo's recursive importer and bolted it
> on my npm importer. After leaving the importer running for a quite
> some hours (and making it more robust in the face of inconsistent npm
> information), it turns out that jQuery has a direct or indirect
> dependcy on about everything. We are talking pretty much all of the
> build systems, all of the testing frameworks and all of the test
> runners. Literally thousands of packages, and multiple (conflicting)
> versions of most.

I’m really impressed that your importer can already grovel this much!
In itself, that’s already a significant achievement, despite the
frustration of not getting “guix package -i jquery” right away.

Do you have figures on the number of vertices and edges on this graph?
Could it be that the recursive importer keeps revisiting the same nodes
over and over again?  :-)

I would suggest publishing the code somewhere so others can try to
import their favorite JS library and give feedback.

> This makes it IMHO a worthwhile goal bootstrap to a working test
> framework, with of course at first tests disabled for the dependencies
> of this test framework.  Test frameworks all have an (indirect)
> dependency on the Coffeescript compiler, of which the first version
> was written in Ruby. Using this initial (alpha) compiler, and the
> awesome git-bisect command, I was able to subsequently compile and use
> the more modern (but still old) Coffeescript-in-coffeescript
> compilers.
>
> I am currently hovering between version 0.6 and 0.7, which can
> properly recompile itself and slightly more contemporary
> version. Getting to version 1.6 from June 2013 should be doable using
> this exact same approach.  This will allow us to package a 2014
> version of the Mocha testing framework.

Impressive.

> For the people more knowledgeable in these matters, how would you deal
> with deprecated functionality in languages such as python, ruby etc?
> Because npm packages are so interdependent, I simply need to start
> somewhere, by packaging things back when they did not have as many
> dependencies. Currently, I have a file containing implementations of
> old Node (pre 1.0) functionality on Node 6.0.  I was thinking of
> releasing this 'hack' as an npm package and then package it in Guix.
>
> The alternative would be to package each version of Node that was used
> to build these ancient packages. For bootstrapping Coffeescript, this
> already forces us to have 3~4 versions of Node, although it is
> conceptually a lot cleaner.
>
> So my current view of our options: * Backport ancient node features to
> a contemporary node version * Package a significant variety of node
> versions Please let me know if anyone has some thoughts, critiques or
> silver bullets for this problem.

People have looked at bootstrapping compilers using their historical
ancestor, and it often looked hard first to find what each
implementation’s ancestor was, and then to actually chain them (I
remember Ricardo going back to an old Haskellish implementation in
Scheme at one point.)

Of the two options you list, packaging several Node versions sounds like
the simplest one.  There may be other stumbling blocks though, so be
prepared to stop the packaging recursion before you get mad.  ;-)  In
practice, we’ve never managed to get this far for the other compilers we
have.

Thanks for the status update!

Ludo’.

Re: [GSoC update] Npm & guix

[Catonano]

[-- Attachment #1: Type: text/plain, Size: 4364 bytes --]

2016-07-25 23:26 GMT+02:00 Ludovic Courtès <ludo@gnu.org>:

> Hello!
>
> Jelle Licht <jlicht@fsfe.org> skribis:
>
> > On Ludo's advice, I snarfed Ricardo's recursive importer and bolted it
> > on my npm importer. After leaving the importer running for a quite
> > some hours (and making it more robust in the face of inconsistent npm
> > information), it turns out that jQuery has a direct or indirect
> > dependcy on about everything. We are talking pretty much all of the
> > build systems, all of the testing frameworks and all of the test
> > runners. Literally thousands of packages, and multiple (conflicting)
> > versions of most.
>
> I’m really impressed that your importer can already grovel this much!
> In itself, that’s already a significant achievement, despite the
> frustration of not getting “guix package -i jquery” right away.
>
> Do you have figures on the number of vertices and edges on this graph?
> Could it be that the recursive importer keeps revisiting the same nodes
> over and over again?  :-)
>



> I would suggest publishing the code somewhere so others can try to
> import their favorite JS library and give feedback.
>
>
I'd like to indicate that in the Guix code base there's a function visiting
the graph of the dependencies of a package with the concern of covering it
all AND of not considering the same node more than one time

It's in guix/packages.scm on line 552, it's called "transitive-inputs" and
takes as an arguments the inputs of a single package.

It could be used as a blueprint for such task of dependencies graphs
covering.

The main observation that I can do is that both versions do check whether a
package being considered is already been seen in a previous passage.

So, it doesn't seem to me that Jelle's version could revisit the same
package (node) over and over

Also, the "official" version clearly distinguishes between the current
depth level in the graph and the next one, using 2 different variables for
containing the packages (nodes) at the current depth and those in the next
level ("inputs" and "propagated") as it does a breadth first visit.

Instead the Jelle's version uses the same variable for the current AND the
next level and it's a list and it conses the nodes for the next level on
the current list

So it seems to me that it does a depth first visit (I might be wrong, I
didn't do a complete trace)

If anything, the "official" version uses a VHash to check if a package has
already been "seen" while Jelle's uses a plain list (again)

So the cost of this check is constant in the official version and linear in
Jelle's version.

Further, in Jelle's version, every package gets checked against the list of
already imported packages (as a plain list) AND against the packages
already present in the store.

Both checks at every iteration. It seems to me that there's space for
improving here ;-)

In fact, in Jelle's guix/import/npm.scm, on line 462 the "and" could be
switched to an "or". It would work anyway and it could save a handful of
checks on a complete run of the algorithm

Anyway, I tried to "import" a random package and it has a ton of
dependencies.

It seems to me that a more systematic approach (like that of a real data
scientist ;-) ) could help here

The whole graph should be imported in some database and then graph
algorithms should be run on it

For example: which are the packages with less or no dependencies (and a lot
of dependants) ?
Because those should be imported first, in my opinion.

Jelle came to the notion that testing frameworks are a dependencies for
almost all the packages but as far as I understand that's a quite empirical
knowledge.


> > I am currently hovering between version 0.6 and 0.7, which can
> > properly recompile itself and slightly more contemporary
> > version. Getting to version 1.6 from June 2013 should be doable using
> > this exact same approach.  This will allow us to package a 2014
> > version of the Mocha testing framework.
>
> Impressive.
>

Indeed. Impressive.

Ok, so these are the first impressions that I came up with. I'd like my
observations on the algorithms to be vetted by more experienced people.

Just to help me verify if I can actually understand someone's else scheme
code

Thanks

[-- Attachment #2: Type: text/html, Size: 5659 bytes --]

Re: [GSoC update] Npm & guix

[Jelle Licht]

[-- Attachment #1: Type: text/plain, Size: 5341 bytes --]

Quick reply from my phone, but thanks for the feedback.

On Jul 29, 2016 16:53, "Catonano" <catonano@gmail.com> wrote:

> 2016-07-25 23:26 GMT+02:00 Ludovic Courtès <ludo@gnu.org>:
>
>> Hello!
>>
>> Jelle Licht <jlicht@fsfe.org> skribis:
>>
>> > On Ludo's advice, I snarfed Ricardo's recursive importer and bolted it
>> > on my npm importer. After leaving the importer running for a quite
>> > some hours (and making it more robust in the face of inconsistent npm
>> > information), it turns out that jQuery has a direct or indirect
>> > dependcy on about everything. We are talking pretty much all of the
>> > build systems, all of the testing frameworks and all of the test
>> > runners. Literally thousands of packages, and multiple (conflicting)
>> > versions of most.
>>
>> I’m really impressed that your importer can already grovel this much!
>> In itself, that’s already a significant achievement, despite the
>> frustration of not getting “guix package -i jquery” right away.
>>
>> Do you have figures on the number of vertices and edges on this graph?
>> Could it be that the recursive importer keeps revisiting the same nodes
>> over and over again?  :-)
>>
>
>
>
>> I would suggest publishing the code somewhere so others can try to
>> import their favorite JS library and give feedback.
>>
>>
> I'd like to indicate that in the Guix code base there's a function
> visiting the graph of the dependencies of a package with the concern of
> covering it all AND of not considering the same node more than one time
>
> It's in guix/packages.scm on line 552, it's called "transitive-inputs" and
> takes as an arguments the inputs of a single package.
>
> It could be used as a blueprint for such task of dependencies graphs
> covering.
>
> The main observation that I can do is that both versions do check whether
> a package being considered is already been seen in a previous passage.
>
> So, it doesn't seem to me that Jelle's version could revisit the same
> package (node) over and over
>
> Also, the "official" version clearly distinguishes between the current
> depth level in the graph and the next one, using 2 different variables for
> containing the packages (nodes) at the current depth and those in the next
> level ("inputs" and "propagated") as it does a breadth first visit.
>
> Instead the Jelle's version uses the same variable for the current AND the
> next level and it's a list and it conses the nodes for the next level on
> the current list
>
> So it seems to me that it does a depth first visit (I might be wrong, I
> didn't do a complete trace)
>
> If anything, the "official" version uses a VHash to check if a package has
> already been "seen" while Jelle's uses a plain list (again)
>
> So the cost of this check is constant in the official version and linear
> in Jelle's version.
>
> Further, in Jelle's version, every package gets checked against the list
> of already imported packages (as a plain list) AND against the packages
> already present in the store.
>
> Both checks at every iteration. It seems to me that there's space for
> improving here ;-)
>
> In fact, in Jelle's guix/import/npm.scm, on line 462 the "and" could be
> switched to an "or". It would work anyway and it could save a handful of
> checks on a complete run of the algorithm
>

You are totally right here! The quick and dirty way of doing it, which I am
currently doing,
is to wrap these checks in a memoized function.
I'll take some of your points into account to properly rewrite this part.


>
> Anyway, I tried to "import" a random package and it has a ton of
> dependencies.
>

Story of my life these past few weeks :-)


>
> It seems to me that a more systematic approach (like that of a real data
> scientist ;-) ) could help here
>
>

> The whole graph should be imported in some database and then graph
> algorithms should be run on it
>
> For example: which are the packages with less or no dependencies (and a
> lot of dependants) ?
> Because those should be imported first, in my opinion.
>
> Jelle came to the notion that testing frameworks are a dependencies for
> almost all the packages but as far as I understand that's a quite empirical
> knowledge.
>

^ This, I like. Does anyone have any suggestions on tools that could help
me do this in guile?
I know of projects like [1] that already do something similar, although I
do no have any experience
with constructing and querying graphs using free software.

>
>
>> > I am currently hovering between version 0.6 and 0.7, which can
>> > properly recompile itself and slightly more contemporary
>> > version. Getting to version 1.6 from June 2013 should be doable using
>> > this exact same approach.  This will allow us to package a 2014
>> > version of the Mocha testing framework.
>>
>> Impressive.
>>
>
> Indeed. Impressive.
>
> Ok, so these are the first impressions that I came up with. I'd like my
> observations on the algorithms to be vetted by more experienced people.
>
> Just to help me verify if I can actually understand someone's else scheme
> code
>
> Thanks
>

I think you definitely got the gist of it.

Thanks
- Jelle

[1]: https://exploringdata.github.io/info/npm-packages-dependencies/

[-- Attachment #2: Type: text/html, Size: 8088 bytes --]

Package graph queries

[Ludovic Courtès]

Jelle Licht <jlicht@fsfe.org> skribis:

> On Jul 29, 2016 16:53, "Catonano" <catonano@gmail.com> wrote:

[...]

>> For example: which are the packages with less or no dependencies (and a
>> lot of dependants) ?
>> Because those should be imported first, in my opinion.

[...]

> ^ This, I like. Does anyone have any suggestions on tools that could help
> me do this in guile?

I understand you want to perform these queries on NPM packages, not Guix
packages, but if you could hook your NPM package representation in (guix
graph)¹, you could do the same as this:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,use(guix graph)
scheme@(guile-user)> ,use(guix scripts graph)
scheme@(guile-user)> ,use(gnu packages)
scheme@(guile-user)> ,use(guix monad-repl)
scheme@(guile-user)> ,enter-store-monad
store-monad@(guile-user) [1]> (node-back-edges %bag-node-type (fold-packages cons '()))
$4 = #<procedure 59ac880 at guix/graph.scm:87:17 (node)>
store-monad@(guile-user) [1]> ,q
scheme@(guile-user)> ,use(srfi srfi-1)
scheme@(guile-user)> (filter (lambda (p)
			       (> (node-reachable-count (list p) $4)
				  600))
			     (fold-packages cons '()))
$5 = (#<package inputproto@2.3.1 gnu/packages/xorg.scm:973 325e3c0> #<package renderproto@0.11.1 gnu/packages/xorg.scm:1690 326c000> #<package xextproto@7.3.0 gnu/packages/xorg.scm:2308 328df00> #<package libxft@2.3.2 gnu/packages/xorg.scm:1302 326ca80> #<package libsm@1.2.2 gnu/packages/xorg.scm:1187 326cd80> #<package xproto@7.0.28 gnu/packages/xorg.scm:4711 32c0240> #<package xorg-rgb@1.0.6 gnu/packages/xorg.scm:4408 32c09c0> #<package libx11@1.6.3 gnu/packages/xorg.scm:5139 32cf9c0> #<package libxdmcp@1.1.2 gnu/packages/xorg.scm:1277 326cb40> #<package libpthread-stubs@0.3 gnu/packages/xorg.scm:1160 326ce40> #<package libxrender@0.9.9 gnu/packages/xorg.scm:4577 32c0540> #<package xcb-proto@1.11 gnu/packages/xorg.scm:2061 327e600> #<package kbproto@1.0.7 gnu/packages/xorg.scm:997 325e300> […])
scheme@(guile-user)> (length $5)
$6 = 119
--8<---------------cut here---------------end--------------->8---

The code above lists all the packages with more than 600 dependents.

It’s inefficient (a bit less than a minute here) because
‘node-reachable-count’ does a complete graph traversal (you could
have a variant of it that caches the value for each node, instead).

HTH,
Ludo’.

¹ All you need is to define a <node-type>, which is essentially 4
  methods.

Re: Package graph queries

[Catonano]

[-- Attachment #1: Type: text/plain, Size: 1641 bytes --]

2016-07-31 0:19 GMT+02:00 Ludovic Courtès <ludo@gnu.org>:

> Jelle Licht <jlicht@fsfe.org> skribis:
>
> > On Jul 29, 2016 16:53, "Catonano" <catonano@gmail.com> wrote:
>
> [...]
>
> >> For example: which are the packages with less or no dependencies (and a
> >> lot of dependants) ?
> >> Because those should be imported first, in my opinion.
>
> [...]
>
> > ^ This, I like. Does anyone have any suggestions on tools that could help
> > me do this in guile?
>
> I understand you want to perform these queries on NPM packages, not Guix
> packages, but if you could hook your NPM package representation in (guix
> graph)¹, you could do the same as this:
>
> --8<---------------cut here---------------start------------->8---
> scheme@(guile-user)> ,use(guix graph)
> scheme@(guile-user)> ,use(guix scripts graph)
> scheme@(guile-user)> ,use(gnu packages)
> scheme@(guile-user)> ,use(guix monad-repl)
> scheme@(guile-user)> ,enter-store-monad
> store-monad@(guile-user) [1]> (node-back-edges %bag-node-type
> (fold-packages cons '()))
> $4 = #<procedure 59ac880 at guix/graph.scm:87:17 (node)>
> store-monad@(guile-user) [1]> ,q
> scheme@(guile-user)> ,use(srfi srfi-1)
> scheme@(guile-user)> (filter (lambda (p)
>                                (> (node-reachable-count (list p) $4)
>                                   600))
>                              (fold-packages cons '()))
>

My concern, here, would be an equivalent of "fold-packages" for
registry.npmjs.com.

The API seems a bit cloudy to me.

See this question
https://stackoverflow.com/questions/34071621/query-npmjs-registry-via-api

[-- Attachment #2: Type: text/html, Size: 2443 bytes --]