Hello Catonano,

2017-01-19 21:48 GMT+01:00 Catonano <catonano@gmail.com>:

> I made a crawler and I let it loose on the jquery 3.1.1 dependencies on
> registry.npmjs.com
> It recursevely fetched the dependencies of jquery 3.1.1, then the
> dependencies of the dependencies, then the dependencies of the dependencies
> of the dependencies... and so on
>
> Until there were no more dependencies to fetch.
>
> It stored all such dependencies in a graph database made by the amazing
> amz3 for their project, Culturia. They wrote about such project several
> times on the guile-users mailing list.
>
> It took days to download. And it took me months to produce a working
> version of this code. Because I don't know Guile very well and maybe I'm
> not that smart overall.
>
> Anyway, now I have a COMPLETE graph of the dependencies of jquery 3.1.1
>

These pictures are very informative indeed. I will try to be brief, but I
quickly wanted to share
that I find your efforts (and results!) amazing.


>
> It's made of
> 47311 vertices and
> 324569 edges
>
> I made a graph of a subset (graphviz chocked a bit on this ;-) )
> It's here
> http://catonano.altervista.org/grafo.svg
>
> There are 448 "broken" packages. Probably I should explain how I
> classified packages as "broken". Maybe not in this email.
>
> Anyway, these broken packages pose a challenge to the mission of porting
> Jquery into Guix, in my opinion, and they should be considered with some
> attention.
> Here's the list
> http://catonano.altervista.org/broken_packages.txt
>

Something of note: a big chunk of the packages you classified as being
'broken' are
making me recall some (unpleasant) memories; From my own crawling
experiments,
which were not nearly as complete as this one, I also ran into a lot of the
same
show-stoppers. In a very real sense, resolving node dependencies quickly
devolves into
resolving dependencies for most of the popular build systems as well as
plugins like
broccoli-funnel. What I am trying to get at here is that fixing and
packaging these 448 packages
will likely contribute a lot towards making an npm-enriched guix possible.


>
>
> There are 1314 packages with NO dependencies that could be used as
> starting points in porting Jquery into Guix.
> Here's the list
> http://catonano.altervista.org/broken_packages.txt
>

These could probably make use of the npm importer I worked on earlier. Do
you make use of your own? Otherwase I'll get to
rebasing my version on the current guix master branch.


>
>
> If there's anyone interested, I can give you the data folder so you can
> try all the queries you want on these data without having to to run this
> thing for a bunch of hours
>

If possible, yes please. What would be the most convenient way you to share
this data?


>
> In the future, I'd like to run this thing on some other package and merge
> the graphs so I will be able to investigate which are the common
> fundamental dependencies for SEVERAL important packages in Nodejs.
>
> So if someone wants to dedicate time to porting Nodejs stuff in Guix they
> will be able to select most urgent packages to start from.
>
> The same could be said of broken packages taht affect several important
> packages.
>
> The porting of Nodejs in Guix cannot be done with brute strength. A data
> oriented approach can help, in my opinion.
>
> Indeed.


> The ideal would be to have something that, like bitcoin, coordinates a
> swarm in such a way that every node can contribute a tiniy bit of data to a
> common data structure, so all the nodes would have a complete copy of the
> database.
>
> Collecting a mantaining of datasets should be freed of the client server
> model too. Not only the social media.
>
> I have no idea what you are referring to. Could you please elaborate a bit
at a later point in time?


> But that's more than I can handle, anyway.
>

I am already thankful

>
> I'd like to talk about the stumbling blocks I run into to discuss Guile
> and my knowledge of it.
>
> For example, I can't use that thing in the autotools that processes
> configure.am files so I just forked amz3's project and added my files in
> there. As guests. Thanks amz3 !
>
> I'd also like to describe te screw ups in the format I put the data into.
> I realized my mistakes when hours of crunching had already been done.
>
> They can be migrated to a better format though
>
> If you don't mind, I will discuss these issue in the future, not now.
>
> The code is here
> https://gitlab.com/humanitiesNerd/Culturia
>
> One last fun fact: while I was watching the output flowing in my terminal,
> I saw a package called
>
> "broccoli-funnel"
>
> No, really. It's here
> https://registry.npmjs.org/broccoli-funnel
>
> Ok, that's all for now.
>

Thanks again for your efforts on this. I am looking forward to working with
your data.

Regards,
Jelle Licht