From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jelle Licht Subject: Re: jquery 3.1.1 Date: Thu, 19 Jan 2017 22:07:57 +0100 Message-ID: References: Mime-Version: 1.0 Content-Type: multipart/alternative; boundary=94eb2c074e7e823bf3054678ed8d Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:35883) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cUJwL-0007bG-Hn for guix-devel@gnu.org; Thu, 19 Jan 2017 16:08:15 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cUJwI-0005rF-6l for guix-devel@gnu.org; Thu, 19 Jan 2017 16:08:13 -0500 Received: from mail.fsfe.org ([2001:aa8:ffed::3:102]:49097) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1cUJwH-0005lB-Oa for guix-devel@gnu.org; Thu, 19 Jan 2017 16:08:10 -0500 Received: from localhost (localhost [127.0.0.1]) by mail.fsfe.org (Postfix) with ESMTP id 1AE2363ABB4 for ; Thu, 19 Jan 2017 22:08:07 +0100 (CET) Received: from mail.fsfe.org ([127.0.0.1]) by localhost (cavendish.fsfeurope.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vQ3SELCWTHJd for ; Thu, 19 Jan 2017 22:08:03 +0100 (CET) Received: from mail-wm0-f47.google.com (mail-wm0-f47.google.com [74.125.82.47]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (not verified)) (Authenticated sender: jlicht) by mail.fsfe.org (Postfix) with ESMTPSA id CA9A663B97A for ; Thu, 19 Jan 2017 22:08:02 +0100 (CET) Received: by mail-wm0-f47.google.com with SMTP id r144so13159061wme.1 for ; Thu, 19 Jan 2017 13:08:02 -0800 (PST) In-Reply-To: List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Catonano Cc: guix-devel --94eb2c074e7e823bf3054678ed8d Content-Type: text/plain; charset=UTF-8 Hello Catonano, 2017-01-19 21:48 GMT+01:00 Catonano : > I made a crawler and I let it loose on the jquery 3.1.1 dependencies on > registry.npmjs.com > It recursevely fetched the dependencies of jquery 3.1.1, then the > dependencies of the dependencies, then the dependencies of the dependencies > of the dependencies... and so on > > Until there were no more dependencies to fetch. > > It stored all such dependencies in a graph database made by the amazing > amz3 for their project, Culturia. They wrote about such project several > times on the guile-users mailing list. > > It took days to download. And it took me months to produce a working > version of this code. Because I don't know Guile very well and maybe I'm > not that smart overall. > > Anyway, now I have a COMPLETE graph of the dependencies of jquery 3.1.1 > These pictures are very informative indeed. I will try to be brief, but I quickly wanted to share that I find your efforts (and results!) amazing. > > It's made of > 47311 vertices and > 324569 edges > > I made a graph of a subset (graphviz chocked a bit on this ;-) ) > It's here > http://catonano.altervista.org/grafo.svg > > There are 448 "broken" packages. Probably I should explain how I > classified packages as "broken". Maybe not in this email. > > Anyway, these broken packages pose a challenge to the mission of porting > Jquery into Guix, in my opinion, and they should be considered with some > attention. > Here's the list > http://catonano.altervista.org/broken_packages.txt > Something of note: a big chunk of the packages you classified as being 'broken' are making me recall some (unpleasant) memories; From my own crawling experiments, which were not nearly as complete as this one, I also ran into a lot of the same show-stoppers. In a very real sense, resolving node dependencies quickly devolves into resolving dependencies for most of the popular build systems as well as plugins like broccoli-funnel. What I am trying to get at here is that fixing and packaging these 448 packages will likely contribute a lot towards making an npm-enriched guix possible. > > > There are 1314 packages with NO dependencies that could be used as > starting points in porting Jquery into Guix. > Here's the list > http://catonano.altervista.org/broken_packages.txt > These could probably make use of the npm importer I worked on earlier. Do you make use of your own? Otherwase I'll get to rebasing my version on the current guix master branch. > > > If there's anyone interested, I can give you the data folder so you can > try all the queries you want on these data without having to to run this > thing for a bunch of hours > If possible, yes please. What would be the most convenient way you to share this data? > > In the future, I'd like to run this thing on some other package and merge > the graphs so I will be able to investigate which are the common > fundamental dependencies for SEVERAL important packages in Nodejs. > > So if someone wants to dedicate time to porting Nodejs stuff in Guix they > will be able to select most urgent packages to start from. > > The same could be said of broken packages taht affect several important > packages. > > The porting of Nodejs in Guix cannot be done with brute strength. A data > oriented approach can help, in my opinion. > > Indeed. > The ideal would be to have something that, like bitcoin, coordinates a > swarm in such a way that every node can contribute a tiniy bit of data to a > common data structure, so all the nodes would have a complete copy of the > database. > > Collecting a mantaining of datasets should be freed of the client server > model too. Not only the social media. > > I have no idea what you are referring to. Could you please elaborate a bit at a later point in time? > But that's more than I can handle, anyway. > I am already thankful > > I'd like to talk about the stumbling blocks I run into to discuss Guile > and my knowledge of it. > > For example, I can't use that thing in the autotools that processes > configure.am files so I just forked amz3's project and added my files in > there. As guests. Thanks amz3 ! > > I'd also like to describe te screw ups in the format I put the data into. > I realized my mistakes when hours of crunching had already been done. > > They can be migrated to a better format though > > If you don't mind, I will discuss these issue in the future, not now. > > The code is here > https://gitlab.com/humanitiesNerd/Culturia > > One last fun fact: while I was watching the output flowing in my terminal, > I saw a package called > > "broccoli-funnel" > > No, really. It's here > https://registry.npmjs.org/broccoli-funnel > > Ok, that's all for now. > Thanks again for your efforts on this. I am looking forward to working with your data. Regards, Jelle Licht --94eb2c074e7e823bf3054678ed8d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hello Catonano,

2017-01-19 21:48 GMT+01:00 = Catonano <catonano@gmail.com>:
I made a crawler and I let it loose on the = jquery 3.1.1 dependencies on registry.npmjs.com
It recursevely fetched the depe= ndencies of jquery 3.1.1, then the dependencies of the dependencies, then t= he dependencies of the dependencies of the dependencies... and so on
Until there were no more dependencies to fetch.
<= br>
It stored all such dependencies in a graph database made by the am= azing amz3 for their project, Culturia. They wrote about such project sever= al times on the guile-users mailing list.

It took days to down= load. And it took me months to produce a working version of this code. Beca= use I don't know Guile very well and maybe I'm not that smart overa= ll.

Anyway, now I have a COMPLETE graph of the dependencies of= jquery 3.1.1

These pictures are very informative indeed. I will = try to be brief, but I quickly wanted to share
that I find your efforts= (and results!) amazing.
=C2=A0

It's made of
47311 vertices and
324569 e= dges

I made a graph of a subset (graphviz chocked a bit on thi= s ;-) )
It's here
http://catonano.altervista.org/grafo.sv= g

There are 448 "broken" packages. Probably I sh= ould explain how I classified packages as "broken". Maybe not in = this email.

Anyway, these broken packages pose a challeng= e to the mission of porting Jquery into Guix, in my opinion, and they shoul= d be considered with some attention.
Here's the list
<= a href=3D"http://catonano.altervista.org/broken_packages.txt" target=3D"_bl= ank">http://catonano.altervista.org/broken_packages.txt
=C2=A0
Something of note: a b= ig chunk of the packages you classified as being 'broken' are
making me recall some (unpleasant) memories; From my own crawling = experiments,
which were not nearly as complete as this one, = I also ran into a lot of the same
show-stoppers. In a very r= eal sense, resolving node dependencies quickly devolves into
= resolving dependencies for most of the popular build systems as well as plu= gins like
broccoli-funnel. What I am trying to get at here is that fixin= g and packaging these 448 packages
will likely contribute a lot to= wards making an npm-enriched guix possible.
=C2=A0


There are 1314 packages with NO dependencies that could be = used as starting points in porting Jquery into Guix.
Here's th= e list
http://catonano.altervista.org/broken_packages.txt=

These could probably make use = of the npm importer I worked on earlier. Do you make use of your own? Other= wase I'll get to
rebasing my version on the current guix= master branch.
=C2=A0


If there's anyone intere= sted, I can give you the data folder so you can try all the queries you wan= t on these data without having to to run this thing for a bunch of hours

If possible, yes please. What would= be the most convenient way you to share this data?
=C2=A0

In the future, I'd like to run this thing on some other packag= e and merge the graphs so I will be able to investigate which are the commo= n fundamental dependencies for SEVERAL important packages in Nodejs.
So if someone wants to dedicate time to porting Nodejs stuff in= Guix they will be able to select most urgent packages to start from.

The same could be said of broken packages taht affect several = important packages.

The porting of Nodejs in Guix cannot be done wit= h brute strength. A data oriented approach can help, in my opinion.

=
Indeed.
=C2=A0
The ideal woul= d be to have something that, like bitcoin, coordinates a swarm in such a wa= y that every node can contribute a tiniy bit of data to a common data struc= ture, so all the nodes would have a complete copy of the database.

<= /div>
Collecting a mantaining of datasets should be freed of the client= server model too. Not only the social media.

=
I have no idea what you are referring to. Cou= ld you please elaborate a bit at a later point in time?
=C2= =A0
But that's more than I can handle, anyway.

I am already thankful
<= /div>

I'd like to talk about the stumbling blocks I run= into to discuss Guile and my knowledge of it.

For exampl= e, I can't use that thing in the autotools that processes configure.am files so I just forke= d amz3's project and added my files in there. As guests. Thanks amz3 !<= br>

I'd also like to describe te screw ups in = the format I put the data into. I realized my mistakes when hours of crunch= ing had already been done.

They can be migrated to a bett= er format though

If you don't mind, I will discuss th= ese issue in the future, not now.

One last f= un fact: while I was watching the output flowing in my terminal, I saw a pa= ckage called

"broccoli-funnel"

No, real= ly. It's here
https://registry.npmjs.org/broccoli-funnel
<= /div>

Ok, that's all for now.

Thanks again for yo= ur efforts on this. I am looking forward to working with your data.

=
Regards,
Jelle Licht
--94eb2c074e7e823bf3054678ed8d--