From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jelle Licht Subject: Re: [GSoC] Integrating npm packages into the Guix ecosystem Date: Thu, 24 Mar 2016 01:45:42 +0100 Message-ID: References: <87shzhbayl.fsf@izanagi.i-did-not-set--mail-host-address--so-tickle-me> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary=001a1142057a1ea7f2052ec0c4b7 Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:43454) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aitPQ-0002RD-MX for guix-devel@gnu.org; Wed, 23 Mar 2016 20:45:58 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aitPL-0004pk-LD for guix-devel@gnu.org; Wed, 23 Mar 2016 20:45:56 -0400 Received: from cavendish.fsfeurope.org ([2001:aa8:ffed::3:102]:54303) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aitPL-0004pe-5v for guix-devel@gnu.org; Wed, 23 Mar 2016 20:45:51 -0400 Received: from localhost (localhost [127.0.0.1]) by cavendish.fsfeurope.org (Postfix) with ESMTP id DA17D63B9E5 for ; Thu, 24 Mar 2016 01:45:48 +0100 (CET) Received: from cavendish.fsfeurope.org ([127.0.0.1]) by localhost (cavendish.fsfeurope.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 06xWmjuiVJQG for ; Thu, 24 Mar 2016 01:45:45 +0100 (CET) Received: from mail-lf0-f51.google.com (mail-lf0-f51.google.com [209.85.215.51]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) (Authenticated sender: jlicht) by cavendish.fsfeurope.org (Postfix) with ESMTPSA id 1B9DA63B9F2 for ; Thu, 24 Mar 2016 01:45:45 +0100 (CET) Received: by mail-lf0-f51.google.com with SMTP id q73so20926679lfe.2 for ; Wed, 23 Mar 2016 17:45:45 -0700 (PDT) In-Reply-To: <87shzhbayl.fsf@izanagi.i-did-not-set--mail-host-address--so-tickle-me> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: David Thompson Cc: guix-devel@gnu.org --001a1142057a1ea7f2052ec0c4b7 Content-Type: text/plain; charset=UTF-8 Thanks for the feedback. You are correct in your interpretation of my description. Some packages only publish a blob, and while this should eventually also be handled gracefully, a first working version of an importer would surely be easier to realize if we focus on the more well-behaved packages at first. Using NODE_PATH could work, but in my mind it would just make managing all these paths more complex, compared to just symlinking the correct packages. Symlinking might lead to some problems with '..' not evaluating to the right place. Some research for an approach that works for us might still be required, but it seems that while the location of where npm places the packages has changed since npm 3, how node loads modules at run-time seems to be more or less the same. Any scheme we come up with that allows node to run the code would suffice, indeed, as we wouldn't use or depend on npm's package management functionalities. The latest folder structure as generated by npm seems as good as any starting point though. Seeing as testing would be an important part to ensure that our builds are functioning correctly, I would propose that a first goal would be the mocha package. As far as I can see, it has only 15 unique dependencies, none of which seem too big to grok. After having a functioning test runner, we could choose a package that makes use of the test runner. Which packages would be most suitable, I do not know right now, but I am open to any suggestions/wish-lists. Another issue that you highlighted is of course the fact that, to us, npm is more of a binary distribution platform, than an actual repository of code artifacts. I am well aware of this fact, but in theory there is no reason a checkout via git should not suffice. Whether it will be possible to properly pinpoint the checkout for a specific version of each project, and if these sources are in a functional state, could worst-case scenario be solved with some elbow-grease when packaging them up. If recursive imports are not deemed a must-have, maybe we can bump this down to the stretch goals? It would make sense to chart new territory after having some experience with Guix under my belt as well ;-) A potential problem with Gulp, and I am assuming the situation with Grunt is not much better, is that it (indirectly) depends on an enormous list of dependencies. I have been keeping half an eye on this mailing-list, especially with regards to what to do with some of the 'bootstrap-binaries' for Haskell and other languages. If possible, I would like to not add more fuel to this fire, but packaging up all the dependencies of the de facto build tools might prove to be too monumental a task for now. In which case, a temporary solution using a bootstrapped Gulp/Grunt/both might work. If possible, some advice from people involved with similar situations would be nice. The node-build-system seems doable, depending on what we need to cover. Node-gyp might take a bit of time to properly support, but I will look into this tomorrow. Most other build steps for npm modules include running some external program, either binary or a globally installed node package, and shuffling some files around. When we get the popular build systems to work, the hardest part (w.r.t. the build system) will have been taken care of. I will send a revised planning tomorrow. ~Jelle [fn:1] https://docs.npmjs.com/misc/scripts --001a1142057a1ea7f2052ec0c4b7 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks for the feedback.

You are c= orrect in your interpretation of my description. Some packages only
publ= ish a blob, and while this should eventually also be handled gracefully, a<= br>first working version of an importer would surely be easier to realize i= f we
focus on the more well-behaved packages at first.

Using NODE= _PATH could work, but in my mind it would just make managing all these
p= aths more complex, compared to just symlinking the correct packages. Symlin= king
might lead to some problems with '..' not evaluating to the= right place. Some
research for an approach that works for us might stil= l be required, but it seems
that while the location of where npm places = the packages has changed since npm
3, how node loads modules at run-time= seems to be more or less the same. Any
scheme we come up with that allo= ws node to run the code would suffice, indeed,
as we wouldn't use or= depend on npm's package management functionalities. The
latest fold= er structure as generated by npm seems as good as any starting point
tho= ugh.

Seeing as testing would be an important part to ensure that our= builds are
functioning correctly, I would propose that a first goal wou= ld be the mocha
package. As far as I can see, it has only 15 unique depe= ndencies, none of which
seem too big to grok.

After having a func= tioning test runner, we could choose a package that makes use
of the tes= t runner. Which packages would be most suitable, I do not know right
now= , but I am open to any suggestions/wish-lists.

Another issue that yo= u highlighted is of course the fact that, to us, npm is
more of a binary= distribution platform, than an actual repository of code
artifacts. I a= m well aware of this fact, but in theory there is no reason a
checkout v= ia git should not suffice. Whether it will be possible to properly
pinpo= int the checkout for a specific version of each project, and if these
so= urces are in a functional state, could worst-case scenario be solved with s= ome
elbow-grease when packaging them up.

If recursive imports ar= e not deemed a must-have, maybe we can bump this down to
the stretch goa= ls? It would make sense to chart new territory after having some
experie= nce with Guix under my belt as well ;-)

A potential problem with Gul= p, and I am assuming the situation with Grunt is not
much better, is tha= t it (indirectly) depends on an enormous list of
dependencies. I have be= en keeping half an eye on this mailing-list, especially
with regards to = what to do with some of the 'bootstrap-binaries' for Haskell andother languages. If possible, I would like to not add more fuel to this fi= re,
but packaging up all the dependencies of the de facto build tools mi= ght prove to
be too monumental a task for now. In which case, a temporar= y solution using a
bootstrapped Gulp/Grunt/both might work. If possible,= some advice from people
involved with similar situations would be nice.=

The node-build-system seems doable, depending on what we need to co= ver. Node-gyp
might take a bit of time to properly support, but I will l= ook into this
tomorrow. Most other build steps for npm modules include r= unning some external
program, either binary or a globally installed node= package, and shuffling some
files around. When we get the popular build= systems to work, the hardest part
(w.r.t. the build system) will have b= een taken care of.

I will send a revised planning tomorrow.

=
~Jelle
--001a1142057a1ea7f2052ec0c4b7--