From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jelle Licht Subject: [GSoC] Integrating npm into the Guix ecosystem Date: Tue, 07 Jun 2016 02:56:34 +0200 Message-ID: <87bn3dizp9.fsf@gmail.com> Mime-Version: 1.0 Content-Type: text/plain Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:46181) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bA5Jx-0000vT-2u for guix-devel@gnu.org; Mon, 06 Jun 2016 20:56:42 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bA5Js-0004bA-Um for guix-devel@gnu.org; Mon, 06 Jun 2016 20:56:40 -0400 Received: from mail-wm0-f43.google.com ([74.125.82.43]:35269) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bA5Js-0004as-Kq for guix-devel@gnu.org; Mon, 06 Jun 2016 20:56:36 -0400 Received: by mail-wm0-f43.google.com with SMTP id n206so11743890wmf.0 for ; Mon, 06 Jun 2016 17:56:36 -0700 (PDT) Received: from veritas (50709B36.static.ziggozakelijk.nl. [80.112.155.54]) by smtp.gmail.com with ESMTPSA id a81sm16689249wma.16.2016.06.06.17.56.34 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 06 Jun 2016 17:56:34 -0700 (PDT) List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: guix-devel Greetings Guix hackers, It has been some time since my last mail to this list, so I wanted to share what I have been up to. For the people who might want to watch along after today, I will be posting the changes that should not break everything immediately to [1]. (Apologies for it being web-only, my sysadminfu is still lacking. If you know how to easily and most of all securely expose a git repo for read-only anonymous checkouts, please do tell). These first two weeks, I have mostly been keeping busy with writing some of the features that will quickly allow me to identify problems with the import and build procedures as I envision them. I was in luck; there were many of these problems as I found out ;-). One of the things that kind of worried me for a while was the fact that npm was broken after my first attempts at writing _anything_. At this moment, still being somehow intimidated by the guix code base, I had quite some trouble finding out what was happening; it turns out somewhere along the way, the way Node was built since version 6 (using guix, at least) has changed, making the `npm' in the PATH of your guix profile a _copy_ instead of a symlink to the actual executable `npm-cli.js' script. A patch to fix this is currently still brewing, so expect to see it soon. In order to continue a bit (and implement something that actually works), up till now I have worked with the 5.10 release of node. After toiling a bit to find a nice way of getting node to find dependencies, I started out with the NODE_PATH variable. Although some alarming messages I read some time ago indicated this method of allowing node to load modules is discouraged, but not deprecated. At first I wanted to make use of a variant of the npm command to actually take care of installing node modules to the store outputs; some of the advantages include: - The 'correct' files are ignored/always included - The 'correct' symlinks are generated to executable scripts, man-pages and other documentation. - Automatically up to date with new conventions in npm-land While that all seemed (and still seems) quite nice, it happens to be the case that npm is quite particular about dependencies that it will accept. Patching the actual dependencies and versions in the `package.json' file was another option, but after some consideration and advice from my mentors I decided this would not work nicely in the long term. Instead, I want to implement the relevant set of behaviors in guix, of course allowing for our particular kind of dependency management. As such, the current installation phase mostly involves copying files to their expected locations. I added support (as an exercise in build system arguments) for 'global' installs[3], which as far as my current bare-bones implementation is concerned means properly symlinking executable scripts. In the (near-)future, I am looking into properly wrapping these scripts to make sure the amount of propagated inputs is kept to a minimum. As a sanity test, I packaged [4] in my own little scratchpad. If you want to follow along, it only takes 4 `guix import npm' invocations ;). The current importer functions as expected for the "90%" of packages that I tried. A problem that I ran into that I could not recognize as easily in other importers is the fact that the npm community only distributes the artifacts that you need to run the npm modules, but not to build them. In most trivial cases, there are literally no differences, but especially for more complicated packages involving transpilation steps, this poses a problem. As such, the importer can not actually 'know' of the location of the source. Right now it uses some limited heuristics to probe GitHub repositories for a tarball release, and if these are not found or the sources are hosted at non-GH sites, it tries to check out a tag according to the npm packaging conventions (SemVer). The most important thing that needs to happen right now would be to extend the range of packages that are buildable by the build system. A combination of working towards having working 'large' packages and test frameworks should help me quickly identify problems. This will be my main focus for the next week. I am aware that the current importer is quite brittle, so this needs to change as well. As I run into problems (or someone brings them to my attention), I will improve the quality of this subsystem as well. If you have some advice, questions or problems, please do not hesitate to reply. The same goes if you have some npm packages you would love to see packaged. Thanks a bunch for reading this WoT :) - Jelle [1]https://gitweb.jlicht.org/?p=guix.git;a=shortlog;h=refs/heads/gsoc-npm [2]https://github.com/nodejs/node/issues/1627 [3]https://docs.npmjs.com/getting-started/installing-npm-packages-globally [4]https://www.npmjs.com/package/package-json-validator