From mboxrd@z Thu Jan 1 00:00:00 1970 From: David Thompson Subject: Re: [GSoC] Integrating npm packages into the Guix ecosystem Date: Wed, 23 Mar 2016 09:06:10 -0400 Message-ID: <87shzhbayl.fsf@izanagi.i-did-not-set--mail-host-address--so-tickle-me> References: Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:34358) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aiiUK-0008Pl-UP for guix-devel@gnu.org; Wed, 23 Mar 2016 09:06:21 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aiiUG-00047h-Ly for guix-devel@gnu.org; Wed, 23 Mar 2016 09:06:16 -0400 Received: from mail-qg0-x22f.google.com ([2607:f8b0:400d:c04::22f]:36746) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aiiUG-00047F-D1 for guix-devel@gnu.org; Wed, 23 Mar 2016 09:06:12 -0400 Received: by mail-qg0-x22f.google.com with SMTP id u110so10898370qge.3 for ; Wed, 23 Mar 2016 06:06:12 -0700 (PDT) In-Reply-To: List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: Jelle Licht , guix-devel@gnu.org Jelle Licht writes: > Hello Guix, > > Seeing as this time next year I'll be finishing up my studies, this was a > now-or-never moment for me. I would love to spend this summer hacking on > both guix, and reading up on npm. I think this project is a great idea! Feedback inline below. > #+AUTHOR: Jelle Licht, jlicht@fsfe.org > #+DATE: 21-03-2016 > #+OPTIONS: toc:nil >=20 > * Overview >=20 > This project will allow Guix hackers to more easily package software that= is > distributed through the Node Package Manager (npm), as well as allowing N= ode > developers on Guix to make use of the reproducible builds guarantee of Gu= ix. >=20 > After completing this project, it should be possible to easily make use o= f the > less-problematic packages in the npm registry on the Guix Software Distri= bution. I assume that by =E2=80=9Cless-problematic=E2=80=9D you are referring to th= e packages that actually upload their corresponding source code and not some minified blob. Maybe also the ones with the least insane dependency graph. Is this correct? > * Project structure >=20 > Depending on findings in the early stages of the project, I foresee the > following distinct parts: >=20 > 1. Extend Guix so it can 'simulate' the dependency graph generation of bo= th the > old and new npm [fn:6]. I don=E2=80=99t quite get this part. Why does it matter how old and new versions of NPM store dependencies? IIRC, there=E2=80=99s a NODE_PATH environment variable that we could set to specify where to find each library. Even if we have to make a similar directory structure as NPM for some reason, shouldn=E2=80=99t we just do what NPM 3 does and forget about older versions? > 2. Extend guix with an algorithm that matches npm's package.json flexible > version specification to a specific version. This sounds like it could be part of =E2=80=99guix import npm=E2=80=99, but= not in the Guix package recipes themselves. To date, we do not have a single importer that recursively imports packages, so this would be somewhat new territory. > 3. Add a ~guix import~ backend for the npm registry Yup. :) > 4. Package npm modules in guix Could you perhaps come up with an application or library to target for packaging that will exercise all of the code you will be adding to Guix? As you probably know, many non-trivial Node modules typically have *massive* dependency graphs (hundreds of nodes), so I think we need to be strategic in what we pick to package. I should also note that NPM, like other language package management systems, obscures the full dependency graph of a module by differentiating between =E2=80=9Cdependencies=E2=80=9D and =E2=80=9Cdevelop= ment dependencies=E2=80=9D. In Guix, we build from source, so we need those =E2=80=9Cdevelopment=E2=80= =9D (misnomer: they are really build-time dependencies, but apparently only a developer would want to build from source) dependencies, too. Ideally, we would like to run test suites for node modules as well. Unfortunately, npm packages are viewed more like binaries, so they strip out things like tests. It=E2=80=99s an open question if the tarballs avail= able on npmjs.com are worth anything at all, or if we should instead build from the upstream source code releases. Unfortunately again, those upstream releases are usually just auto-generated tarballs from git tags on GitHub, which may prove problematic because no one actually uses them. Long story short, we need to be diligent about ensuring that we are truly building from source. :) > 5. Interface >=20 > Right now, dependency resolution in npm is as stateful as can be, with ev= en the > installation order [fn:6] making a difference for where npm expects to fi= nd a > certain dependency. As two different dependency resolution mechanism are = in use, > of which especially the newer one is problematic because of its habit of > propagating dependencies upwards in the folder structure, both have to be > supported by a guix module. As I mentioned earlier, I don=E2=80=99t think that we need to support this = at all. We should just arrange that NODE_PATH be configured correctly. We=E2=80=99re *replacing* NPM, after all, so we should use a more sane architecture if we can. Does that sound doable or are there technical reasons why it wouldn=E2=80=99t work? =20 > npm uses SemanticVersioning range patterns [fn:7] to declare dependencies > between packages. A problem with this approach is that the same package > declaration can lead to an entirely different dependency graph, which def= eats > the purpose of having a system with a focus on reproducible builds. If npm > packages are to be used, these version numbers need to be locked down to a > specific version, corresponding to the version that npm would install if = left to > its devices. A potential problem is that the entire dependency graph has = to be > known ahead of time in order to pinpoint a 'correct' version of the depen= dency. > A consistent ordering for 'installing' dependencies also has to be decide= d upon. This is indeed a problem that needs solving *if* we consider recursive package imports a requirement for this project. Recursive imports haven=E2=80=99t been done before, so maybe this could be a =E2=80=9Cnice to= have=E2=80=9D part of the project rather than a =E2=80=9Cmust have=E2=80=9D. This functionali= ty would be part of =E2=80=99guix import npm=E2=80=99. > After the dependency resolution has been worked out, it should be possibl= e to > create a guix import backend to leverage the code that has been produced = up till > now to allow the packaging of npm modules. The last part of the project > essentially serves as a starting point for packaging up useful npm packag= es. >=20 > A stretch goal for the summer would be to create a guix build system for a > subset of npm packages, such as gulp. I would promote this from a stretch goal to a =E2=80=9Cmust have=E2=80=9D. = One of the first steps of this project should be to create (guix build-system node) and (guix build node-build-system) modules that will do whatever the standard build+test+install process is for node modules. From what I=E2=80= =99ve seen, a lot of node modules need gulp or grunt in order to build, so boostrapping one or both of those tools will probably be necessary in order to package anything substantial. =20 > * Planning >=20 > As I am currently a novice with regards to the internals of guix and the = guile > programming language, up to the start of the actual project I will mostly= be > reading up and hacking on guix. As such, the planning becomes: >=20 > April 22 - May 22: > - Getting to know Guix(SD) > - Package programs using the guix import module > - An informal specification of the npm dependency resolution mechanism > - Getting the know the guix community and what everyone is working on > - Getting familiar with the contributing work flows >=20 > May 23 - Jun 5: > - Formal specification to allow guix to simulate npm dependency resolut= ion (1) >=20 > Jun 6 - Jun 19: > - npm Version pinning should be working (2) > - start working on the guix import backend >=20 > Jun 20 - Jul 3:=20 > - Guix import backend should be finished by now (3) > - Start testing npm packages=20 >=20 > Jul 4 - Jul 10: > - Holidays!=20 >=20 > Jul 11 - Aug 7: > - Solve any problems and corner cases with building and installing npm > packages (4) >=20 > Aug 8 - Aug 23: > - If life goes a planned, Get all contributions ready to be merged back= in the > main Guix codebase. Please add to the timeline when you would expect to have node-build-system implemented. I may have some code laying around that could accelerate this a bit. > * About me >=20 > My name is Jelle Licht, and I am currently studying Data Science at the > University of Technology Delft. I finished my BSc in Computer Science in = 2015, > at the University of Technology Delft as well. Somewhere in the second se= mester > of my studies, my roommate was using Vi in front of me, and since then I'= ve been > falling into the rabbit hole that is GNU/Linux, free software and the 'op= en > source' community. Somewhere along the line I became a huge fan of Clojur= e, and > for a short while was a member of the very much unofficial Lisp Community= Delft. >=20 > I have been a small-time contributor to some free software projects on gi= thub, > as well as having a day job to make ends meet using mostly Node. My online > handles include 'wordempire', as I am quite fond of reading, and 'jlicht'= . For > the past month, I have been lurking on-and-off again in the #guix irc cha= nnel, > as well as reading up on some of the motivations behind reproducible rese= arch > and reproducible builds. >=20 > Besides one week for visiting family and general holidays, this project w= ould be > my full time focus during the summer. Thank you for your proposal! Making JavaScript software packagable in Guix would be a *huge* advance in our quest for reproducible web applications. - Dave