From mboxrd@z Thu Jan 1 00:00:00 1970 From: Timothy Sample Subject: Non-bootstrappable NPM packages Date: Wed, 24 Jul 2019 09:23:42 -0400 Message-ID: <877e87efc1.fsf@ngyro.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:470:142:3::10]:38824) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hqHFE-0004Y3-VL for guix-devel@gnu.org; Wed, 24 Jul 2019 09:23:50 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hqHFD-0005sa-9M for guix-devel@gnu.org; Wed, 24 Jul 2019 09:23:48 -0400 Received: from wout1-smtp.messagingengine.com ([64.147.123.24]:36073) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hqHFC-0005qQ-Uc for guix-devel@gnu.org; Wed, 24 Jul 2019 09:23:47 -0400 List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: guix-devel@gnu.org Hi Guix, (I=E2=80=99m CC=E2=80=99ing swedebugia and Jelle on this, since I promised = both of them off-list that I would publish this work eventually.) Most of us know that the JavaScript situation with Guix is a little dire. Particularly, some JavaScript packages have circular dependencies and most are not bootstrappable. In practice, this has led to Guix having almost no JavaScript packages, making it not suitable for JavaScript-heavy Web development. It also means that packaging high-profile, Web-based programs like GitLab or Mastodon is so difficult as to be impossible. Given these obstacles, I thought it would be interesting to explore how far you can get if you stop worrying about bootstrapping the packages, and just use the pre-built ones. It turns out that you can go as far as you like this way. In fact, I=E2=80=99ve built a recursive importer and build-system for NPM that can import and =E2=80=9Cbuild=E2=80=9D large and = complicated packages such as Babel. It does this by downloading the pre-built tarballs from NPM, and then installing them using the =E2=80=9Cnpm=E2=80=9D= command with some dependency rewriting. It=E2=80=99s actually pretty simple. The main t= rick was finding the right flags to pass to NPM, and figuring out how to trick it into pulling dependencies from the store rather than the Web. The code is at . I know that this is not great for Guix proper, but I thought that this might be a good use-case for channels. In a channel we could build out the JavaScript ecosystem in Guix on top of a bunch of =E2=80=9Cbinary=E2=80= =9D packages, and then try to reduce this set of =E2=80=9Cbinary=E2=80=9D packages gradua= lly. I think that having a useful JavaScript ecosystem, even if it is problematic, might help motivate folks to work out the problems. For example, it would be a lot more exciting to work on bootstrapping TypeScript if it were the only thing standing in the way of getting a Mastodon service in Guix. That=E2=80=99s the high-level overview. The rest of this message is the technical details of how to try out the importer, build packages, etc. Here=E2=80=99s how to make =E2=80=9C@babel/cli=E2=80=9D work with a Babel p= lugin called =E2=80=9Ctransform-arrow-functions=E2=80=9D. Clone the repo and build it like you would build Guix normally, except for where you would usually use =E2=80=9Cguix environment guix=E2=80=9D. I= nstead, use guix environment guix --ad-hoc -l guile-semver.scm to ensure that the Semantic Versioning library required by the importer is available. After this, continue as usual. Once built, create a new file at =E2=80=9Cgnu/packages/node-babel.scm=E2=80= =9D with the following contents: (define-module (gnu packages node-babel) #:use-module (guix build-system other-node) #:use-module (guix download) #:use-module ((guix licenses) #:prefix license:) #:use-module (guix packages)) Then, you can run ./pre-inst-env guix import npm-binary -r @babel/cli \ | tee -a gnu/packages/node-babel.scm (using =E2=80=9Ctee=E2=80=9D is nice =E2=80=99cause you can watch the outpu= t as it goes =E2=80=93 it has a lot of dependencies, so it takes a little while). Get the plugin, too: ./pre-inst-env guix import npm-binary -r \ @babel/plugin-transform-arrow-functions \ | tee -a gnu/packages/node-babel.scm There=E2=80=99s a package in there that is macOS specific, so it has to be removed. It=E2=80=99s called =E2=80=9Cfsevents=E2=80=9D, and it is depende= ncy of =E2=80=9Cchokidar=E2=80=9D. Everything works fine if it is simply deleted. At this point, you can launch an environment with the packages (be sure to include =E2=80=9Cnode=E2=80=9D, as that reminds Guix to set =E2=80=9CNOD= E_PATH=E2=80=9D): ./pre-inst-env guix environment --ad-hoc node node-babel-cli--fiio \ node-babel-plugin-transform-arrow-functions--fiio (For the curious, =E2=80=9Cfiio=E2=80=9D is an initialism for a phrase used= in Christopher Lemmer Webber=E2=80=99s article about JavaScript packaging: . It serves as a reminder that these are not nice packages.) Unfortunately, Babel only looks for plugins relative to the current directory, so you=E2=80=99ll need a symlink: =E2=80=9Cln -s $NODE_PATH=E2= =80=9D. With all this in place, it=E2=80=99s time for a test! echo '() =3D> { return true; };' > test.js babel --plugins=3D@babel/plugin-transform-arrow-functions test.js This prints: (function () { return true; }); which is good, since it is transforming the modern lambda syntax with the arrow =E2=80=9C=3D>=E2=80=9D into the old syntax with the =E2=80=9Cfunc= tion=E2=80=9D keyword. Hooray! Now for some more notes. I=E2=80=99ve tried a handful of packages, and have had good results so far. Sometimes things need a little fiddling (like =E2=80=9Cfsevents=E2=80=9D), = but the results are usable. Both the importer and the build system are mostly new code. For the importer, the main difference is that it uses the NPM version constraints to make sure the recursively imported packages have the expected versions. It does this via a Guile library that I wrote. One limitation is that the recursive importer does not check versions for existing Guix packages. If there is a package with the right name, it is accepted even if it doesn=E2=80=99t satisfy the version constraints. Gu= ix=E2=80=99s =E2=80=9Crecursive-import=E2=80=9D function would have to be changed to fix= this. The build system uses =E2=80=9Cnpm=E2=80=9D for everything, rather than try= ing to emulate =E2=80=9Cnpm=E2=80=9D in Scheme (like the recently added build syst= em does). I=E2=80=99m not sure if this is better or worse, really, but it works nicel= y for this particular use-case. It works in theory to build packages from source, too. One test I did was importing all of jQuery=E2=80=99s developm= ent dependencies as binary packages, and then building jQuery from source. I=E2=80=99ve come to think that bootstrapping JavaScript might be easier th= an it looks. As time goes on, Node gets better at the newer JavaScript features. This removes the need for things like Babel or Rollup, since with some care, Node can run the source directly with out any transformations or bundling. That being said, TypeScript looks to be a major issue, as it is used in many fundamental JavaScript packages and it is not bootstrappable. I=E2=80=99m not sure in what capacity I want to pursue this. It=E2=80=99s = been sitting dormant on my computer for while, so I thought sharing it would be better than letting it fall by the wayside. I hope it proves useful one way or another. If you got this far, thanks for reading! :) -- Tim