bringing npm packages to Guix

bringing npm packages to Guix

[zamfofex]

Hello, Guix!

A few months ago, I picked up the work towards importing npm packages to Guix by Jelle et al. in the hopes of continuing it, and I felt disheartened when I concluded that it does indeed seem like a very large part of npm is necessary to build even a simple package like jQuery.

But, recently, I decided to spend some more time thinking about it, and I realized that not all of the ‘devDependencies’ of a given package are actually necessary to build it. A lot of packages have ‘devDependencies’ for things such as tests and watching the filesystem for changes for ease of development.

If you think about it, what the vast majority of npm packages need to build successfully comes down to (optionally) compile TypeScript, then (optionally) perform some kind of bundling or transpilation. And that doesn’t really require TypeScript or a build tool from npm, since esbuild (which is already packaged) can do both of those things by itself.

Now, taking away the ‘devDependencies’ when recursing the dependency tree makes the whole endeavor of importing an npm package seem *much less* difficult and scary! The biggest unfortunate issue is that for packages with a build step (TypeScript and/or bundling+transpilation), it is necessary to use esbuild ad‐hoc (i.e. in a case-by-case basis), because each package has its own build peculiarities, and esbuild doesn’t acknowledge them. (One solution could be to create an esbuild wrapper that mimics ‘tsc’ and other tools.)

With that approach in mind, I was able to package sucrase as a proof of concept! Note that sucrase doesn’t have a lot of transitive dependencies, so it was easy to just modify the phases of each package in an ad‐hoc way. However, also note that sucrase depends on itself, so I decided to bootstrap it with esbuild, then use the bootstrapped version to build sucrase again. (The bootstrapped sucrase seems to also work as expected!)

The import script I wrote, alongside the sucrase package itself can be found here:
- <https://gist.github.com/zamfofex/eac93bc0e00477a8b79f5ca4dc1a34ff>

Any kinds of thoughts or feedback on this would be appreciated!

A few notes of potential future work:

- Keep tests on packages, after packaging the necessary test libraries.
- Maybe: Write a wrapper for esbuild for mimicking ‘tsc’ and other tools. (As mentioned above.)
- Figure out a way to detect vendored dependencies and other forms of bundles in repositories.
- Polish the import script, and make it fit Guix’ style and formatting practices.
- Use the ‘version’ variable in the source field of package definitions when importing. (An extension of the item above.)

Re: bringing npm packages to Guix

[Ludovic Courtès]

Hi,

zamfofex <zamfofex@twdb.moe> skribis:

> A few months ago, I picked up the work towards importing npm packages to Guix by Jelle et al. in the hopes of continuing it, and I felt disheartened when I concluded that it does indeed seem like a very large part of npm is necessary to build even a simple package like jQuery.

Christine had that great summary of the npm mess at the time:

  https://dustycloud.org/blog/javascript-packaging-dystopia/

With all its shortcomings, I think it would still be useful to merge
Jelle’s npm importer (the “binary” importer), as discussed last year:

  https://lists.gnu.org/archive/html/guix-devel/2021-09/msg00245.html

Thumbs up to whoever submits it for inclusion!  :-)

(I won’t answer on ‘devDependencies’ issues because JS is outside my
area of expertise.)

Thanks,
Ludo’.

Re: bringing npm packages to Guix

[zamfofex]

Hello! Thanks for responding.

The premise of the idea I’m proposing is that it wouldn’t be necessary to traverse into (most of) the ‘devDependencies’ if you build the packages ad‐hoc using esbuild. So, the idea is to sidestep the build process that the package has, and just invoke esbuild on the appropriate files of the package. My hope is that this is sufficient to bring most (or at least a lot of) useful packages from npm to Guix without the need of a binary importer.

I set up a package for sucrase as a proof of concept, since sucrase is a CLI tool that can be run and tried out easily (as opposed to jQuery, which is a library meant to be used in browsers). However, note that following my proposed approach, packaging jQuery becomes entirely trivial — it is a matter of simply running esbuild on jQuery’s input files. For the sake of example, I have included a Guix package I created for jQuery in the GitHub Gist I sent above. (And I can anecdotally verify that it works when tring it on a browser.) <https://gist.github.com/zamfofex/eac93bc0e00477a8b79f5ca4dc1a34ff#file-jquery-scm>

I will admit that I cheated a bit, since I used the currently unreleased 4.0.0 version, as opposed to the latest released 3.x versions, since the realeased 3.x versions use the old and outdated AMD format (“asynchronous module definition”) for representing dependencies between files in a project. (See <https://github.com/evanw/esbuild/issues/1035>) However, most packages nowadays have already moved away from AMD (and jQuery is getting there), so this shouldn’t be an issue in practice.

Re: bringing npm packages to Guix

[Mekeor Melire]

2022-11-19 18:27 zamfofex@twdb.moe:

> Hello, Guix!

Hello :)

> A few months ago, I picked up the work towards importing npm packages to 
> Guix by Jelle et al. in the hopes of continuing it, and I felt disheartened 
> when I concluded that it does indeed seem like a very large part of npm is 
> necessary to build even a simple package like jQuery.

Thanks for bringing this back up!

Although, I work with the node/npm ecosystem on a daily basis, unfortunately, 
I don't really understand why it's so difficult to import npm-packages into 
Guix.

> But, recently, I decided to spend some more time thinking about it, and I 
> realized that not all of the ‘devDependencies’ of a given package are 
> actually necessary to build it. A lot of packages have ‘devDependencies’ for 
> things such as tests and watching the filesystem for changes for ease of 
> development.

Is importing npm-packages into Guix difficult because npm-packages have so 
many dependencies?

> If you think about it, what the vast majority of npm packages need to build 
> successfully comes down to (optionally) compile TypeScript, then 
> (optionally) perform some kind of bundling or transpilation. And that 
> doesn’t really require TypeScript or a build tool from npm, since esbuild 
> (which is already packaged) can do both of those things by itself.

Is importing npm-packages into Guix so difficult because we have a 
chicken-and-egg problem, since many build-time dependencies (like TypeScript) 
have so many dependencies themselves?

Also, is there a reason to choose esbuild over rust-swc (or vice versa)?

> Now, taking away the ‘devDependencies’ when recursing the dependency tree 
> makes the whole endeavor of importing an npm package seem *much less* 
> difficult and scary! The biggest unfortunate issue is that for packages with 
> a build step (TypeScript and/or bundling+transpilation), it is necessary to 
> use esbuild ad‐hoc (i.e. in a case-by-case basis), because each package has 
> its own build peculiarities, and esbuild doesn’t acknowledge them. (One 
> solution could be to create an esbuild wrapper that mimics ‘tsc’ and other 
> tools.)

Could we also build TSC with Esbuild, and then use TSC to build packages that 
depend on it, instead of mimicking TSC with Esbuild? Does that make sense?

With "mimicking TSC", do you mean that Esbuild does not transpile multiple 
.ts-files at once? (By the way, Esbuild seems to make use of the tsconfig.json 
file, according to <https://esbuild.github.io/api/#tsconfig>.)

> With that approach in mind, I was able to package sucrase as a proof of 
> concept! Note that sucrase doesn’t have a lot of transitive dependencies, so 
> it was easy to just modify the phases of each package in an ad‐hoc way. 
> However, also note that sucrase depends on itself, so I decided to bootstrap 
> it with esbuild, then use the bootstrapped version to build sucrase again. 
> (The bootstrapped sucrase seems to also work as expected!)

That's so great!

Sorry for asking nooby questions. I just want to catch up and press ahead with 
the discussion.

Re: bringing npm packages to Guix

[Joshua Branson]

Mekeor Melire <mekeor@posteo.de> writes:

> 2022-11-19 18:27 zamfofex@twdb.moe:
>
>> Hello, Guix!
>
> Hello :)
>
>> A few months ago, I picked up the work towards importing npm packages to Guix
>> by Jelle et al. in the hopes of continuing it, and I felt disheartened when I
>> concluded that it does indeed seem like a very large part of npm is necessary
>> to build even a simple package like jQuery.
>
> Thanks for bringing this back up!
>
> Although, I work with the node/npm ecosystem on a daily basis, unfortunately, I
> don't really understand why it's so difficult to import npm-packages into Guix.
>
>> But, recently, I decided to spend some more time thinking about it, and I
>> realized that not all of the ‘devDependencies’ of a given package are actually
>> necessary to build it. A lot of packages have ‘devDependencies’ for things
>> such as tests and watching the filesystem for changes for ease of development.
>
> Is importing npm-packages into Guix difficult because npm-packages have so many
> dependencies?
>

Yes.  The recursive javascript importer that was written, generates
package definitions that are larger than the source code for the
packages.  It's quite a hairy problem.

>> If you think about it, what the vast majority of npm packages need to build
>> successfully comes down to (optionally) compile TypeScript, then (optionally)
>> perform some kind of bundling or transpilation. And that doesn’t really
>> require TypeScript or a build tool from npm, since esbuild (which is already
>> packaged) can do both of those things by itself.
>
> Is importing npm-packages into Guix so difficult because we have a
> chicken-and-egg problem, since many build-time dependencies (like TypeScript)
> have so many dependencies themselves?
>
> Also, is there a reason to choose esbuild over rust-swc (or vice versa)?
>
>> Now, taking away the ‘devDependencies’ when recursing the dependency tree
>> makes the whole endeavor of importing an npm package seem *much less*
>> difficult and scary! The biggest unfortunate issue is that for packages with a
>> build step (TypeScript and/or bundling+transpilation), it is necessary to use
>> esbuild ad‐hoc (i.e. in a case-by-case basis), because each package has its
>> own build peculiarities, and esbuild doesn’t acknowledge them. (One solution
>> could be to create an esbuild wrapper that mimics ‘tsc’ and other tools.)
>
> Could we also build TSC with Esbuild, and then use TSC to build packages that
> depend on it, instead of mimicking TSC with Esbuild? Does that make sense?
>
> With "mimicking TSC", do you mean that Esbuild does not transpile multiple
> .ts-files at once? (By the way, Esbuild seems to make use of the tsconfig.json
> file, according to <https://esbuild.github.io/api/#tsconfig>.)
>
>> With that approach in mind, I was able to package sucrase as a proof of
>> concept! Note that sucrase doesn’t have a lot of transitive dependencies, so
>> it was easy to just modify the phases of each package in an ad‐hoc way.
>> However, also note that sucrase depends on itself, so I decided to bootstrap
>> it with esbuild, then use the bootstrapped version to build sucrase again.
>> (The bootstrapped sucrase seems to also work as expected!)
>
> That's so great!
>
> Sorry for asking nooby questions. I just want to catch up and press ahead with
> the discussion.