Non-bootstrappable NPM packages

Non-bootstrappable NPM packages

[Timothy Sample]

Hi Guix,

(I’m CC’ing swedebugia and Jelle on this, since I promised both of them
off-list that I would publish this work eventually.)

Most of us know that the JavaScript situation with Guix is a little
dire.  Particularly, some JavaScript packages have circular dependencies
and most are not bootstrappable.  In practice, this has led to Guix
having almost no JavaScript packages, making it not suitable for
JavaScript-heavy Web development.  It also means that packaging
high-profile, Web-based programs like GitLab or Mastodon is so difficult
as to be impossible.

Given these obstacles, I thought it would be interesting to explore how
far you can get if you stop worrying about bootstrapping the packages,
and just use the pre-built ones.  It turns out that you can go as far as
you like this way.  In fact, I’ve built a recursive importer and
build-system for NPM that can import and “build” large and complicated
packages such as Babel.  It does this by downloading the pre-built
tarballs from NPM, and then installing them using the “npm” command with
some dependency rewriting.  It’s actually pretty simple. The main trick
was finding the right flags to pass to NPM, and figuring out how to
trick it into pulling dependencies from the store rather than the Web.

The code is at <https://git.ngyro.com/guix-npm>.

I know that this is not great for Guix proper, but I thought that this
might be a good use-case for channels.  In a channel we could build out
the JavaScript ecosystem in Guix on top of a bunch of “binary” packages,
and then try to reduce this set of “binary” packages gradually.  I think
that having a useful JavaScript ecosystem, even if it is problematic,
might help motivate folks to work out the problems.  For example, it
would be a lot more exciting to work on bootstrapping TypeScript if it
were the only thing standing in the way of getting a Mastodon service in
Guix.

That’s the high-level overview. The rest of this message is the
technical details of how to try out the importer, build packages, etc.

Here’s how to make “@babel/cli” work with a Babel plugin called
“transform-arrow-functions”.

Clone the repo and build it like you would build Guix normally, except
for where you would usually use “guix environment guix”.  Instead, use

    guix environment guix --ad-hoc -l guile-semver.scm

to ensure that the Semantic Versioning library required by the importer
is available.  After this, continue as usual.

Once built, create a new file at “gnu/packages/node-babel.scm” with the
following contents:

    (define-module (gnu packages node-babel)
      #:use-module (guix build-system other-node)
      #:use-module (guix download)
      #:use-module ((guix licenses) #:prefix license:)
      #:use-module (guix packages))

Then, you can run

    ./pre-inst-env guix import npm-binary -r @babel/cli \
        | tee -a gnu/packages/node-babel.scm

(using “tee” is nice ’cause you can watch the output as it goes – it has
a lot of dependencies, so it takes a little while).

Get the plugin, too:

    ./pre-inst-env guix import npm-binary -r \
        @babel/plugin-transform-arrow-functions \
        | tee -a gnu/packages/node-babel.scm


There’s a package in there that is macOS specific, so it has to be
removed.  It’s called “fsevents”, and it is dependency of “chokidar”.
Everything works fine if it is simply deleted.

At this point, you can launch an environment with the packages (be sure
to include “node”, as that reminds Guix to set “NODE_PATH”):

    ./pre-inst-env guix environment --ad-hoc node node-babel-cli--fiio \
        node-babel-plugin-transform-arrow-functions--fiio

(For the curious, “fiio” is an initialism for a phrase used in
Christopher Lemmer Webber’s article about JavaScript packaging:
<http://dustycloud.org/blog/javascript-packaging-dystopia/>.  It serves
as a reminder that these are not nice packages.)

Unfortunately, Babel only looks for plugins relative to the current
directory, so you’ll need a symlink: “ln -s $NODE_PATH”.

With all this in place, it’s time for a test!

    echo '() => { return true; };' > test.js
    babel --plugins=@babel/plugin-transform-arrow-functions test.js

This prints:

    (function () {
      return true;
    });

which is good, since it is transforming the modern lambda syntax with
the arrow “=>” into the old syntax with the “function” keyword.  Hooray!

Now for some more notes.

I’ve tried a handful of packages, and have had good results so far.
Sometimes things need a little fiddling (like “fsevents”), but the
results are usable.

Both the importer and the build system are mostly new code.

For the importer, the main difference is that it uses the NPM version
constraints to make sure the recursively imported packages have the
expected versions.  It does this via a Guile library that I wrote.  One
limitation is that the recursive importer does not check versions for
existing Guix packages.  If there is a package with the right name, it
is accepted even if it doesn’t satisfy the version constraints.  Guix’s
“recursive-import” function would have to be changed to fix this.

The build system uses “npm” for everything, rather than trying to
emulate “npm” in Scheme (like the recently added build system does).
I’m not sure if this is better or worse, really, but it works nicely for
this particular use-case.  It works in theory to build packages from
source, too.  One test I did was importing all of jQuery’s development
dependencies as binary packages, and then building jQuery from source.

I’ve come to think that bootstrapping JavaScript might be easier than it
looks.  As time goes on, Node gets better at the newer JavaScript
features.  This removes the need for things like Babel or Rollup, since
with some care, Node can run the source directly with out any
transformations or bundling.  That being said, TypeScript looks to be a
major issue, as it is used in many fundamental JavaScript packages and
it is not bootstrappable.

I’m not sure in what capacity I want to pursue this.  It’s been sitting
dormant on my computer for while, so I thought sharing it would be
better than letting it fall by the wayside.  I hope it proves useful one
way or another.

If you got this far, thanks for reading!  :)


-- Tim

Re: Non-bootstrappable NPM packages

[Jelle Licht]

Timothy Sample <samplet@ngyro.com> writes:

[snip]

> I’ve come to think that bootstrapping JavaScript might be easier than it
> looks.  As time goes on, Node gets better at the newer JavaScript
> features.  This removes the need for things like Babel or Rollup, since
> with some care, Node can run the source directly with out any
> transformations or bundling.  That being said, TypeScript looks to be a
> major issue, as it is used in many fundamental JavaScript packages and
> it is not bootstrappable.

Very recently (IE about 94 minutes ago), I found out something
interesting that might be helpful; Sucrase[0] is, among other things, a
typescript transpiler that does not do any type checking, and it only
has some runtime dependencies.

I created some “fiio”-packages as well [1] , and I have confirmed that
it actually works! My next step was of course to compile TypeScript
proper, and this worked with one tiny snag that I reported at [2]. After
manually fixing these problems in the TypeScript source tree, I was able
to transpile the TypeScript sources using guix-packaged
`node-sucrase-bootstrap'.

> I’m not sure in what capacity I want to pursue this.  It’s been sitting
> dormant on my computer for while, so I thought sharing it would be
> better than letting it fall by the wayside.  I hope it proves useful one
> way or another.
>
> If you got this far, thanks for reading!  :)
Thank you for sending this informative email :)
>
>
> -- Tim

[0]: https://github.com/alangpierce/sucrase
[1]: https://paste.debian.net/1092893/
[2]: https://github.com/alangpierce/sucrase/issues/464

Re: Non-bootstrappable NPM packages

[Timothy Sample]

Hi Jelle,

Jelle Licht <jlicht@fsfe.org> writes:

> Timothy Sample <samplet@ngyro.com> writes:
>
> [snip]
>
>> I’ve come to think that bootstrapping JavaScript might be easier than it
>> looks.  As time goes on, Node gets better at the newer JavaScript
>> features.  This removes the need for things like Babel or Rollup, since
>> with some care, Node can run the source directly with out any
>> transformations or bundling.  That being said, TypeScript looks to be a
>> major issue, as it is used in many fundamental JavaScript packages and
>> it is not bootstrappable.
>
> Very recently (IE about 94 minutes ago), I found out something
> interesting that might be helpful; Sucrase[0] is, among other things, a
> typescript transpiler that does not do any type checking, and it only
> has some runtime dependencies.

That is exciting!  Did you see that the README links to a similar
project written in Rust <https://swc-project.github.io/>?  There’s also
a Rust-based bundler called Packem <https://packem.github.io/>.  Maybe
those two projects are all that we need.  With “mrustc”, the “rewrite it
in Rust” trend looks good for bootstrapping.  :)

> I created some “fiio”-packages as well [1] , and I have confirmed that
> it actually works! My next step was of course to compile TypeScript
> proper, and this worked with one tiny snag that I reported at [2]. After
> manually fixing these problems in the TypeScript source tree, I was able
> to transpile the TypeScript sources using guix-packaged
> `node-sucrase-bootstrap'.

Cool.  That looks really promising.  I’m glad to see that the current
“node-build-system” can do the same trick I’ve been doing.


-- Tim

Re: Non-bootstrappable NPM packages

[Nicolas Graves via Development of GNU Guix and the GNU System distribution.]

On 2019-07-24 15:41, Jelle Licht wrote:

> Timothy Sample <samplet@ngyro.com> writes:
>
> [snip]
>
>> I’ve come to think that bootstrapping JavaScript might be easier than it
>> looks.  As time goes on, Node gets better at the newer JavaScript
>> features.  This removes the need for things like Babel or Rollup, since
>> with some care, Node can run the source directly with out any
>> transformations or bundling.  That being said, TypeScript looks to be a
>> major issue, as it is used in many fundamental JavaScript packages and
>> it is not bootstrappable.
>
> Very recently (IE about 94 minutes ago), I found out something
> interesting that might be helpful; Sucrase[0] is, among other things, a
> typescript transpiler that does not do any type checking, and it only
> has some runtime dependencies.
>
> I created some “fiio”-packages as well [1] , and I have confirmed that
> it actually works! My next step was of course to compile TypeScript
> proper, and this worked with one tiny snag that I reported at [2]. After
> manually fixing these problems in the TypeScript source tree, I was able
> to transpile the TypeScript sources using guix-packaged
> `node-sucrase-bootstrap'.

Hi Jelle!

Did someone made some progress on the build-system since that to allow
for this to be taken into account? If you still have it, could you share
your "fiio" packages once again? The paste link is expired. Thanks!

>
>> I’m not sure in what capacity I want to pursue this.  It’s been sitting
>> dormant on my computer for while, so I thought sharing it would be
>> better than letting it fall by the wayside.  I hope it proves useful one
>> way or another.
>>
>> If you got this far, thanks for reading!  :)
> Thank you for sending this informative email :)
>>
>>
>> -- Tim
>
> [0]: https://github.com/alangpierce/sucrase
> [1]: https://paste.debian.net/1092893/
> [2]: https://github.com/alangpierce/sucrase/issues/464

-- 
Best regards,
Nicolas Graves

Re: Non-bootstrappable NPM packages

[Timothy Sample]

Heyo,

Nicolas Graves <ngraves@ngraves.fr> writes:

> Did someone made some progress on the build-system since that to allow
> for this to be taken into account? If you still have it, could you share
> your "fiio" packages once again? The paste link is expired. Thanks!

To my knowledge (it’s two years old now, so maybe others have made
progress since then), the best work on bootstrapping TypeScript in Guix
is singpolyma’s work here:

https://git.sr.ht/~whereiseveryone/guixrus/tree/master/item/guixrus/packages/javascript/typescript.scm

There are dozens of JS packages there.  These along with esbuild are
used to build TypeScript.  Everything is built from source (at least
that’s what I remember singpolyma telling me, and it looks like it from
a cursory glance), but no tests are run.  It’s still a mountain of work
to get those packages up to Guix standards, but the fundamentals are
sorted.  It’s a very promising approach; it just needs someone to carry
it across the finish line.


-- Tim

Re: Non-bootstrappable NPM packages

[Jelle Licht]

Nicolas Graves <ngraves@ngraves.fr> writes:

> On 2019-07-24 15:41, Jelle Licht wrote:
>
>> Timothy Sample <samplet@ngyro.com> writes:
>>
>> [snip]
>>
>>> I’ve come to think that bootstrapping JavaScript might be easier than it
>>> looks.  As time goes on, Node gets better at the newer JavaScript
>>> features.  This removes the need for things like Babel or Rollup, since
>>> with some care, Node can run the source directly with out any
>>> transformations or bundling.  That being said, TypeScript looks to be a
>>> major issue, as it is used in many fundamental JavaScript packages and
>>> it is not bootstrappable.
>>
>> Very recently (IE about 94 minutes ago), I found out something
>> interesting that might be helpful; Sucrase[0] is, among other things, a
>> typescript transpiler that does not do any type checking, and it only
>> has some runtime dependencies.
>>
>> I created some “fiio”-packages as well [1] , and I have confirmed that
>> it actually works! My next step was of course to compile TypeScript
>> proper, and this worked with one tiny snag that I reported at [2]. After
>> manually fixing these problems in the TypeScript source tree, I was able
>> to transpile the TypeScript sources using guix-packaged
>> `node-sucrase-bootstrap'.
>
> Hi Jelle!
>
> Did someone made some progress on the build-system since that to allow
> for this to be taken into account? If you still have it, could you share
> your "fiio" packages once again? The paste link is expired. Thanks!

I don't have them anymore, but I could re-generate them.
Are you still interested, or am I too slow in responding, and are your
other node-related emails more relevant by now?

KR,
- Jelle

Re: Non-bootstrappable NPM packages

[Wojtek Kosior via Development of GNU Guix and the GNU System distribution.]

[-- Attachment #1: Type: text/plain, Size: 2763 bytes --]

Hi there

not sure if this was mentioned here but in case it wasn't — there was a
proof of concept packaging of sucrase and jQuery as early as in 2022.
You can see the relevant thread here[1].

So, yes, it's definitely easier than it seems (and easier than some
used to describe it) — the only problem is that nobody seems to have
picked up that work done by zamfofex ¯\_(ツ)_/¯

Wojtek

[1] https://lists.gnu.org/archive/html/guix-devel/2022-11/msg00191.html

-- (sig_start)
website: https://koszko.org/koszko.html
fingerprint: E972 7060 E3C5 637C 8A4F  4B42 4BC5 221C 5A79 FD1A
follow me on Fediverse: https://friendica.me/profile/koszko/profile

♥ R29kIGlzIHRoZXJlIGFuZCBsb3ZlcyBtZQ== | ÷ c2luIHNlcGFyYXRlZCBtZSBmcm9tIEhpbQ==
✝ YnV0IEplc3VzIGRpZWQgdG8gc2F2ZSBtZQ== | ? U2hhbGwgSSBiZWNvbWUgSGlzIGZyaWVuZD8=
-- (sig_end)


On Tue, 20 Feb 2024 20:04:33 +0100 Jelle Licht <jlicht@fsfe.org> wrote:

> Nicolas Graves <ngraves@ngraves.fr> writes:
> 
> > On 2019-07-24 15:41, Jelle Licht wrote:
> >  
> >> Timothy Sample <samplet@ngyro.com> writes:
> >>
> >> [snip]
> >>  
> >>> I’ve come to think that bootstrapping JavaScript might be easier than it
> >>> looks.  As time goes on, Node gets better at the newer JavaScript
> >>> features.  This removes the need for things like Babel or Rollup, since
> >>> with some care, Node can run the source directly with out any
> >>> transformations or bundling.  That being said, TypeScript looks to be a
> >>> major issue, as it is used in many fundamental JavaScript packages and
> >>> it is not bootstrappable.  
> >>
> >> Very recently (IE about 94 minutes ago), I found out something
> >> interesting that might be helpful; Sucrase[0] is, among other things, a
> >> typescript transpiler that does not do any type checking, and it only
> >> has some runtime dependencies.
> >>
> >> I created some “fiio”-packages as well [1] , and I have confirmed that
> >> it actually works! My next step was of course to compile TypeScript
> >> proper, and this worked with one tiny snag that I reported at [2]. After
> >> manually fixing these problems in the TypeScript source tree, I was able
> >> to transpile the TypeScript sources using guix-packaged
> >> `node-sucrase-bootstrap'.  
> >
> > Hi Jelle!
> >
> > Did someone made some progress on the build-system since that to allow
> > for this to be taken into account? If you still have it, could you share
> > your "fiio" packages once again? The paste link is expired. Thanks!  
> 
> I don't have them anymore, but I could re-generate them.
> Are you still interested, or am I too slow in responding, and are your
> other node-related emails more relevant by now?
> 
> KR,
> - Jelle
> 
> 

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Non-bootstrappable NPM packages

[Nicolas Graves via Development of GNU Guix and the GNU System distribution.]

On 2024-02-20 20:04, Jelle Licht wrote:

> Nicolas Graves <ngraves@ngraves.fr> writes:
>
>> On 2019-07-24 15:41, Jelle Licht wrote:
>>
>>> Timothy Sample <samplet@ngyro.com> writes:
>>>
>>> [snip]
>>>
>>>> I’ve come to think that bootstrapping JavaScript might be easier than it
>>>> looks.  As time goes on, Node gets better at the newer JavaScript
>>>> features.  This removes the need for things like Babel or Rollup, since
>>>> with some care, Node can run the source directly with out any
>>>> transformations or bundling.  That being said, TypeScript looks to be a
>>>> major issue, as it is used in many fundamental JavaScript packages and
>>>> it is not bootstrappable.
>>>
>>> Very recently (IE about 94 minutes ago), I found out something
>>> interesting that might be helpful; Sucrase[0] is, among other things, a
>>> typescript transpiler that does not do any type checking, and it only
>>> has some runtime dependencies.
>>>
>>> I created some “fiio”-packages as well [1] , and I have confirmed that
>>> it actually works! My next step was of course to compile TypeScript
>>> proper, and this worked with one tiny snag that I reported at [2]. After
>>> manually fixing these problems in the TypeScript source tree, I was able
>>> to transpile the TypeScript sources using guix-packaged
>>> `node-sucrase-bootstrap'.
>>
>> Hi Jelle!
>>
>> Did someone made some progress on the build-system since that to allow
>> for this to be taken into account? If you still have it, could you share
>> your "fiio" packages once again? The paste link is expired. Thanks!
>
> I don't have them anymore, but I could re-generate them.
> Are you still interested, or am I too slow in responding, and are your
> other node-related emails more relevant by now?

I was going the esbuild route instead of the sucrase one, so that's not
a need anymore. Both are viable however, this choice is
questionnable. There are the packages from Wojtek in answer to this
email that also bootstrap sucrase, so don't bother ;)

I feel tests / native-inputs for Node is a bigger issue than the one of
bundlers now.

>
> KR,
> - Jelle
>

-- 
Best regards,
Nicolas Graves